Skip to content

log_parser job from logsearch/19

Github source: 2503ff99 or master branch

Properties

elasticsearch

cluster_name

The name of the elastic search cluster

config_options

Additional options to append to elasticsearch’s config.yml (YAML format).

discovery

minimum_master_nodes

The minimum number of master eligible nodes a node should “see” in order to operate within the cluster. Recommended to set it to a higher value than 1 when running more than 2 nodes in the cluster.

Default
1

flush_size

Redis queue flush size

Default
100

host

IP / DNS of elasticsearch http endpoint

log_level

The default logging level (e.g. WARN, DEBUG, INFO)

Default
INFO

logging_options

Additional options to append to elasticsearch’s logging.yml (YAML format).

node

tags

A hash of additional tags for the node

port

Port of elasticsearch http endpoint

Default
9200

logstash

metadata_level

Whether to include additional metadata throughout the event lifecycle. NONE = disabled, DEBUG = fully enabled

Default
NONE

logstash_parser

debug

Debug level logging

Default
false

elasticsearch_document_id

Use a specific, dynamic ID rather than an auto-generated identifier.

elasticsearch_index

The specific, dynamic index name to write events to.

Default
logstash-%{+YYYY.MM.dd}

elasticsearch_index_type

The specific, dynamic index type name to write events to.

Default
'%{@type}'

filters

The configuration to embed into the logstash filters section

Default
""

idle_flush_time

How frequently to flush events if the output queue is not full.

message_max_size

Maximum log message length. Anything larger is truncated (TODO: move this to ingestor?)

Default
1.048576e+06

outputs

The configuration to embed into the logstash outputs section

plugins

Plugins to run logstash with (array[] = { plugin-name: install-source }; e.g. [ { logstash-filter-cityindex-acctlookup: ‘https://s3.amazonaws.com/.../logstash-filter-cityindex-acctlookup-1.2.9.gem' } ])

Default
[]

use_local_elasticsearch

Run a local elasticsearch client node

Default
true

workers

The number of worker threads that logstash should use (default: auto = one per CPU)

Default
auto

redis

host

Redis host of queue

key

Name of queue to pull messages from

Default
logstash

port

Redis port of queue

Default
6379

Templates

Templates are rendered and placed onto corresponding instances during the deployment process. This job's templates will be placed into /var/vcap/jobs/log_parser/ directory (learn more).

  • bin/elasticsearch_ctl (from bin/elasticsearch_ctl)
  • bin/log_parser_ctl (from bin/log_parser_ctl)
  • bin/monit_debugger (from bin/monit_debugger)
  • config/elasticsearch/config.yml (from config/elasticsearch/config.yml.erb)
  • config/elasticsearch/logging.yml (from config/elasticsearch/logging.yml.erb)
  • config/filters_override.conf (from config/filters_override.conf.erb)
  • config/filters_post.conf (from config/filters_post.conf.erb)
  • config/filters_pre.conf (from config/filters_pre.conf.erb)
  • config/input_redis_and_output_elasticsearch.conf (from config/input_redis_and_output_elasticsearch.conf.erb)
  • config/logstash-filters-common.conf (from config/logstash-filters-common.conf.erb)
  • data/properties.sh (from data/properties.sh.erb)
  • helpers/ctl_setup.sh (from helpers/ctl_setup.sh)
  • helpers/ctl_utils.sh (from helpers/ctl_utils.sh)
  • logsearch/logs.yml (from logsearch/logs.yml)

Packages

Packages are compiled and placed onto corresponding instances during the deployment process. Packages will be placed into /var/vcap/packages/ directory.