ingestor_syslog job from logsearch/211.0.2

This job runs Logstash process which ingests data by standard Syslog protocol, make some processing, then forward it to Elasticsearch cluster

Github source: a08d4dc or master branch

Properties¶

logstash¶

env¶

a list of arbitrary key-value pairs to be passed on as process environment variables. eg: FOO: 123

Default

[]

heap_percentage¶

The percentage value used in the calculation to set the heap size.

Default

46

heap_size¶

sets jvm heap sized

jvm_options¶

additional jvm options

Default

[]

log_level¶

The default logging level (e.g. WARN, DEBUG, INFO)

Default

info

metadata_level¶

Whether to include additional metadata throughout the event lifecycle. NONE = disabled, DEBUG = fully enabled

Default

NONE

plugins¶

Array of hashes describing logstash plugins to install

Default

[]

Example

- name: logstash-output-cloudwatchlogs
  version: 2.0.0

queue¶

checkpoint¶

acks¶

The maximum number of acked events before forcing a checkpoint.

Default

1024

interval¶

The interval in milliseconds when a checkpoint is forced on the head page.

Default

1000

writes¶

The maximum number of written events before forcing a checkpoint.

Default

1024

max_bytes¶

The total capacity of the queue in number of bytes.

Default

1024mb

max_events¶

The maximum number of unread events in the queue.

Default

0

page_capacity¶

The page data files size. The queue data consists of append-only data files separated into pages.

Default

250mb

type¶

Internal queuing model, “memory” for legacy in-memory based queuing and “persisted” for disk-based acked queueing.

Default

persisted

logstash_ingestor¶

filters¶

Filters to execute on the ingestors

Default

""

relp¶

port¶

Port to listen for RELP messages

Default

2514

syslog¶

port¶

Port to listen for syslog messages

Default

5514

transport¶

Transport protocol to use

Default

tcp

use_keepalive¶

Instruct the socket to use TCP keep alives

syslog_tls¶

port¶

Port to listen for syslog-TLS messages (omit to disable)

skip_ssl_validation¶

Verify the identity of the other end of the SSL connection against the CA.

Default

false

ssl_cert¶

Syslog-TLS SSL certificate (file contents, not a path) - required if logstash_ingestor.syslog_tls.port set

ssl_key¶

Syslog-TLS SSL key (file contents, not a path) - required if logstash_ingestor.syslog_tls.port set

use_keepalive¶

Instruct the socket to use TCP keep alives

logstash_parser¶

debug¶

Debug level logging

Default

false

deployment_dictionary¶

A list of files concatenated into one deployment dictionary file. Each file contains a hash of job name-deployment name keypairs for @source.deployment lookup.

Default

- /var/vcap/packages/logsearch-config/deployment_lookup.yml

elasticsearch¶

data_hosts¶

The list of elasticsearch data node IPs

document_id¶

Use a specific, dynamic ID rather than an auto-generated identifier.

idle_flush_time¶

How frequently to flush events if the output queue is not full.

index¶

The specific, dynamic index name to write events to.

Default

logstash-%{+YYYY.MM.dd}

index_type¶

The specific, dynamic index type name to write events to.

Default

'%{@type}'

routing¶

The routing to be used when indexing a document.

enable_json_filter¶

Toggles the if_it_looks_like_json.conf filter rule

Default

false

filters¶

The configuration to embed into the logstash filters section. Can either be a set of parsing rules as a string or a list of hashes in the form of [{name: path_to_parsing_rules.conf}]

Default

""

inputs¶

A list of input plugins, with a hash of options for each of them. Please refer to example below.

Example

inputs:
- options:
    host: 192.168.1.1
    password: c1oudbunny
    user: logsearch
  plugin: rabbitmq

message_max_size¶

Maximum log message length. Anything larger is truncated (TODO: move this to ingestor?)

Default

1.048576e+06

outputs¶

A list of output plugins, with a hash of options for each of them. Please refer to example below.

Default

  - options: {}
    plugin: elasticsearch

Example

inputs:
- options:
    collection: logs
    database: logsearch
    uri: 192.168.1.1
  plugin: mongodb

timecop¶

reject_greater_than_hours¶

Logs with timestamps greater than this many hours in the future won’t be parsed and will get tagged with fail/timecop

Default

1

reject_less_than_hours¶

Logs with timestamps less than this many hours in the past won’t be parsed and will get tagged with fail/timecop

Default

24

wait_for_templates¶

A list of index templates that need to be present in ElasticSearch before the process starts

Default

- index_template

workers¶

The number of worker threads that logstash should use (default: auto = one per CPU)

Default

auto

Templates¶

Templates are rendered and placed onto corresponding instances during the deployment process. This job's templates will be placed into /var/vcap/jobs/ingestor_syslog/ directory (learn more).

• bin/ingestor_syslog.sh (from bin/ingestor_syslog)
• bin/pre-start (from bin/pre-start)
• config/bpm.yml (from config/bpm.yml.erb)
• config/filters_override.conf (from config/filters_override.conf.erb)
• config/filters_post.conf (from config/filters_post.conf.erb)
• config/filters_pre.conf (from config/filters_pre.conf.erb)
• config/input_and_output.conf (from config/input_and_output.conf.erb)
• config/jvm.options (from config/jvm.options.erb)
• config/logstash.yml (from config/logstash.yml.erb)
• config/syslog_tls.crt (from config/syslog_tls.crt.erb)
• config/syslog_tls.key (from config/syslog_tls.key.erb)

Packages¶

Packages are compiled and placed onto corresponding instances during the deployment process. Packages will be placed into /var/vcap/packages/ directory.

• logsearch-config
• logstash
• openjdk-11