Minions to collect data from each server
This section describes how to install, configure, and use the Beats component within the Elastic Stack, previously called ELK stack before Beats was added in 2016.
Beats is the collective term for Elastic’s utilities that run on servers to read logs from a variety of sources.
Beats are small, lightweight (written in Golang)
- Filebeat collects and sends text log files.
- Metricbeat collects and sends operating system and application data.
- Packetbeat collects and sends network monitoring data.
- Winlogbeat collects and sends Windows Event logs data
- Libbeat collects and sends custom data defined in Golang programs.
Beats usually sends its data to a Logstash receiver, but they can send directly to ElasticSearch database.
Alternatives to get bits onto a server:
C) Microsoft SCCM
D) Direct download
The product marketing page for LogStash is at:
Install on Ubuntu
PROTIP: Beats is written in the Go Language (Golang), which compiles to a static binary containing its own VM. So there is no need to install a run-time such as JVM to run Java.
Add the Elastic public signing key to server:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Install on Ubuntu:
apt-get update && apt-get install filebeat
Look at the tail end (last few lines) in the system log:
Configure the Filebeat configuration file from “/var/log/*.log” for all files to “/var/log/syslog.log” plus add “document_type: syslog”:
sed /etc/filebeat/filebeat.yml “/var/log/*.log” “/var/log/syslog.log \n document_type: syslog”
Before forwarding, Logstash can parse and normalize varying schema and formats.
- System Logs
- Apache web server
- Zookeeper (Puppet)
NOTE: Although Windows machines have Perfmon built-in, Beats collects and forwards the data.
Download and unzip.
winlogbeat.full.ymlcontains ALL specifications.
winlogbeat.template.es2x.ymlshould be ignored since you’re not using v2.
NOTE: Formats changed in v5, so Python scripts/migrate_beat_config_1_x_to_5_0.py is provided to migrate.
winlogbeat.ymlconfiguration file to keep verbosity low and discard debug entries:
winlogbeat.event_logs: - name: Application ignore_older: 72h level: critical, error, warning - name: Security - name: System
Add tags so each shipper can be identified when logs are consolidated:
Add fields to specify which environment:
fields: globo_environment: production
Specify the output destination as a Logstash server (not Elasticsearch server):
#output.elasticsearch: output.logstash: hosts: ["192.168.0.14:5043"]
- Save the file.
Verify whether the server designated can be reached. In PowerShell:
Invoke-WebRequest -Method PUT -InFile .\winlogbeat.template.json -Uri http://192.168.0.12:9200/_template
Install as a service within PowerShell CLI:
Run in PowerShell:
winlogbeat.exe -c winlogbeat.yml
Start in PowerShell:
View logs in PowerShell:
Get-Content .\logs\winlogbeat -Wait
Errors will be reported if Logstash has not been setup.
Beats in Logstash
This is one of a series on Elastic Stack and monitoring:
- Elastic Stack ecosystem of people, websites, tutorials
- Elastic Stack architecture and installation
- Elastic Scaling (the database engine)
- Elastic Query (via REST API)
- Elastic Kibana (the visualization engine, like Grafana)
- Elastic Logstash to assemble and filter data from Beats
- Elastic Beats to collect data from servers