Elastic Beats

Minions to collect data from each server

Overview

Beats Marketing
Install on Ubuntu
- beats.yml
- Download Beats
Processing
Metricbeat modules
Winlogbeat
Beats in Logstash
Resources
More

This section describes how to install, configure, and use the Beats component within the Elastic Stack, previously called ELK stack before Beats was added in 2016.

Beats is the collective term for Elastic’s utilities that run on servers to read logs from a variety of sources.

Beats are small, lightweight (written in Golang)

Filebeat collects and sends text log files.
Metricbeat collects and sends operating system and application data.
Packetbeat collects and sends network monitoring data.
Winlogbeat collects and sends Windows Event logs data
Libbeat collects and sends custom data defined in Golang programs.
Heartbeat
Auditbeat

Beats usually sends its data to a Logstash receiver, but they can send directly to ElasticSearch database.

Alternatives to get bits onto a server:

A) Ansible

B) Chef

C) Microsoft SCCM

D) Direct download

Beats Marketing

The product marketing page for LogStash is at:

https://www.elastic.co/products/beats

Install on Ubuntu

PROTIP: Beats is written in the Go Language (Golang), which compiles to a static binary containing its own VM. So there is no need to install a run-time such as JVM to run Java.

Add the Elastic public signing key to server:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Install on Ubuntu:

apt-get update && apt-get install filebeat
Look at the tail end (last few lines) in the system log:

tail /var/logs/sys.log
Configure the Filebeat configuration file from “/var/log/*.log” for all files to “/var/log/syslog.log” plus add “document_type: syslog”:

sed /etc/filebeat/filebeat.yml “/var/log/*.log” “/var/log/syslog.log \n document_type: syslog”

See https://www.elastic.co/guide/logstash/current/index.html

beats.yml

Download Beats

https://www.elastic.co/downloads/beats

Processing

Before forwarding, Logstash can parse and normalize varying schema and formats.

Reading
Filtering
Enhancing
Forwarding

Metricbeat modules

System Logs
Apache web server
NginX
HAProxy
MongoDB
MySQL
PostgreSQL
Redis
Zookeeper (Puppet)

Winlogbeat

Based on this

NOTE: Although Windows machines have Perfmon built-in, Beats collects and forwards the data.

Download and unzip.

File winlogbeat.full.yml contains ALL specifications.

File winlogbeat.template.es2x.yml should be ignored since you’re not using v2.

NOTE: Formats changed in v5, so Python scripts/migrate_beat_config_1_x_to_5_0.py is provided to migrate.

Edit the winlogbeat.yml configuration file to keep verbosity low and discard debug entries:

winlogbeat.event_logs:
- name: Application
  ignore_older: 72h
  level: critical, error, warning
- name: Security
- name: System

Add tags so each shipper can be identified when logs are consolidated:
```
tags: ["us-east-01"]"
```

Add fields to specify which environment:

fields:
  globo_environment: production

Specify the output destination as a Logstash server (not Elasticsearch server):
```
#output.elasticsearch:
output.logstash:
  hosts: ["192.168.0.14:5043"]
```
Save the file.
Verify whether the server designated can be reached. In PowerShell:

Invoke-WebRequest -Method PUT -InFile .\winlogbeat.template.json -Uri http://192.168.0.12:9200/_template
Install as a service within PowerShell CLI:

.\install-sevice-winlogbeat.ps1
Run in PowerShell:

winlogbeat.exe -c winlogbeat.yml
Start in PowerShell:

start-service winlogbeat
View logs in PowerShell:

Get-Content .\logs\winlogbeat -Wait

Errors will be reported if Logstash has not been setup.

scripts/import_dashboards.exe

Beats in Logstash

File beats.conf

Resources

This is one of a series on Elastic Stack and monitoring:

Wilson Mar