Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

Minions to collect data from each server


Overview

This section describes how to install, configure, and use the Beats component within the Elastic Stack, previously called ELK stack before Beats was added in 2016.

Beats is the collective term for Elastic’s utilities that run on servers to read logs from a variety of sources.

Beats are small, lightweight (written in Golang)

  • Filebeat collects and sends text log files.
  • Metricbeat collects and sends operating system and application data.
  • Packetbeat collects and sends network monitoring data.
  • Winlogbeat collects and sends Windows Event logs data
  • Libbeat collects and sends custom data defined in Golang programs.
  • Heartbeat
  • Auditbeat

Beats usually sends its data to a Logstash receiver, but they can send directly to ElasticSearch database.

Beats Marketing

The product marketing page for LogStash is at:

https://www.elastic.co/products/beats

Install

Beats is written in the Go Language (Golang), which compiles to a static binary containing its own VM. So there is no need to install a run-time such as JVM to run Java.

Alternatives to get bits onto a server:

A) Ansible

B) Chef

C) Microsoft SCCM

D) Direct download

Download Beats

  1. https://www.elastic.co/downloads/beats

Processing

Before forwarding, Logstash can parse and normalize varying schema and formats.

  • Reading
  • Filtering
  • Enhancing
  • Forwarding

Metricbeat modules

  • System Logs
  • Apache web server
  • NginX
  • HAProxy

  • MongoDB
  • MySQL
  • PostgreSQL
  • Redis

  • Zookeeper (Puppet)

Winlogbeat

Based on this

NOTE: Although Windows machines have Perfmon built-in, Beats collects and forwards the data.

  1. Download and unzip.

    File winlogbeat.full.yml contains ALL specifications.

    File winlogbeat.template.es2x.yml should be ignored since you’re not using v2.

    NOTE: Formats changed in v5, so Python scripts/migrate_beat_config_1_x_to_5_0.py is provided to migrate.

  2. Edit the winlogbeat.yml configuration file to keep verbosity low and discard debug entries:

    winlogbeat.event_logs:
    - name: Application
      ignore_older: 72h
      level: critical, error, warning
    - name: Security
    - name: System
    
  3. Add tags so each shipper can be identified when logs are consolidated:

    tags: ["us-east-01"]"
    
  4. Add fields to specify which environment:

    fields:
      globo_environment: production
    
  5. Specify the output destination as a Logstash server (not Elasticsearch server):

    #output.elasticsearch:
    output.logstash:
      hosts: ["192.168.0.14:5043"]
    
  6. Save the file.
  7. Verify whether the server designated can be reached. In PowerShell:

    Invoke-WebRequest -Method PUT -InFile .\winlogbeat.template.json -Uri http://192.168.0.12:9200/_template

  8. Install as a service within PowerShell CLI:

    .\install-sevice-winlogbeat.ps1

  9. Run in PowerShell:

    winlogbeat.exe -c winlogbeat.yml

  10. Start in PowerShell:

    start-service winlogbeat

  11. View logs in PowerShell:

    Get-Content .\logs\winlogbeat -Wait

    Errors will be reported if Logstash has not been setup.

scripts/import_dashboards.exe

Beats in Logstash

File beats.conf

Resources

More

This is one of a series on Elastic Stack and monitoring:

  1. Elastic Stack ecosystem of people, websites, tutorials
  2. Elastic Stack architecture and installation
  3. Elastic Scaling (the database engine)
  4. Elastic Query (via REST API)
  5. Elastic Kibana (the visualization engine, like Grafana)
  6. Elastic Logstash to assemble and filter data from Beats
  7. Elastic Beats to collect data from servers