Elastic Stack Installation

Here are several ways to get it up and running quickly

This page describes the different options to get going with an Elastic Stack of your own.

“Elasticsearch is a real-time distributed, scalable, real-time search and analytics engine. It enables you to search, analyze, and explore your data, often in ways that you did not anticipate at the start of a project.”

First of all, each component of the Elastic Stack (previously called ELK Stack) are typically on a different set of servers:

• Beats components are installed on servers being monitored.
• Logstash servers should ideally be near the same subnet as servers being monitored.
• ElasticSearch database nodes (usually a cluster of them) can be in the same or different data centers than servers being monitored.
• Kibana servers would be near those who request visualizations
  
  

Elastic also offers cloud services as paid (licensed) software to manage and protect stacks:

1. Found
2. Shield to Secure data in Elasticsearch.
3. Marvel to Monitor Elasticsearch deployments.
4. Watcher Alerting for Elasticsearch.
5. Packetbeat to Analyze network packet data.

Versions for download

https://www.elastic.co/downloads/elasticsearch
presents the lastest version from Elastic.co.

Notice several images are available:

• .ZIP
• .TAR for generic Linux
• .DEB for Debian
• .RPM for Ruby
• .MSI for Windows

Only one previous back release is available.

The folder unzipped contains these folders:

|-- bin     = executables and shell scripts 
|-- config  = settings to tweak
|-- lib     = library jar files for Java
|-- modules = jar files and properties for optional modules
`-- plugins = empty
   

The config folder contains settings to tweak: log4j2.properties, and elasticsearch.yml,

1. On Windows 64-bit machines, edit the jvm.options file to add the stack size for Java (under -Xmx2g):

   -Xss1m

Ways to setup

A) Run on Elastic’s own Cloud Service. This is a quickest way.

B) Run on Amazon’s Elastic Service. This is a quick way, but not optimal.

C) Use a Docker image.

D) Download installer onto a local or virtual machine.

E) Build from source code in GitHub.

Elastic Cloud

See the webinar

Other SaaS service providers include:

https://logz.io/elk_as_a_service/

Elastic from Amazon

Amazon announced its ElastiSearch Service offering October 1st 2015. See: https://aws.amazon.com/blogs/aws/new-amazon-elasticsearch-service/

After a month working with it, http://kirankoduru.github.io/elasticsearch/moving-from-aws-elasticsearch-service.html identified reasons why he got away from it:

• AWS is not supported by Elastic Support.

• AWS does not include Shield, the Elastic commercial security plugin that handles RBAC of cluster and indexes.

• AWS controls the elasticsearch.yml file containing settings to tweak. The AWS elasticsearch service makes sure everything works perfectly.

• AWS’s service does not consume credits companies accumulate volume discounts for Amazon usage.

• AWS does not provide a selection of region, which may cause some latency.

• AWS’s services uses an older version of Elasticsearch, such as 1.5 when version 2.0 is available directly from Elastic.co. This is important for those who don’t want to miss out on those bug fixes and shiny new feature releases.

• AWS’s IAM policies are the only way to configure access to its Elasticsearch service. That is a good way to secure inbound connections. But some prefer using security groups.

• AWS turns off dynamic scripting and does not allow script upload into the scripts directory.

• AWS performs backups only after receiving an email to AWS support rather than allowing the elasticsearch-aws-cloud plugin to configure s3 repositories from the elasticsearch service dashboard.

The ironic thing is that Elastic’s own Found service is run on AWS.’

With that said, let’s try it out anyway:

AWS Install Steps</a>

Below are the steps I took to create a public instance within AWS.

1. Use Firefox to get in EC2 dashboard at http://aws.amazon.com/ec2/

2. Select a region appropriate to you.

   CAUTION: If you intend on creating an image for the Marketplace, you must use US Northern Virginia (us-east-1).

The default for examples on this page is US-West-2, which is in Oregon.

1. Scroll down to the bottom of the Analytics section to click on the Elasticsearch Service link.

   https://us-west-2.console.aws.amazon.com/es/home?region=us-west-2#
   

2. If you want to use a different region, click on “Oregon” (or another) at the upper right corner.

3. Clicking on the Getting Starte Guide opens a new window tab.

4. Click on the blue Get Started button.

5. PROTIP: When specifying domain, keep to lower case characters and perhaps a number at the end for versioning. For example:

   test1
   

6. For Instance type, since I don’t have much data yet during experimentation, I use Free Tier eligible t2.micro.elasticsearch.

7. Hover over the (i) icon to the right of Enable dedicated master. It says: “We recommend that you allocate at least three dedicated master nodes for each production Elastisearch domain.”

   I leave blank anyways during experimentation.

8. Hover over the (i) icon to the right of Enable zone awareness. It says: “Distributes nodes across two Availability Zones…”

9. For Storage, I select EBS with 10 GB of General Purpose (SSD) drives.

   PROTIP: Free tier users gets up to 10 gigabytes of Magnetic or SSD-Backed EBS storage at no charge for up to 750 hours per month.

10. After clicking Next, for access policy I select “Allow open access to the domain” so anyone can upload documents.

    The version in the file is of AWS access code API version:

    "Version": "2012-10-17",
    

11. Click Next.
12. Confirm and create. It takes several minutes for Domain status to go from Loading to Ready. Below is a sample screen after provisioning:

These sample links are no longer active, of course. But when it was for me, clicking on https://search-test1-da54anmy3esch22sskcuukwf6i.us-west-2.es.amazonaws.com/ got me this:

  {
  "status" : 200,
  "name" : "Ikthalon",
  "cluster_name" : "495629083449:test1",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Instead of the web UI, use AWS ES CLI.

1. Click on the Kibana link, such as:

   https://search-test1-da54anmy3esch22sskcuukwf6i.us-west-2.es.amazonaws.com/_plugin/kibana/
   

1. To make use of the instance using Python in my Mac terminal window:

   pip install elasticsearch
   

2. For more about using Python for Elastisearch, this article (from November 2014):

http://bitquabit.com/post/having-fun-python-and-elasticsearch-part-1/

Use Docker images

WARNING: Prior Docker images for Elastic servers on Docker hub (https://hub.docker.com/_/elasticsearch/) have been deprecated.

1. Install Docker.

   See https://docs.docker.com/docker-for-mac/

   See https://docs.docker.com/compose/install/

2. Run the Docker daemon.

3. Retrieve the official Docker image for Elastic server for a specific release:

   docker pull docker.elastic.co/elasticsearch/elasticsearch:5.6.0
   

   NOTE: Source code for it can be found on GitHub at
   https://github.com/elastic/elasticsearch-docker/tree/5.6

   max_memory_account

   DOC

4. Before running in production mode, which

   Run in dev mode

5. Run in development mode (on a different port if you prefer):

   docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.6.0
   

   Alternately, bring up a cluster of several Elasticsearch nodes based on specifications in the docker-compose.yml file:

   docker-compose up
   

   NOTE: The image is built with X-Pack and uses centos:7 as the base image.

Download installer from GitHub on your local machine

Pre-requisites

1. Instantiate an Ubunto server image.

2. Verify the Linux operating system:

   cat /etc/issue.net; ifconfig
   

3. Get IP address assigned, to later put in the elasticsearch.yml file:

   ifconfig
   

4. Install the Java runtime:

   apt-get install openjdk-8-jre-headless && 
   java -version
   

5. Get curl:

   For Windows: From http://curl.haxx.se/download.html download the Win64 x86_64 zip.

   Get Elasticsearch

6. Make a folder:

   mkdir pkg
   

7. QUESTION: Identify the version you want to use.

8. Download Debian package from Elastic:

   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.0.deb
   

   PROTIP: RPM and Debian packages are available. Satellite versions from the OS ES.co does not recommend apt-get which may be older than those directly from Elastic.co.

9. De-package:

   dpkg -i elasticsearch-5.6.0.deb
   

   Configure defaults

10. PROTIP: Some copy over the file with one from your own GitHub repository.

11. Edit using the nano text editor:

    nano /etc/elasticsearch/elasticsearch.yml

12. Remove the # to change cluster.name: to yours.
13. Make the node.name the same.
14. Uncomment network.host to the IP address of the server.

15. Save and exit the text editor.

    Memory maps

    PROTIP: Elasticsearch makes extensive use of memory maps. So give it the maximum.

    To set the maximum value permanently on a Linux server:

    grep vm.max_map_count /etc/sysctl.conf </strong</tt>

    Alternately, to set the value for just the current session on a Linux server:

    sysctl -w vm.max_map_count=262144 </strong</tt>

16. Start the Elasticsearch service:

    service elasticsearch start

    Alternately, on a Windows machine:

    .\elasticsearch-service.bat install Elasticsearch

17. Verify access using a headless API call to its port, for example:

    curl http://192.168.0.12:9200

    On Windows, bring up PowerShell and:

    Invoke-WebRequest -Uri http://192.168.0.12:9200

    The response should be JSON reflecting its most recent build metadata.

    Alternately, invoke:

    bin/elasticsearch --cluster.name=my-application --node.name=node-1 --path.repo=/
    

18. Make it come up after a reboot. On Linux:

    systemctl enable elasticsearch

Resources

• Andrew Puch @gmail https://www.linkedin.com/in/apuch who runs http://andrewpuch.com/ and the #devopsengineers Slack channel at http://devopsengineers.com/ created as https://twitter.com/awstutseries https://www.youtube.com/watch?v=ge8uHdmtb1M which details how to setup Elasticsearch.

• How to Install and Configure Elasticsearch

• ObjectRocker has a KnowledgeBase of articles at https://kb.objectrocket.com/category/elasticsearch

More

This is one of a series on Elastic Stack and monitoring:

1. Elastic Stack ecosystem of people, websites, tutorials
2. Elastic Stack architecture and installation
3. Elastic Scaling (the database engine)
4. Elastic Query (via REST API)
5. Elastic Kibana (the visualization engine, like Grafana)
6. Elastic Logstash to assemble and filter data from Beats
7. Elastic Beats to collect data from servers