Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Instagram Youtube

Github Stackoverflow Pinterest

Your robot butler is simple, but not stupid


Overview

The object of this tutorial is to succintly present, with step-by-step instructions but without much marketing hype, how to make use of Ansible to install software on servers.

Popularity

Ansible was written in Python 2.6 and open-sourced under the ansible organization on GitHub. So to build Ansible from source (containing sub-modules):

git clone https://github.com/ansible/ansible.git --recursive
cd ./ansible
make rpm
sudo rpm -Uvh ./rpm-build/ansible-*.noarch.rpm

The GitHub Octoverse report has dentified Ansible as being among the top 10 projects with most contributors and most discussed projects.

Its documentation is at ansible.com.

The @ansible Twitter account is titled “Red Hat Ansible” because it’s initiating author, Michael DeHaan from North Carolina (@laserllama), began writing Ansible in his spare time while working at RedHat. Quotes from his lightning talk at All Things Open Dec 3, 2014:

  • “Your IT infrastructure should be boring”
  • “How do we get sysadmins and developers together to cheat off each other, even at competing companies”
  • “Automation should not be your day job”
  • “Build early and often. Build a culture of testing.”
  • “Have Dev/QA/Stage environments that mirror production to see what can go wrong early.”

See the slides to his video “Python-Powered Radically Simple IT Automation” at PyCon 2014:

  • https://www.youtube.com/watch?time_continue=80&v=Qi0AhK7PMCI

How Ansible works

Ansible’s Control Server (acs) communicates with servers to download and provision software locally in them.

The name “ansible” is popularized by the science-fiction book and movie Enders Game which uses an “ansible” to communicate, in real-time, with many ships at once, galaxies away.

One of the distinguishing technologies Ansible uses the SSH (secure shell) protocol (OpenSSH program) in all Linux distributions. Windows Remote Management (WinRM) is used to connect with Windows (from Vista SP1 or Windows 2008 R1 and up).

With SSH on port 22, one does not need to beg for special ports to be opened through the enterprise firewall, which one needs to do with Chef and Puppet agents.

Use of SSH does require use of the ssh-keygen program to create private/public key pairs.

By default, JSON messages are communicated back to the Control Server’s API listening on standard port 80. Internally, Ansible uses the Django-REST framework, PyYAML.

But plug-ins can be installed so Ansible can communicate via ZeroMQ “fireball mode” or other means.

Ansible works with only text files, so it does not add a database nor the daemons to start or keep a database running.

New major releases of Ansible come out approximately every two months, with release cycles of about four months.

Python Ansible Control Server on Linux, not Windows

Ansible Control Server core is written in Python 2.6+. Thus, it can run natively on NIX (Linux/Unix/Mac), but Windows not currently supported nor recommended.

The “ansible_python_interpreter” variable in inventory points to the Python executable folder.

However, run virtual instances on a Windows, Mac, or other native OS if you want to use them to run Ansible.

Ansible Tower

Additionally, licensed product Ansible Tower runs playbooks for enterprises.

.yml files in Git vs. database

Instead of a database server, Ansible stores declarations in text files of yml (yamil) format that are both human and machine readable.

These Playbooks can be edited by any text editor.

Playbooks from others are available as roles in the Ansible Galaxy community website.

Being text files, most enterprises put Ansible configuration files in a Git repository (such as GitHub or Bitbucket) to maintain back versions for the team.

Modules do the work within the server are invoked by tasks specified in plays.

Modules can apply plays on several servers defined in an inventory file which can be dynamically generated from an enterprise CMDB (Configuration Management DataBase) cataloguing assets in AWS, Azure, GCP, or private cloud.

Additional modules can be defined, such as for building assets within AWS using CloudFormation.

Windows modules include win_feature (to installs and uninstall Windows Features) and win_regedit (Add, Edit, or Remove Registry Keys and Values). WinRM python module

To recap:


Let’s look at a playbook with full annotations:

Ansible works under the concept of “idempotance”, where repeated executions of the same script results in the same state at the end of each run. If something doesn’t exist, it is created. If something does exist already, it is left alone and another isn’t created.

Ansible reads declarations of desired state (what is wanted after processing) rather than imperative programming commands (to do this and that in a specified sequence).

This is like when you get in a taxi and you provide a destination address rather than directions to that location.

This makes definitions more reusable.

Windows

This yaml file launches the hello.ps1 PowerShell script:

- name: Run Powershell Scripts
  hosts: test
  tasks:
    - name: run a powershell script
      script: scripts/hello.ps1
      register: out
    - debug: var=out
   

The script:

https://github.com/dstamen/Ansible/tree/master/ansible_powershell

See http://davidstamen.com/ansible/using-ansible-to-run-powershell-scripts/

To execute the script, run:

ansible-playbook powershell.yml -i hosts

Videos:

Alternatives

The “Comparison of open source configuration management software on Wikipedia

This mentions the free Remote Execution Enabler for PowerShell tool from Solarwinds.

Ansible with Cloud Formation

Yan Kurniawan’s book provides Ansible playbook these procedural examples:

  • vpc_create.yml
  • sg_empty.yml to create empty security groups.
  • sg_modify.yml to modify security groups for each type of server
  • sg_delete.yml
  • ec2_vpc_web_create.yml to launch an instance in a particular subnet
  • ec2_vpc_db_create.yml without assigning a public IP address
  • nat_launch.yml to launch a “staging_nat” paravirtual t1.micro instance (with AMI name that includes “amzn-ami-vpc-nat”)
  • vpc_delete.yml
  • vpc_create_multi_az.yml
  • sg_jumpbox.yml
  • ec2_vpc_jumpbox.yml to launch jump box instance in public subnet A
  • ansible -i ec2.py tag_class_jumpbox -m ping
  • sg_openvpn.yml still requires manual retrieval of the AMI ID on https://openvpn.net/index.php/access-server/docs.html

The book provides an Ansible module in library/vpc_lookup

  • an update of https://github.com/edx/configuration/blob/master/playbooks/library/vpc_lookup (from John Jarvis) to lookup a VPC or subnets ID stored in local (safe) folder based on a particular filter specified in a script.

The suggested hashtag for the book is Tweet #ansible4aws.

PROTIP: Disable host key checking in ssh configuration so ssh will automatically add new host keys to the user known hosts files without asking (the default is “ask”).

  • Disable host key checking with StrictHostKeyChecking set to “no” in /etc/ssh/ssh_config file.

View sample configurations

  1. Use an internet browser to open galaxy.ansible.com/explore

  2. Search.

  3. Open a sample playbook.

    Playbooks are defined in .yml files, which begin with three dashes in the first line.

    Playbooks define plays. consisting of one or a set of tasks.

    tasks invoke modules.

    Tasks trigger handlers which are run on some condition, such as once at the end of plays.

    Spaces after dashes and colons are required.

An Ansible Config define Ansible control server configuration.

Notice the repos downloaded more than anyone is from @geerlingguy, Jeff Geerling (jeffgeerling.com) has been contributing to Ansible community since early 2013, and wrote ansiblefordevops.com.

Encrypted data within playbooks stored in GitHub can be unencrypted in memory using Ansible Vault.

Roles

Role files encapsulate instructions on how do a discrete unit of work, such as building a webserver. A role file contains for each type of resource tasks, variables, configuration templates, etc.

acme/
  webserver/
    README.md
    defaults/
    files/
    handlers/
    meta/main.yml
    tasks/
    templates/
    tests/
    vars/

The main.yml in meta defines dependencies:

---
galaxy_info:
  author: John Doe
  description: Quick and easy acme web installer.
  company: Acme
  license: MIT
  min_ansible_version: 1.9
  platforms:
  - name: EL
    versions:
    - all
  galaxy_tags:
    - acme
    - installer
    - web
dependencies:
  - { role: username.common, some_parameter: 3 }
  - { role: username.iptables, open_port: 80 }

Setup Vagrant and Virtualbox

  1. Download and install:

    • A virtual image manager from VagrantUp.com (87.9 MB for vagrant_1.8.1.dmg).
    • A vm provider (hypervisor) to run virtual machines from Oracle’s VirtualBox

     

  2. Verify availability from a command-line Terminal:

    vagrant
    vboxmanager

  3. Create a folder (of any name) for Ansible configuration files. This is typically for a project. This can be in a git folder if you’d like version management.

    cd ~
    mkdir ansible
    cd ansible

    The ~ (tilde character above) refers to your home folder.

  4. Switch to an internet browser to open a repository of Vagrant server base images:

    http://vagrantcloud.com (which redirects to a site owned by hashicorp, who owns Vagrant, thus the advert for the Atlas licensed product)

    NOTE: Many enterprises instead use an internal repository.

  5. In the box under “Discover Vagrant Boxes”, search for ubuntu or CentOS, etc.

  6. Choose one and copy its text in blue, such as “nrel/CentOS-6.5-x86_64” from contributor nrel or “ubuntu/trusty64”.

  7. Close down any process making use of port 8080, as that’s Vagrant’s default port. (Jenkins also uses port 8080 by default)

  8. Initialize a Vagrantfile for use by Vagrant:

    vagrant init

    Sample response:

    A `Vagrantfile` has been placed in this directory. You are now
    ready to `vagrant up` your first virtual environment! Please read
    the comments in the Vagrantfile as well as documentation on
    `vagrantup.com` for more information on using Vagrant.
    
  9. If you have a file named Vagrantfile from another source, copy it into the folder to replace the file generated.

    Alternately, open a text editor to create a file name Vagrantfile in end up with this sample content to specific the acs (Ancible Control Server), web, and db servers:

   Vagrant.configure(2) do |config|

     config.vm.define "acs" do |acs|
       acs.vm.box = "nrel/CentOS-6.5-x86_64"
       acs.vm.hostname = "acs"
       acs.vm.network "private_network", ip: "192.168.33.10"
     end

     config.vm.define "web" do |web|
       web.vm.box="nrel/CentOS-6.5-x86_64"
       web.vm.hostname = "web"
       web.vm.network "private_network", ip: "192.168.33.20"
       web.vm.network "forwarded_port", guest: 80, host: 8080
     end

     config.vm.define "db" do |db|
       db.vm.box = "nrel/CentOS-6.5-x86_64"
       db.vm.hostname = "db"
       db.vm.network "private_network", ip: "192.168.33.30"
     end
   end
  

The (2) in Vagrant.configure(2) configures the configuration version.

Names between | (pipe) characters provide handles to identify each server.

Two spaces are used to indent.

Internal IP addresses (192.168.33.xxx) are used in this example.

Change 8080 to another port if it is already used by another process on your computer.

Vagrant up

  1. Navigate to a folder containing a Vagrantfile specification file.
  2. Bring up a machine based on the Vagrantfile in the folder:

    vagrant up

    This can take several minutes if this is the first time, since images for servers specified need to be downloaded.

  3. Switch to a Finder to see that a .vagrant (hidden) folder has been added. Under the machines folder is a folder for each type specified between pipe characters (acs, web, db, etc).

  4. Open another terminal shell to check what is running:

    vboxmanage list runningvms

    The response is are hashes:

   "ansible_acs_1463860205025_4852" {128ce450-8384-4adb-a4fd-7f4ac5c1f0b8}
   "ansible_web_1463862332570_44406" {dd044db3-ecf1-4b9b-9c42-96952172bd4d}
   "ansible_db_1463882256962_22323" {411c8704-f220-4188-8b94-d1bfb093e1b4}
   

Provision Ansible Control Server

  1. SSH into the acs server via vagrant:

    vagrant ssh acs

    This takes several seconds to connect.

    This adds the ey to the known_hosts file within the .ssh folder for future reference.

  2. When you’re done:

    exit

  3. Use a package manager to download bits. On a CentOS or RHEL server:

    sudo yum -y install ansible

    Alternately, on a Debian Ubuntu server:

    sudo apt-get -y install ansible

    Notice the log says Python is installed as well.

  4. Verify:

    ansible --version

    Provision web server

  5. SSH into the web server via vagrant:

    vagrant ssh web

  6. Use a package manager to download bits:

    sudo yum -y install epel-release

Install by Compiling Source Code

  1. Install the C compiler used with Python:

    sudo yum install gcc

    sudo yum install python-setuptools

    sudo easy_install pip

    sudo yum install python-devel

    sudo pip install ansible

Tasks

Ansible tasks are commands executed from command line terminals.

Tasks are shereable and repeatable.

Steps Modules do

  1. Gather facts on hosts into variables such as ansible_os_family.
  2. Fetch md5 checksum from remote to verify downloaded file
  3. Create and manage local users and groups
  4. Enable and disable OS features and preferences

  5. Fetch files from remote sites
  6. Install software (web server, app server, database, virus scanner, etc.)
  7. Update software security patches

  8. Copy app configurations
  9. Copy files into server
  10. Call databases to retrieve data

  11. Enable service to start on reboot
  12. Start web service
  13. Deploy load balancer configurations (put in or take out server on rotation)

Install sample environment

Install:

Ansible covers more functionality:

  • Provisioning - install software, patch security, copy files in, customize configurations, start web service.
  • Change management of configurations with configuration remediation.
  • Automation - make decisions. A single change can impact several machines.
  • Complex Orchestration of dependencies.

Ansible evaluates to mark changed states.

A function is “idempotent” if repeated applications has the same affect as a single application.

Inventory file

The inventory of hosts can be defined within /etc/ansible/hosts containing .ini format Microsoft uses or yml format:

An example of ini format:

   [webservers]
   192.168.33.20
   192.168.33.30  ansible_connection=ssh ansible_user=mpdehaan

   [db]
   db-a.example.com

   [lbservers]
   lbserver  http_port=80 maxRequestsPerChild=808

   [monitoring]
   nagios
   

Items in square brackets define group names.

To get the status of servers under [webservers] in the inventory file above:

ansible webservers -m ping

In addition to this ad-hoc run, Ansible can be run based on the contents of Playbooks with a command such as:

ansible-playbook file.yml

Add -v for more detailed response.

Such inventory files are typically kept in a Git repository.

Modules in various languages

Unlike Puppet, Ansible does not require agent software to be installed and thus potentially leave residual bits on servers.

Modules are the “brains” of Ansible.

Various modules running on remote hosts provide the plumbing for other networking protocols, such as HTTP, runing on remote machines.

List of available modules, or locally:

ansible-doc -l

Press q to quit list, cursor up/down individual line, or space bar to page down.

Responses returned to the Ansible Control Server are in JSON messages.

Modules (hopefully written by following Module Development Guide) can be selected from various sources:

Ansible Module development can be in any dynamic language, not just Python on the server.

  • Simplejson library on *NIX.

Windows support

Ansible’s native Windows support uses Windows PowerShell remoting to manage Windows like Windows in the same Ansible agentless way that Ansible manages Linux like Linux.

  • Windows Remote PowerShell 2.0 enabled.

  • Windows modules

    • Push and execute any PowerShell scripts you write

Playbooks

Play behavior can be controlled several ways:

   with_items, 
   failed_when, 
   changed_when, 
   until, 
   ignore_errors
   

Register Output to Variable

To capture the result or output of a task so that follow-on tasks can act accordingly:

  tasks:
    - shell: /usr/bin/whoami
      register: username
    - debug: msg="Host=, User="
    - file: path=/home/myfile.txt
            owner=

Conditional Handlers

An example:

  tasks:
    - name: Deploy configuration file
      template: src=templates/httpd.j2 dest=/etc/httpd/conf/httpd.conf
      notify:
        - Restart Apache
  handlers:
    - name: Restart Apache
      service: name=httpd state=restarted

NOTE: .j2 files are processed by Jinja2, the template engine for Python, which replace variables with data values in static files.

To set a register to put result in a variable, then if the debug sees that a previous task failed, it would send a message.

  tasks:
    - command: ls /bad/path
      register: result
      ignore_errors: yes

    - debug: msg="Failure!"
      when: result|failed

NOTE: Handlers don’t run until all playbook tasks have executed.

  tasks:
    - copy: src=files/httpd.conf
            dest=/etc/httpd/conf/
      notify:
        - Apache Restart
  handlers:
    - name: Apache Restart
      service: name=httpd state=restarted

NOTE: A particular handler only executes once if needed.

NOTE: Handlers don’t run until all playbook tasks have executed.

Config. settings

The precedence Ansible looks for configuration variables. (stop searching once it finds one):

  1. $ANSIBLE_CONFIG environment variable
  2. ./ansible_cfg in current directory
  3. ~/ansible.cfg (home directory of currently logged in account)
  4. /etc/ansible/ansible.cfg global config. file

An example $ANSIBLE_CONFIG environment variable from the full list is:

   $ANSIBLE_FORKS=5
   

This sets the maximum number of parallel operations allowed on an Ansible server, determined through performance and capacity testing.

Include files

  tasks:
    - include: wordpress.yml
      vars:
        sitename: My Site
    - include_vars: variables.yml

Roles

  • https://bitbucket.org/fquffio/ansible-elasticsearch/src
  • https://bitbucket.org/fquffio/ansible-kibana/src
  • https://bitbucket.org/fquffio/ansible-iptables

Daemon Sets

Scalyr.com has a DaemonSet for Kubernetes monitoring.

Rolling updates

Ansible achieves zero-downtime deployments with multi-tear rolling updates to each specific node in a cluster.

This specifies taking 5 machines at a time out of a cluster:

   - hosts: webservers
     serial: 5

   pre_tasks:

   - name: take out of load balancer pool
     local_action: command /usr/bin/take_out_of_pool 

  roles:
   - common
   - webserver
   - monitored

  post_tasks:
   - name: add back to load balancer pool
     local_action: command /usr/bin/add_back_to_pool 
   

Social Community

  • Twitter: @ansible by Red Hat, @robynbergeron
  • https://groups.google.com/forum/#!forum/ansible-announce
  • On a IRC client, select Destination: Freenode, and add channel #ansible.
  • AnsibleFest (SF July 28, 2016)
  • Ansible-Galaxy.com/explore/ is the community hub to find and share reusable Ansible content.
  1. Link to GitHub https://galaxy.ansible.com/accounts/github/login/
  2. Confirm email.

From Ansible.com

Within Ansible’s YouTube channel

  • ansible.com/quick-start-video provide your email because it is a high-level, high-flautin’ marketing pitch which introduces Ansible Tower proprietary software.

Tim Gerla of Ansibleworks:

Videos

This tutorial presents related material in a different sequence for better understanding and updated.

Gwyn Price:

Others:

Misc

To enable Python to talk with Windows WinRM:

sudo pip install pywinrm

  1. Test whether a connection can be made:

    Test-WsMan 192.168.5.3

    https://github.com/PowerShell/PowerShell/issues/1883

https://github.com/PowerShell/PowerShell/blob/master/docs/KNOWNISSUES.md#remoting-support (WinRM does not run within MacOS 10) PowerShell https://quizlet.com/178078947/ansible-devops-automation-mamun-flash-cards/

https://github.com/PowerShell/psl-omi-provider

https://github.com/sthulb-attic/laptop-osx by https://twitter.com/sthulb an Amazon Solution Architect

https://github.com/monfresh/laptop Those are my personal playbooks and scripts to install a laptop from scratch including some dotfiles. Based on Fedora 27 Not idempotent

https://github.com/vaskas/laptop-ansible for Fedora 26

https://github.com/ansible/ansible-examples

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud regions
  16. AWS Virtual Private Cloud
  17. Azure Cloud Onramp
  18. Azure Cloud
  19. Azure Cloud Powershell
  20. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  21. Digital Ocean
  22. Cloud Foundry

  23. Packer automation to build Vagrant images
  24. Terraform multi-cloud provisioning automation

  25. Powershell Ecosystem
  26. Powershell on MacOS
  27. Powershell Desired System Configuration

  28. Jenkins Server Setup
  29. Jenkins Plug-ins
  30. Jenkins Freestyle jobs
  31. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  32. Dockerize apps
  33. Docker Setup
  34. Docker Build

  35. Maven on MacOSX

  36. Ansible

  37. MySQL Setup

  38. SonarQube static code scan

  39. API Management Microsoft
  40. API Management Amazon

  41. Scenarios for load