Terraform from Hashicorp

I declare! Client-only immutable multi-cloud provisioning, with open-sourced Enterprise support

Overview

Competitors to Terraform
Installation
Get Sample Terraform scripts
Cloud Providers
Terraform Console
Automation
Modules
Bootstrap Terraform
Plugins into Terraform
Learning Resources
AWS Cloud Formation
References
More on DevOps

This tutorial is a step-by-step hands-on introduction to use of Terraform for creating a cluster of web servers through a load balancer on AWS, Azure, and Google Cloud.

Terraform, at terraform.io, is a tool for building, changing, and versioning infrastructure in the cloud.

Repeatable. Versioned. Documented. Automated. Testable. Shareable.

Competitors to Terraform

NOTE: Other IAC (Infrastructure as Code) tools include Chef, Puppet, Ansible, SaltStack, AWS CloudFormation.

Click to pop-up full screen image

“Immutable” means servers are treated like “cattle” (removed from the herd) and not as “pets” (kept alive as long as possible).

Feature	CloudFormation	Terraform
Source code	closed-source	open source
Configuration format	JSON	HCL JSON
State management	JSON	HCL JSON
Cloud Providers support	AWS only	AWS, GCE, Azure (20+)
Execution control	No	Yes
Iterations	No	Yes
Manage already created resources	No	Yes (hard)
Failure handling	Optional rollback	Fix & retry
Open Source contributions?	No	Yes (GitHub issues)
Logical comparisons	No	Limited
Extensible Modules	No	Yes

Terraform also provides execution control, iterations, and (perhaps most of all) management of resources already created (desired state configuration) over several cloud providers (not just AWS).

Installation

PROTIP: Terraform is written in the Go language, so there is no JVM to download as well.

Consider tfenv

If you plan on frequently switching among several versions installed of Terraform:

https://github.com/kamatama41/tfenv

Install on MacOS

In a Terminal window:

brew install -g terraform

The response at time of writing:

==> Downloading https://homebrew.bintray.com/bottles/terraform-0.10.6.sierra.bot
######################################################################## 100.0%
==> Pouring terraform-0.10.6.sierra.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
🍺  /usr/local/Cellar/terraform/0.10.6: 7 files, 63.6MB

Proceed to Get sample Terraform scripts.

Install on Windows

In a Run command window as Administrator.
Install Chocolatey cmd:

Install Terraform using Chocolatey:

choco install terraform -y

The response at time of writing:

Chocolatey v0.10.8
Installing the following packages:
terraform
By installing you accept licenses for the packages.
Progress: Downloading terraform 0.10.6... 100%
 
terraform v0.10.6 [Approved]
terraform package files install completed. Performing other installation steps.
The package terraform wants to run 'chocolateyInstall.ps1'.
Note: If you don't run this script, the installation will fail.
Note: To confirm automatically next time, use '-y' or consider:
choco feature enable -n allowGlobalConfirmation
Do you want to run the script?([Y]es/[N]o/[P]rint): y
 
Removing old terraform plugins
Downloading terraform 64 bit
  from 'https://releases.hashicorp.com/terraform/0.10.6/terraform_0.10.6_windows_amd64.zip'
Progress: 100% - Completed download of C:\Users\vagrant\AppData\Local\Temp\chocolatey\terraform\0.10.6\terraform_0.10.6_windows_amd64.zip (12.89 MB).
Download of terraform_0.10.6_windows_amd64.zip (12.89 MB) completed.
Hashes match.
Extracting C:\Users\vagrant\AppData\Local\Temp\chocolatey\terraform\0.10.6\terraform_0.10.6_windows_amd64.zip to C:\ProgramData\chocolatey\lib\terraform\tools...
C:\ProgramData\chocolatey\lib\terraform\tools
 ShimGen has successfully created a shim for terraform.exe
 The install of terraform was successful.
  Software installed to 'C:\ProgramData\chocolatey\lib\terraform\tools'
 
Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Get Sample Terraform scripts

Install a Git client if you haven’t already.
Use an internew browser (Chrome) to my sample PowerShell DSC scripts at:

https://github.com/wilsonmar/terraform-starter.git

(I would be honored if you click Star)
Create a GitHub account for yourself if you haven’t already.
Click the Fork button to make it yours, since you may be making changes.
Create or navigate to a container folder where new repositories are added. For example:

~/gits/wilsonmar/terraform
Get my sample PowerShell scripts onto your laptop (substituting “wilsonmar” with your own account name):

git clone https://github.com/wilsonmar/terraform-starter.git --depth=1 && cd terraform-starter

The above is one line, but may be word-wrapped on your screen.

The response at time of writing:
```
Cloning into 'terraform-starter'...
remote: Counting objects: 12, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 12 (delta 1), reused 9 (delta 0), pack-reused 0
Unpacking objects: 100% (12/12), done.
```
Other scripts

These scripts is a combination of scripts prepared by several helpful people:
- https://github.com/gruntwork-io/intro-to-terraform.git
- https://github.com/brikis98/infrastructure-as-code-talk/tree/master/terraform-configurations
https://github.com/gruntwork-io/intro-to-terraform.git

TODO: The sample scripts referenced by this tutorial contain moustache variable mark-up so that you can generate a set for your organization.

https://www.terraform.io/docs/providers/azurerm/r/virtual_machine_scale_set.html

Environment folders Validate

PROTIP: Separate Terraform configurations by a folder for each environment.
- base
- dev
- loadtest (performance testing)
- stage
- uat (User Acceptance Testing)
- prod
Navigate into the base folder.

PROTIP: Terraform commands act only on the current directory, and does not recurse into sub directories.

View the development.tfvars file:

environment_tag = "dev"
tenant_id = "223d"
billing_code_tag = "DEV12345"
dns_site_name = "dev-web"
dns_zone_name = "mycorp.xyz"
dns_resource_group = "DNS"
instance_count = "2"
subnet_count = "2"

The production.tfvars file usually instead contain more instances and thus subnets that go through a load balancer for auto-scaling:

environment_tag = "prod"
tenant_id = "223d"
billing_code_tag = "PROD12345"
dns_site_name = "marketing"
dns_zone_name = "mycorp.com"
dns_resource_group = "DNS"
instance_count = "6"
subnet_count = "3"

All these would use main_config.tf and variables.tf files that are commonly used for all environments:

Plug-in Initialization

Providers are not included with the installer, so…

Initialize Terraform plug-ins:

terraform init

Response:

Initializing provider plugins...
       - Checking for available provider plugins on https://releases.hashicorp.com...
       - Downloading plugin for provider "aws" (0.1.4)...   
 
The following providers do not have any version constraints in configuration,
so the latest version was installed.
 
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
 
       * provider.aws: version = "~> 0.1"
 
Terraform has been successfully initialized!
 
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
 
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

This creates a hidden .terraform\plugins" folder path containing a folder for your os - darwin_amd64` for MacOS.

.tf files

Navigate into one of the folders (in the case of the example repo):

cd single-web-server
Open folder using Atom or list files:

ls -al

https://www.terraform.io/docs/commands/validate.html
Navigate into
Validate the folder:

terraform validate

If no issues are identified, no message appears. (no news is good news)

pre-commit hook to validate in your Git repository

Main.tf

PROTIP: There should be only one main.tf per folder.

NOTE: Terraform files coded end with .tf file type.
Edit file main.tf. Widen the screen width to avoid wrapping.

Both CloudFormation and Terraform work with JSON, but Terraform works with HCL (Hashicorp Configuratio Language) that is both human and machine friendly. https://github.com/hashicorp/hcl and described at https://www.terraform.io/docs/configuration/syntax.html

NOTE: Terraform code is written in a language called HCL (HashiCorp Configuration Language). It’s less verbose than JSON and more concise than YML.

Unlike JSON and YML, HCL allows annotations as in bash scripts: Single line comments start with # (pound sign).
Multi-line comments are wrapped between /* and */.

Values can be interpolated usning syntax wrapped in ${}, called interpolation syntax, in the format of ${type.name.attribute}. Literal $ are coded by doubling up $$. For example, ${aws.instance.base.id} is interpolated to i-28978a2.

Back-slashes specify continuation.

More importantly, tf files are declarative, meaning that they define the desired end-state (outcomes). If 15 servers are declared, Terraform automatically adds or removes servers to end up with 15 servers rather than specifying procedures to invoke (such as add 5 servers). Terraform know how many servers it has setup already.

Paid Pro and Premium licenses of Terraform add version control integration, MFA security, and other enterprise features.

HCL does not have conditional if/else logic, which is why modules are necessary.

Cloud Providers

Terraform providers reference APIs. Examples are AWS, Google, Azure, Kubernetes, GitLab, DigitalOcean, Heroku, GitHub, OpenStack.

https://github.com/terraform-providers

https://github.com/hashicorp/terraform/tree/master/builtin/providers
https://aws.amazon.com/

Metadata related to each provider are defined like this:
```
provider "aws" {
  alias = "NorthEast"
  region = "us-east-1"
  instance_type = "t1.micro"
}
```
resource definitions specify the desired state of resources.

“t1.micro” qualifies for the Amazon free tier available to first-year subscribers.

NOTE: Components of Terrform are: provider, resource, provision.

Provisioners

Provisioners configurations are also plugins:

Provisioner definitions define the properties of each resource, such as initialization commands. For example, this installs an nginx web server and displays a minimal HTML page:
```
provisioner "remote-exec" {
  inline = [
 "sudo yum install nginx -y",
 "sudo service nginx start",
 "echo "<title>NGINX server</head><body style=\"background-color"></body></html>"
  ]
}
</pre>
```
PROTIP: Make sure that the AWS region is what you want.

https://www.terraform.io/docs/providers/aws/r/instance.html AWS provider

NOTE: The contents of the repo were written based on Terraform 0.7.x.

Variables file:

vars.tf file contains specifcation of values to variables, such as the server port (8080).

Defaults and lookup function

PROTIP: Variables can be assigned multiple default values selected by a lookup function:

# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
# export AWS_DEFAULT_REGION=xx-yyyy-0
 
variable "server_port" {
  description = "The port the server will use for HTTP requests"
  default = 8080
}
variable "amis" {
  type = "map”"
  default = {
 us-east-1 = "ami-1234"
 us-west-1 = "ami-5678"
  }
}
ami = ${lookup(var.amis, "us-east-1")}

PROTIP: With AWS EC2, the us-east-1 region must be used as the basis for creating others.

NOTE: Amazon has an approval process for making AMIs available on the public Amazon Marketplace.

CIDR Subnet function

variable network_info {
default = “10.0.0.0/8” #type, default, description
}
cidr_block = ${cidrsubnet(var.network_info, 8, 1)} # returns 10.1.0.0/16
cidr_block = ${cidrsubnet(var.network_info, 8, 2)} # returns 10.2.0.0/16

variable network_info {
default = “10.0.0.0/8” #type, default, description
}
cidr_block = ${cidrsubnet(var.network_info, 8, 1)} # returns 10.1.0.0/16
cidr_block = ${cidrsubnet(var.network_info, 8, 2)} # returns 10.2.0.0/16

In this example terraform.tfvars file are credentials for both AWS EC2 and Azure ARM providers:

bucket_name = "mycompany-sys1-v1"
arm_subscription_id = "???"
arm_principal = "???"
arm_passsord = "???"
tenant_id = "223d"
aws_access_key = "insert access key here>"
aws_secret_key = "insert secret key here"
private_key_path = "C:\\MyKeys1.pem"

The private_key_path should be a full path, containing \\ so that the single slash is not interpreted as a special character.

bucket_name must be globally unique within all of the AWS provider customers.

Terraforming AWS Configuration

PROTIP: Install from https://github.com/dtan4/terraforming a Ruby script that enables a command such as:

terraforming s3 --profile dev

You can pass profile name by –profile option.

Terraform Plan

Have Terrform evaluate based on vars in a different (parent) folder:

terraform plan -var-file=’..\terraform.tfvars’ -var-file=’.\Development\development.tfvars’ -state=’.\Development\dev.state’ -out base-date-+'%s'.plan

The two dots in the command specifies to look above the current folder.

The -out parameter outputs to a file name that begins with base- and ends with .plan. The date is like 147772345 which is the numer of seconds since 1/1/1970.

A sample response:
```
"<computered>" means Terraform figures it out.
```
Output

outputs.tf file
```
output "aws_elb_public_dns" {
  value = "${aws_elb.web.dns_name}"
}
output "public_ip" {
  value = "${aws_instance.example.public_ip}"
}
output "azure_rm_dns_cname" {
  value = "${azurerm_dns_cname_record.elb.id}"
}
```
PROTIP: If the AMI is no longer available, you will get an error message.

Terraform apply

terraform apply -state=”.\develop\dev.state” -var=”environment_name=development”
1. Generate model from logical definition (the Desired State).
2. Load current model (preliminary source data).
3. Refresh current state model by querying remote provider (final source state).
4. Calculate difference from source state to target state (plan).
5. Apply plan.
Ignore state files

Terraform apply generates .tfstate files (containing JSON) to persist the state of runs. It contains a maps resources IDs to their data.
View the .gitignore file:
```
terraform.tfstate*
*.tfstate
*.tfstate.backup
.terraform/
*.iml
*.plan
vpc
```
CAUTION: State files can contain secrets.

.terraform/ specifies that the folder is ignored when pushing to GitHub.

Terraform apply creates a dev.state.lock.info file as a way to signal to other processes to stay away while changes to the environment are underway.

tfstate.backup is created from the most recent previous execution before the current tfstate file contents.

Remote state

NOTE terraform.tfstate can be stored over the network in S3, etcd distributed key value store (used by Kubernetes), or a Hashicorp Atlas or Consul server.

Hashicorp Atlas is a licensed solution.

State can be obtained using command:

terraform remote pull
vars.tf file contains specifcation of values to variables, such as the server port (8080).
```
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
```

variable “server_port” { description = “The port the server will use for HTTP requests” default = 8080 } </pre>

outputs.tf file

output "public_ip" {
  value = "${aws_instance.example.public_ip}"
}

PROTIP: Make sure that the AWS region is what you want.

https://www.terraform.io/docs/providers/aws/r/instance.html AWS provider

NOTE: The contents of the repo were written based on Terraform 0.7.x.
PROTIP: If the AMI is no longer available, you will get an error message.
```
ami = "ami-2d39803a"
```
Terraform Plan
Have Terrform evaluate based on vars in a different (parent) folder:

terraform plan -var-file=’..\terraform.tfvars’

In the sample response:

Pluses and minuses flag additions and deletions. This is a key differentiator for Terraform as a “”

“&LT;computered>” means Terraform figures it out.

Terraform creates a dependency graph (specfically, a Directed Acyclic Graph). This is so that nodes are built in the order they are needed.

Version
Obtain version installed:

terraform version

Alternately:

terraform --version

WARNING: The response at time of writing, Terraform is not even “1.0” release, meaning it’s in beta maturity.:
```
Terraform v0.10.6
```
QUESTION: Pace of change in Terraform?

Release 0.6.8 (2.12.2015)

TODO: Update terraform

Commands list & help

For a list of commands:

terraform

The response at time of writing:

Usage: terraform [--version] [--help] <command> [args]
 
The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.
 
Common commands:
 apply              Builds or changes infrastructure
 console            Interactive console for Terraform interpolations
 destroy            Destroy Terraform-managed infrastructure
 env                Workspace management
 fmt                Rewrites config files to canonical format
 get                Download and install modules for the configuration
 graph              Create a visual graph of Terraform resources
 import             Import existing infrastructure into Terraform
 init               Initialize a Terraform working directory
 output             Read an output from a state file
 plan               Generate and show an execution plan
 providers          Prints a tree of the providers used in the configuration
 push               Upload this Terraform module to Atlas to run
 refresh            Update local state file against real resources
 show               Inspect Terraform state or plan
 taint              Manually mark a resource for recreation
 untaint            Manually unmark a resource as tainted
 validate           Validates the Terraform files
 version            Prints the Terraform version
 workspace          Workspace management
 
All other commands:
 debug              Debug output management (experimental)
 force-unlock       Manually unlock the terraform state
 state              Advanced state management

Help on a specific command:

terraform plan --help

Environment variables
Define enviornment variables:
```
TF_VAR_first_name=John terraform apply
```
Define Terraform variable:
```
terraform apply -var 'first_name=John' -var 'last_name=Bunyan'
```
Values to Terraform variables define inputs such as run-time DNS/IP addresses into Terraform modules.

NOTE: Built-in functions:

https://terraform.io/docs/configuration/interpolation.html

Apps to install

NOTE: Software can be specified for installation using Packer’s local-exec provisioner which has Terraform on host machines executes commands. For example, on a Ubuntu machine:
```
resource "null_resource" "local-software" {
  provisioner "local-exec" {
 command = <<EOH
sudo apt-get update
sudo apt-get install -y ansible
EOH
  }
}
```
NOTE: apt-get is in-built within Ubuntu Linux distributions.

PROTIP: Use this to bootstrap automation such as assigning permissions and running Ansible or PowerShell DSC, then use DSC scripts for more flexibility and easier debugging.

Output variables
Output Terraform variable:
```
output "loadbalancer_dns_name" {
  value = "${aws_elb.loadbalancer.dns_name}"
}
```
Processing flags

HCL can contain flags that affect processing. For example, within a resource specification, force_destroy = true forces the provider to delete the resource when done.

Verify websites
The website accessible?
In the provider’s console (EC2), verify

Destroy to clean up
Destroy instances so they don’t rack up charges unproductively:

terraform destroy

PROTIP: Amazon charges by the hour while others charge by the minute.
Verify in the provider’s console.

Terraform Console

On Windows, open the Terraform Console from a command Prompt rather than from within PowerShell.

terraform console

element(list(“one”,”two”,”three”),0,2)

The response is (because counting begins from zero):

three

element(list(“one”,”two”,”three”),0,2)

one, two

Automation

NOTE: Automating infrastructure deployment consists of:

Provisioning resources
Planning updates
Using source control
Reusing templates

PROTIP: Terrform files are “idempotent” (repeat runs don’t change anything if nothing is changed). Thus Terraform defines the “desired state configuration”.

NOTE: Terraform remote configures remote state storage with Terraform.

Modules

https://github.com/terraform-community-modules

https://github.com/objectpartners/tf-modules

module "rancher" {
  source = "github.com/objectpartners/tf-modules//rancher/server-standalone-elb-db&ref=9b2e590"
}

The double slashes separate the repo from the subdirectory.

The ref is a commit ID

Bootstrap Terraform

https://forge.puppet.com/inkblot/terraform

https://supermarket.chef.io/cookbooks/terraform

https://github.com/migibert/terraform-role Ansible role to install Terraform on Linux machines

https://github.com/hashicorp/docker-hub-images/tree/master/terraform builds Docker containers for using the terraform command line program.

Plugins into Terraform

All Terraform providers are plugins - multi-process RPC (Remote Procedure Calls).

https://github.com/hashicorp/terraform/plugin

https://terraform.io/docs/plugins/index.html

Terraform expect plugins to follow a very specific naming convention of terraform-TYPE-NAME. For example, terraform-provider-aws, which tells Terraform that the plugin is a provider that can be referenced as “aws”.

PROTIP: Establish a standard for where plugins are located:

For *nix systems, ~/.terraformrc

For Windows, %APPDATA%/terraform.rc

https://www.terraform.io/docs/internals/internal-plugins.html

PROTIP: When writing your own terraform plugin, create a new Go project in GitHub, then locally use a directory structure:

$GOPATH/src/github.com/USERNAME/terraform-NAME

where USERNAME is your GitHub username and NAME is the name of the plugin you’re developing. This structure is what Go expects and simplifies things down the road.

TODO:

Grafana or Kibana monitoring
PagerDuty alerts
DataDog metrics

Learning Resources

IRC
Google Group terraform-tool
StackOverflow
Official Getting Started docs at Hashicorp focus on individual elements (i.e. resources, input variables, output variables, etc).
Manage AWS infrastructure as code using Terraform talk in Norway 14 Dec 2015 by Anton Babenko https://github.com/antonbabenko linkedin.com/in/antonbabenko

Video courses at Pluralsight.com:

Terraform - Getting Started (Beginner level) Sep 14 2017 [3:11] by Ned Bellavance (@ned1313, nedinthecloud), MS MVP.
Automating AWS and vSphere with Terraform (Intermediate level) Jun 12 2017[ 1:22] by Nick Colyer

YouTube videos:

Terraform w/ Lee Trout Chadev
Automating Infrastructure Management with Terraform at SF CloudOps Meetup

Rock Stars

James Turnbull

The Terraform Book ($8 on Kindle) is based on Terraform v0.10.3. Files referenced are at:

https://github.com/turnbullpress/tfb-code

https://github.com/jason-azze/tf-web-exercise

James Nugent

Engineer at Hashicorp

Yevgeniy (Jim) Brikman (ybrikman.com), co-founder of DevOps as a Service Gruntwork.io :

Gruntwork’s Introduction
Terraform Up & Running (OReilly book $11.99 on Amazon)
terraformupandrunning.com
https://github.com/brikis98/terraform-up-and-running-code contains a Bash script that installs Apache, PHP, and a sample PHP app on an Ubuntu server. It also has automated tests written in Ruby script to make sure it returns “Hello, World”.

zero-downtime deployment, are hard to express in purely declarative terms.

Comprehensive Guide to Terraform includes:

dtan4

http://terraforming.dtan4.net/

https://github.com/dtan4/terraforming is a Ruby

AWS Cloud Formation

Puppet, Chef, Ansible, Salt AWS API libraries Boto, Fog

References

https://github.com/brikis98/infrastructure-as-code-talk Infrastructure-as-code: running microservices on AWS with Docker, ECS, and Terraform

https://www.youtube.com/channel/UCgWfCzNeAPmPq_1lRQ64JtQ/videos SignalWarrant’s videos on PowerShell by David Keith Hall includes:

Automate Creating Lab Virtual Machines in Azure with PowerShell July 12, 2017 by taking input from a CSV file.

Wilson Mar

Terraform from Hashicorp

Competitors to Terraform

Installation

Consider tfenv

Install on MacOS

Install on Windows

Get Sample Terraform scripts

Other scripts

Environment folders Validate

Plug-in Initialization

.tf files

Main.tf

Cloud Providers

Provisioners

Defaults and lookup function

CIDR Subnet function

Terraforming AWS Configuration

Terraform Plan

Output

Terraform apply

Ignore state files

Remote state

Terraform Plan

Version

Commands list & help

Environment variables

Apps to install

Output variables

Processing flags

Verify websites

Destroy to clean up

Terraform Console

Automation

Modules

Bootstrap Terraform

Plugins into Terraform

Learning Resources

Video courses at Pluralsight.com:

YouTube videos:

Rock Stars

AWS Cloud Formation

References

More on DevOps

You might also enjoy (View all posts)