to animate Templates in StackSets of Stacks in AWS
Overview
The object of this tutorial is to succinctly present step-by-step instructions to use Cloud Formation to automate work within AWS (instead of AWS GUI Console).
NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.
- IaC (Infrastructure as Code)
-
Templates
- IAM
- Security Groups
- Helper Scripts in Python
-
Drift and Nested Stacks
- Resource Attibutes - e.g., Deletion Policy
- Intrinsic Functions - Built-in functions (Fn::Base64, Ref, Sub, etc.)
- Intrinsic Conditional Functions (And, Equals, Id, Not, Or)
- Pseudo Parameters - Predefined parameters (
AWS Account and region
- Log into a AWS account (LinuxAcademy)
-
Select region N. Virginia (
us-east-1
).Create a Key Pair
- In the AWS Management Console dashboard.
- Find Services: “EC2” (“Virtual servers in the cloud”).
- Click Key Pairs in the left sidebar under “NETWORK & SECURITY”.
- Click “Create Key Pair” (blue button).
-
Name the key pair “helperdemo”. Click Create for a pop-up dialog containing the key text.
Define IAM role
CloudFormation needs permissions to do each of the services defined in its templates.
Cloud Formation Console
-
Use this URL to get in
https://console.aws.amazon.com/cloudformation/home
Notice Amazon automatically adds your default region.
https://console.aws.amazon.com/cloudformation/home?region=us-east-1
Menu
On the left menu are the major organizing “objects”:
Stacks are a collection of resources managed by the AWS service, which is aware of events.
CloudFormation handles Stack creation and update as well as error detection and rollback.
StackSets are a collection of stacks.
Change Sets
Designer is a graphic tool for creating, viewing, and modifying AWS CloudFormation templates. Docs
https://console.aws.amazon.com/cloudformation/designer
- Canvas Pane: Displays template resources as diagram.
- Resource types pane: Lists all of the template resources that you can add to your template.
- JSON Editor: Here you specify the details of your template, such as resource properties or template parameters
- Errors Pane: Errors pane displays validation errors.
- Full screen and Split screen buttons: Buttons to select different views of Designer.
- Fit to window button: to resize the canvas pane to fit your template’s diagram.
- Toolbar: Provides quick access to commands for common actions, such as opening and closing/saving templates.
Exports
-
Click on Stacks in the menu.
-
Click “CREATE STACK”.
If you select “Use a sample template” from AWS (LAMP, Ruby on Rails, WorkPress) for single and multi-AZ configurations.
If you select “Create template in Designer”, you can drag and drop the ever-expanding list of resources from the left menu.
Template files are stored in AWS S3. So PROTIP: create a bucket hierarchy ahead.
PROTIP: Define Tags.
Stack policy
Rollback configuration (CloudWath alarm ARN)
Notification options
Stack creation options
IaC (Infrastructure as Code)
VIDEO: #19 DevOpsPlayground Hands On with AWS CloudFormation Apr 16, 2018 [1:05:26] by Sunil Tailor of https://devopsplayground.co.uk/ references https://github.com/DevOpsPlayground/Hands-on-with-AWS-Cloudformation
- Lab-001 - Create a CloudFormation Stack to Provision a S3 bucket Resource Goal 2: with Change Set
- Lab-002 - Updating Stacks with Security Groups
- Lab-003 - Provisioning Resources with cnf-init and Userdata
Terraform from Hashicorp is multi-vendor.
Templates
Sections of a Template
- Description
- Parameters [In sample]
- Mappings [In sample]
- Metadata (Properties) [In sample]
- Conditions [In sample???]
- Transform ???
- Outputs [In sample]
Below is a rather full (production-worthy?) sample template from here
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Create a LAMP stack using a single EC2 instance and a local MySQL database for storage. This template demonstrates using the AWS CloudFormation bootstrap scripts to install the packages and files necessary to deploy the Apache web server, PHP and MySQL at instance launch time.", "Parameters" : { "KeyName": { "Description" : "Name of an EC2 KeyPair to enable SSH access to the instance", "Type": "AWS::EC2::KeyPair::KeyName", "ConstraintDescription" : "must be the name of an existing EC2 KeyPair." }, "DBName": { "Default": "DatabaseNameExample", "Description" : "MySQL database name", "Type": "String", "MinLength": "1", "MaxLength": "64", "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*", "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters." }, "DBUser": { "NoEcho": "true", "Description" : "Username for MySQL database access", "Type": "String", "MinLength": "1", "MaxLength": "16", "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*", "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters." }, "DBPassword": { "NoEcho": "true", "Description" : "Password for MySQL database access", "Type": "String", "MinLength": "1", "MaxLength": "41", "AllowedPattern" : "[a-zA-Z0-9]*", "ConstraintDescription" : "must contain only alphanumeric characters." }, "DBRootPassword": { "NoEcho": "true", "Description" : "Root password for MySQL", "Type": "String", "MinLength": "1", "MaxLength": "41", "AllowedPattern" : "[a-zA-Z0-9]*", "ConstraintDescription" : "must contain only alphanumeric characters." }, "InstanceType" : { "Description" : "WebServer EC2 instance type", "Type" : "String", "Default" : "t2.micro", "AllowedValues" : ["t2.nano", "t2.micro", "t2.small", "t2.medium"], "ConstraintDescription" : "must be a valid EC2 instance type." }, "SSHLocation" : { "Description" : "The IP address range that can be used to SSH to the EC2 instances", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } }, "Mappings" : { "AWSInstanceType2Arch" : { "t2.nano" : { "Arch" : "HVM64" }, "t2.micro" : { "Arch" : "HVM64" }, "t2.small" : { "Arch" : "HVM64" }, "t2.medium" : { "Arch" : "HVM64" } }, "AWSRegionArch2AMI" : { "us-east-1" : {"PV64" : "ami-2a69aa47", "HVM64" : "ami-6869aa05", "HVMG2" : "ami-2e5e9c43"}, "us-west-2" : {"PV64" : "ami-7f77b31f", "HVM64" : "ami-7172b611", "HVMG2" : "ami-83b770e3"} }, "SubnetConfig" : { "VPC" : { "CIDR" : "10.0.0.0/16" }, "Public" : { "CIDR" : "10.0.0.0/24" } } }, "Resources" : { "WebServerInstance": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "configSets" : { "InstallAndRun" : [ "Install", "Configure" ] }, "Install" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "mysql-libs" : [], "httpd" : [], "php" : [], "php-mysql" : [], "git" : [] } }, "files" : { "/var/www/html/index.php" : { "content" : { "Fn::Join" : [ "", [ "\n", " \n", "AWS CloudFormation PHP Sample \n", " <meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\n", " \n", " \n", " <h1>Welcome to the AWS CloudFormation PHP Sample</h1>\n", " <p/>\n", " <?php\n", " // Print out the current data and time\n", " print \"The Current Date and Time is: <br/>\";\n", " print date(\"g:i A l, F j Y.\");\n", " ?>\n", " <p/>\n", " <?php\n", " $Database = \"localhost\";\n", " $DBUser = \"", {"Ref" : "DBUser"}, "\";\n", " $DBPassword = \"", {"Ref" : "DBPassword"}, "\";\n", " print \"Database = \" . $Database . \"
\";\n", " $dbconnection = mysql_connect($Database, $DBUser, $DBPassword)\n", " or die(\"Could not connect: \" . mysql_error());\n", " print (\"Connected to $Database successfully\");\n", " mysql_close($dbconnection);\n", " ?>\n", " \n", "\n" ]]}, "mode" : "000600", "owner" : "apache", "group" : "apache" }, "/tmp/setup.mysql" : { "content" : { "Fn::Join" : ["", [ "CREATE DATABASE ", { "Ref" : "DBName" }, ";\n", "GRANT ALL ON ", { "Ref" : "DBName" }, ".* TO '", { "Ref" : "DBUser" }, "'@localhost IDENTIFIED BY '", { "Ref" : "DBPassword" }, "';\n" ]]}, "mode" : "000400", "owner" : "root", "group" : "root" }, "/etc/cfn/cfn-hup.conf" : { "content" : { "Fn::Join" : ["", [ "[main]\n", "stack=", { "Ref" : "AWS::StackId" }, "\n", "region=", { "Ref" : "AWS::Region" }, "\n", "interval=6", "\n", ]]}, "mode" : "000400", "owner" : "root", "group" : "root" }, "/etc/cfn/hooks.d/cfn-auto-reloader.conf" : { "content": { "Fn::Join" : ["", [ "[cfn-auto-reloader-hook]\n", "triggers=post.update\n", "path=Resources.WebServerInstance.Metadata.AWS::CloudFormation::Init\n", "action=/opt/aws/bin/cfn-init -v ", " --stack ", { "Ref" : "AWS::StackName" }, " --resource WebServerInstance ", " --configsets InstallAndRun ", " --region ", { "Ref" : "AWS::Region" }, "\n", "runas=root\n" ]]} } }, "services" : { "sysvinit" : { "mysqld" : { "enabled" : "true", "ensureRunning" : "true" }, "httpd" : { "enabled" : "true", "ensureRunning" : "true" }, "cfn-hup" : { "enabled" : "true", "ensureRunning" : "true", "files" : ["/etc/cfn/cfn-hup.conf", "/etc/cfn/hooks.d/cfn-auto-reloader.conf"]} } } }, "Configure" : { "commands" : { "01_set_mysql_root_password" : { "command" : { "Fn::Join" : ["", ["mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "'"]]}, "test" : { "Fn::Join" : ["", ["$(mysql ", { "Ref" : "DBName" }, " -u root --password='", { "Ref" : "DBRootPassword" }, "' >/dev/null 2>&1 </dev/null); (( $? != 0 ))"]]} }, "02_create_database" : { "command" : { "Fn::Join" : ["", ["mysql -u root --password='", { "Ref" : "DBRootPassword" }, "' < /tmp/setup.mysql"]]}, "test" : { "Fn::Join" : ["", ["$(mysql ", { "Ref" : "DBName" }, " -u root --password='", { "Ref" : "DBRootPassword" }, "' >/dev/null 2>&1 </dev/null); (( $? != 0 ))"]]} } } } } }, "Properties": { "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] }, "InstanceType" : { "Ref" : "InstanceType" }, "KeyName" : { "Ref" : "KeyName" }, "NetworkInterfaces" : [{ "GroupSet" : [{ "Ref" : "EC2SecurityGroup" }], "PrivateIpAddress" : "10.0.0.100", "AssociatePublicIpAddress" : "true", "DeviceIndex" : "0", "DeleteOnTermination" : "true", "SubnetId" : { "Ref" : "PublicSubnet" } }], "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash -xe\n", "yum update -y aws-cfn-bootstrap\n", "# Install the files and packages from the metadata\n", "/opt/aws/bin/cfn-init -v ", " --stack ", { "Ref" : "AWS::StackName" }, " --resource WebServerInstance ", " --configsets InstallAndRun ", " --region ", { "Ref" : "AWS::Region" }, "\n", "# Signal the status from cfn-init\n", "/opt/aws/bin/cfn-signal -e $? ", " --stack ", { "Ref" : "AWS::StackName" }, " --resource WebServerInstance ", " --region ", { "Ref" : "AWS::Region" }, "\n" ]]}}, }, "CreationPolicy" : { "ResourceSignal" : { "Timeout" : "PT5M" } } }, "VPC" : { "Type" : "AWS::EC2::VPC", "Properties" : { "EnableDnsSupport" : "true", "EnableDnsHostnames" : "true", "CidrBlock" : { "Fn::FindInMap" : [ "SubnetConfig", "VPC", "CIDR" ]}, "Tags" : [ { "Key" : "Application", "Value" : { "Ref" : "AWS::StackName" } }, { "Key" : "Network", "Value" : "Public" } ] } }, "PublicSubnet" : { "Type" : "AWS::EC2::Subnet", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "AvailabilityZone": { "Fn::Select": [ "0", { "Fn::GetAZs": "" } ] }, "CidrBlock" : { "Fn::FindInMap" : [ "SubnetConfig", "Public", "CIDR" ]}, "Tags" : [ { "Key" : "Application", "Value" : { "Ref" : "AWS::StackName" } }, { "Key" : "Network", "Value" : "Public" } ] } }, "InternetGateway" : { "Type" : "AWS::EC2::InternetGateway", "Properties" : { "Tags" : [ { "Key" : "Application", "Value" : { "Ref" : "AWS::StackName" } }, { "Key" : "Network", "Value" : "Public" } ] } }, "GatewayToInternet" : { "Type" : "AWS::EC2::VPCGatewayAttachment", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "InternetGatewayId" : { "Ref" : "InternetGateway" } } }, "PublicRouteTable" : { "Type" : "AWS::EC2::RouteTable", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "Tags" : [ { "Key" : "Application", "Value" : { "Ref" : "AWS::StackName" } }, { "Key" : "Network", "Value" : "Public" } ] } }, "PublicRoute" : { "Type" : "AWS::EC2::Route", "DependsOn" : "GatewayToInternet", "Properties" : { "RouteTableId" : { "Ref" : "PublicRouteTable" }, "DestinationCidrBlock" : "0.0.0.0/0", "GatewayId" : { "Ref" : "InternetGateway" } } }, "PublicSubnetRouteTableAssociation" : { "Type" : "AWS::EC2::SubnetRouteTableAssociation", "Properties" : { "SubnetId" : { "Ref" : "PublicSubnet" }, "RouteTableId" : { "Ref" : "PublicRouteTable" } } }, "PublicNetworkAcl" : { "Type" : "AWS::EC2::NetworkAcl", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "Tags" : [ { "Key" : "Application", "Value" : { "Ref" : "AWS::StackName" } }, { "Key" : "Network", "Value" : "Public" } ] } }, "InboundHTTPPublicNetworkAclEntry" : { "Type" : "AWS::EC2::NetworkAclEntry", "Properties" : { "NetworkAclId" : { "Ref" : "PublicNetworkAcl" }, "RuleNumber" : "100", "Protocol" : "6", "RuleAction" : "allow", "Egress" : "false", "CidrBlock" : "0.0.0.0/0", "PortRange" : { "From" : "80", "To" : "80" } } }, "InboundHTTPSPublicNetworkAclEntry" : { "Type" : "AWS::EC2::NetworkAclEntry", "Properties" : { "NetworkAclId" : { "Ref" : "PublicNetworkAcl" }, "RuleNumber" : "101", "Protocol" : "6", "RuleAction" : "allow", "Egress" : "false", "CidrBlock" : "0.0.0.0/0", "PortRange" : { "From" : "443", "To" : "443" } } }, "InboundSSHPublicNetworkAclEntry" : { "Type" : "AWS::EC2::NetworkAclEntry", "Properties" : { "NetworkAclId" : { "Ref" : "PublicNetworkAcl" }, "RuleNumber" : "102", "Protocol" : "6", "RuleAction" : "allow", "Egress" : "false", "CidrBlock" : "0.0.0.0/0", "PortRange" : { "From" : "22", "To" : "22" } } }, "InboundEmphemeralPublicNetworkAclEntry" : { "Type" : "AWS::EC2::NetworkAclEntry", "Properties" : { "NetworkAclId" : { "Ref" : "PublicNetworkAcl" }, "RuleNumber" : "103", "Protocol" : "6", "RuleAction" : "allow", "Egress" : "false", "CidrBlock" : "0.0.0.0/0", "PortRange" : { "From" : "1024", "To" : "65535" } } }, "OutboundPublicNetworkAclEntry" : { "Type" : "AWS::EC2::NetworkAclEntry", "Properties" : { "NetworkAclId" : { "Ref" : "PublicNetworkAcl" }, "RuleNumber" : "100", "Protocol" : "6", "RuleAction" : "allow", "Egress" : "true", "CidrBlock" : "0.0.0.0/0", "PortRange" : { "From" : "0", "To" : "65535" } } }, "PublicSubnetNetworkAclAssociation" : { "Type" : "AWS::EC2::SubnetNetworkAclAssociation", "Properties" : { "SubnetId" : { "Ref" : "PublicSubnet" }, "NetworkAclId" : { "Ref" : "PublicNetworkAcl" } } }, "EC2SecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Enable access to the EC2 host", "VpcId" : { "Ref" : "VPC" }, "SecurityGroupIngress" : [ { "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0" }, { "IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0" }, { "IpProtocol" : "tcp", "FromPort" : "443", "ToPort" : "443", "CidrIp" : "0.0.0.0/0" }, { "IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1", "CidrIp" : "0.0.0.0/0" } ] } }, "SGBaseIngress": { "Type": "AWS::EC2::SecurityGroupIngress", "Properties": { "GroupId": { "Ref": "EC2SecurityGroup" }, "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "SourceSecurityGroupId": { "Ref": "EC2SecurityGroup" } } } }, [ "Outputs" : { "WebsiteURL" : { "Description" : "URL for newly created LAMP stack", "Value" : { "Fn::Join" : ["", ["http://", { "Fn::GetAtt" : [ "WebServerInstance", "PublicDnsName" ]}]] } } } }
Jason vs. Yaml
ECMA-404 JSON standard and YAML version 1.1, excluding aliases, hash merges, and (binary, omap, pairs, sets, and timestam) tags
https://www.json2yaml.com ONLINE TOOL
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/working-with-templates-cfn-designer-json-editor.html?icmpid=docs_cfn_console_designer Components view
Stacks
stacker is a tool and library used to create & update multiple CloudFormation stacks. It was originally written at Remind and released to the open source community.
-
Install AWS CLI under ~/.aw </pre>
http://docs.aws.amazon.com/cli/latest/userguide/installing.html
--upgrade
option tells pip to upgrade any requirements that are already installed.--user
option tells pip to install the program to a subdirectory of your user directory to avoid modifying libraries used by your operating system.
Templates
$template = “cpv-1-1.yaml”
aws cloudformation validate-template –template-body “file://cpv-1-1.yaml”
http://docs.aws.amazon.com/cli/latest/reference/cloudformation/validate-template.html
Helper Scripts in Python
AWS CloudFormation provides Python helper scripts to install software and start services on an Amazon EC2 instance that you create as part of your stack. They reside (by default) in folder /opt/aws/bin/
-
cfn-init retrieves and interprets resource metadata, install packages, create files, and start services.
-
cfn-signal signals a CreationPolicy or WaitCondition, so you can synchronize other resources in the stack when the prerequisite resource or application is ready.
-
cfn-get-metadata retrieves metadata for a resource or path to a specific key.
-
cfn-hup checks for updates to metadata and execute custom hooks when changes are detected.
this hands-on lab uses CloudFormation helper scripts from here to manage the provisioning of LAMP stack within EC2 instances in Auto Scaling Groups.
### Design a CloudFormation Stack
</a>
- Click “Services” at the top menu to Find Services: “CloudFormation” (“Create and Manage Services with Templates”).
-
Select “Designer” on the left (previously “Design Template” button).
https://console.aws.amazon.com/cloudformation/designer/home?region=us-east-1
- Switch to copy the CloudFormation helper script (445 lines) from below.
- In “new.template”, select to overwrite all existing (including “AWSTemplateFormatVersion” required on all templates) and paste from Clipboard.
- Next, navigate to the CloudFormation service in the AWS Management Console.
-
Click the checkbox icon to validate.
Error messages, if any, appear on the lower-right corner.
- Click the cloud icon with the up arrow to “Create stack”.
- Specify the Stack Name “HelperDemo”.
-
Generate a password such as “KCsLsVmJoofkgVXdkHgFs3Jhokv” for DBPassword.
PROTIP: DBPassword must contain only alphanumeric characters. (No special characters)
- For DBUser “helperdemo”.
- Click on empty KeyName to select “helperdemo” created earlier.
- Leave all of the default settings on the Select Template page, and click Next.
- On the Specify Details page, name the stack “HelperDemo”.
- Enter the password from MySQL for DBPassword and DBRootPassword.
- For KeyName, select helperdemo from the dropdown.
- Click Next, then click Next again on the Options page.
-
Scroll down to “Create Stack”.
List of stacks
- Refresh the page, and select the HelperDemo stack from the list.
- Scroll through the Events tab at the bottom of the screen to monitor the stack creation process.
- Click the Events tab, and change it to Outputs.
- Right-click the URL next to WebsiteURL, and open the link in a new tab.
Resources
https://aws.amazon.com/cloudformation/faqs/
https://www.udemy.com/join/login-popup/?next=/aws-cloudformation-master-class/learn/v4/overview
AWS CloudFormation Master Class on Udemy by Stephane Maarek [March 2019 Update]: Added Drift and Nested Stacks
More on DevOps
This is one of a series on DevOps:
- DevOps_2.0
- ci-cd (Continuous Integration and Continuous Delivery)
- User Stories for DevOps
- Git and GitHub vs File Archival
- Git Commands and Statuses
- Git Commit, Tag, Push
- Git Utilities
- Data Security GitHub
- GitHub API
- Choices for DevOps Technologies
- Pulumi Infrastructure as Code (IaC)
- Java DevOps Workflow
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- AWS server deployment options
- Cloud services comparisons (across vendors)
- Cloud regions (across vendors)
- Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
- Azure Certifications
- Azure Cloud Powershell
- Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
- Azure Networking
- Azure Storage
- Azure Compute
- Digital Ocean
- Packer automation to build Vagrant images
- Terraform multi-cloud provisioning automation
-
Hashicorp Vault and Consul to generate and hold secrets
- Powershell Ecosystem
- Powershell on MacOS
- Jenkins Server Setup
- Jenkins Plug-ins
- Jenkins Freestyle jobs
- Docker (Glossary, Ecosystem, Certification)
- Make Makefile for Docker
- Docker Setup and run Bash shell script
- Bash coding
- Docker Setup
- Dockerize apps
- Ansible
- Kubernetes Operators
- Threat Modeling
- API Management Microsoft
- Scenarios for load
- Chaos Engineering