Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Instagram Youtube

Github Stackoverflow Pinterest

Get certified in Developer Operations on AWS


You are at https://wilsonmar.github.io/aws-devops/

This tutorial describes information to get certified as a AWS Certified DevOps Engineer - Professional, which focuses on continuous integration and deployment in the AWS EC2 cloud. It costs $300 USD to write out 80 essay (not multiple-choice) questions in 170 minutes (with no breaks). That’s 2.1 seconds per question. Those who fail the exam must wait 30 days before being allowed to retake the exam, and only 3 times in a 12 month period.

Its pre-requisite is passing either one:

  • https://aws.amazon.com/certification/certified-sysops-admin-associate/
  • https://aws.amazon.com/certification/certified-developer-associate/

Amazon defines “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity.”

CodeCommit, CodePipeline, Code Deploy

aws devops 201606-650x209-i11

Setup instances

  1. In IAM Service, create Role “codedeploy”.

  2. Create CDInstanceRole

  3. In Compute EC2 service, launch Amazon Linux, t2.micro, 2 instances, using the role created above. In Advanced Details, paste script from https://gist.github.com/mikepfeiffer/4d9386afdcceaf29493a

    EC2 UserData script to install CodeDeploy agent:

    yum install -y aws-cli
    cd /home/ec2-user/
    aws s3 cp 's3://aws-codedeploy-us-east-1/latest/codedeploy-agent.noarch.rpm' . --region us-east-1
    yum -y install codedeploy-agent.noarch.rpm

    CUSTOMIZE folder, region mentionedtwice.

  4. Tag instances with name “Dev” for Development.
  5. Add a Security Group Role for HTTP. No SSH.

AWS CodeDeploy Setup

  1. Got to AWS CodeDeploy service, Get Started Now.
  2. Custom deployment.
  3. Specify an Application Name and Deployment Group Name according to your organization’s naming standards.
  4. Select Tag Type “Amazon EC2” value “Dev” specified for 2 instances in a step above.
  5. Deployment Config - AllAtOnce (instead of Half at a time).
  6. No Triggers.
  7. Select a Service Role ARN defined in a prior step. Create Application.

    The Console provides code to deploy from a S3 bucket.

AWS CodePipeline

  1. Go to AWS CodePipeline service, Get Started.
  2. Specify a Pipeline name according to your organization’s naming standards. (“Pipeline1”)
  3. Select Source Provider: GitHub (NOT Amazon S3). Click Connect to GitHub.
  4. Select a Repository and Branch from the GitHub account authenticated.
  5. Select Deployment provider AWS CodeDeploy (NOT AWS Elastic Beanstalk).
  6. Supply AWS CodeDeploy Application Name and Deployment group from earlier.
  7. Do not define Build Stage (until we have a build).
  8. Create Service Role using default name “AWS-CodePipeline-Service”. View Policy Document to review Actions allowed the role:
  9. Review Pipeline summary.

View app deployed

  1. In EC2, get the Public DNS address (such as “ec2-11-222-177-132-us-west-2-compute.amazonaws.com”).
  2. Paste URL in an internet browser.

    It should respond with “Congratualations”.

Make Change

  1. Commit.
  2. Detect a change.
  3. View app deployed again.

  4. Create a Deployment Group or Autoscaling Group
  5. CodePipeline

  6. appspec.yml file in the root folder in source code repo

    version: 0.0
    os: linux
             - source: /index.html
     destination: /var/www/html/
     - location: scripts/install_dependencies
       timeout: 300
       runas: root
     - location: scripts/start_server
       timeout: 300
       runas: root
     - location: scripts/stop_server
       timeout: 300
       runas: root
  7. For sample application, it’s just a single index.html file containing CSS, no JavaScript.


  8. Install dependencies

    yum install -y httpd
  9. Start server

    service httpd start
  10. Stop server:

    isExistApp = `pgrep httpd`
    if [[ -n  $isExistApp ]]; then
        service httpd stop        

using AWS Code Services

CodeDeploy agent in EC2 Deploy Group

  • Amazon Route 53 globomantics.com

  • https://github.com/mikepfeiffer/PowerShell

Snapshot AMIs

Building a server from installers in S3 can be time-consuming because it take so much I/O and network bandwidth.

Michael Tripoli & Karate Vick at Netflix open-sourced on GitHub their Python tool called “Animator” for creating EBS-backed AMIs. The tool create a Base AMI by taking a snapshot of the root volume and making it available as an EBS volume that can be used to launch an EC2 instance.

It’s described on Medium and in this blog from 2013.

They said “We knew that application startup latency would be very important, especially during scale-up operations.”

This is simpler than Packer from Hashicorp.

Learning Resources

Australian Nick Triantafillou (@xelfer) recorded 6 hours of videos that covers 60 lessons in his course which can be purchased separately for $99.

Mike Pfeiffer created at Pluralsight a series of video courses, one for each “domain” of the AWS Certified DevOps Engineer Professional exam:

  1. Continuous Delivery and Process Automation 4h 16m Apr 28, 2016

  2. Monitoring, Metrics, and Logging 2h 46m June 2, 2016

    • CloudWatch
    • CloudTrail
  3. Security, Governance, and Validation 2h 11m July 26, 2016

    • Delegation & Federation
    • Corporate Identity Federation
    • Web Identity Federation
  4. High Availability and Elasticity 2h 51m Sept 26, 2016

    • Auto Scaling
    • GSI/LSI
    • RDS
    • Dynamo
    • Aurora
    • SQS
    • Kinesis

Justin Manga (@jmenga, pseudo.co.de) in video course Continuous Delivery using Docker and Ansible shows learners how to create a Python app, then Jenkins 2 Pipeline plugin in a container using Cloud Formation files. deploy to a EC2 Container Services holding Debian instances.

Since ECS has Group of 497:

RUN groupadd -g ${DOCKER_GID:-497} docker
  • http://blog.serverbooter.com/blog/2013/10/24/simple-clouformation-with-multiple-aws-accounts/

Boto Python

https://github.com/boto/boto3 The README page says “Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2.”

NOTE: Boto 3 is built on top of Botocore used by the AWS CLI:

  • Sessions
  • Clients: low level service connections

Boto 3 consists of these major features:

  • Resources: a high level, object oriented interface
  • Collections: a tool to iterate and manipulate groups of resources
  • Paginators: automatic paging of responses
  • Waiters: a way to block until a certain state has been reached

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud regions
  16. AWS Virtual Private Cloud
  17. Azure Cloud Onramp
  18. Azure Cloud
  19. Azure Cloud Powershell
  20. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  21. Digital Ocean
  22. Cloud Foundry

  23. Packer automation to build Vagrant images
  24. Terraform multi-cloud provisioning automation

  25. Powershell Ecosystem
  26. Powershell on MacOS
  27. Powershell Desired System Configuration

  28. Jenkins Server Setup
  29. Jenkins Plug-ins
  30. Jenkins Freestyle jobs
  31. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  32. Dockerize apps
  33. Docker Setup
  34. Docker Build

  35. Maven on MacOSX

  36. Ansible

  37. MySQL Setup

  38. SonarQube static code scan

  39. API Management Microsoft
  40. API Management Amazon

  41. Scenarios for load

More on Amazon

This is one of a series on Amazon: