Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

Get certified in Developer Operations on AWS

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This tutorial contains my notes about getting certified as a AWS Certified DevOps Engineer - Professional – able to setup and manage continuous integration and deployment in the AWS cloud – after paying $300 USD to write out 80 essay (not multiple-choice) questions in 170 minutes (3 hours with no breaks). That’s 2.1 seconds per essay question. Those who fail the exam must wait 30 days before being allowed to retake the exam (at additional cost), and only 3 times in a 12 month period.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

The previous pre-requisite is passing either one:

  • https://aws.amazon.com/certification/certified-sysops-admin-associate/
  • https://aws.amazon.com/certification/certified-developer-associate/

References

Building a CI/CD Pipeline

AWS Artifact competes with DockerHub, JFrog Artifactory, etc. to hold Docker containers and other artifacts built for loadeding into Kubernetes and other run-time systems.

From AWS YouTube playlist:

  • https://www.youtube.com/playlist?list=PLhr1KZpdzukeH9VMPbNHMCXl_NrVc1JGe
  • https://www.youtube.com/playlist?list=PLhr1KZpdzuke5pqzTvI2ZxwP8-NwLACuU
  • https://www.youtube.com/playlist?list=PLhr1KZpdzukeMbjRqGswHX38DCqOHZ5GA
  • https://www.youtube.com/playlist?list=PLhr1KZpdzukfVW6NrpDzdT6Sej0p5POkN

  • http://cantrill.io
  • http://ozaws.com
  • https://serverlesscode.com/
  • https://paulwakeford.info/
  • https://aws.amazon.com/blogs/aws/
  • https://www.awsarchitectureblog.com
  • http://blogs.aws.amazon.com/application-management
  • http://blogs.aws.amazon.com/security/
  • https://aws.amazon.com/blogs/compute/
  • https://aws.amazon.com/whitepapers/

https://aws.amazon.com/documentation/

  • https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html

  • AWS Certified DevOps Engineer - Professional 6-hour video by Nick Triantafillou covers:

    • Core Concepts
    • CI/CD Automation
    • Monitoring/Metrics/Logging
    • Security/Governance/Validation
    • High Availability and Elasticity
    • Operations

https://wilsonmar.github.io/build-load-balanced-servers-in-AWS-EC2/

Projects on AWS: Set Up a CI/CD Pipeline on AWS

From https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials.html

A pipeline helps you automate steps in your software delivery process, such as initiating automatic builds and then deploying to Amazon EC2 instances. You will use AWS CodePipeline, a service that builds, tests, and deploys your code every time there is a code change, based on the release process models you define. Use CodePipeline to orchestrate each step in your release process. As part of your setup, you will plug other AWS services into CodePipeline to complete your software delivery pipeline. This guide will show you how to create a very simple pipeline that pulls code from a source repository and automatically deploys it to an Amazon EC2 instance.


What is DevOps (DevSecOps)?

Amazon defines “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity.”

CodeCommit, CodeBuild, CodePipeline, CodeDeploy

aws devops 201606-650x209-i11

CodeCommit setup

aws-codecommit-left-menu-252x481-8146.jpg

AWS CodeCommit competes with GitHub, GitLab, BitBucket, and other cloud-based text code repositories.

Repositories in AWS CodeCommit have a URL that contains a region, such as:

https://git-codecommit.us-east-1.amazonaws.com/v1/repos/MyRepo

CodeBuild Setup

CodeDeploy Setup instances

  1. In IAM Service, create Role “codedeploy”.
  2. Create CDInstanceRole
  3. In Compute EC2 service, launch Amazon Linux, t2.micro, 2 instances, using the role created above. In Advanced Details, paste script from https://gist.github.com/mikepfeiffer/4d9386afdcceaf29493a

    EC2 UserData script to install CodeDeploy agent:

    #!/bin/bash
    yum install -y aws-cli
    cd /home/ec2-user/
    aws s3 cp 's3://aws-codedeploy-us-east-1/latest/codedeploy-agent.noarch.rpm' . --region us-east-1
    yum -y install codedeploy-agent.noarch.rpm
    

    CUSTOMIZE folder, region mentionedtwice.

  4. Tag instances with name “Dev” for Development.
  5. Add a Security Group Role for HTTP. No SSH.

AWS CodeDeploy Setup

  1. Got to AWS CodeDeploy service, Get Started Now.
  2. Custom deployment.
  3. Specify an Application Name and Deployment Group Name according to your organization’s naming standards.
  4. Select Tag Type “Amazon EC2” value “Dev” specified for 2 instances in a step above.
  5. Deployment Config - AllAtOnce (instead of Half at a time).
  6. No Triggers.
  7. Select a Service Role ARN defined in a prior step. Create Application.

    The AWS Console provides code to deploy from a S3 bucket.

AWS CodePipeline

  1. Go to AWS CodePipeline service, Get Started.
  2. Specify a Pipeline name according to your organization’s naming standards. (“Pipeline1”)
  3. Select Source Provider: GitHub (NOT Amazon S3). Click Connect to GitHub.
  4. Select a Repository and Branch from the GitHub account authenticated.
  5. Select Deployment provider AWS CodeDeploy (NOT AWS Elastic Beanstalk).
  6. Supply AWS CodeDeploy Application Name and Deployment group from earlier.
  7. Do not define Build Stage (until we have a build).
  8. Create Service Role using default name “AWS-CodePipeline-Service”. View Policy Document to review Actions allowed the role:
  9. Review Pipeline summary.
  10. Cleanup: Delete the pipeline you just created.

PROTIP: Each pipeline costs about $1 per month, and charges only if a deployment occurs.

View app deployed

  1. In EC2, copy the Public DNS address, such as:

    "ec2-11-222-177-132-us-west-2-compute.amazonaws.com
  2. Paste URL in an internet browser.

    It should respond with “Congratualations”.

Make Change

  1. Commit.
  2. Detect a change.
  3. View app deployed again.

  4. Create a Deployment Group or Autoscaling Group
  5. CodePipeline

  6. appspec.yml file in the root folder in source code repo

    version: 0.0
    os: linux
    files:
             - source: /index.html
     destination: /var/www/html/
    hooks:
      BeforeInstall:
     - location: scripts/install_dependencies
       timeout: 300
       runas: root
     - location: scripts/start_server
       timeout: 300
       runas: root
      ApplicationStop:
     - location: scripts/stop_server
       timeout: 300
       runas: root
    
  7. For sample application, it’s just a single index.html file containing CSS, no JavaScript.

    https://github.com/mikepfeiffer/aws-codedeploy-linux/blob/master/index.html

  8. Install dependencies

    #!/bin/bash
    yum install -y httpd
    
  9. Start server

    #!/bin/bash
    service httpd start
    
  10. Stop server:

    #!/bin/bash
    isExistApp = `pgrep httpd`
    if [[ -n  $isExistApp ]]; then
        service httpd stop        
    fi
    

using AWS Code Services

CodeDeploy agent in EC2 Deploy Group

  • Amazon Route 53 globomantics.com

  • https://github.com/mikepfeiffer/PowerShell

Snapshot AMIs

Building a server from installers in S3 can be time-consuming because it take so much I/O and network bandwidth.

Michael Tripoli & Karate Vick at Netflix open-sourced on GitHub their Python tool called “Animator” for creating EBS-backed AMIs. The tool create a Base AMI by taking a snapshot of the root volume and making it available as an EBS volume that can be used to launch an EC2 instance.

It’s described on Medium and in this blog from 2013.

They said “We knew that application startup latency would be very important, especially during scale-up operations.”

This is simpler than Packer from HashiCorp.

Learning Resources

Australian Nick Triantafillou (@xelfer) for $99 provides 6 hours of videos covering 60 lessons in his ACloud.guru course.

Mike Pfeiffer created at Pluralsight a series of video courses, one for each “domain” of the AWS Certified DevOps Engineer Professional exam:

  1. Continuous Delivery and Process Automation 4h 16m Apr 28, 2016

  2. Monitoring, Metrics, and Logging 2h 46m June 2, 2016

    • CloudWatch
    • CloudTrail

  3. Security, Governance, and Validation 2h 11m July 26, 2016

    • Delegation & Federation
    • Corporate Identity Federation
    • Web Identity Federation

  4. High Availability and Elasticity 2h 51m Sept 26, 2016

    • Auto Scaling
    • GSI/LSI
    • RDS
    • Dynamo
    • Aurora
    • SQS
    • Kinesis

Justin Manga (@jmenga, pseudo.co.de) in Pluralsight video course Continuous Delivery using Docker and Ansible shows learners how to create a Python app, then Jenkins 2 Pipeline plugin in a container using Cloud Formation files. deploy to a EC2 Container Services holding Debian instances.

Since ECS has Group of 497:

ARG DOCKER_GID=497
RUN groupadd -g ${DOCKER_GID:-497} docker
ARG DOCKER_ENGINE=1.10.2
ARG DOCKER_COMPOSE=1.6.2
   
  • http://blog.serverbooter.com/blog/2013/10/24/simple-clouformation-with-multiple-aws-accounts/

Boto Python

The README page says “Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2.”

https://github.com/boto/boto3

NOTE: Boto 3 is built on top of Botocore used by the AWS CLI:

  • Sessions
  • Clients: low level service connections

Boto 3 consists of these major features:

  • Resources: a high level, object oriented interface
  • Collections: a tool to iterate and manipulate groups of resources
  • Paginators: automatic paging of responses
  • Waiters: a way to block until a certain state has been reached

References

Amazon’s own DevOps Engineering on AWS 3-day classroom course covers:

  • Use the principal concepts and practices behind the DevOps methodology
  • Design and implement an infrastructure on AWS that supports one or more DevOps development projects
  • Use AWS CloudFormation and AWS OpsWorks to deploy the infrastructure necessary to create development, test, and production environments for a software development project
  • Use AWS CodeCommit and AWS CodeBuild to understand the array of options for enabling a continuous integration (CI) environment on AWS
  • Use AWS CodePipeline to design and implement a continuous integration and continuous delivery (CI/CD) pipeline on AWS
  • Use AWS CodeStar to manage all software development activities in one place
  • Implement several common continuous deployment (CD) use cases using AWS technologies, including blue/green deployment and A/B testing

  • Distinguish between the array of application deployment technologies available on AWS, including AWS CodeDeploy, AWS OpsWorks, AWS Elastic Beanstalk, Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Container Registry (Amazon ECR), and decide which technology best fits a given scenario
  • Use Amazon EC2 Systems Manager for patch management
  • Leverage automated testing in different stages of a CI/CD pipeline
  • Fine-tune the applications you deliver on AWS for high performance, and use AWS tools and technologies to monitor your application and environment for potential issues

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators
  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling
  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft
  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering

More on Amazon

This is one of a series on Amazon: