Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Featuring a Bash shell script that does everything, explained

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This is a hands-on deep dive into the coding of various assets in an end-to-end DevOps workflow. Examples are modified from Rob van der Leek’s Apr 9, 2017 Medium article and “buzz phrase generator” in his “cicd-buzz” open-source repo.

Here we first work backwards, leveraging the outcome of Robert’s work (to make sure that it’s not vaporware ;).

        Production usage: the buzz phrase

  1. Manually use an internet browser to visit the web page generated from within Heroku (under Robert’s account):

    https://fathomless-inlet-53225.herokuapp.com

    If the site is still alive, you should see a random phrase generated, such as:

    “Complete Continuous Improvement Enormously Boosts Continuous Integration”

    Below is a break-down of each sub-phrase above:

    “Complete” is an adjective that also includes
    ‘modern’, ‘self-service’, ‘integrated’, ‘end-to-end’

    “Continuous Improvement” is a buzz word that also includes
    ‘continuous testing’, ‘continuous integration’, ‘continuous deployment’, ‘devops’

    “Enormously” is an adverb that also includes
    ‘remarkably’, ‘substantially’, ‘significantly’, ‘seriously’

    “Boosts” is a verb that also includes
    ‘accelerates’, ‘improves’, ‘enhances’, ‘revamps’

    “Continuous Integration” is another buzz word.

    TODO: If you want to change these, edit your copy of the program code and build another Docker image.

    generator.py coding

  2. Click the URL below to view the sample (“trivial”) Python program code that generates the buzz phrases above:

    https://github.com/robvanderleek/cicd-buzz/blob/master/buzz/generator.py

    Below is an example of how one would explain the program during a live walkthrough:

    The program begins at bottom of the code with the print(generate_buzz()) command under the if __name__ == "__main__": entry point.

    The generate_buzz() function returns out the program the phrase variable after using the Python built-in title() method that capitalizes the first character of each word.

    The value for the phrase variable is obtained by joining together a samples within arrays of adjectives, buzz (terms), adverbs, and verbs.

    The variable list_ in the signature of the sample function stands in for the variables specified in the join function.

    The sample function returns the output fromrandom.sample because it is an inbuilt function brought in via the module random specified by the import statement at the top of the code file.

    Run the Python program

  3. Open a Text Editor program. Navigate into the “buzz” folder to edit “generator.py”.
  4. Open a Terminal on your Mac and navigate into the buzz folder.
  5. Run the generator.py program using the Python interpreter (either version 2 or 3 should work).

    python generator.py

    NOTE: I would prefer to use from random import sample because import random brings in the whole module, which this custom code doesn’t use. However, running it results in:

    NameError: global name 'random' is not defined

    The variable n is a commonly used name for a temporary variable containing the limit. It is defined in the function’s signature specification.

    Within the if statement, the return clause is indented because that’s Python.

  6. Repeat the call and another set of values should appear.

    Local use of Docker

    I created the run.sh script so that you have an example of a Bash script to run locally.

  7. Open a Terminal on your Mac.
  8. Install Docker if you haven’t already.
  9. Make a containing folder to add the repository. I personally use ~/gits/wilsonmar.
  10. A Git clone command creates the “cicd-buzz” repo folder:

    git clone https://github.com/wilsonmar/cicd-buzz
    cd cicd-buzz
    ls

    My version of the repo contains a shell file named “run.sh”.

  11. Run the run.sh Bash script I’ve added to the repo:

    echo $PWD
    chmod +x run.sh
    ./run.sh
    

    BTW The chmod +x run.sh prevents the Permission denied error.

    Unlike other similar scripts, this one does it all: downloads the image and cleans up after itself like a good Boyscout. All it leaves behind is the Console log.

Below is an examplation of each step in the shell script:

  1. Define values within variables:

    NAME="cicd-buzz"
    IMAGE="robvanderleek/cicd-buzz"
    CONTAINER_PORT="8082"
    
  2. Pull the latest image from DockerHub:

    docker image pull "${IMAGE}:latest"
    

    Login to DockerHub is not needed if the Docker image is open to the public, which the above is.

    Sample response:

    latest: Pulling from robvanderleek/cicd-buzz
    8cae0e1ac61c: Pull complete 
    08e039a98597: Pull complete 
    a25e0df325b7: Pull complete 
    a3fcc9a668be: Pull complete 
    87841fbb21e8: Pull complete 
    3f38a4622442: Pull complete 
    Digest: sha256:82992e5e8069af9664cc2f88428b4cd813752f91dfc1130fe232bd070c6b8f10
    Status: Downloaded newer image for robvanderleek/cicd-buzz:latest
    

    If the image is already there, you’ll see:

    Status: Image is up to date for robvanderleek/cicd-buzz:latest
    
  3. Verify the Docker image size:

    docker images "${IMAGE}"   # 61.8MB
    IMAGE_ID=$(docker images --format="&7B;&7B;.Repository}} &7B;&7B;.ID}}" | grep "^$IMAGE " | cut -d' ' -f2)
    echo "$IMAGE IMAGE_ID=$IMAGE_ID"
    

    As of the time of writing, the image SIZE was “61.8MB”.

  4. Remove the previous process (container) if it’s still running:

    docker ps
    CONTAINER_ID=$(docker ps -aqf "name=$NAME")
    echo "$CONTAINER_ID for $IMAGE"
    if ! [[ -z "${CONTAINER_ID// }"  ]]; then  #it's blank
    	  echo_f "Stopping CONTAINER_ID=$CONTAINER_ID ... (takes a few seconds)"
       docker stop "${CONTAINER_ID}" > /dev/null 2>&1
       docker rm   "${CONTAINER_ID}" > /dev/null 2>&1
    fi
    
  5. The run.sh invokes Dockerfile:

    docker run --name ${NAME} -p "$CONTAINER_PORT:5000" -i ${IMAGE}
    

    ${NAME} references the Dockerfile in the same directory:

    FROM python:alpine:3.5
    EXPOSE 5000
    RUN apk add --update python py-pip
    COPY requirements.txt /src/requirements.txt
    RUN pip install -r /src/requirements.txt
    COPY app.py /src
    COPY buzz /src/buzz
    CMD ["python", "/src/app.py"]
    

    alphine:3.5 is the operating system running within the Docker container, as provided by those who own the python account on DockerHub.

    QUESTION: What is the version of Python installed by the line: RUN apk add --update python py-pip

    Sample response:

           * Serving Flask app "app" (lazy loading)
           * Environment: production
    WARNING: Do not use the development server in a production environment.
    Use a production WSGI server instead.
           * Debug mode: off
           * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
    

    app.py

    NOTE: The “0.0.0.0” and port 5000 is specified in this line within program app.py:

    app.run(host=’0.0.0.0’, port=int(os.getenv(‘PORT’, 5000)))

    requirements.txt

    The requirements.txt file contains Python version dependency specifications for the Python Flash library and pytest library:

    pytest==4.2.0
    Flask==1.0.2
    

    Flask is used by the app.py program which listens and responds to HTTP requests by wrapping HTML tags around the output from the text output from program generator.py.

    HTML response

    PROTIP: Normally, the Terminal session would not take any more interactive commands, but the Bash script is written to call docker run with & in the background. The stop command.

    Rather than opening a browser instance, we use curl utility to show the HTML response in the Console.

    RESPONSE=$(curl "localhost:$CONTAINER_PORT")
    echo "RESPONSE=$RESPONSE"
    

    Example response (the app randomly varies words output):

    <html><body><h1>Self-Service Devops Seriously Accelerates Continuous Deployment</h1></body></html>
    

    https://stackoverflow.com/questions/37139786/is-init-py-not-required-for-packages-in-python-3

  6. Clean-up within run.sh

    After running a single curl command, the script stops, then removes the docker process based on capturing the CONTAINER_ID in a variable.

    CONTAINER ID        IMAGE                     COMMAND                CREATED              STATUS              PORTS               NAMES
    8e5bc7f37e78        robvanderleek/cicd-buzz   "python /src/app.py"   About a minute ago   Up About a minute                       cicd-buzz
    

    The image file downloaded is also removed as well to conserve disk space.


TravisCI

Travis CI is a hosted service for Continuous Integration work. It’s free for public GitHub repositories.

  1. Use a browser to visit https://travis-ci.org and get a Travis CI account

    To setup Travis CI to continuously automate tests:

  2. Sign up with your GitHub credentials.
  3. Click your profile icon and select <a target=”_blank” href=”https://github.com/settings/installations”?Applications</a> from the left menu.
  4. Click “Configure” to the right of “Travis CI”. cicd-buzz-travis-github-349x409-10048.jpg

  5. Select your repository. Click Save.
  6. In Travis-ci.org, click “Sync”.
  7. Read through https://docs.travis-ci.com/user/tutorial

    PROTIP: Travis only runs builds on commits pushed only if there is a .travis.yml file in the repo.

  8. In the sample repo, the .travis.yml file specifies the language in the pytest script run by Travis:

    sudo: required
    services:
             - docker
    language: python
    script:
             - python -m pytest -v 
    after_success:
             - sh .travis/deploy_dockerhub.sh
             - test "$TRAVIS_BRANCH" = "master" && "$TRAVIS_PULL_REQUEST" = "false" && sh .travis/deploy_heroku.sh
    

    && combines several commands in sequence.

    WARNING: If the file is not valid YAML, Travis CI will ignore it.

after_success of pytest, Travis is told to run the script .travis/deploy_dockerhub.sh (described below).

Build in DockerHub

In the .travis folder deploy_dockerhub.sh

#!/bin/sh
docker login -u $DOCKER_USER -p $DOCKER_PASS
if [ "$TRAVIS_BRANCH" = "master" ]; then
    TAG="latest"
else
    TAG="$TRAVIS_BRANCH"
fi
docker build -f Dockerfile -t $TRAVIS_REPO_SLUG:$TAG .
docker push $TRAVIS_REPO_SLUG
   

$DOCKER_PASS is the password into DockerHub account $DOCKER_USER.

$TRAVIS_BRANCH is the Git branch name.

$TRAVIS_REPO_SLUG

$TAG

Heroku

Heroku hosts over the public internet applications such as the described above.

  1. Get an account on heroku.com
  2. Identify the GitHub repo.
  3. Use the assigned host name (such as “fathomless-inlet-53225”) or specify your own such as “devops-cert-activity-wilsonmar” as in “https://devops-cert-activity-wilsonmar.herokuapp.com”.

    PROTIP: The host name need not be the same as your repo’s name. Hereoku imposes a 32 character limit to host names (not counting the “herokuapp.com”).

  4. Make note of the assigned host name.
  5. Assign a key and paste the string in Heroku’s env as HEROKU_API_KEY.
  6. In Git, set the repository’s remote to heroku so that the repository can be sent to Heroku after changes occur, such as:

    git remote add heroku https://git.heroku.com/devops-cert-activity-wilsonmar.git
    git remote -v
    

    The response:

    heroku	https://git.heroku.com/devops-cert-activity-wilsonmar.git (fetch)
    heroku	https://git.heroku.com/devops-cert-activity-wilsonmar.git (push)
    origin	https://github.com/wilsonmar/devops-cert-activity-wilsonmar2.git (fetch)
    origin	https://github.com/wilsonmar/devops-cert-activity-wilsonmar2.git (push)
    

    install-ubuntu.sh

Travis run https://toolbelt.heroku.com/install-ubuntu.sh

#!/bin/sh
{
    set -e
    SUDO=''
    if [ "$(id -u)" != "0" ]; then
      SUDO='sudo'
      echo "This script requires superuser access to install apt packages."
      echo "You will be prompted for your password by sudo."
      # clear any previous sudo permission
      sudo -k
    fi
 
    # run inside sudo
    $SUDO sh <<SCRIPT
  set -ex
 
  # if apt-transport-https is not installed, clear out old sources, update, then install apt-transport-https
  dpkg -s apt-transport-https 1>/dev/null 2>/dev/null || \
    (echo "" > /etc/apt/sources.list.d/heroku.list \
      && apt-get update \
      && apt-get install -y apt-transport-https)

  # add heroku repository to apt
  echo "deb https://cli-assets.heroku.com/apt ./" > /etc/apt/sources.list.d/heroku.list
 
  # remove toolbelt
  (dpkg -s heroku-toolbelt 1>/dev/null 2>/dev/null && (apt-get remove -y heroku-toolbelt heroku || true)) || true
 
  # install heroku's release key for package verification
  curl https://cli-assets.heroku.com/apt/release.key | apt-key add -
 
  # update your sources
  apt-get update
 
  # install the toolbelt
  apt-get install -y heroku
 
SCRIPT
  # test the CLI
  LOCATION=$(which heroku)
  echo "heroku installed to $LOCATION"
  heroku version
}

deploy_heroku.sh

Travis then runs deploy_heroku.sh containing:

#!/bin/sh
wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
heroku plugins:install heroku-container-registry
docker login -e _ -u _ --password=$HEROKU_API_KEY registry.heroku.com
heroku container:push web --app $HEROKU_APP_NAME

Git Commit to Transfer

  1. Push a commit from your local git history to GitHub and hooks in GitHub will trigger a build.
  2. Travis dashboard

To enable Travis CI to start a build at each Push and Pull Request for a repository, flip the switch in front of your GitHub repository (click the ‘Sync account’ button in case your repository is not yet visible) :

This blog post documents how to, on every push to the repository, Travis builds a new image (for testing) and when a branch is merged, the built image is pushed to Docker Hub.

This blog talks about automatically syncing README.

Resources on Travis:

Test-first

The script does the following steps:

  1. Show run environment time, etc.
  2. Create a folder.
  3. Write a little Python program (not Hello World)
  4. Add some automated tests for the program
  5. Push your code to GitHub
  6. Setup Travis CI to continuously run your automated tests
  7. Setup Better Code Hub to continuously check your code quality
  8. Turn the Python program into a web app
  9. Create a Docker image for the web app
  10. Push the Docker image to Docker Hub as https://hub.docker.com/r/robvanderleek/cicd-buzz/tags/
  11. Deploy the Docker image to Heroku, which for Robert is https://fathomless-inlet-53225.herokuapp.com/

https://www.youtube.com/watch?v=Z3S2gMBUkBo Integrate with GitHub: build after each commit (Get started with Jenkins, part 13)

https://hackernoon.com/ci-cd-continuous-integration-tools-delivery-react-web-travis-github-example-tutorial-javascript-vue-db8afe9f9a81 $0, Free CICD and Web hosting integration. Travis-ci + Github.page Go to the profile of Peter Chang Peter Chang Aug 19, 2018

https://medium.com/vaidikkapoor/managing-open-source-docker-images-on-docker-hub-using-travis-7fd33bc96d65 Jul 9, 2018

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering