CICD pipeline (Docker of Python Console program being tested within TravisCI)

Featuring a Bash shell script that does everything, explained

Overview

Production usage: the buzz phrase
generator.py coding
Run the Python program
Local use of Docker
- app.py
- requirements.txt
HTML response
TravisCI
Build in DockerHub
Heroku
Test-first
More on DevOps

This is a hands-on deep dive into the coding of various assets in an end-to-end DevOps workflow. Examples are modified from Rob van der Leek’s Apr 9, 2017 Medium article and “buzz phrase generator” in his “cicd-buzz” open-source repo.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Here we first work backwards, leveraging the outcome of Robert’s work (to make sure that it’s not vaporware ;).

Production usage: the buzz phrase

Manually use an internet browser to visit the web page generated from within Heroku (under Robert’s account):
```
https://fathomless-inlet-53225.herokuapp.com
```
If the site is still alive, you should see a random phrase generated, such as:

“Complete Continuous Improvement Enormously Boosts Continuous Integration”

Below is a break-down of each sub-phrase above:

“Complete” is an adjective that also includes
‘modern’, ‘self-service’, ‘integrated’, ‘end-to-end’

“Continuous Improvement” is a buzz word that also includes
‘continuous testing’, ‘continuous integration’, ‘continuous deployment’, ‘devops’

“Enormously” is an adverb that also includes
‘remarkably’, ‘substantially’, ‘significantly’, ‘seriously’

“Boosts” is a verb that also includes
‘accelerates’, ‘improves’, ‘enhances’, ‘revamps’

“Continuous Integration” is another buzz word.

TODO: If you want to change these, edit your copy of the program code and build another Docker image.

generator.py coding
Click the URL below to view the sample (“trivial”) Python program code that generates the buzz phrases above:
```
https://github.com/robvanderleek/cicd-buzz/blob/master/buzz/generator.py
```
Below is an example of how one would explain the program during a live walkthrough:

The program begins at bottom of the code with the print(generate_buzz()) command under the if __name__ == "__main__": entry point.

The generate_buzz() function returns out the program the phrase variable after using the Python built-in title() method that capitalizes the first character of each word.

The value for the phrase variable is obtained by joining together a samples within arrays of adjectives, buzz (terms), adverbs, and verbs.

The variable list_ in the signature of the sample function stands in for the variables specified in the join function.

The sample function returns the output fromrandom.sample because it is an inbuilt function brought in via the module random specified by the import statement at the top of the code file.

Run the Python program
Open a Text Editor program. Navigate into the “buzz” folder to edit “generator.py”.
Open a Terminal on your Mac and navigate into the buzz folder.
Run the generator.py program using the Python interpreter (either version 2 or 3 should work).
```
python generator.py
```
NOTE: I would prefer to use from random import sample because import random brings in the whole module, which this custom code doesn’t use. However, running it results in:

NameError: global name 'random' is not defined

The variable n is a commonly used name for a temporary variable containing the limit. It is defined in the function’s signature specification.

Within the if statement, the return clause is indented because that’s Python.
Repeat the call and another set of values should appear.

Local use of Docker

I created the run.sh script so that you have an example of a Bash script to run locally.
Open a Terminal on your Mac.
Install Docker if you haven’t already.
Make a containing folder to add the repository. I personally use ~/gits/wilsonmar.
A Git clone command creates the “cicd-buzz” repo folder:
```
git clone https://github.com/wilsonmar/cicd-buzz
cd cicd-buzz
ls
```
My version of the repo contains a shell file named “run.sh”.
Run the run.sh Bash script I’ve added to the repo:
```
echo $PWD
chmod +x run.sh
./run.sh
```
BTW The chmod +x run.sh prevents the Permission denied error.

Unlike other similar scripts, this one does it all: downloads the image and cleans up after itself like a good Boyscout. All it leaves behind is the Console log.

Below is an examplation of each step in the shell script:

Define values within variables:

NAME="cicd-buzz"
IMAGE="robvanderleek/cicd-buzz"
CONTAINER_PORT="8082"

Pull the latest image from DockerHub:

docker image pull "${IMAGE}:latest"

Sample response:

latest: Pulling from robvanderleek/cicd-buzz
8cae0e1ac61c: Pull complete 
08e039a98597: Pull complete 
a25e0df325b7: Pull complete 
a3fcc9a668be: Pull complete 
87841fbb21e8: Pull complete 
3f38a4622442: Pull complete 
Digest: sha256:82992e5e8069af9664cc2f88428b4cd813752f91dfc1130fe232bd070c6b8f10
Status: Downloaded newer image for robvanderleek/cicd-buzz:latest

If the image is already there, you’ll see:

Status: Image is up to date for robvanderleek/cicd-buzz:latest

Verify the Docker image size:

docker images "${IMAGE}"   # 61.8MB
IMAGE_ID=$(docker images --format="&7B;&7B;.Repository}} &7B;&7B;.ID}}" | grep "^$IMAGE " | cut -d' ' -f2)
echo "$IMAGE IMAGE_ID=$IMAGE_ID"

As of the time of writing, the image SIZE was “61.8MB”.

Remove the previous process (container) if it’s still running:

docker ps
CONTAINER_ID=$(docker ps -aqf "name=$NAME")
echo "$CONTAINER_ID for $IMAGE"
if ! [[ -z "${CONTAINER_ID// }"  ]]; then  #it's blank
	  echo_f "Stopping CONTAINER_ID=$CONTAINER_ID ... (takes a few seconds)"
   docker stop "${CONTAINER_ID}" > /dev/null 2>&1
   docker rm   "${CONTAINER_ID}" > /dev/null 2>&1
fi

The run.sh invokes Dockerfile:
```
docker run --name ${NAME} -p "$CONTAINER_PORT:5000" -i ${IMAGE}
```
${NAME} references the Dockerfile in the same directory:
```
FROM python:alpine:3.5
EXPOSE 5000
RUN apk add --update python py-pip
COPY requirements.txt /src/requirements.txt
RUN pip install -r /src/requirements.txt
COPY app.py /src
COPY buzz /src/buzz
CMD ["python", "/src/app.py"]
```
alphine:3.5 is the operating system running within the Docker container, as provided by those who own the python account on DockerHub.

QUESTION: What is the version of Python installed by the line: RUN apk add --update python py-pip

Sample response:
```
       * Serving Flask app "app" (lazy loading)
       * Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
       * Debug mode: off
       * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
```
app.py

NOTE: The “0.0.0.0” and port 5000 is specified in this line within program app.py:

app.run(host=’0.0.0.0’, port=int(os.getenv(‘PORT’, 5000)))

requirements.txt

The requirements.txt file contains Python version dependency specifications for the Python Flash library and pytest library:
```
pytest==4.2.0
Flask==1.0.2
```
Flask is used by the app.py program which listens and responds to HTTP requests by wrapping HTML tags around the output from the text output from program generator.py.

HTML response

PROTIP: Normally, the Terminal session would not take any more interactive commands, but the Bash script is written to call docker run with & in the background. The stop command.

Rather than opening a browser instance, we use curl utility to show the HTML response in the Console.
```
RESPONSE=$(curl "localhost:$CONTAINER_PORT")
echo "RESPONSE=$RESPONSE"
```
Example response (the app randomly varies words output):
```
<html><body><h1>Self-Service Devops Seriously Accelerates Continuous Deployment</h1></body></html>
```
https://stackoverflow.com/questions/37139786/is-init-py-not-required-for-packages-in-python-3

Clean-up within run.sh

After running a single curl command, the script stops, then removes the docker process based on capturing the CONTAINER_ID in a variable.

CONTAINER ID        IMAGE                     COMMAND                CREATED              STATUS              PORTS               NAMES
8e5bc7f37e78        robvanderleek/cicd-buzz   "python /src/app.py"   About a minute ago   Up About a minute                       cicd-buzz

The image file downloaded is also removed as well to conserve disk space.

TravisCI

Travis CI is a hosted service for Continuous Integration work. It’s free for public GitHub repositories.

Use a browser to visit https://travis-ci.org and get a Travis CI account

To setup Travis CI to continuously automate tests:
Sign up with your GitHub credentials.
Click your profile icon and select <a target=”_blank” href=”https://github.com/settings/installations”?Applications</a> from the left menu.
Click “Configure” to the right of “Travis CI”.
Select your repository. Click Save.
In Travis-ci.org, click “Sync”.
Read through https://docs.travis-ci.com/user/tutorial

PROTIP: Travis only runs builds on commits pushed only if there is a .travis.yml file in the repo.

In the sample repo, the .travis.yml file specifies the language in the pytest script run by Travis:

sudo: required
services:
         - docker
language: python
script:
         - python -m pytest -v 
after_success:
         - sh .travis/deploy_dockerhub.sh
         - test "$TRAVIS_BRANCH" = "master" && "$TRAVIS_PULL_REQUEST" = "false" && sh .travis/deploy_heroku.sh

&& combines several commands in sequence.

WARNING: If the file is not valid YAML, Travis CI will ignore it.

after_success of pytest, Travis is told to run the script .travis/deploy_dockerhub.sh (described below).

Build in DockerHub

In the .travis folder deploy_dockerhub.sh

#!/bin/sh
docker login -u $DOCKER_USER -p $DOCKER_PASS
if [ "$TRAVIS_BRANCH" = "master" ]; then
    TAG="latest"
else
    TAG="$TRAVIS_BRANCH"
fi
docker build -f Dockerfile -t $TRAVIS_REPO_SLUG:$TAG .
docker push $TRAVIS_REPO_SLUG

$DOCKER_PASS is the password into DockerHub account $DOCKER_USER.

$TRAVIS_BRANCH is the Git branch name.

$TRAVIS_REPO_SLUG

$TAG

Heroku

Heroku hosts over the public internet applications such as the described above.

Get an account on heroku.com
Identify the GitHub repo.
Use the assigned host name (such as “fathomless-inlet-53225”) or specify your own such as “devops-cert-activity-wilsonmar” as in “https://devops-cert-activity-wilsonmar.herokuapp.com”.

PROTIP: The host name need not be the same as your repo’s name. Hereoku imposes a 32 character limit to host names (not counting the “herokuapp.com”).
Make note of the assigned host name.
Assign a key and paste the string in Heroku’s env as HEROKU_API_KEY.

In Git, set the repository’s remote to heroku so that the repository can be sent to Heroku after changes occur, such as:

git remote add heroku https://git.heroku.com/devops-cert-activity-wilsonmar.git
git remote -v

The response:

heroku	https://git.heroku.com/devops-cert-activity-wilsonmar.git (fetch)
heroku	https://git.heroku.com/devops-cert-activity-wilsonmar.git (push)
origin	https://github.com/wilsonmar/devops-cert-activity-wilsonmar2.git (fetch)
origin	https://github.com/wilsonmar/devops-cert-activity-wilsonmar2.git (push)

install-ubuntu.sh

Travis run https://toolbelt.heroku.com/install-ubuntu.sh

#!/bin/sh
{
    set -e
    SUDO=''
    if [ "$(id -u)" != "0" ]; then
      SUDO='sudo'
      echo "This script requires superuser access to install apt packages."
      echo "You will be prompted for your password by sudo."
      # clear any previous sudo permission
      sudo -k
    fi
 
    # run inside sudo
    $SUDO sh <<SCRIPT
  set -ex
 
  # if apt-transport-https is not installed, clear out old sources, update, then install apt-transport-https
  dpkg -s apt-transport-https 1>/dev/null 2>/dev/null || \
    (echo "" > /etc/apt/sources.list.d/heroku.list \
      && apt-get update \
      && apt-get install -y apt-transport-https)

  # add heroku repository to apt
  echo "deb https://cli-assets.heroku.com/apt ./" > /etc/apt/sources.list.d/heroku.list
 
  # remove toolbelt
  (dpkg -s heroku-toolbelt 1>/dev/null 2>/dev/null && (apt-get remove -y heroku-toolbelt heroku || true)) || true
 
  # install heroku's release key for package verification
  curl https://cli-assets.heroku.com/apt/release.key | apt-key add -
 
  # update your sources
  apt-get update
 
  # install the toolbelt
  apt-get install -y heroku
 
SCRIPT
  # test the CLI
  LOCATION=$(which heroku)
  echo "heroku installed to $LOCATION"
  heroku version
}

deploy_heroku.sh

Travis then runs deploy_heroku.sh containing:

#!/bin/sh
wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
heroku plugins:install heroku-container-registry
docker login -e _ -u _ --password=$HEROKU_API_KEY registry.heroku.com
heroku container:push web --app $HEROKU_APP_NAME

Git Commit to Transfer

Push a commit from your local git history to GitHub and hooks in GitHub will trigger a build.
Travis dashboard

To enable Travis CI to start a build at each Push and Pull Request for a repository, flip the switch in front of your GitHub repository (click the ‘Sync account’ button in case your repository is not yet visible) :

This blog post documents how to, on every push to the repository, Travis builds a new image (for testing) and when a branch is merged, the built image is pushed to Docker Hub.

This blog talks about automatically syncing README.

Resources on Travis:

https://docs.travis-ci.com/user/tutorial/#to-get-started-with-travis-ci lists .travis.yml files for various languages. https://docs.travis-ci.com/user/language-specific/

The example for Node was presented with this slidedeck by David Reeve in VIDEO: Travis CI Tutorial - How to Use Travis CI with Github for Continuous Integration Jan 22, 2016 account FullStack Academy.

Test-first

The script does the following steps:

Show run environment time, etc.
Create a folder.
Write a little Python program (not Hello World)
Add some automated tests for the program
Push your code to GitHub
Setup Travis CI to continuously run your automated tests
Setup Better Code Hub to continuously check your code quality
Turn the Python program into a web app
Create a Docker image for the web app
Push the Docker image to Docker Hub as https://hub.docker.com/r/robvanderleek/cicd-buzz/tags/
Deploy the Docker image to Heroku, which for Robert is https://fathomless-inlet-53225.herokuapp.com/

https://www.youtube.com/watch?v=Z3S2gMBUkBo Integrate with GitHub: build after each commit (Get started with Jenkins, part 13)

https://hackernoon.com/ci-cd-continuous-integration-tools-delivery-react-web-travis-github-example-tutorial-javascript-vue-db8afe9f9a81 $0, Free CICD and Web hosting integration. Travis-ci + Github.page Go to the profile of Peter Chang Peter Chang Aug 19, 2018

https://medium.com/vaidikkapoor/managing-open-source-docker-images-on-docker-hub-using-travis-7fd33bc96d65 Jul 9, 2018

More on DevOps

This is one of a series on DevOps:

Wilson Mar

CICD pipeline (Docker of Python Console program being tested within TravisCI)

Production usage: the buzz phrase

generator.py coding

Run the Python program

Local use of Docker

app.py

requirements.txt

HTML response