Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

This sample Bash script contains multiple features: install, configure, and run (then remove) a web app within Docker on macOS and Linux, with one copy/paste

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This article describes a Bash script that, with a single command does all this:

BTW content here are my personal opinions, and not intended to represent any employer (past or present).

  1. Define initial lines to:
    • First line file type “shebang”
    • Define bash ShellCheck rules that need to be disabled.
    • Echo time, name, version metadata about run
    • Metadata about the script in comments.
    • Capture a time stamp to later calculate how long the script runs.
  2. Display a menu if no parameter is specified in the command line
  3. Define variables for use as “feature flags” to control specific features run.
  4. Set variables associated with each parameter flag.
  5. Define custom functions to echo text to screen
  6. Detect the operating system in use to install the install appropriate to the OS.
  7. Upgrade to the latest version of bash
  8. Set Bash traps to display information if script is interrupted.
  9. Print run Operating environment information and set “Strict Mode” based on parameters specified for the run.
  10. Install installers (XCode, HomeBrew, apt-get), depending on operating system
  11. Define shell utility functions, such as ShellCheck and the function to kill process by name, etc.
  12. Install basic utilities: Git, jq

    Run configuration:

  13. Get secrets (and other run-time variables) from a clear-text file in $HOME folder or from a crypto program.
  14. Configure project folder location where files are created during the run.
  15. Obtain repository from GitHub.
  16. Reveal secrets stored within .gitsecret folder within repo from GitHub (after installing gnupg and git-secret)
  17. Pipenv and Pyenv to install Python and its modules.

    Connect to cloud (to get secrets):

  18. Connect to Google Comput Cloud (GCP), if requested, to get secrets
  19. Connect to AWS
  20. Connect to Azure

    Apps:

  21. Install K8S minikube
  22. Install EKS using eksctl
  23. Read secrets from a configuration file in clear text, encrypted file, Vault API using govaultenv
  24. Use CircleCI
  25. Use Yubikey
  26. Use HashiCorp Vault
  27. Use NodeJs
  28. Run Virtualenv
  29. Configure Pyenv with virtualenv
  30. Use Anaconda
  31. Use GoLang
  32. Use Python
  33. Use Tensorflow
  34. Use Ruby
  35. Setup Eggplant

  36. Use Docker
  37. Run within Docker
  38. Update GitHub

    Clean-Up:

  39. -C to remove GitHub folder after run
  40. -K to Kill processes after run (to save CPU)
  41. -D to Delete containers and other files after run (to save disk space)
  42. -M to remove Docker iMages downloaded from DockerHub (to save disk space)

Each of the above are preceded by “###” comment tags in the script.

I’ve refined the script over the years to be a “Swiss Army Knife” that enables me to very quickly get stuff done. So it contains most of the coding tricks one would need to use. The script use includes all the above features for apps in NodeJs, Ruby, and Python (Anacodna and Tensorflow, and a program cloned from GitHub) so that we can avoid some of the toil and human error of manually typing commands on each new instance.

If this is too much for you, just cut out the features you don’t want, and enjoy the rest.

Copy and paste invocation for menu

  1. Open a Terminal on your Mac or instantiate a Linux machine on VMWare, EC2, or other cloud.

  2. Execute the script just to get a short description of the parameters controlling what features are invoked, copy this command into your Clipboard by triple-clicking “bash” to turn this command line gray, then press command+C to copy:

    bash -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/bash/sample.sh)"
    

    =========================== 2020-06-28T10:26:41-0600-347 ./sample.sh v0.72
    OPTIONS:
    -E            continue (NOT stop) on error
    -v            run -verbose (list space use and each image to console)
    -q           -quiet headings for each step
    -x            set -x to trace command lines
     
    -I           -Install jq, brew, docker, docker-compose, etc.
    -U           -Upgrade installed packages
     
    -s           -secrets retrieve
    -S "~/.alt.secrets.sh"  -Secrets full file path
    -H           install/use -HashiCorp Vault secret manager
    -m           Setup Vault SSH CA cert
     
    -L           use CircleCI
    -aws         -AWS cloud
    -eks         -eks (Elastic Kubernetes Service) in AWS cloud
    -g "abcdef...89" -gcloud API credentials for calls
    -p "cp100"   -project in cloud
     
    -d           -delete GitHub and pyenv from previous run
    -c           -clone from GitHub
    -N           -Name of GitHub Repo folder
    -n "John Doe"            GitHub user -name
    -e "john_doe@gmail.com"  GitHub user -email
     
    -k           -k install and use Docker
    -k8s         -k8s (Kubernetes) minikube
    -b           -build Docker image
    -dc           use docker-compose.yml file
    -w           -write image to DockerHub
    -r           -restart (Docker) before run
     
    -py          run with Pyenv
    -V           to run within VirtualEnv (pipenv is default)
    -tf          -tensorflow
    -A           run with Python -Anaconda
    -y            install Python Flask
     
    -i           -install Ruby and Refinery
    -j            install -JavaScript (NodeJs) app with MongoDB
     
    -G           -GitHub is the basis for program to run
    -F "abc"     -Folder inside repo
    -f "a9y.py"  -file (program) to run
    -P "-v -x"   -Parameters controlling program called
    -u           -update GitHub
    -a           -actually run server (not dry run)
    -t           setup -test server to run tests
    -o           -open/view app or web page in default browser
     
    -K           stop OS processes at end of run (to save CPU)
    -D           -Delete files after run (to save disk space)
    -C           remove -Cloned files after run (to save disk space)
    -M           remove Docker iMages pulled from DockerHub
    USAGE EXAMPLE during testing:
    ./sample.sh -v -W -r -k -a -o -K -D  # WebGoat Docker with Contrast agent
    ./sample.sh -v -s -eggplant -k -a -K -D  # eggplant use docker-compose of selenium-hub images
    ./sample.sh -v -S "$HOME/.mck-secrets.sh" -eks -D
    ./sample.sh -v -S "$HOME/.mck-secrets.sh" -H -m -t    # Use SSH-CA certs with -H HashiCorp Vault -test actual server
    ./sample.sh -v -g "abcdef...89" -p "cp100-1094"  # Google API call
    ./sample.sh -v -n -a  # NodeJs app with MongoDB
    ./sample.sh -v -i -o  # Ruby app
    ./sample.sh -v -I -U -c -s -y -r -a -AWS   # Python Flask web app in Docker
    ./sample.sh -v -I -U    -s -H    -t        # Initiate Vault test server
    ./sample.sh -v          -s -H              #      Run Vault test program
    ./sample.sh -q          -s -H    -a        # Initiate Vault prod server
    ./sample.sh -v -I -U -c    -H -G -N "python-samples" -f "a9y-sample.py" -P "-v" -t -AWS -C  # Python sample app using Vault
    ./sample.sh -v -V -c -T -F "section_2" -f "2-1.ipynb" -K  # Jupyter anaconda Tensorflow in Venv
    ./sample.sh -v -V -c -L -s    # Use CircLeci based on secrets
    ./sample.sh -v -D -M -C
    ./sample.sh -G -v -f "challenge.py" -P "-v"  # to run a program in python-samples
    ./sample.sh -v -s -H -m -o -t  # Vault SSH keygen
    

Edit sample.sh

  1. Use a text editor or IDE to open the sample.sh file.

    File metadata

    Metadata about the file, such as

Utility functions

Shell functions are defined near the beginning of the script for use later in the script.

QUESTION: What are good Bash libraries with common functions? Libraries for bash are not common. One is /etc/rc.d/functions on RedHat-based systems. The file contains functions commonly used in sysV init script.

NOTE: Bash libraries are scarce is due to limitation of Bash functions.

NOTE: Bash’s “functions” have several issues:

Code reusability: Bash functions don’t return anything; they only produce output streams. Every reasonable method of capturing that stream and either assigning it to a variable or passing it as an argument requires a SubShell, which breaks all assignments to outer scopes. (See also BashFAQ/084 for tricks to retrieve results from a function.) Thus, libraries of reusable functions are not feasible, as you can’t ask a function to store its results in a variable whose name is passed as an argument (except by performing eval backflips).

Scope: Bash has a simple system of local scope which roughly resembles “dynamic scope” (e.g. Javascript, elisp). Functions see the locals of their callers (like Python’s “nonlocal” keyword), but can’t access a caller’s positional parameters (except through BASH_ARGV if extdebug is enabled). Reusable functions can’t be guaranteed free of namespace collisions unless you resort to weird naming rules to make conflicts sufficiently unlikely. This is particularly a problem if implementing functions that expect to be acting upon variable names from frame n-3 which may have been overwritten by your reusable function at n-2. Ksh93 can use the more common lexical scope rules by declaring functions with the “function name { … }” syntax (Bash can’t, but supports this syntax anyway).

Script run environment

These commands obtain information about the script’s environment:

HOSTNAME=$( hostname )
PUBLIC_IP=$( curl -s ifconfig.me )

PROTIP: The alternative to curl is wget, which follows redirects.

This script code prints information about the script’s running environment:

      note "Running $0 in $PWD"  # $0 = script being run in Present Wording Directory.
      note "Bash $BASH_VERSION at $LOG_DATETIME"  # built-in variable.
      note "OS_TYPE=$OS_TYPE using $PACKAGE_MANAGER from $DISK_PCT_FREE disk free"
      note "on hostname=$HOSTNAME at PUBLIC_IP=$PUBLIC_IP"
   if [ -f "$OS_DETAILS" ]; then
      note "$OS_DETAILS"
   fi
   

PROTIP: “$0” within Bash scripts returns the script file name.

PROTIP: “$PWD” returns the “Present Working Directory” (current folder path).

Sample response:

   Running ./sample.sh in /Users/wilson_mar/gits/wilsonmar/DevSecOps/bash
  Bash 5.0.11(1)-release at 2020-01-20T00:23:03-0700-1000
  OS_TYPE=macOS using brew from 27% disk free
  on hostname=12345 at PUBLIC_IP=162.142.245.98
  

wilson_mar is my user name on my macOS laptop.

ShellCheck

Getting Initial Secrets

Keeping secrets from being exposed is the bane of developers’ existance.

We need to retieve secrets in order to have credentials to access services on the web, such as AWS, Azure, GCP, etc.

Some think that specifying .gitignore or keeping a repo as private is enough to keep secrets safe. But anytime something is on the internet, it can be exposed.

retrieve a .secrets file in your user $HOME folder. Edit the file to contain something like:

# Used by https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/bash/sample.sh
# Explained in https://wilsonmar.github.io/bash-scripts/#KeepingSecrets
GitHub_USER_NAME="John Doe"
GitHub_USER_EMAIL="john_doe@a.com"

-s specified in run parameters for the script to make use of this file.

If that is not specified, or if the file is not found or variable not found, the script falls back to asking for manual input of the variables every run.

In a forthcoming refactoring, we may add use of HashiCorp Vault, which puts another secret in place of the real secret.

Copy Sample files

The particular application has sample files which should be copied, then edited for use.

  • .env.example to .env
  • docker-compose.override.example.yml to docker-compose.override.yml

The script looks for the file name copied by a previous run.

File names on the local machine are specified in the repo’s .gitignore file so they don’t get pushed into GitHub.

GitHub and .gitsecret

If a .gitsecret folder is found in the repo, the script installs gpg and git-secret brew.

TODO: Also detect if https://www.passwordstore.org using brew install pass.

Package Manager install

This script installs the packages managers needed for the operating system under use. brew first requires HomeBrew to be installed (using Ruby).

Read this README which provides someone new to Macs specific steps to configure and run scripts to install apps on Macs. So first finish reading that about “shbangs” and grep for Bash shell versions.

On Macs, XCode needs to be installed for utilities needed by the HomeBrew installer.

Windows on Linux can use either a fork of Homebrew (Linuxbrew) or apt-get/yum. But Linuxbrew installs packages to a unique folder, so that path needs to be added to the search PATH in ~/.bash_profile.

brew –prefix yields “/usr/local”

Ruby Gemfile of versions

The Ruby Gemfile specifies the packages mentioned in the import statement within Ruby programs. The latest version of each package is specified by default. Or a specific version can be specified.

The Gemfile.lock file reflects what Bundler records as the exact versions installed. This way, when the same library/project is loaded on another machine, running bundle install will look at the Gemfile.lock and sinstall the exact same versions, rather than just using the Gemfile and installing the most recent versions. (Running different versions on different machines could lead to broken tests, etc.)

Docker and docker-compose

This script can get you up and running with a DockerHub image, but with the ability to get listings of containers and images without much typing.

This is the case when running -eggplant.

-k installs and uses Docker and docker-compose. It restarts the Docker daemon if it’s already running. Either way, the Docker daemon is started.

-a

Clean-up

-D stops and removes Docker containers still running.

-M removes the images pulled from DockerHub.

-R removes the cloned app repository.


References

https://www.tutorialspoint.com/unix/unix-file-operators.htm

Qwiklabs.com: Automating AWS Services with Scripting and the AWS CLI

Sander van Vugt (LivingOpenSource.com) https://github.com/sandervanvugt/cool-bash

Ian Miell, author of Bash the Hard Way, has a “Bash Next Steps” video course on OReilly which covers Bash 5 features.

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering