This sample Bash script contains multiple features: install, configure, and run (then remove) a web app within Docker on macOS and Linux, with one copy/paste
This article describes a Bash script that, with a single command does all this:
NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.
- Define initial lines to:
- First line file type “shebang”
- Define bash ShellCheck rules that need to be disabled.
- Echo time, name, version metadata about run
- Metadata about the script in comments.
- Capture a time stamp to later calculate how long the script runs.
- Display a menu if no parameter is specified in the command line
- Define variables for use as “feature flags” to control specific features run.
- Set variables associated with each parameter flag.
- Define custom functions to echo text to screen
- Detect the operating system in use to install the install appropriate to the OS.
- Upgrade to the latest version of bash
- Set Bash traps to display information if script is interrupted.
- Print run Operating environment information and set “Strict Mode” based on parameters specified for the run.
- Install installers (XCode, HomeBrew, apt-get), depending on operating system
- Define shell utility functions, such as ShellCheck and the function to kill process by name, etc.
Install basic utilities: Git, jq
- Get secrets (and other run-time variables) from a clear-text file in $HOME folder or from a crypto program.
- Configure project folder location where files are created during the run.
- Obtain repository from GitHub.
- Reveal secrets stored within .gitsecret folder within repo from GitHub (after installing gnupg and git-secret)
Pipenv and Pyenv to install Python and its modules.
Connect to cloud (to get secrets):
- Connect to Google Comput Cloud (GCP), if requested, to get secrets
- Connect to AWS
Connect to Azure
- Install K8S minikube
- Install EKS using eksctl
- Read secrets from a configuration file in clear text, encrypted file, Vault API using govaultenv
- Use CircleCI
- Use Yubikey
- Use HashiCorp Vault
- Use NodeJs
- Run Virtualenv
- Configure Pyenv with virtualenv
- Use Anaconda
- Use GoLang
- Use Python
- Use Tensorflow
- Use Ruby
- Use Docker
- Run within Docker
- -C to remove GitHub folder after run
- -K to Kill processes after run (to save CPU)
- -D to Delete containers and other files after run (to save disk space)
- -M to remove Docker iMages downloaded from DockerHub (to save disk space)
Each of the above are preceded by “###” comment tags in the script.
I’ve refined the script over the years to be a “Swiss Army Knife” that enables me to very quickly get stuff done. So it contains most of the coding tricks one would need to use. The script use includes all the above features for apps in NodeJs, Ruby, and Python (Anacodna and Tensorflow, and a program cloned from GitHub) so that we can avoid some of the toil and human error of manually typing commands on each new instance.
If this is too much for you, just cut out the features you don’t want, and enjoy the rest.
Copy and paste invocation for menu
Open a Terminal on your Mac or instantiate a Linux machine on VMWare, EC2, or other cloud.
Execute the script just to get a short description of the parameters controlling what features are invoked, copy this command into your Clipboard by triple-clicking “bash” to turn this command line gray, then press command+C to copy:
bash -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/bash/sample.sh)"
Shell functions are defined near the beginning of the script for use later in the script.
QUESTION: What are good Bash libraries with common functions? Libraries for bash are not common. One is /etc/rc.d/functions on RedHat-based systems. The file contains functions commonly used in sysV init script.
NOTE: Bash libraries are scarce is due to limitation of Bash functions.
NOTE: Bash’s “functions” have several issues:
Code reusability: Bash functions don’t return anything; they only produce output streams. Every reasonable method of capturing that stream and either assigning it to a variable or passing it as an argument requires a SubShell, which breaks all assignments to outer scopes. (See also BashFAQ/084 for tricks to retrieve results from a function.) Thus, libraries of reusable functions are not feasible, as you can’t ask a function to store its results in a variable whose name is passed as an argument (except by performing eval backflips).
Script run environment
These commands obtain information about the script’s environment:
HOSTNAME=$( hostname ) PUBLIC_IP=$( curl -s ifconfig.me )
PROTIP: The alternative to curl is wget, which follows redirects.
This script code prints information about the script’s running environment:
note "Running $0 in $PWD" # $0 = script being run in Present Wording Directory. note "Bash $BASH_VERSION at $LOG_DATETIME" # built-in variable. note "OS_TYPE=$OS_TYPE using $PACKAGE_MANAGER from $DISK_PCT_FREE disk free" note "on hostname=$HOSTNAME at PUBLIC_IP=$PUBLIC_IP" if [ -f "$OS_DETAILS" ]; then note "$OS_DETAILS" fi
PROTIP: “$0” within Bash scripts returns the script file name.
PROTIP: “$PWD” returns the “Present Working Directory” (current folder path).
Running ./sample.sh in /Users/wilson_mar/gits/wilsonmar/DevSecOps/bash Bash 5.0.11(1)-release at 2020-01-20T00:23:03-0700-1000 OS_TYPE=macOS using brew from 27% disk free on hostname=12345 at PUBLIC_IP=188.8.131.52
wilson_mar is my user name on my macOS laptop.
Getting Initial Secrets
Keeping secrets from being exposed is the bane of developers’ existance.
We need to retieve secrets in order to have credentials to access services on the web, such as AWS, Azure, GCP, etc.
Some think that specifying
.gitignore or keeping a repo as private is enough to keep secrets safe.
But anytime something is on the internet, it can be exposed.
retrieve a .secrets file in your user $HOME folder. Edit the file to contain something like:
# Used by https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/bash/sample.sh # Explained in https://wilsonmar.github.io/bash-scripts/#KeepingSecrets GitHub_USER_NAME="John Doe" GitHub_USER_EMAIL="email@example.com"
-s specified in run parameters for the script to make use of this file.
If that is not specified, or if the file is not found or variable not found, the script falls back to asking for manual input of the variables every run.
In a forthcoming refactoring, we may add use of HashiCorp Vault, which puts another secret in place of the real secret.
Copy Sample files
The particular application has sample files which should be copied, then edited for use.
- .env.example to .env
- docker-compose.override.example.yml to docker-compose.override.yml
The script looks for the file name copied by a previous run.
File names on the local machine are specified in the repo’s .gitignore file so they don’t get pushed into GitHub.
GitHub and .gitsecret
If a .gitsecret folder is found in the repo, the script installs gpg and git-secret brew.
TODO: Also detect if https://www.passwordstore.org using brew install pass.
Package Manager install
This script installs the packages managers needed for the operating system under use. brew first requires HomeBrew to be installed (using Ruby).
Read this README which provides someone new to Macs specific steps to configure and run scripts to install apps on Macs. So first finish reading that about “shbangs” and grep for Bash shell versions.
On Macs, XCode needs to be installed for utilities needed by the HomeBrew installer.
Windows on Linux can use either a fork of Homebrew (Linuxbrew) or apt-get/yum. But Linuxbrew installs packages to a unique folder, so that path needs to be added to the search PATH in ~/.bash_profile.
brew –prefix yields “/usr/local”
Ruby Gemfile of versions
The Ruby Gemfile specifies the packages mentioned in the import statement within Ruby programs. The latest version of each package is specified by default. Or a specific version can be specified.
The Gemfile.lock file reflects what Bundler records as the exact versions installed. This way, when the same library/project is loaded on another machine, running bundle install will look at the Gemfile.lock and sinstall the exact same versions, rather than just using the Gemfile and installing the most recent versions. (Running different versions on different machines could lead to broken tests, etc.)
Docker and docker-compose
This script can get you up and running with a DockerHub image, but with the ability to get listings of containers and images without much typing.
This is the case when running -eggplant.
-k installs and uses Docker and docker-compose. It restarts the Docker daemon if it’s already running.
Either way, the Docker daemon is started.
-D stops and removes Docker containers still running.
-M removes the images pulled from DockerHub.
-R removes the cloned app repository.
Qwiklabs.com: Automating AWS Services with Scripting and the AWS CLI
Sander van Vugt (LivingOpenSource.com) https://github.com/sandervanvugt/cool-bash
Ian Miell, author of Bash the Hard Way, has a “Bash Next Steps” video course on OReilly which covers Bash 5 features.
More on DevOps
This is one of a series on DevOps:
- ci-cd (Continuous Integration and Continuous Delivery)
- User Stories for DevOps
- Git and GitHub vs File Archival
- Git Commands and Statuses
- Git Commit, Tag, Push
- Git Utilities
- Data Security GitHub
- GitHub API
- Choices for DevOps Technologies
- Pulumi Infrastructure as Code (IaC)
- Java DevOps Workflow
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- AWS server deployment options
- Cloud services comparisons (across vendors)
- Cloud regions (across vendors)
- Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
- Azure Certifications
- Azure Cloud Powershell
- Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
- Azure Networking
- Azure Storage
- Azure Compute
- Digital Ocean
- Packer automation to build Vagrant images
- Terraform multi-cloud provisioning automation
Hashicorp Vault and Consul to generate and hold secrets
- Powershell Ecosystem
- Powershell on MacOS
- Jenkins Server Setup
- Jenkins Plug-ins
- Jenkins Freestyle jobs
- Docker (Glossary, Ecosystem, Certification)
- Make Makefile for Docker
- Docker Setup and run Bash shell script
- Bash coding
- Docker Setup
- Dockerize apps
- Kubernetes Operators
- Threat Modeling
- API Management Microsoft
- Scenarios for load
- Chaos Engineering