Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

So I don’t fear losing my macOS laptop

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Here are steps for (what I think is) the fastest way to configure a Mac for software development. This has speeded up onboarding of new team members, so soon after joining they can do pair programming without distraction.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

On a new laptop :

  1. Obtain a macOS laptop (from Apple, Amazon, or 2nd-hand). Unbox it. Order Apple hardware accessories. Arrange it all on your desk for convenient access.

  2. Get familiar using the keyboard, Trackpad, and mouse.

  3. Save baseline data to Time Machine (before making changes), as part of a full backup and restore strategy

  4. Open a Terminal by pressing command+spacebar, then type enough of Terminal.app to select it.

  5. Commands make use of the curl utility to download. So make you don’t get any error about that on the Terminal:

    curl --version

    You should see several lines such as:

    curl 7.79.1 (x86_64-apple-darwin13.4.0) libcurl/7.79.1 OpenSSL/1.1.1l zlib/1.2.11 libssh2/1.10.0 nghttp2/1.43.0
    Release-Date: 2021-09-22
    Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
    Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets
    

    mac-setup.zsh menu options

  6. Use your mouse to triple-click zsh in the command below to highlight the line, then press command+C to copy it to your Clipboard:

    zsh -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/mac-setup/main/mac-setup.zsh)" -v
    

    CAUTION: Do not click on the URL (starting with httpd) since the terminal program opens a browser to that URL.

    This page presents a guided tour of that shell script file.

  7. Press command+Tab to switch to the Terminal.app.

  8. Click anywhere in the Terminal window and Press command+V to paste the command from your Clipboard. Notice at the end of the command:

    -v specifies optional verbose log output.

    We will cover additional parameters later in this article.

  9. Press Return/Enter on your keyboard to begin execution.

    The design of the script is that, except for the mac-setup.env exception, a parameter must be specified with the command in order for each feature to be procssed. With no parameters specified, the script returns a menu of options and feature parameters.

    This modular approach is designed to enable quick experimentation.

  10. Expand the width of your Terminal window by dragging the right edge so you see any more line wrapping.

  11. Paste the command again and add more parameters to additional command executions. See “Customize mac-setup specifications to your liking”

    -I specifies install of base utilities and app most popular among developers using macOS.

    -go specifies install of Go language.

    etc.

  12. Again press Return to start the script running using the new parameters.

    If each utility program in the list is not installed, the script installs it.

    We created the script to automatically take care of workarounds to issues.

    Edit mac-setup.env parameter values

    The behavior described below can be stopped by specifying the
    -nenv run-time parameter.

  13. Here’s that <a exception noted earlier: WARNING: The mac-setup.zsh script, by default, downloads (using curl) a mac-setup.env file in your user $HOME to hold configuration settings. That’s needed so that, in future runs, the script makes use of values in variables within that file rather than default values hard-coded in the mac-setup.zsh script file. View it online:

    https://github.com/wilsonmar/mac-setup/blob/master/mck-setup.env

    NOTE: Secrets are kept in a different file.

    Once downloaded, the script does not replace the file with another download.

    The script loads this file. However, at run-time, an alternative name can be specified dynamically (for a particular run), such as:
    -env “~/alt-mac-setup.env”

  14. After your favorite text editor is installed, edit that mac-setup.env file and change some values:

    We expect that you change each “wilson” sample value, since that is never applicable to you.

    The script uses a folder based on what is specified in the PROJECTS_CONTAINER_PATH variable.

    ~/Projects/mac-setup folder on your machine and downloads the mac-setup.zsh script file into it.

  15. If you have not created an account on GitHub.com, please do so before continuing. Most actions by developers make use of the GitHub versioned change management utility on the web. It is reached by the git utility the script installs.

  16. If you (and your team) want a different set of configurations (utilities and apps to install), save the script.

  17. Alternately, edit the configuration files to run Ansible, a whole other approach.


Steps in script mac-setup.zsh

Click each section for an explanation of the coding:

The first lines:

01. Capture time stamps to later calculate how long the script runs, no matter how it ends.

02. Display a menu if no parameter is specified in the command line

03. Custom functions to echo text to screen. Each echo function displays in different colors. -vv causes sample output to be displayed.

04. Define variables for use as “feature flags”

05. Download config settings file to $HOME/mac-setup.env (away from GitHub)

06. Set variables dynamically based on each parameter flag

07. Display run variables

08. Obtain and show information about the operating system in use to define which package manager to use

09. Set traps to display information if script is interrupted.

10. Set Continue on Error and Trace

11. Print run Operating environment information

12. Keep-alive: update existing sudo time stamp until .osx has finished

13. Install installers (brew, apt-get), depending on operating system

14. Install ShellCheck

15. Install basic utilities (git, jq, tree, etc.) used by many

16. Override defaults in Apple macOS System Preferences:”

17. Image SD card

  1. Reserved.

19. Configure project folder location where files are created by the run

20. Obtain repository from GitHub

21. Reveal secrets stored within .gitsecret folder within repo from GitHub

22. Pipenv and Pyenv to install Python and its modules

23. Connect to Google Cloud, if requested

24. Connect to AWS

25. Install Azure

26. Install K8S minikube

27. Install EKS using eksctl

  1. Open

29. Use CircleCI SaaS

30. Use Yubikey

31. Use GitHub

32. Use HashiCorp Vault

33. Use HashiCorp Vault

34. Put secret in HashiCorp Vault

35. Install NodeJs

36. Install Virtualenv

37. Configure Pyenv with virtualenv

38. Install Anaconda

39. RUN_GOLANG

40. Install Python

41. RUN_TERRAFORM

42. RUN_TENSORFLOW

43. Finish RUN_VIRTUALENV

44. USE_TEST_ENV

45. RUBY_INSTALL

46. RUN_EGGPLANT

47. USE_DOCKER

48. RUN_ACTUAL within Docker

49. UPDATE_GITHUB

50. REMOVE_GITHUB_AFTER folder after run

51. KEEP_PROCESSES after run

52. Delete Docker containers in memory after run …

53. REMOVE_DOCKER_IMAGES downloaded

54. Report Timings


Now a technical dive into each topic:

First line Shebang and comments

#!/usr/bin/env zsh

# is a comment in Bash scripts.

#! is called the “Shebang”.

/usr/bin is the folder that holds the executable program env.

/env is the name of the program that obtains the appropriate shell based on the next nugget. It select the appropriate version.

zsh and bash specify the interpreter. (python is used in Python scripts.)

Unlike the Windows operating system, which decides what program is used to open a file based on the file name “extension” behind the dot, Linux systems ignores the file name and looks into the file to see the first line.

Historically, up to MacOS High Sierra, the “Bourne-compliant” shebang for the Bash v3.2 shell installed in folder /bin by default on . Thus:

#!/bin/bash

/ means the folder is from the root level, above where the operating system stores home files for specific users.

However, Bash v4 is installed (for parallel operation) in another folder:

#!/usr/local/bin/bash

This blog describes what is improved by version 4, such as “associative arrays”.

Disable Shellcheck Linting Rules

  1. This comment line disables (excludes) ShellCheck linter check SC2034 in the file:

    shellcheck disable=SC2001 # See if you can use ${variable//search/replace} instead.
    

    After ShellCheck version 0.4.6, the line can be added anywhere for the next line in the script.

    Alternately, in the script define the code associated with each rule to -exclude from checking:

    export SHELLCHECK_OPTS="-e SC2001 -e SC2059 -e SC2034 -e SC1090"

    Also, the entire script can be copied and pasted online for checking at shellcheck.net, but that can be a security violation. So we install it for local running:

    Install ShellCheck from https://github.com/koalaman/shellcheck

    bash install shellcheck
  2. This script runs ShellCheck to lint itself.

    shellcheck sample.sh

    No response text is issued if no errors were found.

    PROTIP: If ShellCheck finds an issue, the script stops.

Program Version and change description

SCRIPT_VERSION=”v0.82” # Renumber steps

PROTIP: Each time I edit the file, I increment the SCRIPT_VERSION and type a short description of what I intend to change. That text is often used in the git commit message.

Time end - start = elapsed

  • To determine elapsed time, START time stamps are captured as soon as the script starts.

    When the script ends, END time stamps are captured to calculate the elapsed time.

    There are two time stamp formats.

    EPOCH_START="$(date -u +%s)"  # such as 1572634619

    captures the number of minutes since the Jan 1, 1970 epoch point in time.

    LOG_DATETIME=$(date +%Y-%m-%dT%H:%M:%S%z)-$((1 + RANDOM % 1000))

    captures the date in a human-readable year-month-day-hour-minutes “ISO 8601” format which also includes the hours and minutes from UTC/GMT, such as “-600” (for US Central Time) in this sample output:

    sample.sh.2018-04-22T19:26:20-0600-18

    An additional RANDOM number is added to ensure uniqueness among several instances running.

    PROTIP: Values stored in variables during a run do not persist.

    The number of seconds is rounded DOWN, so a run that takes less than a second is measured as 0 seconds.

Indent code 3 spaces

It’s an asthetic choice.

Google’s Style Guide calls for two spaces.

But three spaces make the line indent under if align better. And the if statement is the most common in the script.

Arguments into script

  • The args_prompt() function defines text that is echoed to the console if the script is invoked with no arguments, such as:

    ./mac-setup.zsh -h -v -I -U -c -s -r -a -o

    Checking for whether parameters were added is done by this code:

    if [ $# -eq 0 ]; then  # display if no parameters are provided:
     args_prompt
    fi

    A sample response was shown above.

    The USAGE example shows the various parameters that need to be added to specific actions taken by the script.

    This design ensures the flexibility of the script.

    Flags not associated with a text string specification (such as Verbose) default to false and get switched to true when specified.

    Text variables are defined first, then exported in a separate step as recommended by Shellcheck.

Text color codes in messages

The Unix operating system (on which today’s Linux distributions are based) “streams” text to the Console. Colors (colours) and other effects are specified by inserting “toggles” (attributes) that change the appearing of text following it. A reset sets all text to display in the default appearance.

The color and other text attributes described above are specified within functions called to display message text to the console.

On macOS the approach is to define variables containing ANSI escape numbers:

blink="\e[5m"
blue="\e[34m"
bold="\e[1m"
dim="\e[2m"
green="\e[32m"
red="\e[31m"
reset="\e[0m"
underline="\e[4m"
   

Different Linux distributions and platforms recognize different toggle codes. So use the alternate approach using the tput utility which works on all *nix systems to display attribute variables.

# Set less cryptic color attributes names using tput common to all Linux distributions: 
   blink=$(tput blink)         # 5 as in ANSI 5 in "\e[5m"
   bold=$(tput bold)           # 1
   dim=$(tput dim)             # 2 (faint)
   underline=$(tput smul)      # 4
   end_underline=$(tput rmul)
   reverse=$(tput rev)         # 7
# Foreground colors:
   red=$(tput setaf 1)         # 31
   green=$(tput setaf 2)       # 32
   yellow=$(tput setaf 3)      # 33
   blue=$(tput setaf 4)        # 34
   purple=$(tput setaf 5)      # 35
   cyan=$(tput setaf 6)        # 36
   white=$(tput setaf 7)       # 37
   reset=$(tput setaf 0)       # 39 default
# Background colors:
   b_red=$(tput setb 1)        # 41
   b_green=$(tput setb 2)      # 42
   b_yellow=$(tput setb 3)     # 43
   b_blue=$(tput setb 4)       # 44
   b_purple=$(tput setb 5)     # 45
   b_cyan=$(tput setb 6)       # 46
   b_white=$(tput setb 7)      # 47
   b_reset=$(tput setb 0)      # 49 default
# Reset all to defaults:
   reset=$(tput sgr0)
   

BTW To test how the codes, put this in a script:

echo "${green}Success! ${dim}dimmed${reset} "
echo "${red}Failure ${bold}bolded${reset}"
echo "${blink}${f_yellow}Caution ${bold}bolded${reset} bad"
echo "${blue}Note${reset} blue on black is annoying"
echo "${underline}${purple}Alert${reset} magenta underlined"
echo "${reverse}${cyan}Info${reset} cyan reversed"
echo "${white}Whatever white${reset} this is"
   

Custom functions to echo text to screen

To format output, this code is used:

h2() {     # heading
   printf "\n${bold}>>> %s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
}
info() {   # output on every run
   printf "${dim}\n➜ %s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
}
note() { if [ "${RUN_VERBOSE}" = true ]; then
   printf "${bold}${cyan} ${reset} ${cyan}%s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
   fi
}
success() {
   printf "${green}✔ %s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
}
error() {       # ☓
   printf "${red}${bold}✖ %s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
}
warnNotice() {  # ☛
   printf "${cyan}☛ %s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
}
warnError() {   # Skull: ☠  # Star: ★ ★ U+02606  # Toxic: ☢
   printf "${red}☢ %s${reset}\n" "$(echo "$@" | sed '/./,$!d')"
}
   

“h2” is a homage to HTML heading names. The other functions correspond to the different levels of verbosity used by the log4j library (in the npm aws-code-deploy repo).

The printf command is used instead of echo for compatibility with all versions of Bash.

PROTIP: Notice there are HTML/CSS icons within text, so the file must be stored in UTF-8 format.

bash-scripts-171x139.png -vv sets debugging on to print how the codes look:

h2 "Header here"
info "info"
note "note"
success "success!"
error "error"
warning "warning (warnNotice)"
fatal "fatal (warnError)"
   

## Feature Flags (Run Parameters)

  1. Change what each run of the script does by changing parameters invoking the command, such as -v -I -U -c -s -r -a -o

    bash -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/bash/sample.sh)" -v -I -U -c -s -r -a -o
    

    The script ends with a message like this:

    ✔ End of script after 1883 seconds and 677960 bytes of disk space.
    

    -v for -verbosity adds additional notes.

    -vv for debugging verbosity such as a display example log types.

    -q for -quiet suppression of headers and footers that appear by default, such as when running in production mode.

    -t for -testing mode, which runs local Vault and app servers.

    -I runs installers, but installs each only if it is not already installed.

    -I and -U updates installers even though each is installed. Some installers are invoked only if the feature is also specified. But Homebrew and git are updated if no other utilities are specified.

    -o -opens the sample app in your default browser.


The rest of this article describes coding tricks used and how you might customize the script.

Set variables associated with each parameter flag

There are several ways for shell script to process parameters. The one shown seems the easiest for me to read.

Parameters are processed in the order coded. Multi-character options are coded first.

Clear screen echo

  1. To show responses at the top of the terminal, delete the comment # to enable clear # screen (but not history).

    However, a lot of output would scroll past, so it is rather useless. Better to print a long string as a visual marker to differentiate between different runs:

    echo "========================= $SCRIPT_VERSION"

    The SCRIPT_VERSION is shown at the beginning and the end to detect whether a cached version of the script was used. That happens.

Set “Strict Mode”

At the beginning of the script file is:

set -e  # exits script when a command fails
# set -eu pipefail  # pipefail counts as a parameter
   

Others are there for convenience, to copy and paste to a specific point in the script where commands need to be visible for debugging:

# set -x to show commands for specific issues.
# set -o nounset
   

Some put them all in one line:

set -o nounset -o pipefail -o errexit  # "strict mode"

pipefail means that when the program encounters an exit code != 0, the exit code for the pipeline (Bash script) becomes != 0. E.g. pipefail can be useful to ensure curl does-not-exist-aaaaaaa.com | wc -c doesn’t exit with exit code 0..!>

Some toggle tracing on and off by defining export DEBUG=TRUE and add in the code:

DEBUG="TRUE"
...
if [[ "${DEBUG:-FALSE}" != "FALSE" ]]; then
  set -o xtrace
fi

Save config flag

settings file to $HOME/mac-setup.env (away from GitHub)

Operating System Detection

We code shell scripts to operate in macOS and various distributions of Linux so that developers can focus on processing sequence which are similar on all platforms.

uname is supposed to be available on all versions of Linux and macOS.

Darwin is the internal name of the current macOS operating system. It is based on the NeXTSTEP operating system Steve Jobs brought into Apple upon his return to Apple in 1998. [Wikipedia explains its roots in BSD]

brew is the command used by the Homebrew package manager used by macOS.

Different Linux distributions use different file names to store its version information. And different Linux distributions have their own package manager. Thus we need to obtain the PACKAGE_MANAGER used by the script.

# Check what operating system is in use:
   OS_TYPE="$( uname )"
   OS_DETAILS=""  # default blank.
if [ "$(uname)" == "Darwin" ]; then  # it's on a Mac:
      OS_TYPE="macOS"
      PACKAGE_MANAGER="brew"
elif [ "$(uname)" == "Linux" ]; then  # it's on a Mac:
   if command -v lsb_release ; then
      lsb_release -a
      OS_TYPE="Ubuntu"
      PACKAGE_MANAGER="apt-get"

      silent-apt-get-install(){  # "$1" refers to parameter of package to install:
         sudo DEBIAN_FRONTEND=noninteractive apt-get install -qq "$1" < /dev/null > /dev/null
      }
   elif [ -f "/etc/os-release" ]; then
      OS_DETAILS=$( cat "/etc/os-release" )  # ID_LIKE="rhel fedora"
      OS_TYPE="Fedora"
      PACKAGE_MANAGER="yum"
   elif [ -f "/etc/redhat-release" ]; then
      OS_DETAILS=$( cat "/etc/redhat-release" )
      OS_TYPE="RedHat"
      PACKAGE_MANAGER="yum"
   elif [ -f "/etc/centos-release" ]; then
      OS_TYPE="CentOS"
      PACKAGE_MANAGER="yum"
   else
      error "Linux distribution not anticipated. Please update script. Aborting."
      exit 0
   fi
else 
   error "Operating system not anticipated. Please update script. Aborting."
   exit 0
fi

Version of Bash installed

Some commands make use of a more recent version of Bash than the operating system may have by default. So the script updates the bash processor if the -U flag is specified. Follow along manually:

  1. Be at a macOS Terminal.
  2. Test what version of Bash is installed on your Mac by typing this:

    bash --version
    

    If you see the below, you are using Bash version 3.x, which macOS first released in 2007.

    GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)

    In macOS Mojave version, Apple still ships that old thing due to licensing issues.

  3. Install the latest version of the Bash shell, using Homebrew:

    brew install bash
    

    Bash 4.0 was released in 2009.

    As of this writing, the response is:

    GNU bash, version 5.0.11(1)-release (x86_64-apple-darwin18.6.0)
    Copyright (C) 2019 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
     
    This is free software; you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
  4. If you want to see just the version line, pipe the response to the grep utility built into macOS:

    bash --version | grep 'bash'
    

    Hold down the left Shift key to press the | (called pipe) key at the upper-right of the keyboard.

    grep ‘bash’ filters out lines that do not contain the word “bash” in the response.

apt-get install function

apt-get install commands in this script use a custom function which feeds in the package name to be installed:

silent-apt-get-install "git"

The function is defined where the operating system and package manager is recognized:

silent-apt-get-install(){  # "$1" refers to parameter of package to install:
sudo DEBIAN_FRONTEND=noninteractive apt-get install -qq git htop < /dev/null > /dev/null

DEBIAN_FRONTEND=noninteractive gets rid of “(Reading database … 5%” output.

-qq is there to not request manual confirmation messages such as:

Need to get 260 MB of archives.
After this operation, 308 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
   

The -qq parameter combines the effect of the -y (yes) and -q (quiet) parameter, plus more suppression.

The output that remains is from dpkg which was forked by apt-get. So > /dev/null pipes the standard output (stdout) to nothing so you don’t see them. However, you’ll still see error messages, which go out thru stderr.

< /dev/null pipes stdin standard output to nothing. Explained here.

Bash Traps

The Bash trap command catches signals so it can execute some commands when appropriate, such as cleaning up temp files before the script finishes, called an exit trap.

cleanup() {
    err=$?
    echo "Cleaning stuff up..."
    trap '' EXIT INT TERM
    exit $err 
}
sig_cleanup() {
    trap '' EXIT # some shells will call EXIT after the INT handler
    false # sets $?
    cleanup
}
   

The above cleanup function is invoked when INT TERM occurs to trigger the function, at the bottom of the script:

trap cleanup EXIT
trap sig_cleanup INT QUIT TERM
   

This statement in the script…

trap 'ret=$?; test $ret -ne 0 && printf "failed\n\n" >&2; exit $ret' EXIT

Display OS Info

Define utility functions: Keep Alive, pskill

pskill is not a native function.

Install Installers

For online pair programming where several people can change a file at once, this article notes that there is a plug-in for each of the popular text editors:

Install utilities

ShellCheck Shell script linter

Get secrets from a clear-text file in $HOME folder

Configure project folder

This provides a place for files to be created away from the GitHub folder. This prevents inadvertant upload to public GitHub repo.

Obtain repository from GitHub

Unencrypt GitSecret

This involves storing secrets in GitHub within files encrypted with long-running secrets. This is not very secure because bad actors would have time to break the encryption.

Pipenv

Connect to Google Cloud

Connect to AWS Cloud

Install Azure</a>

Install K8S minikube

Install EKS using eksctl

<hr /. >

Report Timings : Disk space free capacity

We want to know how much disk space is available at the beginning of the run, and the amount of space taken during the run.

On macOS and other BSD operating systems, the “disk free” command df -P / outputs a “standardized” number of 512 byte blocks in the “/” mount:

Filesystem   512-blocks      Used  Available Capacity  Mounted on
/dev/disk1s1 1953595632 521869880 1417651624    27%    /

We use this methodology to obtain the percentage of disk free, which obtains the 12th text item -delimited by a space (which include heading items):

DISK_PCT_FREE=$(read -d '' -ra df_arr < <(LC_ALL=C df -P /); echo "${df_arr[11]}" )

The blocks Available is the 10th text item.

FREE_DISKBLOCKS_START=$(read -d '' -ra df_arr < <(LC_ALL=C df -P /); echo "${df_arr[10]}" )

This uses bash arrays which became available since Bash version 4.

NOTE: We don’t use “-m” for megabytes or “-k” for kilobytes which result in mesuring small amounts of space used as zero.

This captures the starting count:

FREE_DISKBLOCKS_START="$( df . | cut -d' ' -f 6 )"   # e.g. 254781 MB Used
   

TODO: Within cloud environments such as Amazon AWS EC2 or Azure, this may still be relevant.

df is the disk free command used to obtain the number of blocks Used and Available for each storage device mounted.

. specifies calculation of the number of 512 byte blocks in the current device:

Filesystem   512-blocks      Used  Available Capacity iused               ifree %iused  Mounted on
/dev/disk1s1 1953595632 521825264 1417696240    27% 1480293 9223372036853295514    0%   /

| cut -d' ' -f 6 pipes to cut using a space to -demarkate the 6th column. The response is an integer such as “254781”. Divided by 1024 means 248 Gigabytes.

At the end of the script, another variable is obtained when the END variable is obtained for use in calculating the time and disk space used during the script run.


Alternatives to automation

If you’ve already run this, click here to skip to view and edit the install specifications further down this tutorial.

To automate preparation of third-party software for mass deployment onto managed clients:

  • Autopkg at https://github.com/autopkg/autopkg (requies Git) aims to define the steps in a “Recipe” which can be defined in an understood (plist XML-based) format, run automatically instead of by hand, and shared with others.

  • Software distribution server software: Munki (pronounced like “monkey”), Casper, Absolute Manage, etc.

There are two phases to this:

  1. Phase one - supply your password to install Xcode, Homebrew, Tap, and Cask, and Ansible

  2. Phase two - supply your password again for Ansible to install what is not commented out in specification files:

Dot files and Ansible

I haven’t seen it elsewhere on the internet, so I created an explanation of MacOS customizations and “dot files” that set them automatically.

  1. Copy from below this shell script call (highlight the line, then hold down command and press C to the invisible Clipboard):

    
    sh -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/ansible-macos-setup/master/install.sh)"
    
  2. Make a folder and navigate into it:

    
    cd ~
    mkdir gits
    cd gits
    mkdir siyelo
    cd siyelo
    

    You can create another folder name if you prefer, but folder names in this tutorial will be wrong.

  3. In the Terminal window, paste the call from clipboard (hold down command and press V).
  4. Press the enter/return key to invoke the command which causes files to be downloaded and folders created:

     

    The command also downloads folder laptop from https://github.com/siyelo/laptop

    NOTE: Download and installation is skipped if it’s already installed.

  5. When SUDO password: appears, hold down the control key and press C. This message should appear (in red):

    “[ERROR]: User interrupted execution failed”

  6. While still in the Terminal window from above, change to the directory just created:

    cd laptop
    pwd

    You should now be at:

    /Users/mac/gits/siyelo/laptop
    

    Instead of “mac” above, it would show your account name.

    The siyelo playbook.yml file lists the applications to be loaded.

    The scripts/system_settings.sh file are commands to configure the Mac operating system for better security and productivity. It would take hours to set them all manually.

Mac Programs Google Sheet

Switch to use an internet browser to see my list of programs for Mac OSX at this Google spreadsheet online.

The _installer column identifies the technique to install to install each program named:

  • brew (formulas) are installed using the brew command. Programs installed this way are invoked using a text-based Terminal command-line interface.

  • tap uses the brew tap command to install a package within another repository. interesting brew tap.

  • cask packages are installed using the brew cask command which is an extension to Homebrew that allows management of applications with a graphical user interface (GUI).

  • pip are Python plug-ins.
  • npm are Node modules (such as Express).

  • store indicates manual installation using the Apple iTunes program accessing the Apple Store.

  • zip programs are installed after a download, requiring an Ansible task to replace manual clicks and data entry.

WARNING: Each program added uses up more disk space.

WARNING: This may make obsolete documentation based on default Mac settings.

Ansible configurations are “idempotent”, which means that if you run it again, the result is the same. With declarative specifications, items specified for install are not re-installed.

The other columns in the spreadsheet:

_want contains Y for the generator to include in Ansible file.

Homebrew

Cask

Dotfiles

GitHub’s Dotfiles manage files containing preferences for command-line programs.

For example, the configuration file for Zsh is .zshrc. The SSH configuration folder is .ssh. And on.

Such files have a dot because they are hidden.

Famous collections of dotfiles from:

  • https://github.com/holman/dotfiles from Zach Holman
  • Matthias Bynens
  • Paul Irish
  • https://github.com/mattstauffer/ohmyzsh-dotfiles

Features added by dotfiles:

  • Color grep output
  • Boost ls commands
  • Adding plugins to vim

Symlinks to files allow you to git pull and watch your dotfiles instantly update.

One stores .ssh-config file in his Dropbox folder which he symlinks:

   touch ~/Dropbox/.ssh-config
   ln -s ~/Dropbox/.ssh-config ~/.ssh/config
   

The .ssh-config file contains:

# My Awesome Web Site
Host awesome
    Hostname 141.141.141.141
    User me_duh
    IdentityFile=/Users/me/.ssh/id_for_awesome_site.rsa

The above enables him to invoke:


   ssh awesome
   

No more remembering ip addresses, fumbling with command line switches for multiple SSH IDs, or even remembering your ssh usernames.

  • https://mattstauffer.co/blog/setting-up-a-new-os-x-development-machine-part-3-dotfiles-rc-files-and-ssh-config

Edit playbook.yml

  1. Use the vi text editor program to edit the file:

    vi ~/gits/siyelo/laptop/playbook.yml

    • To quit without editing type :q
    • To save changes type :wq (colon, w, and q)
    • To enter insert mode, press I.
    • To escape insert mode, press the Esc key.

     

  2. Have a # (pound sign) in front of each package based on your evaluation.

    Click on each link to open a web page about each. Some links take you to a section lower in this tutorial.

    NOTE: The playbook.yml file contains code to run the system_settings.sh file.

Edit System Settings

  1. List the files and change directory into the scripts folder:

    ls -al cd scripts

    The above list the files and change directory into the scripts folder.

  2. Use the vi text editor program to edit the file:

    vi ~/gits/siyelo/laptop/scripts/system_settings.sh

    • To quit without editing type :q
    • To save changes type :wq (colon, w, and q)

NPM

  1. Search for packages at this site where resources are downloaded for install:

Example:

npm install -g yo

The -g globally installs for access from any folder.

Inventory of Hosts

The inventory of hosts can be defined within /etc/ansible/hosts containing .ini format Microsoft uses:

mail.example.com

[webservers]
foo.example.com
bar.example.com:5309

[dbservers]
one.example.com
two.example.com
three.example.com

Items in square brackets define group names.

Alternately, a YAML format:

Other Ansible Playbooks

Consider other Ansible playbooks for Mac:

|-- LICENSE
|-- README.md
|-- files
|   |-- etc
|   |   `-- sudoers
|   |-- sublime
|   |   `-- Library
|   |       `-- Packages
|   |           `-- User
|   |               |-- DashDoc.sublime-settings
|   |               |-- Package\ Control.sublime-settings
|   |               `-- Preferences.sublime-settings
|   `-- terminal
|       `-- JJG-Term.terminal
|-- inventory
|-- main.yml
|-- requirements.txt
|-- tasks
|   |-- ansible-setup.yml
|   `-- preferences.yml
`-- vars
    `-- main.yml

The requirements.txt file specifies files to download from the Ansible Gaxlaxy sharing site.

Use a text editor to edit the main.yml file at the root. Change user: value from “jgeeling” to your Mac account name.

---
- hosts: localhost
  user: jgeerling
  connection: local

  vars_files:
    - vars/main.yml

  roles:
    - geerlingguy.homebrew
    - geerlingguy.dotfiles

  tasks:
    - include: tasks/ansible-setup.yml
    - include: tasks/preferences.yml

    # TODO: Use sudo once .osx can be run via root with no user interaction.
    - name: Run .osx dotfiles.
      shell: ~/.osx --no-restart
      changed_when: false

Under the vars folder main.yml file is a list of several Homebrew packages.

Run Installation

  1. Open a Termial, go to the directory:

    cd ~/gits/siyelo/laptop/

  2. Run Ansible:

    ansible-playbook playbook.yml -i hosts –ask-sudo-pass -vvvv

  3. Press the return key to invoke the command.

  4. A restart is needed at the end.

Remove app

QUESTION: How to remove apps

Apple Store programs

Programs are installed into the Applications folder. Two different ones:

  • cd /Applications

  • cd ~/Applications to see programs installed by clicking browser links, such as GoToMeeting, etc.

To get rid of the GarageBand music program, which comes with Apple MacOS:

  1. In a Terminal, go to where Apple stores its default apps at:

    cd "/Library/Application Support/GarageBand/"
    

    The quotes are needed because there is a space in the folder name.

  2. See how much disk space it takes.

    ls -al
    

    The number of bytes is to the left of the date:

    -rw-rw-r--   1 root  admin  31211520 Oct 25  2013 Themes.db
    
  3. PROTIP: Delete the contents, but leave the folder name.

    rm -rf *.* /s
    

Ansible Tower

Ansible Tower is a hub for automation tasks. It is a commercial product supported by Red Hat, Inc. but open sourced as AWX in September 2017 at https://github.com/ansible/awx.

An open source alternative to Tower is Semaphore, written in Go.

Others like this

The script described here was based on reviews of other similar “Mac booststrap” scripts:

  • https://gist.github.com/zeekay/7394565

  • https://github.com/fs/osx-bootstrap is a setup for Rails developement, including http://fs.github.io/fs-tool/ for creating pull-requests from command line and Ruby in Heroku.

    ImageMagick

  • https://github.com/divio/osx-bootstrap also asks for RSA keys to GitHub

  • https://github.com/monfresh/laptop provides a set of shell scripts.

Certification

Since Ansible was acquired by Red Hat, Inc. in October 2015, the company also controls certification and provides three training for it.

Resources

https://en.wikipedia.org/wiki/Ansible_(software)

Red Hat’s YouTube channel has all products together.

https://code.tutsplus.com/tutorials/setting-up-a-mac-dev-machine-from-zero-to-hero-with-dotfiles–net-35449

To uninstall, select it and press Command+Delete or drag the program into the Trash.

http://osxdaily.com/2014/01/02/delete-garageband-imovie-iphoto-mac-os-x/

http://osxdaily.com/2014/01/01/4-free-os-x-utilities-mac/

https://www.ansible.com/videos-ansiblefest-sf-2016

5 minutes Kubernetes Cluster on Mac

Ansible on Azure

READ: Ansible & Azure: Automating the Basic Building Blocks of the Azure Cloud

More on OSX

This is one of a series on Mac OSX:

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering