Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Instagram Youtube

Github Stackoverflow Pinterest

Walk though the tricks used in a script to install, configure, and run many programs.


Overview

This page dives into the technical ideosycracies of the “mac-setup-all.sh” bash script file which installs apps on Macs.

This tutorial picks up from this README to the mac-setup script tutorial, which provides someone new to Macs specific steps to configure and run scripts to install apps on Macs. So first finish reading that about “shbangs” and grep for Bash shell versions.

NOTE: This page is still actively under construction (as of July 1, 2018).

A Question of Style

The best professionals I know who work as a team try more to be clear rather than to be clever.

What I try to avoid is:

  • Using a single line when several is more clear. I think it’s OK to use more lines
  • Using complex commands when simple ones do the same
  • Using a lesser number of comparisons if it makes the code unreadable

The more people who can understand the code and make changes without error, the more valuable that script is. Elegance is as elegance does.

Version with Grep

  1. Test what version of Bash is installed on your Mac by typing this:

    bash --version | grep 'bash'
    
    Hold the Shift key to press the (called pipe) key at the upper-right of the keyboard.

    The grep ‘bash’ is needed to filter out lines that do not contain the word “bash” in the response:

    GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)

    Apple still ships version 3.x, which first released in 2007. Bash 4.0 was released in 2009. So you have a more recent version of Bash if you see:

    GNU bash, version 4.4.19(1)-release (x86_64-apple-darwin17.3.0)

    Shebang

    If you have Bash v3 (that comes with MacOS), you can only use this first line used to specify Bash v3 scripts:

    #!/bin/bash

    The above specifies that the script be processed by the bash program in the /bin folder.

    If you have Bash v4, you can use the Bash program installed in folder /usr/local/bin:

    #!/usr/local/bin/bash

    this blog describes what is improved by version 4.

Traps

The Bash trap command catches signals so it can execute some commands when appropriate, such as cleaning up temp files before the script finishes, called an exit trap.

cleanup() {
    err=$?
    echo "Cleaning stuff up..."
    trap '' EXIT INT TERM
    exit $err 
}
sig_cleanup() {
    trap '' EXIT # some shells will call EXIT after the INT handler
    false # sets $?
    cleanup
}
   

The above cleanup function is invoked when INT TERM occurs to trigger the function, at the bottom of the script:

trap cleanup EXIT
trap sig_cleanup INT QUIT TERM
   

This statement in the script…


trap 'ret=$?; test $ret -ne 0 && printf "failed\n\n" >&2; exit $ret' EXIT

Set “Strict Mode”

set -o nounset -o pipefail -o errexit  # "strict mode"

pipefail means that when the program encounters an exit code != 0, the exit code for the pipeline (Bash script) becomes != 0. E.g. pipefail can be useful to ensure curl does-not-exist-aaaaaaa.com | wc -c doesn’t exit with exit code 0..!>

Indent 3 spaces

It’s an asthetic choice.

Google’s Style Guide calls for two spaces.

But three spaces make the line indent under if align better. And the if statement is the most common in the script.

Homebrew

Lint Shellcheck

Coding in this script is linted using ShellCheck online at shellcheck.com or installed from https://github.com/koalaman/shellcheck

To override the triggering of one of its particular rules so that it does not appear as an error, a line like this is added

# shellcheck disable=SC2059

Time start and elapsed

To determine elapsed time, time stamps are captured and the start and end of the script:

Near the script’s beginning, the MacOS date command is used to obtain a starting time stamp:

TIME_START="$(date -u +%s)"

This yields a number counting the number of seconds since the Jan 1, 1970 epoch point in time.

The output is like “1524256274”, which is the number of seconds since the “epoch” of January 1, 1970.

At the end of the script, the END timestamp is obtained for use in calculating the time elapsed during the script run.

Since there may be relationships among several files, all files changed in the same run have the same timestamp.

The file name of the backup contains a date and time stamp in ISO 8601 format such as:

mac-setup-all.sh.2018-04-22T19:26:20-0600-18.log

The coding uses the bash date and RANDOM commands (for microseconds):

LOG_DATETIME=$(date +%Y-%m-%dT%H:%M:%S%z)-$((1 + RANDOM % 1000))
LOGFILE="$HOME/$THISPGM.$LOG_DATETIME.log"
   

Logging to file

The script is designed so the historical record of each run pops up at the end of the script. This is so you can easily scroll and search through the document, something that is difficult on the Console.

What sends statements to a file is this:

echo "something" >>$LOGFILE
   

The file name of the log file contains a date in ISO 8601 format, so multiple editions of the file can be sorted by date.

The file is put in the user’s $HOME folder so that logs don’t accumulate unnoticed in an obscure location.

Logs are not sent to the Linux /var/log folder because on a Mac its default owner for permissions is root:

ls -ld /var/log
drwxr-xr-x  54 root  wheel  1728 Apr 21 05:01 /var/log
   

The script outputs logs to a file.

This is so that during runs, what appears on the command console are only what is relevant to debugging the current issue.

At the end of the script, the log is shown in an editor to enable search through the whole log.

Sudo and Password

To disable inputting password, add this below line in sudoers file:

sudo visudo
yourusername ALL=(root) NOPASSWD: /usr/sbin/installer install
   

NOTE: An XML file can be used to specify inputs.

Disk Space Free and Used

Near the script’s beginning, the MacOS df command is used to obtain the number of blocks available at the start of run:

FREE_DISKBLOCKS_START="$(df | sed -n -e '2{p;q}' | cut -d' ' -f 6)"
   

At the end of the script, the END variable is obtained for use in calculating the space used during the script run.

Functions for dependencies

QUESTION: What are good Bash libraries with common functions? Libraries for bash are not common. One is /etc/rc.d/functions on RedHat-based systems. The file contains functions commonly used in sysV init script.

NOTE: Bash libraries are scarce is due to limitation of Bash functions.

NOTE: Bash’s “functions” have several issues:

Code reusability: Bash functions don’t return anything; they only produce output streams. Every reasonable method of capturing that stream and either assigning it to a variable or passing it as an argument requires a SubShell, which breaks all assignments to outer scopes. (See also BashFAQ/084 for tricks to retrieve results from a function.) Thus, libraries of reusable functions are not feasible, as you can’t ask a function to store its results in a variable whose name is passed as an argument (except by performing eval backflips).

Scope: Bash has a simple system of local scope which roughly resembles “dynamic scope” (e.g. Javascript, elisp). Functions see the locals of their callers (like Python’s “nonlocal” keyword), but can’t access a caller’s positional parameters (except through BASH_ARGV if extdebug is enabled). Reusable functions can’t be guaranteed free of namespace collisions unless you resort to weird naming rules to make conflicts sufficiently unlikely. This is particularly a problem if implementing functions that expect to be acting upon variable names from frame n-3 which may have been overwritten by your reusable function at n-2. Ksh93 can use the more common lexical scope rules by declaring functions with the “function name { … }” syntax (Bash can’t, but supports this syntax anyway).

GITS_PATH

NOTE on testing if a variable is blank.

   if [[ ! -z "${newdir// }" ]]; then  #it's not blank
   

Disk space used

Available space on disk is obtained using command:

FREE_DISKBLOCKS_START=$(df | sed -n -e '2{p;q}' | cut -d' ' -f 6)

The avaiable value is captured from 2nd line, 6th item: 190920080 of the response:

Filesystem   1024-blocks      Used Available Capacity iused               ifree %iused  Mounted on
/dev/disk1s1   488245284 294551984 190920080    61% 2470677 9223372036852305130    0%   /

The number of blocks needs to be converted to MB (megabytes).

File Descriptors

Most services such as databases need more file descriptors than the 2048 default on MacOS. We can see the current limit with this command:

ulimit -n

Use the tee command to concatenate to the bottom of the /etc/profile file:

'ulimit -n 10032' | sudo tee -a /etc/profile

A reboot is necessary for this to take.

http://bencane.com/2013/09/16/understanding-a-little-more-about-etcprofile-and-etcbashrc/

Save backup

See “Reliable Writes” in https://www.greenend.org.uk/rjk/tech/shellmistakes.html

A script overwrites files only after saving a backup copy, naming the file with a suffix of “.bak”

Brew Package Path Linkages

When we run brew install, it adds new packages to a folder under:

/usr/local/Cellar/

For example, running brew install sonar creates a new folder:

/usr/local/Cellar/sonarqube/7.1

(Notice in this case the path contains “sonarqube” rather than “sonar”.)

PROTIP: It’s best that version numbers not be hard-coded into scripts.

QUESTION: How do we access the folder using a path that doesn’t have a version number?

One of the great reasons for using Homebrew is that it automatically creates a symlink from where it installs so that you can execute a command from any folder, such as:

sonar console

Also, when my bash script wants to know if brew install sonar has been run already, it runs:

command -v sonar

If it has not be installed, nothing is returned. If it has been installed, the response is a path such as “/usr/local/bin/sonar”.

BLAH: However, we cannot visit that path using ls /usr/local/bin/sonar:

-bash: cd: /usr/local/bin/sonar: Not a directory

When we execute brew link sonar, it reports:

Warning: Already linked: /usr/local/Cellar/sonarqube/7.1

When we look for “sonar” in the file system using this:

find / -name sonar 2>/dev/null

two paths are reported:

/usr/local/bin/sonar
   /usr/local/Cellar/sonarqube/7.1/bin/sonar
   

However, a find for sonarqube yields:

/usr/local/opt/sonarqube
    /usr/local/var/homebrew/linked/sonarqube
   

PROTIP: So we use that URL found in the full path to the conf file to edit.

MySQL config files

After brew installs mysql, this message appears:

A "/etc/mysql/my.cnf" from another install may interfere with a Homebrew-built
server starting up correctly.
   

There is no folder, but there is folder /usr/local/etc/my.cnf.

NOTE: By default, the OS X installation does not use a my.cnf, and MySQL just uses the default values. To set up your own my.cnf, you could just create a file straight in /etc.

OS X provides example configuration files at /usr/local/mysql/support-files/

BLAH: Thus I do not recommend using MySQL and to avoid installing it. The same errors are occuring for MariaDB as well. So let’s just use Postgresql instead to avoid wasting time and headache. Go to another blog for advice on this terrible program.

No Brew, No problem

Editors

Other apps look to the enviornment variable EDITOR for the commnand to use for displaying text.

export EDITOR='subl -w'
   

“-w” causes the command to not exit until the file is closed.

Scape web page for URL

Most packages setup their installer for easy installation by creating an entry in Homebrew website.

Some don’t do that, such as Gatling. So we have to “scrape” their webpage to obtain the URL that is downloaded when a user manually clicks “DOWNLOAD” on the webpage. An analysis of the page (at gatling.io/download) shows it is one of two URLs which download the bundle.zip file.

DOWNLOAD_URL="gatling-version.html"; wget -q "https://gatling.io/download/" -O $outputFile; cat "$outputFile" | sed -n -e '/<\/header>/,/<\/footer>/ p' | grep "Format:" | sed -n 's/.*href="\([^"]*\).*/\1/p' ; rm -f $outputFile
   

The code above (from Wisdom Hambolu) pulls down the page. The grep filters out all but the line containing “Format:” which has a link to “zip bundle”. The sed function extracts out the URL between the “href=”.

Alternately, Python programmers have a utility called “Beautiful Soup” https://medium.freecodecamp.org/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe installed by pip install BeautifulSoup4

Within the Python program:

import urllib2
from bs4 import BeautifulSoup
quote_page = ‘http://www.bloomberg.com/quote/SPX:IND'
page = urllib2.urlopen(quote_page)
soup = BeautifulSoup(page, ‘html.parser’)
name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})
name = name_box.text.strip() # strip() is used to remove starting and trailing
print name
# get the index price
price_box = soup.find(‘div’, attrs={‘class’:’price’})
price = price_box.text
print price

Others:

  • https://www.joyofdata.de/blog/using-linux-shell-web-scraping/
  • http://www.gregreda.com/2013/03/03/web-scraping-101-with-python/
  • http://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/
  • https://github.com/ContentMine/quickscrape

Install from GitHub Releases

Some apps do not have Homebrew but have installers in GitHub. An example is:

https://github.com/pact-foundation/pact-go/releases

Unlike a brew command, which downloads into its own folder no matter what folder you’re in, the destination folder for downloads using curl needs to be specified in the command or it goes to the current folder.

To download outside Homebrew, right-click the link for “Darwin” to copy for pasting it in a curl command such as:

curl "$DOWNLOAD_URL" -o "$GATLING_HOME/gatling.zip"  # 55.3M renamed
 

DOWNLOAD_URL=”https://github.com/pact-foundation/pact-go/releases/download/v0.0.12/pact-go_darwin_amd64.tar.gz”

The trouble is that

Alternately, the wget command can be used if it has been installed earlier.

Either way, the “gunzip” file needs to be unzipped and verified.

unzip pact-go_darwin_amd64.tar.gz
    
Unzip the package into a known location, 
 ensuring the pact-go binary is on the PATH, next to the pact folder.
Run pact-go to see what options are available.
Run go get -d github.com/pact-foundation/pact-go to install the source packages

Process start/stop/kill

There are several ways to start processes:

  • Invoke the program’s cli command, such as redis-cli start
  • Invoke the brew services start command
  • Invoke launchctl load $HOME/Library/LaunchAgents/homebrew.mxcl.mongodb.plist

Examples to stop processes:

  • redis-cli stop
  • brew services stop command
  • launchctl unload -w /System/Library/LaunchDaemons/org.apache.httpd.plist 2>/dev/null

The typical caveats from a brew install is, for example:

To have launchd start sonarqube now and restart at login:
  brew services start sonarqube
Or, if you don't want/need a background service you can just run:
  sonar console
   

During installation, few installers create a Plist file to define a service that MacOS brings up automatically on boot-up.

The preference is to use an approach which enables orderly saving of what may be in memory. This is preferrable to using a kill command which can be too abrupt.

To kill a process within Terminal stopped while presenting a process, press command+. (period). But in a script, first obtain the PID:


PID="$(ps -A | grep -m1 '/postgresql' | grep -v "grep" | awk '{print $1}')"
kill $PID
   

PROTIP: The “/” in the process name is sometime added if there are several (“child”) processes related to the one PID. Bringing that down would bring down the others automatically. For example:

86700   ??  Ss     0:00.00 postgres: checkpointer process     
86701   ??  Ss     0:00.07 postgres: writer process     
86702   ??  Ss     0:00.05 postgres: wal writer process     
86703   ??  Ss     0:00.04 postgres: autovacuum launcher process     
86704   ??  Ss     0:00.05 postgres: stats collector process     
86705   ??  Ss     0:00.00 postgres: bgworker: logical replication launcher     
86698 s000  S      0:00.04 /usr/local/Cellar/postgresql/10.3/bin/postgres -D /usr/local/var/postgres
87259 s001  S+     0:00.00 grep postgres
   

PROTIP: The grep -v “grep” filters out the grep process itself. Killing/stopping the “postgresql” process also stops several “postres:” automatically.

Variables

  1. Define a variable:

    MY_VARIABLE="x"
    
  2. Clear out a variable as if it was not defined:

    unset MY_VARIABLE
    
  3. Test a variable:

    \# PROTIP: -z tests for zero value (empty).  -v is only available on new versions of Bash.
    if [ -z "$MY_ZONE" ]; then  # not empty
    MY_ZONE="us-central1-b"  # set default value.
    fi
    echo "**** MY_ZONE=\"$MY_ZONE\""
    
  4. Define new verbs so different colors are displayed at various levels of concern:

    function echo_ok { echo -e '\033[1;32m'"$1"'\033[0m'; }
    function echo_warn { echo -e '\033[1;33m'"$1"'\033[0m'; }
    function echo_error  { echo -e '\033[1;31mERROR: '"$1"'\033[0m'; }
    echo_ok "Install starting. You may be asked for your password (for sudo)."
    
  5. If Xcode is not installed, exit the program (quit):

    # Require xcode or quit out:
    xcode-select -p || exit "XCode must be installed! (use the app store)"
    
  6. Set permissions

    cd ~
    mkdir -p tmp
    echo_ok "Setting permissions..."
    for dir in "/usr/local /usr/local/bin /usr/local/include /usr/local/lib /usr/local/share"; do
     sudo chgrp admin $dir
     sudo chmod g+w $dir
    done
    
  7. Make sure Homebrew is installed:

    # homebrew
    export HOMEBREW_CASK_OPTS="--appdir=/Applications"
    if hash brew &> /dev/null; then
    echo_ok "Homebrew already installed"
    else
    echo_warn "Installing homebrew..."
    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    fi
    
  8. The meanjs sample app requires MongoDB to be running, so this code makes that so by concatenating a control keyword text to a variable:

    if echo "$TRYOUT_KEEP" | grep -q "mongodb"; then
       echo "mongodb already in string";
    else
       echo "$TRYOUT_KEEP,mongodb";  # add to string
    fi
    if echo "$TRYOUT" | grep -q "mongodb"; then
       echo "mongodb already in string";
    else
       echo "$TRYOUT,mongodb";  # add to string
    fi
    MONGODB_INSTALL
    

    Before calling MONGO_INSTALL, we mark the strings that brings up the MongoDB service and keeps it running rather than shutting it down (the default action).


Multiple terminals

When a service is started by a shell script within a terminal, no additional commands can be entered while the service runs.

So additional terminal sessions are needed to invoke the client or other services.

The options to do that depend on the operating system.

On Mac, the “open” command is unique to Macs.

   open -a Terminal.app crazy.sh
   
  • See https://stackoverflow.com/questions/19440007/mac-gnome-terminal-equivalent-for-shell-script

To make the terminal stay when the command exits:

In xterm, there is a -hold flag.

In Fedora:

gnome-terminal -e command

  • https://help.gnome.org/users/gnome-terminal/stable/

  • See https://askubuntu.com/questions/46627/how-can-i-make-a-script-that-opens-terminal-windows-and-executes-commands-in-the

  • https://askubuntu.com/questions/484993/run-command-on-anothernew-terminal-window

  • https://stackoverflow.com/questions/42444615/how-to-write-a-shell-script-to-open-four-terminals-and-execute-a-command-in-each

In gnome-terminal, go to Edit -> Profile Preferences -> Title. Click the Command tab. Select Hold the terminal from the drop-down menu labelled When command exits. You should create a new profile for that and execute with

gnome-terminal –window-with-profile=NAMEOFTHEPROFILE -e command

Alternately:

konsole -e command

In konsole there is a –noclose flag.

Pretty much

terminal -e command

Eclips IDE plug-ins

http://download.eclipse.org/releases/juno

Within Eclipse IDE, get a list of plugins at Help -> Install New Software -> Select a repo -> select a plugin -> go to More -> General Information -> Identifier

eclipse -application org.eclipse.equinox.p2.director \
-destination d:/eclipse/ \
-profile SDKProfile  \
-clean -purgeHistory  \
-noSplash \
-repository http://download.eclipse.org/releases/juno/ \
-installIU org.eclipse.cdt.feature.group, \
   org.eclipse.egit.feature.group
   

“Equinox” is the runtime environment of Eclipse, which is the reference implementation of OSGI. Thus, Eclipse plugins are architectually the same as bundles in OSGI.

Notice that there are different versions of Eclipse repositories, such as “juno”.

PROTIP: Although one can install several at once, do it one at a time to see if you can actually use each one. Some of them:

   org.eclipse.cdt.feature.group, \
   org.eclipse.egit.feature.group, \
   org.eclipse.cdt.sdk.feature.group, \
   org.eclipse.linuxtools.cdt.libhover.feature.group, \
   org.eclipse.wst.xml_ui.feature.feature.group, \
   org.eclipse.wst.web_ui.feature.feature.group, \
   org.eclipse.wst.jsdt.feature.feature.group, \
   org.eclipse.php.sdk.feature.group, \
   org.eclipse.rap.tooling.feature.group, \
   org.eclipse.linuxtools.cdt.libhover.devhelp.feature.feature.group, \
   org.eclipse.linuxtools.valgrind.feature.group, \
   

NOTE: A feature group is a list of plugins and other features which can be understood as a logical separate project unit for the updates manager and for the build process.

Testing

https://medium.com/wemake-services/testing-bash-applications-85512e7fe2de

https://github.com/sstephenson/bats

Java tools via Maven, Ant

Apps added by specifying in JAVA_TOOLS are GUI apps.

Most other Java dependencies are specified by manually added in each custom app’s pom.xml file to specify what Maven downloads from the Maven Central online repository of installers at

http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.dbunit%22

Popular in the Maven Repository are:

  • yarn for code generation. JHipster uses it as an integrated tool in Java Spring development.
  • DbUnit extends the JUnit TestCase class to put databases into a known state between test runs. Written by Manuel Laflamme, DbUnit is added in the Maven pom.xml (or Ant) for download from Maven Central. See http://dbunit.wikidot.com/
  • mockito enables calls to be mocked as if they have been creted. Insert file java-mockito-maven.xml as a dependency to maven pom.xml See https://www.youtube.com/watch?v=GKUlQMrbtHE - May 28, 2016 and https://zeroturnaround.com/rebellabs/rebel-labs-report-go-away-bugs-keeping-your-code-safe-with-junit-testng-and-mockito/9/

  • TestNG See http://testng.org/doc/download.html and https://docs.mendix.com/howto/testing/create-automated-tests-with-testng

When using Gradle, insert file java-testng-gradle as a dependency to gradle working within Eclipse plug-in Build from source git://github.com/cbeust/testng.git using ./build-with-gradle

TODO: The Python edition of this will insert specs such as this in pom.xml files.

Pkg silent install

Although Microsoft’s DotNet Core for Mac has a brew formula for the CLI, its SDK is installer is only available using a Pkg.

So we use a command for installing that silently:

echo "$SUDO_PASS" | sudo installer -store -verbose -verboseR -allowUntrusted -pkg "$DOTNET_PROJ/$PKG" -target /
   

The -target value of / is not a path but a device listed by the df command.

There is also the platypus or pkginastall libraries.

Extract version from webpage

To automate “Download .NET SDK” for Mac from page https://www.microsoft.com/net/learn/get-started/macos#install

curl -s https://www.microsoft.com/net/learn/get-started/macos#macos 

extracts the entire page.

grep -B1 "Download .NET SDK"

obtains one line before the text “Download .NET SDK”, which is the two lines:

<a onclick="recordDownload('.NET Core', 'dotnet-sdk-2.1.105-macos-x64-getstarted-installer')"
href="https://download.microsoft.com/download/2/E/C/2EC018A0-A0FC-40A2-849D-AA692F68349E/dotnet-sdk-2.1.105-osx-gs-x64.pkg"
class="btn btn-primary">Download .NET SDK</a>
   
grep href

filters just the lines with href.

grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*"

specifies Extended functionality using the Regular Expression to obtain the URLs (one URL for .exe and one URL for .pkg).

grep -E "*.pkg"

filters the lines to just the one ending with “pkg” file extension (not .exe).

The “basename” command obtains the file name from a file path variable.

PKG_LINK=$(curl -s https://www.microsoft.com/net/learn/get-started/macos#macos | grep -B1 "Download .NET SDK" | grep href | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | grep -E ".pkg")

Jenkins server

To start the Jenkins server to a specified port:

jenkins --httpPort=$JENKINS_PORT  &

The “&” puts the process in the background so that the script can continue running.

The response is a bunch of lines ending with “INFO: Jenkins is fully up and running”.

Several other methods (which don’t work now) are presented on the internet:

sudo defaults write /Library/Preferences/org.jenkins-ci httpPort "$JENKINS_PORT"
   sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist
   sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
   

The command “jenkins” above is actually a bash script that invokes Java:

#!/bin/bash
   JAVA_HOME="$(/usr/libexec/java_home --version 1.8)" \
   exec java  -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war "$@"
   

The code within “$(…)” is run to obtain the value. In this case, it’s:

/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home

The link above is the folder where MacOS keeps the Java SDK. Java executables (java, javac, etc.) are in the bin folder below that location.

The path to jenkins.war and jenkins-cli.war executable files are physcally at:

ls /usr/local/opt/jenkins/libexec

Mac Plist file for Jenkins

Instead of specifying the port in the command, change the configuration file.

On MacOS, services are defined by plist files containing XML, such as this for Jenkins server:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.jenkins</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/libexec/java_home</string>
      <string>-v</string>
      <string>1.8</string>
      <string>--exec</string>
      <string>java</string>
      <string>-Dmail.smtp.starttls.enable=true</string>
      <string>-jar</string>
      <string>/usr/local/opt/jenkins/libexec/jenkins.war</string>
      <string>--httpListenAddress=127.0.0.1</string>
      <string>--httpPort=8080</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>
   

The “1.8” is the version of Java, described below.

The “httpPort=8080” default is customized using this variable in secrets.sh:

  JENKINS_PORT="8082"  # default 8080

The above is file homebrew.mxcl.jenkins.plist within folder /usr/local/opt/jenkins installed by brew. The folder is a symlink created by brew to the physical path where brew installed it:

  /usr/local/Cellar/Jenkins/2.113/homebrew.mxcl.jenkins.plist

The “2.113” means that several versions of Jenkins can be installed side-by-side. This version number changes over time. So it is captured by command:

JENKINS_VERSION=$(jenkins --version)  # 2.113

The folder is actually a symlnk which points to the physical folder defined by: JENKINS_CONF=”/usr/local/Cellar/Jenkins/$JENKINS_VERSION/homebrew.mxcl.jenkins.plist”

The path is defined in a variable so simplify the sed command to make the change:

sed -i "s/httpPort=8080/httpPort=$JENKINS_PORT/g" $JENKINS_CONF
    # --httpPort=8080 is default.

Jenkins GUI in browser

The command to view the server in the default internet browser (such as Safari, Chrome, etc.) is:

open "http://localhost:$JENKINS_PORT"

It’s “http” and not “https” because a certificate has not been established yet.

When executed the first time, Jenkins displays this screen:

However, we don’t want to open it from the command line script, but from a GUI automation script.

Jenkins GUI automation

The script invokes a GUI automation script that opens the file mentioned on the web page above:

/Users/wilsonmar/.jenkins/secrets/initialAdminPassword

“/Users/wilsonmar” is represented by the environment variable named $HOME or ~ symbol, which would be different for you, with your own MacOS account name. Thus, the generic coding is:

JENKINS_SECRET=$(<$HOME/.jenkins/secrets/initialAdminPassword)

The file (and now $JENKINS_SECRET) contains a string in clear-text like “851ed535fd3249ab95a274d23242655c”.

We then call a GUI automation script to get that string to paste it in the box labeled “Administrator Password” based on the id “security-token” defined in this HTML:

<input id="security-token" class="form-control" type="password" name="j_password">
   

This was determined by obtaining the outer HTML from Chrome Developer Tools.

The call is:

python tests/jenkins_secret_chrome.py  chrome  $JENKINS_PORT  $JENKINS_SECRET
   

We use Selenium Python because it reads and writes system environment variables.

Use of Selenium and Python this way requires them to be installed before Jenkins and other web servers.

Jenkins shutdown (kill)

To shut down Jenkins,

PID="ps -A | grep -m1 'jenkins' | awk '{print $1}'"
   fancy_echo "Shutting downn jenkins $PID ..."
   kill $PID

The above is the automated approach to the manual on recommended by many blogs on the internet:

Some say in Finder look for Applications -> Utilities -> Activity Monitor

Others say use command:

ps -el | grep jenkins

Two lines would appear. One is the bash command to do the ps command.

The PID desired is the one that lists the path used to invoke Jenkins, described above:

/usr/bin/java -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war
kill 2134

That is the equivalent of Windows command “taskkill /F /PID XXXX”

There is also:

sudo service jenkins stop

Either way, the response expected is:

INFO: JVM is terminating. Shutting down Winstone

Python GUI Automation

If the title is not found an error message like this appears on the console:

  File "tests/jenkins_secret_chrome.py", line 30, in <module>
    assert "Jenkins [Jenkins]" in driver.title  # bail out if not found.
AssertionError
   

Delay to view

Some put in a 5 second delay:

time.sleep(5)

Use of this feature requires a library to be specified at the top of the file:

import sys

Screen shot picture

Some also take a photo to “prove” that the result was achieved:

driver.save_screenshot('jenkins_secret_chrome.py' +utc_offset_sec+ '.png')

We put the name of the script file in the picture name to trace back to its origin. We put a time stamp in ISO 8601 format so that several png files sort by date.

utc_offset_sec = time.altzone if time.localtime().tm_isdst else time.timezone datetime.datetime.now().replace(tzinfo=datetime.timezone(offset=utc_offset_sec)).isoformat()

The long explanation is https://docs.python.org/2/library/datetime.html

End of script

NOTE:

  • webDriver.Close() - Close the browser window that currently has focus
  • webDriver.Quit() - Calls Dispose()
  • webDriver.Dispose() Closes all browser windows and safely ends the session

driver.quit() means that someone watching the script execute would only see the web app’s screen for a split second.

We prefer to use id rather than name fields because the HTML standard states that id’s are supposed to be unique in each web page.


Groovy

Other similar scripts (listed in “References” below) run

http://groovy-lang.org/install.html

Scape for Fonts in GitHub

Some developers have not put their stuff from GitHub into Homebrew. So we need to read (scrape) the website and see what is listed, then grab the text and URL to download.

Such is the situation with font files at https://github.com/adobe-fonts/source-code-pro/releases/tag/variable-fonts The two files desired downloaded using the curl command are:

  • https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Italic.ttf
  • https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Roman.ttf

The files are downloaded into where MacOS holds fonts available to all users: /Library/Fonts/

ITerm2 can make use of these font files.

Say text out loud

At the bottom of the script is a MacOS command that translates text into voice through the spearker:

say “script ended.” # through speaker

References

https://jasonjwilliamsny.github.io/wrangling-genomics/01-automating_a_workflow.html

https://www.greenend.org.uk/rjk/tech/shellmistakes.html

http://wiki.bash-hackers.org/commands/builtin/set

http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html Debugging bash scripts

https://www.tldp.org/LDP/abs/html/abs-guide.html

http://www.linuxjournal.com/article/9001 on CPU Load Averages

https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes

https://github.com/Bash-it/bash-it a collection of community Bash commands and scripts for Bash 3.2+. (And a shameless ripoff of oh-my-zsh 😃) Includes autocompletion, themes, aliases, custom functions, a few stolen pieces from Steve Losh, and more.

https://github.com/denysdovhan/bash-handbook

See https://scriptingosx.com/2017/10/on-the-shebang/

TODO’s

For AWS there is a cloudping.info website which tells people which region is the quickest. Do you know if there is one like it for Azure and Google. If not we can write a Serverless program to do that.

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud regions
  16. AWS Virtual Private Cloud
  17. Azure Cloud Onramp
  18. Azure Cloud
  19. Azure Cloud Powershell
  20. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  21. Digital Ocean
  22. Cloud Foundry

  23. Packer automation to build Vagrant images
  24. Terraform multi-cloud provisioning automation

  25. Powershell Ecosystem
  26. Powershell on MacOS
  27. Powershell Desired System Configuration

  28. Jenkins Server Setup
  29. Jenkins Plug-ins
  30. Jenkins Freestyle jobs
  31. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  32. Dockerize apps
  33. Docker Setup
  34. Docker Build

  35. Maven on MacOSX

  36. Ansible

  37. MySQL Setup

  38. SonarQube static code scan

  39. API Management Microsoft
  40. API Management Amazon

  41. Scenarios for load