Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Walk though the tricks (Bashisms) used in a script to install, configure, and run many programs on macOS and Linux

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This page is a deep dive into the technical ideosycracies of shell script files.

This tutorial picks up from this README which provides someone new to Macs specific steps to configure and run scripts to install apps on Macs. So first finish reading that about “shbangs” and grep for Bash shell versions.

NOTE: This page is still actively under construction.

A prime example of a production shell script is Prowler: AWS CIS Benchmark Tool at https://github.com/toniblyx/prowler for AWS Security Best Practices Assessment, Auditing, Hardening and Forensics Readiness.


A Question of Style

The best professionals I know who work as a team try to be more clear rather than be more clever.

The more people who can understand the code and make changes without error, the more valuable that script is. Elegance is as elegance does.

What I try to avoid is:

  • Squeezing several commands into a single line when several lines is more clear. I think it’s OK to use more lines.
  • Using complex commands when simple ones do the same
  • Using syntax not recognized by multiple platforms (recognized by both Bourne and Bash shells on Mac and Linux)

Disk Space of folder

For the script to remove a folder (as in git-patch), we want to provide a feature flag so that is controllable during a particular run, with variable REMOVE_REPO_FROM_WHEN_DONE.

After the folder is supposed to be removed, we want to verify whether it has. There could have been a typo in the command.

If we don’t want it removed, we want to know how much disk space is taken. For that we use the command du -hs which returns something like 319M . which we pipe thru this:

FOLDER_DISK_SPACE="$( du -hs | tr -d '\040\011\012\015\056' )"

The tr -d command gets rid of special characters, specifed in ASCII such as \040 for space, \011 for tabs, \012\015 for Line Feed Carriage return, and \056 for period.

The full logic:

   if [ "$REMOVE_REPO_FROM_WHEN_DONE" -eq "1" ]; then  # 0=No (default), "1"=Yes
      echo_f "Removing $URL_FROM/$PATCH_FILE as REMOVE_REPO_FROM_WHEN_DONE=$REMOVE_REPO_FROM_WHEN_DONE"
      rm -rf  "$REPO_TO_CONTAINER/$REPO_NAME_FROM"
      if [ -d "$REPO_FROM_CONTAINER/$REPO_NAME_FROM" ]; then
         FOLDER_DISK_SPACE="$(du -hs | tr -d '\040\011\012\015\056')"
         echo_f "WARNING: $FOLDER_DISK_SPACE folder still at $REPO_FROM_CONTAINER/$REPO_NAME_FROM."
         ls -al
      fi
   else
      if [ -d "$REPO_FROM_CONTAINER/$REPO_NAME_FROM" ]; then
         FOLDER_DISK_SPACE="$(du -hs | tr -d '\040\011\012\015\056')"
         echo_f "WARNING: $FOLDER_DISK_SPACE folder remains at $REPO_FROM_CONTAINER/$REPO_NAME_FROM."
      else
         echo_f "Folder no longer at $REPO_FROM_CONTAINER/$REPO_NAME_FROM."
      fi
   fi

Logging to file

The script is designed so the historical record of each run pops up at the end of the script. This is so you can easily scroll and search through the document, something that is difficult on the Console.

What sends statements to a file is this:

echo "something" >>$LOGFILE
   

The file name of the log file contains a date in ISO 8601 format, so multiple editions of the file can be sorted by date.

The file is put in the user’s $HOME folder so that logs don’t accumulate unnoticed in an obscure location.

Logs are not sent to the Linux /var/log folder because on a Mac its default owner for permissions is root:

ls -ld /var/log
drwxr-xr-x  54 root  wheel  1728 Apr 21 05:01 /var/log
   

The script outputs logs to a file.

This is so that during runs, what appears on the command console are only what is relevant to debugging the current issue.

At the end of the script, the log is shown in an editor to enable search through the whole log.

Sudo and Password

To disable inputting password, add this below line in sudoers file:

sudo visudo
yourusername ALL=(root) NOPASSWD: /usr/sbin/installer install
   

NOTE: An XML file can be used to specify inputs.

GITS_PATH

NOTE on testing if a variable is blank.

if [[ ! -z "${newdir// }" ]]; then  #it's not blank
   

File Descriptors

Most services such as databases need more file descriptors than the 2048 default on MacOS. We can see the current limit with this command:

ulimit -n

Use the tee command to concatenate to the bottom of the /etc/profile file:

'ulimit -n 10032' | sudo tee -a /etc/profile

A reboot is necessary for this to take. *

Save backup

See “Reliable Writes” in https://www.greenend.org.uk/rjk/tech/shellmistakes.html

A script overwrites files only after saving a backup copy, naming the file with a suffix of “.bak”

Brew Package Path Linkages

When we run brew install, it adds new packages to a folder under:

/usr/local/Cellar/

For example, running brew install sonar creates a new folder:

/usr/local/Cellar/sonarqube/7.1

(Notice in this case the path contains “sonarqube” rather than “sonar”.)

PROTIP: It’s best that version numbers not be hard-coded into scripts.

QUESTION: How do we access the folder using a path that doesn’t have a version number?

One of the great reasons for using Homebrew is that it automatically creates a symlink from where it installs so that you can execute a command from any folder, such as:

sonar console

Also, when my bash script wants to know if brew install sonar has been run already, it runs:

command -v sonar

If it has not be installed, nothing is returned. If it has been installed, the response is a path such as “/usr/local/bin/sonar”.

BLAH: However, we cannot visit that path using ls /usr/local/bin/sonar:

-bash: cd: /usr/local/bin/sonar: Not a directory

When we execute brew link sonar, it reports:

Warning: Already linked: /usr/local/Cellar/sonarqube/7.1

When we look for “sonar” in the file system using this:

find / -name sonar 2>/dev/null

two paths are reported:

/usr/local/bin/sonar
   /usr/local/Cellar/sonarqube/7.1/bin/sonar
   

However, a find for sonarqube yields:

/usr/local/opt/sonarqube
    /usr/local/var/homebrew/linked/sonarqube
   

PROTIP: So we use that URL found in the full path to the conf file to edit.

MySQL config files

After brew installs mysql, this message appears:

A "/etc/mysql/my.cnf" from another install may interfere with a Homebrew-built
server starting up correctly.
   

There is no folder, but there is folder /usr/local/etc/my.cnf.

NOTE: By default, the OS X installation does not use a my.cnf, and MySQL just uses the default values. To set up your own my.cnf, you could just create a file straight in /etc.

OS X provides example configuration files at /usr/local/mysql/support-files/

BLAH: Thus I do not recommend using MySQL and to avoid installing it. The same errors are occuring for MariaDB as well. So let’s just use Postgresql instead to avoid wasting time and headache. Go to another blog for advice on this terrible program.

No Brew, No problem

Editors

Other apps look to the enviornment variable EDITOR for the commnand to use for displaying text.

export EDITOR='subl -w'
   

“-w” causes the command to not exit until the file is closed.

Scape web page for URL

Most packages setup their installer for easy installation by creating an entry in Homebrew website.

Some don’t do that, such as Gatling. So we have to “scrape” their webpage to obtain the URL that is downloaded when a user manually clicks “DOWNLOAD” on the webpage. An analysis of the page (at gatling.io/download) shows it is one of two URLs which download the bundle.zip file.

DOWNLOAD_URL="gatling-version.html"; wget -q "https://gatling.io/download/" -O $outputFile; cat "$outputFile" | sed -n -e '/<\/header>/,/<\/footer>/ p' | grep "Format:" | sed -n 's/.*href="\([^"]*\).*/\1/p' ; rm -f $outputFile
   

The code above (from Wisdom Hambolu) pulls down the page. The grep filters out all but the line containing “Format:” which has a link to “zip bundle”. The sed function extracts out the URL between the “href=”.

Alternately, Python programmers have a utility called “Beautiful Soup” https://medium.freecodecamp.org/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe installed by pip install BeautifulSoup4

Within the Python program:

import urllib2
from bs4 import BeautifulSoup
quote_page = ‘http://www.bloomberg.com/quote/SPX:IND'
page = urllib2.urlopen(quote_page)
soup = BeautifulSoup(page, ‘html.parser’)
name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})
name = name_box.text.strip() # strip() is used to remove starting and trailing
print name
# get the index price
price_box = soup.find(‘div’, attrs={‘class’:’price’})
price = price_box.text
print price

Others:

  • https://www.joyofdata.de/blog/using-linux-shell-web-scraping/
  • http://www.gregreda.com/2013/03/03/web-scraping-101-with-python/
  • http://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/
  • https://github.com/ContentMine/quickscrape

Install from GitHub Releases

Some apps do not have Homebrew but have installers in GitHub. An example is:

https://github.com/pact-foundation/pact-go/releases

Unlike a brew command, which downloads into its own folder no matter what folder you’re in, the destination folder for downloads using curl needs to be specified in the command or it goes to the current folder.

To download outside Homebrew, right-click the link for “Darwin” to copy for pasting it in a curl command such as:

curl "$DOWNLOAD_URL" -o "$GATLING_HOME/gatling.zip"  # 55.3M renamed
 

DOWNLOAD_URL=”https://github.com/pact-foundation/pact-go/releases/download/v0.0.12/pact-go_darwin_amd64.tar.gz”

The trouble is that

Alternately, the wget command can be used if it has been installed earlier.

Either way, the “gunzip” file needs to be unzipped and verified.

unzip pact-go_darwin_amd64.tar.gz
    
Unzip the package into a known location, 
 ensuring the pact-go binary is on the PATH, next to the pact folder.
Run pact-go to see what options are available.
Run go get -d github.com/pact-foundation/pact-go to install the source packages

Process start/stop/kill

There are several ways to start processes:

  • Invoke the program’s cli command, such as redis-cli start
  • Invoke the brew services start command
  • Invoke launchctl load $HOME/Library/LaunchAgents/homebrew.mxcl.mongodb.plist

Examples to stop processes:

  • redis-cli stop
  • brew services stop command
  • launchctl unload -w /System/Library/LaunchDaemons/org.apache.httpd.plist 2>/dev/null

The typical caveats from a brew install is, for example:

To have launchd start sonarqube now and restart at login:
  brew services start sonarqube
Or, if you don't want/need a background service you can just run:
  sonar console
   

During installation, few installers create a Plist file to define a service that MacOS brings up automatically on boot-up.

The preference is to use an approach which enables orderly saving of what may be in memory. This is preferrable to using a kill command which can be too abrupt.

To kill a process within Terminal stopped while presenting a process, press command+. (period). But in a script, first obtain the PID:


PID="$(ps -A | grep -m1 '/postgresql' | grep -v "grep" | awk '{print $1}')"
kill $PID
   

PROTIP: The “/” in the process name is sometime added if there are several (“child”) processes related to the one PID. Bringing that down would bring down the others automatically. For example:

86700   ??  Ss     0:00.00 postgres: checkpointer process     
86701   ??  Ss     0:00.07 postgres: writer process     
86702   ??  Ss     0:00.05 postgres: wal writer process     
86703   ??  Ss     0:00.04 postgres: autovacuum launcher process     
86704   ??  Ss     0:00.05 postgres: stats collector process     
86705   ??  Ss     0:00.00 postgres: bgworker: logical replication launcher     
86698 s000  S      0:00.04 /usr/local/Cellar/postgresql/10.3/bin/postgres -D /usr/local/var/postgres
87259 s001  S+     0:00.00 grep postgres
   

PROTIP: The grep -v “grep” filters out the grep process itself. Killing/stopping the “postgresql” process also stops several “postres:” automatically.

ANSI Color Escape Codes

Bash colors and highlights characters using Escape Codes https://en.wikipedia.org/wiki/ANSI_escape_code

NC='\033[0m' # No Color
Black='\033[0;30m'
Black        0;30     Dark Gray     1;30
Red          0;31     Light Red     1;31
Green        0;32     Light Green   1;32
Brown/Orange 0;33     Yellow        1;33
Blue         0;34     Light Blue    1;34
Purple       0;35     Light Purple  1;35
Cyan         0;36     Light Cyan    1;36
Light Gray   0;37     White         1;37

Variables

  1. Define a variable:

    MY_VARIABLE="x"
    
  2. Clear out a variable as if it was not defined:

    unset MY_VARIABLE
    
  3. Test a variable:

    \# PROTIP: -z tests for zero value (empty).  -v is only available on new versions of Bash.
    if [ -z "$MY_ZONE" ]; then  # not empty
    MY_ZONE="us-central1-b"  # set default value.
    fi
    echo "**** MY_ZONE=\"$MY_ZONE\""
    
  4. If Xcode is not installed, exit the program (quit):

    # Require xcode or quit out:
    xcode-select -p || exit "XCode must be installed! (use the app store)"
    
  5. Set permissions

    cd ~
    mkdir -p tmp
    echo_ok "Setting permissions..."
    for dir in "/usr/local /usr/local/bin /usr/local/include /usr/local/lib /usr/local/share"; do
     sudo chgrp admin $dir
     sudo chmod g+w $dir
    done
    
  6. Make sure Homebrew is installed:

    # homebrew
    export HOMEBREW_CASK_OPTS="--appdir=/Applications"
    if hash brew &> /dev/null; then
    echo_ok "Homebrew already installed"
    else
    echo_warn "Installing homebrew..."
    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    fi
    
  7. The meanjs sample app requires MongoDB to be running, so this code makes that so by concatenating a control keyword text to a variable:

    if echo "$TRYOUT_KEEP" | grep -q "mongodb"; then
       echo "mongodb already in string";
    else
       echo "$TRYOUT_KEEP,mongodb";  # add to string
    fi
    if echo "$TRYOUT" | grep -q "mongodb"; then
       echo "mongodb already in string";
    else
       echo "$TRYOUT,mongodb";  # add to string
    fi
    MONGODB_INSTALL
    

    Before calling MONGO_INSTALL, we mark the strings that brings up the MongoDB service and keeps it running rather than shutting it down (the default action).


Multiple terminals

When a service is started by a shell script within a terminal, no additional commands can be entered while the service runs.

So additional terminal sessions are needed to invoke the client or other services.

The options to do that depend on the operating system.

On Mac, the “open” command is unique to Macs.

open -a Terminal.app crazy.sh
   
  • See https://stackoverflow.com/questions/19440007/mac-gnome-terminal-equivalent-for-shell-script

To make the terminal stay when the command exits:

In xterm, there is a -hold flag.

In Fedora:

gnome-terminal -e command

  • https://help.gnome.org/users/gnome-terminal/stable/

  • See https://askubuntu.com/questions/46627/how-can-i-make-a-script-that-opens-terminal-windows-and-executes-commands-in-the

  • https://askubuntu.com/questions/484993/run-command-on-anothernew-terminal-window

  • https://stackoverflow.com/questions/42444615/how-to-write-a-shell-script-to-open-four-terminals-and-execute-a-command-in-each

In gnome-terminal, go to Edit -> Profile Preferences -> Title. Click the Command tab. Select Hold the terminal from the drop-down menu labelled When command exits. You should create a new profile for that and execute with

gnome-terminal –window-with-profile=NAMEOFTHEPROFILE -e command

Alternately:

konsole -e command

In konsole there is a –noclose flag.

Pretty much

terminal -e command

Eclips IDE plug-ins

http://download.eclipse.org/releases/juno

Within Eclipse IDE, get a list of plugins at Help -> Install New Software -> Select a repo -> select a plugin -> go to More -> General Information -> Identifier

eclipse -application org.eclipse.equinox.p2.director \
-destination d:/eclipse/ \
-profile SDKProfile  \
-clean -purgeHistory  \
-noSplash \
-repository http://download.eclipse.org/releases/juno/ \
-installIU org.eclipse.cdt.feature.group, \
   org.eclipse.egit.feature.group
   

“Equinox” is the runtime environment of Eclipse, which is the reference implementation of OSGI. Thus, Eclipse plugins are architectually the same as bundles in OSGI.

Notice that there are different versions of Eclipse repositories, such as “juno”.

PROTIP: Although one can install several at once, do it one at a time to see if you can actually use each one. Some of them:

   org.eclipse.cdt.feature.group, \
   org.eclipse.egit.feature.group, \
   org.eclipse.cdt.sdk.feature.group, \
   org.eclipse.linuxtools.cdt.libhover.feature.group, \
   org.eclipse.wst.xml_ui.feature.feature.group, \
   org.eclipse.wst.web_ui.feature.feature.group, \
   org.eclipse.wst.jsdt.feature.feature.group, \
   org.eclipse.php.sdk.feature.group, \
   org.eclipse.rap.tooling.feature.group, \
   org.eclipse.linuxtools.cdt.libhover.devhelp.feature.feature.group, \
   org.eclipse.linuxtools.valgrind.feature.group, \
   

NOTE: A feature group is a list of plugins and other features which can be understood as a logical separate project unit for the updates manager and for the build process.

Testing

https://medium.com/wemake-services/testing-bash-applications-85512e7fe2de

https://github.com/sstephenson/bats

Java tools via Maven, Ant

Apps added by specifying in JAVA_TOOLS are GUI apps.

Most other Java dependencies are specified by manually added in each custom app’s pom.xml file to specify what Maven downloads from the Maven Central online repository of installers at

http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.dbunit%22

Popular in the Maven Repository are:

  • yarn for code generation. JHipster uses it as an integrated tool in Java Spring development.
  • DbUnit extends the JUnit TestCase class to put databases into a known state between test runs. Written by Manuel Laflamme, DbUnit is added in the Maven pom.xml (or Ant) for download from Maven Central. See http://dbunit.wikidot.com/
  • mockito enables calls to be mocked as if they have been creted. Insert file java-mockito-maven.xml as a dependency to maven pom.xml See https://www.youtube.com/watch?v=GKUlQMrbtHE - May 28, 2016 and https://zeroturnaround.com/rebellabs/rebel-labs-report-go-away-bugs-keeping-your-code-safe-with-junit-testng-and-mockito/9/

  • TestNG See http://testng.org/doc/download.html and https://docs.mendix.com/howto/testing/create-automated-tests-with-testng

When using Gradle, insert file java-testng-gradle as a dependency to gradle working within Eclipse plug-in Build from source git://github.com/cbeust/testng.git using ./build-with-gradle

TODO: The Python edition of this will insert specs such as this in pom.xml files.

Pkg silent install

Although Microsoft’s DotNet Core for Mac has a brew formula for the CLI, its SDK is installer is only available using a Pkg.

So we use a command for installing that silently:

echo "$SUDO_PASS" | sudo installer -store -verbose -verboseR -allowUntrusted -pkg "$DOTNET_PROJ/$PKG" -target /
   

The -target value of / is not a path but a device listed by the df command.

There is also the platypus or pkginastall libraries.

Extract version from webpage

To automate “Download .NET SDK” for Mac from page https://www.microsoft.com/net/learn/get-started/macos#install

curl -s https://www.microsoft.com/net/learn/get-started/macos#macos 

extracts the entire page.

grep -B1 "Download .NET SDK"

obtains one line before the text “Download .NET SDK”, which is the two lines:

<a onclick="recordDownload('.NET Core', 'dotnet-sdk-2.1.105-macos-x64-getstarted-installer')"
href="https://download.microsoft.com/download/2/E/C/2EC018A0-A0FC-40A2-849D-AA692F68349E/dotnet-sdk-2.1.105-osx-gs-x64.pkg"
class="btn btn-primary">Download .NET SDK</a>
   
grep href

filters just the lines with href.

grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*"

specifies Extended functionality using the Regular Expression to obtain the URLs (one URL for .exe and one URL for .pkg).

grep -E "*.pkg"

filters the lines to just the one ending with “pkg” file extension (not .exe).

The “basename” command obtains the file name from a file path variable.

PKG_LINK=$(curl -s https://www.microsoft.com/net/learn/get-started/macos#macos | grep -B1 "Download .NET SDK" | grep href | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | grep -E ".pkg")

cd / pushd / popd / pwd

To temporarily push into a folder then pop back up to the original folder:

echo $PWD
pushd   # remembers a directory stack.
pwd   # displays the present workding directory (folder).
popd   # returns to the previous directory.

NOTE: Command pwd obtains its value from the variable $PWD which the operating system maintains.

return numbers to 255

PROTIP: CAUTION: bash’s return can only return numbers, and only integers between 0 and 255.

If you have large numbers, put it in a global variable or return a pre-agreed value.

For a shell that can return anything (lists of things), consider es:

es -c "fn f {return (a 'b c' d \$*)}; printf '%s\n' <={f x y}"
a
b c
d
x
y

Jenkins server

To start the Jenkins server to a specified port:

jenkins --httpPort=$JENKINS_PORT  &

The “&” puts the process in the background so that the script can continue running.

The response is a bunch of lines ending with “INFO: Jenkins is fully up and running”.

Several other methods (which don’t work now) are presented on the internet:

sudo defaults write /Library/Preferences/org.jenkins-ci httpPort "$JENKINS_PORT"
   sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist
   sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
   

The command “jenkins” above is actually a bash script that invokes Java:

#!/bin/bash
   JAVA_HOME="$(/usr/libexec/java_home --version 1.8)" \
   exec java  -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war "$@"
   

The code within “$(…)” is run to obtain the value. In this case, it’s:

/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home

The link above is the folder where MacOS keeps the Java SDK. Java executables (java, javac, etc.) are in the bin folder below that location.

The path to jenkins.war and jenkins-cli.war executable files are physcally at:

ls /usr/local/opt/jenkins/libexec

Mac Plist file for Jenkins

Instead of specifying the port in the command, change the configuration file.

On MacOS, services are defined by plist files containing XML, such as this for Jenkins server:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.jenkins</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/libexec/java_home</string>
      <string>-v</string>
      <string>1.8</string>
      <string>--exec</string>
      <string>java</string>
      <string>-Dmail.smtp.starttls.enable=true</string>
      <string>-jar</string>
      <string>/usr/local/opt/jenkins/libexec/jenkins.war</string>
      <string>--httpListenAddress=127.0.0.1</string>
      <string>--httpPort=8080</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>
   

The “1.8” is the version of Java, described below.

The “httpPort=8080” default is customized using this variable in secrets.sh:

  JENKINS_PORT="8082"  # default 8080

The above is file homebrew.mxcl.jenkins.plist within folder /usr/local/opt/jenkins installed by brew. The folder is a symlink created by brew to the physical path where brew installed it:

  /usr/local/Cellar/Jenkins/2.113/homebrew.mxcl.jenkins.plist

The “2.113” means that several versions of Jenkins can be installed side-by-side. This version number changes over time. So it is captured by command:

JENKINS_VERSION=$(jenkins --version)  # 2.113

The folder is actually a symlnk which points to the physical folder defined by: JENKINS_CONF=”/usr/local/Cellar/Jenkins/$JENKINS_VERSION/homebrew.mxcl.jenkins.plist”

The path is defined in a variable so simplify the sed command to make the change:

sed -i "s/httpPort=8080/httpPort=$JENKINS_PORT/g" $JENKINS_CONF
    # --httpPort=8080 is default.

Jenkins GUI in browser

The command to view the server in the default internet browser (such as Safari, Chrome, etc.) is:

open "http://localhost:$JENKINS_PORT"

It’s “http” and not “https” because a certificate has not been established yet.

When executed the first time, Jenkins displays this screen:

However, we don’t want to open it from the command line script, but from a GUI automation script.

Jenkins GUI automation

The script invokes a GUI automation script that opens the file mentioned on the web page above:

/Users/wilsonmar/.jenkins/secrets/initialAdminPassword

“/Users/wilsonmar” is represented by the environment variable named $HOME or ~ symbol, which would be different for you, with your own MacOS account name. Thus, the generic coding is:

JENKINS_SECRET=$(<$HOME/.jenkins/secrets/initialAdminPassword)

The file (and now $JENKINS_SECRET) contains a string in clear-text like “851ed535fd3249ab95a274d23242655c”.

We then call a GUI automation script to get that string to paste it in the box labeled “Administrator Password” based on the id “security-token” defined in this HTML:

<input id="security-token" class="form-control" type="password" name="j_password">
   

This was determined by obtaining the outer HTML from Chrome Developer Tools.

The call is:

python tests/jenkins_secret_chrome.py  chrome  $JENKINS_PORT  $JENKINS_SECRET
   

We use Selenium Python because it reads and writes system environment variables.

Use of Selenium and Python this way requires them to be installed before Jenkins and other web servers.

Jenkins shutdown (kill)

To shut down Jenkins,

PID="ps -A | grep -m1 'jenkins' | awk '{print $1}'"
   fancy_echo "Shutting downn jenkins $PID ..."
   kill $PID

The above is the automated approach to the manual on recommended by many blogs on the internet:

Some say in Finder look for Applications -> Utilities -> Activity Monitor

Others say use command:

ps -el | grep jenkins

Two lines would appear. One is the bash command to do the ps command.

The PID desired is the one that lists the path used to invoke Jenkins, described above:

/usr/bin/java -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war
kill 2134

That is the equivalent of Windows command “taskkill /F /PID XXXX”

There is also:

sudo service jenkins stop

Either way, the response expected is:

INFO: JVM is terminating. Shutting down Winstone

Python GUI Automation

If the title is not found an error message like this appears on the console:

  File "tests/jenkins_secret_chrome.py", line 30, in <module>
    assert "Jenkins [Jenkins]" in driver.title  # bail out if not found.
AssertionError
   

Delay to view

Some put in a 5 second delay:

time.sleep(5)

Use of this feature requires a library to be specified at the top of the file:

import sys

Screen shot picture

Some also take a photo to “prove” that the result was achieved:

driver.save_screenshot('jenkins_secret_chrome.py' +utc_offset_sec+ '.png')

We put the name of the script file in the picture name to trace back to its origin. We put a time stamp in ISO 8601 format so that several png files sort by date.

utc_offset_sec = time.altzone if time.localtime().tm_isdst else time.timezone datetime.datetime.now().replace(tzinfo=datetime.timezone(offset=utc_offset_sec)).isoformat()

The long explanation is https://docs.python.org/2/library/datetime.html

End of script

NOTE:

  • webDriver.Close() - Close the browser window that currently has focus
  • webDriver.Quit() - Calls Dispose()
  • webDriver.Dispose() Closes all browser windows and safely ends the session

driver.quit() means that someone watching the script execute would only see the web app’s screen for a split second.

We prefer to use id rather than name fields because the HTML standard states that id’s are supposed to be unique in each web page.


Groovy

Other similar scripts (listed in “References” below) run

http://groovy-lang.org/install.html

Scape for Fonts in GitHub

Some developers have not put their stuff from GitHub into Homebrew. So we need to read (scrape) the website and see what is listed, then grab the text and URL to download.

Such is the situation with font files at https://github.com/adobe-fonts/source-code-pro/releases/tag/variable-fonts The two files desired downloaded using the curl command are:

  • https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Italic.ttf
  • https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Roman.ttf

The files are downloaded into where MacOS holds fonts available to all users: /Library/Fonts/

ITerm2 can make use of these font files.

“User Data” for EC2 instance bootup

Here is code to install, enable, and start Apache web server:

#!/bin/sh
yum -y install https
chkconfig httpd on
/etc/init.d/httpd start

Say text out loud

At the bottom of the script is a MacOS command that translates text into voice through the spearker:

say “script ended.” # through speaker

Configure location to create new files

Because the script command can be pasted onto any folder, files

Expect for managing manual input

https://www.linuxcloudvps.com/blog/how-to-automate-shell-scripts-with-expect-command/

References

https://dev.to/awwsmm/101-bash-commands-and-tips-for-beginners-to-experts-30je

Rockstars

$18 Mastering Linux Shell Scripting: A practical guide to Linux command-line, Bash scripting, and Shell programming, 2nd Edition Paperback – April 19, 2018 by Mokhtar Ebrahim and Andrew Mallett

Andrew Mallett (@theurbanpenguin, theurbanpenguin.com, Udemy) also created Pluralsight video course Creating Shell Scripts in Enterprise Linux July 31, 2019 [2h 10m]

Books you pay for

Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting 2012 by Mendel Cooper

https://www.greenend.org.uk/rjk/tech/shellmistakes.html

http://wiki.bash-hackers.org/commands/builtin/set

http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html Debugging bash scripts

https://www.tldp.org/LDP/abs/html/abs-guide.html

http://www.linuxjournal.com/article/9001 on CPU Load Averages

https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes

https://github.com/Bash-it/bash-it a collection of community Bash commands and scripts for Bash 3.2+. (And a shameless ripoff of oh-my-zsh ) Includes autocompletion, themes, aliases, custom functions, a few stolen pieces from Steve Losh, and more.

https://github.com/denysdovhan/bash-handbook

See https://scriptingosx.com/2017/10/on-the-shebang/

https://jasonjwilliamsny.github.io/wrangling-genomics/01-automating_a_workflow.html

https://www.amazon.com/Introduction-Unix-Shell-Programming-Venkateshmurthy-ebook-dp-B00B5P724G/dp/B00B5P724G/ref=mt_kindle?_encoding=UTF8&me=&qid=

https://dev.to/awwsmm/101-bash-commands-and-tips-for-beginners-to-experts-30je contrasts commands that are similar (such as which, whereis, whatis)

https://medium.com/the-code-review/top-10-bash-file-system-commands-you-cant-live-without-4cd937bd7df1

https://medium.freecodecamp.org/sh-silence-your-bash-scripts-by-coding-your-own-silent-flag-c7e9f8b668a4

https://www.udemy.com/git-bash/ paid course

Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly)

If you have an O’Reilly subscription: Bash Shel Scripting in 4 hours by Sander van Vugt, who has been teaching Linux since 1995 and has written more than 60 books about different Linux related topics. Sander is the author of the best-selling previous editions of the RHCSA Complete Video Course, as well as many other Red Hat related video courses. He is also a regular speaker on major Linux conferences all over the world.

  1. Writing a shell script with all basic elements (30 minutes), including best practices for writing readable shell scripts.
  2. Working with Variables (40 minutes)
  3. Using Positional Parameters (35 minutes) and how to process their values within the scripts.
  4. Applying Pattern Matching substituion (35 minutes), to shape them exactly as required for specific tasks.
  5. Using looping structures (70 minutes) with common looping structures, such as if, case, for and while.

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud services comparisons (across vendors)
  16. Cloud regions (across vendors)
  17. AWS Virtual Private Cloud

  18. Azure Cloud Onramp
  19. Azure Cloud
  20. Azure Cloud Powershell
  21. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  22. Digital Ocean
  23. Cloud Foundry

  24. Packer automation to build Vagrant images
  25. Terraform multi-cloud provisioning automation
  26. Hashicorp Vault and Consul to generate and hold secrets

  27. Powershell Ecosystem
  28. Powershell on MacOS
  29. Powershell Desired System Configuration

  30. Jenkins Server Setup
  31. Jenkins Plug-ins
  32. Jenkins Freestyle jobs
  33. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  34. Docker (Glossary, Ecosystem, Certification)
  35. Make Makefile for Docker
  36. Docker Setup and run Bash shell script
  37. Bash coding
  38. Docker Setup
  39. Dockerize apps
  40. Docker Registry

  41. Maven on MacOSX

  42. Ansible

  43. MySQL Setup

  44. SonarQube static code scan

  45. API Management Microsoft
  46. API Management Amazon

  47. Scenarios for load