MacOS (Command-Line) Utilities

kill, grep, sed, regex, cron, etc. vs. Linux commands

Overview

This article compares and contrasts macOS utilites vs. Linux utilities – the Swiss Army Knife for almost every need.

Uname = Darwin (BSD)

The command that is common to all Linux/BSD variants is the one that returns the operating system name:

uname

On macOS, the response is “Darwin”.

MacOS (Mac OS X) comes with the BSD (Berkeley Standard Distribution) version of command line tools which are slightly different from the Linux version (in Red Hat, Debian, Ubuntu, CoreOS, etc.) even though both are compliant with POSIX standards.

apt (Advanced Packaging Tool)

On Debian and its derivatives is the apt-cache utility that goes with the apt-get package manager (like Homebrew and MacPorts). They are ported to Mac via https://github.com/KubaKaszycki/mac-apt

Use it to search within Python libraries:
```
apt-cache search python3 | wc -l
```
See https://www.dedoimedo.com/computers/aptitude.html

There is also http://rudix.org/ which is a collection of “the hassle-free way to get Unix programs on OS X”. Its packages include zshell and tig (Git spelled backwards), the Text-Mode Interface for Git.

System Preferences GUI

macOS provides a GUI to manage system configuration settings.

Mouse to the upper-left corner of the screen and click the Apple icon that appears to choose “System Preferences…”.

Alternately, you can also open the dialog with this command:
```
open /Applications/System\ Preferences.app
```

BTW, instead of typing out the whole line above, you can simply type “prefs” if you use a text editor to add this line in the ~/.bash_profile file:

alias prefs='open /Applications/System\ Preferences.app'

Apple stores its apps in folder “/Applications”. The “back-slash” character needs to precede every space character in the name because a space usually separates parts of commands.

BTW There is also a folder at “~/Applications” for user-level apps.

Open a Finder window to view files in both folders.

System information

In folder “/Applications/Utilities” are several apps which include “System Information.app”.

macOS provides a GUI to display detailed information about system Hardware, Software, and Networks. Mouse to the upper-left corner of the screen and hold down Option while you click the Apple icon. “System Information…” appears. But you can also open it without the Option key by selecting “About This Mac” then pressing “System Report”.

Instead of examining various Linux config files (/etc/*elease, /proc/meminfo for memory, /proc/cpuinfo for number of cores), macOS has a “system_profiler” utility presenting many data types.

You hot?

Get CPU thermal data on MacOS
```
sysctl machdep.xcpm.cpu_thermal_level
```
PROTIP: Add this among your keyboard shortcuts in aliases.sh.

system_profiler

Internally, the display can be output as text using this command:

system_profiler SPHardwareDataType

PROTIP: Use the command above to obtain your serial number for Apple Support.

“SPHardwareDataType” is one of several DataTypes or items of information listed by:

system_profiler -listDataTypes

The list output:

SPParallelATADataType
SPUniversalAccessDataType
SPApplicationsDataType
SPAudioDataType
SPBluetoothDataType
SPCameraDataType
SPCardReaderDataType
SPComponentDataType
SPDeveloperToolsDataType
SPDiagnosticsDataType
SPDisabledSoftwareDataType
SPDiscBurningDataType
SPEthernetDataType
SPExtensionsDataType
SPFibreChannelDataType
SPFireWireDataType
SPFirewallDataType
SPFontsDataType
SPFrameworksDataType
SPDisplaysDataType
SPHardwareDataType
SPHardwareRAIDDataType
SPInstallHistoryDataType
SPNetworkLocationDataType
SPLogsDataType
SPManagedClientDataType
SPMemoryDataType
SPNVMeDataType
SPNetworkDataType
SPPCIDataType
SPParallelSCSIDataType
SPPowerDataType
SPPrefPaneDataType
SPPrintersSoftwareDataType
SPPrintersDataType
SPConfigurationProfileDataType
SPRawCameraDataType
SPSASDataType
SPSerialATADataType
SPSPIDataType
SPSmartCardsDataType
SPSoftwareDataType
SPStartupItemDataType
SPStorageDataType
SPSyncServicesDataType
SPThunderboltDataType
SPUSBDataType
SPNetworkVolumeDataType
SPWWANDataType
SPAirPortDataType
SPiBridgeDataType

There is a lot of information, so it takes time to generate output.

PROTIP: Save these files to provide to Support:

system_profiler -detailLevel basic > ~/Desktop/system_report.txt

Instead of “basic”, there is also “mini” and “full” scope of output. When providing full scope, output in .spx file extension so that it opens automatically using the GUI:

system_profiler -detailLevel full -xml > ~/Desktop/system_report_mini.spx

CPU Utilization uptime

Was your Mac rebooted recently?

uptime

The response also shows whether CPU utilization is increasing recently:

6:35  up 9 days, 11:56, 7 users, load averages: 2.09 2.24 2.06

“load averages” numbers are calculations of the average system load over three periods of times: the last one-, five-, and fifteen-minute periods. These count the number of processes either using or waiting for CPU (the ready queue or run queue) increments the load number by 1.

In a system with four CPU cores, a load average of 3.73 would indicate that there were, on average, 3.73 processes ready to run, and each one could be scheduled into a CPU.

To identify number of CPU cores

sysctl hw.physicalcpu 
   hw.logicalcpu

Response:

hw.physicalcpu: 6
hw.logicalcpu: 12

Memory Statistics

To obtain a new reading every 5 seconds, the command on macOS is:

vm_stat 5

The number of most interest is pageout, the “3459” in this sample response:

Mach Virtual Memory Statistics: (page size of 4096 bytes)
  free active inac wire   faults     copy zerofill reactive  pageins pageout
 49314  97619 154001 26746 42361341   320895 24148787   145786    50308 3459
 49341  97814 153974 26551     2902       21     1429        0        0 0

On Linux systems, the “vmstat” command is similar but not identical.

To cancel the display, press control+C.
PROTIP: The memory page size is obtained using getconf PAGESIZE, which is “4096”.

See mac-diskspace

Top processes

Is your computer fan loud?

To list the top hungry processes consuming the CPU:
```
top
```
This refreshes automatically.
To cancel the display, press control+C keys (which works with any process you want to kill).
Alternately, install the htop utility using Homebrew:
```
brew install htop
```
The response includes:
```
/usr/local/Cellar/htop/2.2.0_1: 11 files, 188KB
```
Invoke it:
```
htop
```

Processes

So that we can kill it for fun, create a background process (by specifying &) which sleep for 999 seconds:
```
sleep 999 &
```
Get the process identifier:
```
pgrep sleep
```

List background processes:

jobs

[1]+  Running                 sleep 999 &

The + shows the focus.

To list all processes with a niceness (NI) column:
```
ps -l
```
Expand the terminal window width to avoid line wrapping.

UID   PID  PPID        F CPU PRI NI       SZ    RSS WCHAN     S             ADDR TTY           TIME CMD
501  2752  2749     4006   0  31  0  4320804     44 -      Ss                  0 ttys001    0:00.19 /Users/wilsonmar

### Niceness of priority

Default niceness of zero, but can be -20 to +19.

PROTIP: A niceness of +19 is a priority of 99, which is lowest.

nice -n 5 sleep 1000&

root permissions are needed to set nice below zero.

Reset nicer:
```
renice -n 5 sleep 1000&
```
Kill
To kill a single program by name:
```
pkill sleep
```
To kill several progams by name:
```
killall sleep
```

Command Line Tools

To install additional utilities:
```
xcode-select --install
```
List what it installed to folder /Library/Developer/CommandLineTools (containing folders Library, SDKs, and usr):
```
ls /Library/Developer/CommandLineTools/usr/bin/
```

USB info

Install the equivalent of Linux usbutils</stong>:
```
brew install mikhailai/misc/usbutils
usbutils
```
Get the state of kernel objects that the kernel has matched to devices:
```
system_profiler SPUSBDataType
```
This has the same problem as Linux ioreg - it reflects the state of kernel objects that the kernel has matched to devices, not the devices themselves.*
Alternately, to work same problem as ioreg
```
ioreg -p IOUSB -l -w 0
```

lsmod (modules) vs. Apple Kernel Extensions

macOS has Kernel Extensions (kexts) to handle hardware*. Developers and software use the low-level kextload utility to load, kextunload to unload kexts, and kextstat to diagnose problems. There is also the kextutil command.

Although there are no direct equivalent in Linux, the Linux lsmod command lists operating system kernel modules defined in a folder containing “.ko” files:

ls /lib/modules/$(uname -r) -type -f -iname “*.ko”

Such udev rules are loaded in this sequence:

/etc/udev/rules.d
/run/udev/rules.d
/usr/lib/udev/rules.d

coreutils (Core Utilities)

Many who work with Linux distribution avoid minor (but annoying) differences by replacing OS X commands based on BSD with the GNU (Linux) version by installing the “coreutils” family of commands. It’s among Daniel Missler’s The First 10 Things I Do on a New Mac. This is about more than having the same toolset as on Linux machines. See The difference between Linux vs. Mac:

Native capability to search for Unicode strings are not in the Mac (BSD) version of strings.
ANSI-C escape sequences (e.g., \r, \t) beyond \n are not suppoed by the Mac sed command.

See https://en.wikipedia.org/wiki/GNU_Core_Utilities

Get the set of utilities from GNU Linux, but for Mac:
```
brew install coreutils
```

Then you can link:

ln -s /usr/local/bin/gtac /usr/local/bin/tac

NOTE:

NOTE: Add these lines to your .bashrc or .zshrc:

/usr/local/opt/coreutils
```
export PATH="$(brew --prefix coreutils)/libexec/gnubin:/usr/local/bin:$PATH"
```
NOTE: brew --prefix coreutils retrieves the path to the executable in the Homebrew package, which is, at time of writing:

/usr/local/Cellar/coreutils/8.31

PROTIP: Using a command to retrieve the path ensures that the version number from being hard-coded and thus possibly wrong.

Thus, the command is used in ~/.bash_profile to define variables for compilers to find:
```
export CFLAGS="-I$(brew --prefix readline)/include -I$(brew --prefix openssl)/include -I$(xcrun --show-sdk-path)/usr/include"
export LDFLAGS="-L$(brew --prefix readline)/lib -L$(brew --prefix openssl)/lib"
```
Bash shell
Update Bash to version 4:
```
brew install bash
brew link --overwrite bash
```
See https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x/

Zsh

Switching to Zsh from Bash is a rather person choice. But it’s done thus:
```
brew install zsh  # in /bin/zsh
```
Update Mac utilities
Several utilities come installed on macOS, but can be upgraded to the (newest?) version known by brew:
```
brew reinstall m4
brew reinstall make
brew reinstall unzip
```
Because macOS already provides this software, when brew install runs, as its response message says, formulas for them are installed as keg-only, which means brew did not symlink it into /usr/local and installing another version in parallel can cause all kinds of trouble.”

So their installation would require placing their location in front of the default program’s location. That’s why many don’t bother.

Updates specific to text editors:
```
brew install nano  # text editor
brew install emacs
brew install emacs --cocoa --srgb
brew linkapps emacs
brew install vim --override-system-vi
brew install macvim --override-system-vim --custom-system-icons
```

GNU Debugger

Although with its Mavericks version, Apple (along with the transition from GCC to Clang) substituted GDB (the GNU interactive debugger) with LLDB (the standalone LLVM debugger).

Unfortunately, the Eclipse IDE was not capable of communicating with any interactive debugger other than gdb.

NOTE: Install Xcode (version 7.3.1 is known to work). The simplest way is to get is from the App Store. Once it is installed, lldb-mi will reside somewhere under the Xcode folder (it normally is /Applications/Xcode.app/Contents/Developer/usr/bin/lldb-mi). CDT will initialize the default LLDB path to this value if it is present. Note that if you had previous debug configurations with a non-default path for LLDB or if you changed the path in the preferences, the path to lldb will not be automatically set for you. You will have to edit the LLDB path manually in the debug configuration and/or you need to reset the preferences to defaults (if it was modified).

Restore GDB back on your Mac:
```
brew install gdb
```
The above does not create a ~/.gdbinit folder.
Notice in the response “On 10.12 (Sierra) or later with SIP, “brew info gdb” says you need to run:”
```
echo "set startup-with-shell off" >> ~/.gdbinit
```
NOTE: https://sourceware.org/gdb/wiki/BuildingOnDarwin
1. download the most recent GDB from https://www.sourceware.org/gdb/download/
2. expand the gdb-7.12.1.tar.xz file: tar xopf gdb-7.12.1.tar.xz
3. cd gdb-7.12.1 in terminal to open the gdb folder
4. Follow the instructions in the README file in the gdb folder, or simply follow the following steps:
5. ./configure, wait for the terminal
6. make and wait again (which can take some time)
7. sudo make install
8. csrutil enable –without debug
This describes how to code-sign the GDB executable so that macOS will allow it to control other processes. It involves some manual steps.

codesign -s gdb-cert /usr/local/Cellar/gdb/7.12_1/bin/gdb

NOTE: To start dbg, use sudo or define alias gdb=”sudo gdb”

GNU Not pre-installed on macOS

Tutorials make use of some commands, so install them:

brew install gawk # in /usr/local/bin/gawk brew install gzip # in /usr/bin/gzip brew install wget # /usr/local/bin/wget brew install screen # in /usr/bin </pre>

brew install guile  # GNU Ubiquitious Language for Extensions https://www.gnu.org/software/guile/
brew install gpatch
brew install binutils  # https://en.wikipedia.org/wiki/GNU_Binutils

Below are GNU packages on https://www.gnu.org/software but not on macOS:

brew install grep --with-default-names
brew install gnu-indent --with-default-names   # C code prettifier
brew install gnu-sed --with-default-names
brew install gnu-tar --with-default-names
brew install gnu-which --with-default-names
brew install gnutls
brew install ed --with-default-names # in /bin/ed
brew install watch
brew install wdiff --with-gettext
   
   These are search 

   
brew install diffutils
brew install wdiff --with-gettext

–with-default-names prevents Homebrew from prepending a “g” to each command, so they can be used instead of the ones shipped by OS X.

### findutils

Missing from the list above is brew install findutils –with-default-names because that causes ‘brew doctor’ to issue warning: “Putting non-prefixed findutils in your path can cause python builds to fail.”

### GPG

https://superuser.com/questions/655246/are-gnupg-1-and-gnupg-2-compatible-with-each-other

brew install gnupg  # gpg
brew install gnupg2 # gpg2

### Add to MacOS Non-GNU commands

brew install file-formula
brew install less
brew install openssh
brew install perl518   # must run "brew tap homebrew/versions" first!
brew install rsync  # to backup

These are handled by specific request:

brew install git
brew install python
brew install svn  # in /usr/bin/svn

### Utilities for Mac only

brew install htop  # like GNU top
brew install nmap
brew install tmux

### MoreUtils and parallel

brew install moreutils --without-parallel

The Debian description for it is “Unix tools that nobody thought to write long ago, when Unix was young.”

Read about each utility command at https://rentes.github.io/unix/utilities/2015/07/27/moreutils-package/

chronic runs a command quietly unless it fails
combine combines lines in two files using boolean operations
errno look up errno names and descriptions
ifdata get network interface info without parsing ifconfig output
ifne run a program if the standard input is not empty
isutf8 check if a file or standard input is utf-8
lckdo execute a program with a lock held
mispipe pipe two commands, returning the exit status of the first
parallel run multiple jobs at once (conflicts with brew install parallel, so don’t install that stand-alone)
pee tee standard input to pipes
sponge soak up standard input and write to a file. See this
ts timestamp standard input
vidir edit a directory in your text editor
vipe insert a text editor into a pipe
zrun automatically uncompress arguments to command

Its home page at https://joeyh.name/code/moreutils/ says more are on the way.

Riff on it at https://news.ycombinator.com/item?id=9013570

Openssl

To generate a random set of 32 upper and lower case characters (with special characters) for use as a password:
```
openssl rand -base64 32
```
Example:
```
sUZmmvWZ5u+jbpHqdYRMQ2jn0wCrDx2IBc1LcJsyy4w=
```

Certificates

http://sourceware.org/gdb/wiki/BuildingOnDarwin

Cron Launchd Background Jobs

This mentions that Apple has deprecated cron in favor of launchd (a daemon running under the System context). to automatically start (after reboots) service programs at boot time.

If the system is turned off or asleep, cron jobs do not execute until the next designated time occurs.

However, a launchd job will run when the computer wakes up if the computer was asleep when the job should have run (if the StartCalendarInterval key has been set).

Since it’s a background process, launchd doesn’t present a user interface. So you get “launchd cannot be run directly.” when you run launchd like other commands.

The standard way now to run a service on Mac OS X is to use launchd , a program that starts, stops and manages daemons and scripts in Apple OS X environments.

An XML document named with file extension .plist defines its properties. The sample file below defines the Nexus Repository Manager from Sonatype.com installed in /opt:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.sonatype.nexus</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/nexus/bin/nexus</string>
        <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

The above is the com.sonatype.nexus.plist file in /Library/LaunchDaemons/

Change its ownership and access rights:


sudo chown root:wheel /Library/LaunchDaemons/com.sonatype.nexus.plist
sudo chmod 644 /Library/LaunchDaemons/com.sonatype.nexus.plist

PROTIP: Consider setting up a different user to run the repository manager and adapt permissions and the RUN_AS_USER setting in the nexus startup script.

To manually start it after the configuration:


sudo launchctl load /Library/LaunchDaemons/com.sonatype.nexus.plist

Install appium_console gem

gem uninstall -aIx appium_lib
gem uninstall -aIx appium_console
gem install --no-rdoc --no-ri appium_console

Install flaky gem:
https://github.com/appium/flaky
(posix-spawn)

gem uninstall -aIx flaky
gem install --no-rdoc --no-ri flaky

Add wi-fi network

NOTE To avoid the manual effort to add a wi-fi, use this command:


   /usr/sbin/networksetup -addpreferredwirelessnetworkatindex Airport my_ssid 0 my_security my_passkey

my_ssid is the SSID of your network.
my_security is the level of encryption (WEP, WPA, WPA2, etc)
my_passkey is your encryption passkey for your wireless network.

NOTE:


   networksetup -setairportnetwork [interface] [router SSID] [password]

Shells

nix-shell environment on top of nixos/nixpkgs

Mac Message Reset

https://github.com/mattgraham/dotfiles/blob/master/bash/message_reset

References

https://apple.stackexchange.com/questions/69223/how-to-replace-mac-os-x-utilities-with-gnu-core-utilities/69332

https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x/

http://clarkgrubb.com/diagnostic-tools compares Linux, Darwin, and Windows utilities

http://www.brendangregg.com/USEmethod/use-macosx.html

https://support.apple.com/kb/DL75?viewlocale=en_US&locale=en_US Apple’s Common Criteria Tools for 10.5

More on macOS

This is one of a series on macOS (Mac OSX):

Wilson Mar