Make (from) Makefile

How the venerable utility is used in Jenkins and GoCD to invoke shell commands around building Docker images for Kubernetes

Overview

Install locally on macOS
Executing make
- Multiple make files
Comments about usage
Rules, recipies, actions, variables
Shell version
Context consistent
Docker example
Simply expanded variable
Include another file
Phony targets
File Globbing
Stops on error
Target Dependencies
Processing dependencies
Make file Linting
References
More on DevOps

NOTE: This page is still actively under construction.

The contribution of this article is the logical ordering of concepts presented in a succinct way, as a hands-on narrated scenic tour.

A Makefile contains a set of directives which the “make” program reads to automate compilation of source code (such as C and java) into binary files (such as class and jar object files).

There is an “mk” program that offers a light version of make.

Install locally on macOS

Install on macOS using Homebrew:
```
brew install make
```
In the response, notice that make is installed for a specific version of macOS:
```
Downloading https://homebrew.bintray.com/bottles/make-4.3.mojave.bottle.tar.gz
```
That means after installed a new version of macOS, upgrade the program.

If it’s already installed, upgrade it:
```
brew upgrade make
```
Now your make program should behave as defined in the latest version of the documentation in a pdf at:

https://www.gnu.org/software/make/manual/make.pdf

At time of writing in January 2020, the 299 manual is for GNU make version 4.3.

Notice GNU make is published by the Free Software Foundation.
For some reason, as of this writing, the version is inextricably reported as “GNU Make 3.81” (not the “4.3” installed) in:
```
make --version
```
```
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
 
This program built for i386-apple-darwin11.3.0
```
The history of versions is at http://git.savannah.gnu.org/cgit/make.git/refs/tags. The first version was over 32 years ago (in the 1970’s), used to build C and the Linux kernel.

Bugs in make source are reported and managed at http://savannah.gnu.org/projects/make

Executing make
Navigate to a folder that doesn’t contain a Makefile.
```
make
```
The message:
```
make: *** No targets specified and no Makefile found.  Stop.
```
“targets” are executable or object files to be made by the program.
Navigate to a folder containing a Makefile.
Executing the make program without a parameter causes the program to process a file specifically named “Makefile” (usually with the capital M) in the same directory folder where the program is invoked.
```
make
```
Multiple make files
If you create more than one Makefile, create a different directory to store each or specify the specific Makefile name:[5]
```
make -f Makefile1
```
For a full list of options:
```
man make
```
At the “:” prompt, type q to quit out.
Edit the sample makefile.

Comments about usage

# (pound characters) mark the beginning of comments, such as these comments about options to invoke make to process a particular Makefile:
```
# Usage:
\# make        # compile all binary
\# make clean  # remove ALL binaries and objects
\# Run on GNU bash, version 5.0.11(1)-release (x86_64-apple-darwin18.6.0)
```
If no parameters are specified, all targets are performed.

Rules, recipies, actions, variables

In the above comment, “clean” refers to a set of rules located at the bottom of the Makefile:
```
clean:
     @echo "Cleaning up..."
     rm -rvf *.o $\{BINS}
```
The colon (:) and the positioning in column 1 on the line defines what is called a “target” under where coding for it is defined.

Lines under each target are called a recipe consisting of action lines to achieve its rule.

PROTIP: Make requires that each action line be indented using a tab, which is usally 4 characters, but can be more if configured that way.

BTW Make has some built-in variables such as “$(RM)” that takes the place of “rm -f” to remove files in a -forced way. Its use enables RM to be redefined with other parameters, such as “-rvf”. Use of variables for commands enable a single file to server multiple platforms. For example, “$(CC)” is the gcc program in one platform but some other program on another platform.

Shell version

Some call Make a kinda shell extension.

Action lines are typically shell script commands such as echo, rm (remove), etc.

PROTIP: Some shell commands are specific to specific versions of the shell program (such as Bash version 4). A Makefile that works well in one shell may not execute properly in another shell. So it helps if the shell assumed being used is part of the comments at the top of the file.
Get the version of Bash to paste as your Makefile’s comment line:
```
bash --version
```
Context consistent

PROTIP: The big difference between how Makefiles run vs. a Bash script is that in a Makefile, each action is evaluated from the same folder. When a cd is issued within an action, the next action does not operate from the changed directory. So put commands for another directory behind a semi-colon on the same line after the cd.

Docker example
```
login:
 docker login -u="${USERNAME}" -p="${PASSWORD}" $(REGISTRY)
 
logout:
 docker logout $(REGISTRY)
```
Variables between curly braces, such as “USERNAME” and “PASSWORD” above, are environment variables.

Replacement operations “$(REGISTRY)” with parentheses are defined instead of hard-coded text to avoid typos - to ensure that values are the same when referenced different times.

Simply expanded variable

The Make program begins by parsing through the Makefile to create an internal dependency tree before taking whatever action is necessary.

Variable assignment code near the top of the Makefile use the := operator to define what are called “simply expanded variables” to associate with the text indicated. The operator is used to avoid infinite loops when referenced [2]. This is in contrast to the “==” recursive expansion which first expands variables inside[11]

So this code:
```
BUILD_BASE := tags/
TAGS := $(shell ls $(BUILD_BASE))
```
after parsing has the variable TAGS to contain shell ls tags/ which, when executed, yields a list of tags. Thus,

$(TAGS) can stand in for several items processed by the rule:
```
$(TAGS):
 docker build -t $(REGISTRY)/$(DOCKER_IMAGE):$(@) -f $(BUILD_BASE)$(@)/Dockerfile --build-arg OWNER=$(OWNER) .
```
Include another file

The variable OWNER above is defined in the file included, by this command, which enable lines in another file to be inserted.
```
include metadata.make
```
The file is located where the agent processing the file is located. ???

Phony targets

Historically, the make program was created to automate compilation of source code (such as C and java) into executables (such as class and jar files). So rules handle files.

But it is not necessary for the target to be a file; it could be just a name for the recipe, as in our example. We call them “phony targets.”

A phony target is one that is not really the name of a file. Rather, it is just a name for a recipe to be executed when you make an explicit request. There are two reasons to use a phony target: to avoid a conflict with a file of the same name, and to improve performance.[10]

Declare Phony targets by a line such as:
```
.PHONY: login logout scan $(TAGS) $(addsuffix .scan, $(TAGS)) $(addsuffix .push, $(TAGS))
```
PROTIP: Not all targets are actually executed. Individual targets (such as .push) can be invoked or not.

File Globbing

A big reason for needing to use a Makefile is to iterate through several similar files specified by wildcard symbols. In this sample:[6]
```
CC=gcc
WFLAGS=-Wall
OBJ=project.o test.o
Exec: $(OBJ)
    $(CC) -o $@  $^ $(WFLAGS)
%.o: %.c
    $(CC) -o $@  -c $< $(WFLAGS)
```
$^ refers to the filenames of all dependencies. It is one of the “automatic variables” defined with a dollar sign.

The .o files depend on the .c files. So, to generate the .o file, represented by the $@ automatic variable which stands in for the filename of the target, Make needs to first -compile the first dependency (prerequisite) file, represented by $<.

Ohter Automatic variables:

$(@) refers to the target file above the action line using it.

$* refers to the target filename without suffix.

$? refers to the prerequisite files with changes.

Stops on error

Unless directed otherwise, make stops when it encounters an error during the construction process. That is why make is used within CI/CD processing within Jenkins, GoCD, etc.

The objective of the whole file is to build a Docker image and push into a Docker Registry:
```
$(addsuffix .push, $(TAGS)):
 docker push $(REGISTRY)/$(DOCKER_IMAGE):$(basename $(@))
```
BTW basename is a built-in Linux command that returns the path without but not the filename after the last slash in the path.

However, that is not invoked if any rules above that fails, such as when vulnerabilities are found while processing this rule:
```
$(addsuffix .scantrivy, $(TAGS)):
 docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
 	-v trivy_db:/root/.cache/ $(TRIVY_SCANNER) $(REGISTRY)/$(DOCKER_IMAGE):$(basename $(@))
```
The docker run command references the Dockerfile in the same folder.

PROTIP: /var/run/docker.sock is the Unix socket file the Docker daemon listens on by default. It is used to communicate with the Docker container by commands such as this to start a container inside Docker:[4]
```
docker run -v /var/run/docker.sock:/var/run/docker.sock -ti alpine sh
apk update && apk add curl
curl -XPOST --unix-socket /var/run/docker.sock http://localhost/events
```
PROTIP: Note: Bind mounting the Docker daemon socket gives a lot of power to a container as it can control the daemon. It must be used with caution, and only with containers we can trust.

Target Dependencies

Each target rule has two parts:

RULE: DEPENDENCY LINE [tab]ACTION LINE(S)

The first line is called a “dependency line”.

Each dependency line is made of two parts.

DEPENDENCY LINE: TARGET FILES: SOURCE FILES

The first part (before the colon) are target files and the second part (after the colon) are called source files. It is called a dependency line because the first part depends on the second part.

Make uses spaces as delimiters between items.

Multiple target files must be separated by a space.

Multiple source files must also be separated by a space.

Processing dependencies

Makes does not necessarily process all rules in the Makefile as all dependencies may not need updating. Make rebuilds only target files which are missing or older than dependency files. It can do that because it keeps track of the last time files (normally object files) were updated.

??? If you have a large program with many source and/or header files, when you change a file on which others depend, you must recompile all the dependent files. Without a Makefile, this is a very time-consuming task.

Make file Linting

There is an “experimental” linter for Makefiles at https://github.com/mrtazz/checkmake

Install the linter’s dependency:
```
brew install pandoc
brew install go
```

Use Golang to clone the repo in the $GOPATH ($/gopkgs):

go get github.com/mrtazz/checkmake
cd $GOPATH/src/github.com/mrtazz/checkmake

build the binary and man page yourself:
```
make
```
WARNING: This is not working for me.

Perform linting

cd location of Makefile
checkmake Makefile

TODO: Add linting to kick off on Git commit.

References

[1] https://www.gnu.org/software/make/manual/make.pdf is the canonical definition, make version 4.3 as of January 2020.

[2] “What is a Makefile and how does it work?” by Sachin Patil (Red Hat)

[3] https://www.slideshare.net/zakariaelktaoui/how-to-make-a-simple-make-file
Introduction to Makefile by Zakaria El ktaoui, Consultant SAP SuccessFactors chez Value Pass Consulting

[4] https://medium.com/better-programming/about-var-run-docker-sock-3bfd276e12fd
Docker Tips : about /var/run/docker.sock

[5] https://getintodevops.com/blog/the-simple-way-to-run-docker-in-docker-for-ci
The simple way to run Docker-in-Docker for CI

[6] http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/

[7] https://scene-si.org/2019/12/04/make-dynamic-makefile-targets/

[8] VIDEO: Makefile Tutorials Mar 7 2017

[9] Makefile Youtube playlist

[10] Gnu make documentation

[11] “Intermediate Project Management with GNU Make”

[12] Wikipedia: Makefile

[13] Using make and writing Makefiles

POSIX standard?

More on DevOps

This is one of a series on DevOps:

Wilson Mar

Make (from) Makefile

Install locally on macOS

Executing make

Multiple make files

Comments about usage

Rules, recipies, actions, variables

Shell version

Context consistent

Docker example

Simply expanded variable

Include another file

Phony targets

File Globbing

Stops on error

Target Dependencies

Processing dependencies

Make file Linting

References

More on DevOps

You might also enjoy (View all posts)