Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

How the venerable utility is used in Jenkins and GoCD to invoke shell commands around building Docker images for Kubernetes

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

NOTE: This page is still actively under construction.

The contribution of this article is the logical ordering of concepts presented in a succint way, as a hands-on narrated scenic tour.

A Makefile contains a set of directives which the “make” program reads to automate compilation of source code (such as C and java) into binary files (such as class and jar object files).

There is an “mk” program that offers a light version of make.

Install locally on macOS

  1. Install on macOS using Homebrew:

    brew install make

    In the response, notice that make is installed for a specific version of macOS:

    Downloading https://homebrew.bintray.com/bottles/make-4.3.mojave.bottle.tar.gz

    That means after installed a new version of macOS, upgrade the program.

    If it’s already installed, upgrade it:

    brew upgrade make
  2. Now your make program should behave as defined in the latest version of the documentation in a pdf at:

    https://www.gnu.org/software/make/manual/make.pdf

    At time of writing in January 2020, the 299 manual is for GNU make version 4.3.

    Notice GNU make is published by the Free Software Foundation.

  3. For some reason, as of this writing, the version is inextricably reported as “GNU Make 3.81” (not the “4.3” installed) in:

    make --version
    GNU Make 3.81
    Copyright (C) 2006  Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.
    There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
    PARTICULAR PURPOSE.
     
    This program built for i386-apple-darwin11.3.0
    

    The history of versions is at http://git.savannah.gnu.org/cgit/make.git/refs/tags. The first version was over 32 years ago (in the 1970’s), used to build C and the Linux kernel.

    Bugs in make source are reported and managed at http://savannah.gnu.org/projects/make

    Executing make

  4. Navigate to a folder that doesn’t contain a Makefile.

    make

    The message:

    make: *** No targets specified and no Makefile found.  Stop.

    “targets” are executable or object files to be made by the program.

  5. Navigate to a folder containing a Makefile.

  6. Executing the make program without a parameter causes the program to process a file specifically named “Makefile” (usually with the capital M) in the same directory folder where the program is invoked.

    make

    Multiple make files

  7. If you create more than one Makefile, create a different directory to store each or specify the specific Makefile name:[5]

    make -f Makefile1
  8. For a full list of options:

    man make

    At the “:” prompt, type q to quit out.

  9. Edit the sample makefile.

    Comments about usage

    # (pound characters) mark the beginning of comments, such as these comments about options to invoke make to process a particular Makefile:

    # Usage:
    \# make        # compile all binary
    \# make clean  # remove ALL binaries and objects
    \# Run on GNU bash, version 5.0.11(1)-release (x86_64-apple-darwin18.6.0)
    

    If no parameters are specified, all targets are performed.

    Rules, recipies, actions, variables

    In the above comment, “clean” refers to a set of rules located at the bottom of the Makefile:

    clean:
         @echo "Cleaning up..."
         rm -rvf *.o $\{BINS}
    

    The colon (:) and the positioning in column 1 on the line defines what is called a “target” under where coding for it is defined.

    Lines under each target are called a recipe consisting of action lines to achieve its rule.

    PROTIP: Make requires that each action line be indented using a tab, which is usally 4 characters, but can be more if configured that way.

    BTW Make has some built-in variables such as “$(RM)” that takes the place of “rm -f” to remove files in a -forced way. Its use enables RM to be redefined with other parameters, such as “-rvf”. Use of variables for commands enable a single file to server multiple platforms. For example, “$(CC)” is the gcc program in one platform but some other program on another platform.

    Shell version

    Some call Make a kinda shell extension.

    Action lines are typically shell script commands such as echo, rm (remove), etc.

    PROTIP: Some shell commands are specific to specific versions of the shell program (such as Bash version 4). A Makefile that works well in one shell may not execute properly in another shell. So it helps if the shell assumed being used is part of the comments at the top of the file.

  10. Get the version of Bash to paste as your Makefile’s comment line:

    bash --version

    Context consistent

    PROTIP: The big difference between how Makefiles run vs. a Bash script is that in a Makefile, each action is evaluated from the same folder. When a cd is issued within an action, the next action does not operate from the changed directory. So put commands for another directory behind a semi-colon on the same line after the cd.

    Docker example

    login:
     docker login -u="${USERNAME}" -p="${PASSWORD}" $(REGISTRY)
     
    logout:
     docker logout $(REGISTRY)
    

    Variables between curly braces, such as “USERNAME” and “PASSWORD” above, are environment variables.

    Replacement operations “$(REGISTRY)” with parentheses are defined instead of hard-coded text to avoid typos - to ensure that values are the same when referenced different times.

    Simply expanded variable

    The Make program begins by parsing through the Makefile to create an internal dependency tree before taking whatever action is necessary.

    Variable assignment code near the top of the Makefile use the := operator to define what are called “simply expanded variables” to associate with the text indicated. The operator is used to avoid infinite loops when referenced [2]. This is in contrast to the “==” recursive expansion which first expands variables inside[11]

    So this code:

    BUILD_BASE := tags/
    TAGS := $(shell ls $(BUILD_BASE))
    

    after parsing has the variable TAGS to contain shell ls tags/ which, when executed, yields a list of tags. Thus,

    $(TAGS) can stand in for several items processed by the rule:

    $(TAGS):
     docker build -t $(REGISTRY)/$(DOCKER_IMAGE):$(@) -f $(BUILD_BASE)$(@)/Dockerfile --build-arg OWNER=$(OWNER) .
    

    Include another file

    The variable OWNER above is defined in the file included, by this command, which enable lines in another file to be inserted.

    include metadata.make

    The file is located where the agent processing the file is located. ???

    Phony targets

    Historically, the make program was created to automate compilation of source code (such as C and java) into executables (such as class and jar files). So rules handle files.

    But it is not necessary for the target to be a file; it could be just a name for the recipe, as in our example. We call them “phony targets.”

    A phony target is one that is not really the name of a file. Rather, it is just a name for a recipe to be executed when you make an explicit request. There are two reasons to use a phony target: to avoid a conflict with a file of the same name, and to improve performance.[10]

    Declare Phony targets by a line such as:

    .PHONY: login logout scan $(TAGS) $(addsuffix .scan, $(TAGS)) $(addsuffix .push, $(TAGS))
    

    PROTIP: Not all targets are actually executed. Individual targets (such as .push) can be invoked or not.

    File Globbing

    A big reason for needing to use a Makefile is to iterate through several similar files specified by wildcard symbols. In this sample:[6]

    CC=gcc
    WFLAGS=-Wall
    OBJ=project.o test.o
    Exec: $(OBJ)
        $(CC) -o $@  $^ $(WFLAGS)
    %.o: %.c
        $(CC) -o $@  -c $< $(WFLAGS)
    

    $^ refers to the filenames of all dependencies. It is one of the “automatic variables” defined with a dollar sign.

    The .o files depend on the .c files. So, to generate the .o file, represented by the $@ automatic variable which stands in for the filename of the target, Make needs to first -compile the first dependency (prerequisite) file, represented by $<.

    Ohter Automatic variables:

    $(@) refers to the target file above the action line using it.

    $* refers to the target filename without suffix.

    $? refers to the prerequisite files with changes.

    Stops on error

    Unless directed otherwise, make stops when it encounters an error during the construction process. That is why make is used within CI/CD processing within Jenkins, GoCD, etc.

    The objective of the whole file is to build a Docker image and push into a Docker Registry:

    $(addsuffix .push, $(TAGS)):
     docker push $(REGISTRY)/$(DOCKER_IMAGE):$(basename $(@))
    

    BTW basename is a built-in Linux command that returns the path without but not the filename after the last slash in the path.

    However, that is not invoked if any rules above that fails, such as when vulnerabilities are found while processing this rule:

    $(addsuffix .scantrivy, $(TAGS)):
     docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
     	-v trivy_db:/root/.cache/ $(TRIVY_SCANNER) $(REGISTRY)/$(DOCKER_IMAGE):$(basename $(@))
    

    The docker run command references the Dockerfile in the same folder.

    PROTIP: /var/run/docker.sock is the Unix socket file the Docker daemon listens on by default. It is used to communicate with the Docker container by commands such as this to start a container inside Docker:[4]

    docker run -v /var/run/docker.sock:/var/run/docker.sock -ti alpine sh
    apk update && apk add curl
    curl -XPOST --unix-socket /var/run/docker.sock http://localhost/events
    

    PROTIP: Note: Bind mounting the Docker daemon socket gives a lot of power to a container as it can control the daemon. It must be used with caution, and only with containers we can trust.

    Target Dependencies

    Each target rule has two parts:

    RULE: DEPENDENCY LINE
    [tab]ACTION LINE(S)

    The first line is called a “dependency line”.

    Each dependency line is made of two parts.

    DEPENDENCY LINE: TARGET FILES: SOURCE FILES

    The first part (before the colon) are target files and the second part (after the colon) are called source files. It is called a dependency line because the first part depends on the second part.

    Make uses spaces as delimiters between items.

    Multiple target files must be separated by a space.

    Multiple source files must also be separated by a space.

    Processing dependencies

    Makes does not necessarily process all rules in the Makefile as all dependencies may not need updating. Make rebuilds only target files which are missing or older than dependency files. It can do that because it keeps track of the last time files (normally object files) were updated.

    ??? If you have a large program with many source and/or header files, when you change a file on which others depend, you must recompile all the dependent files. Without a Makefile, this is a very time-consuming task.

Make file Linting

There is an “experimental” linter for Makefiles at https://github.com/mrtazz/checkmake

  1. Install the linter’s dependency:

    brew install pandoc
    brew install go
  2. Use Golang to clone the repo in the $GOPATH ($/gopkgs):

    go get github.com/mrtazz/checkmake
    cd $GOPATH/src/github.com/mrtazz/checkmake
    
  3. build the binary and man page yourself:

    make
    

    WARNING: This is not working for me.

  4. Perform linting

    cd location of Makefile
    checkmake Makefile
    
  5. TODO: Add linting to kick off on Git commit.

References

[1] https://www.gnu.org/software/make/manual/make.pdf is the canonical definition, make version 4.3 as of January 2020.

[2] “What is a Makefile and how does it work?” by Sachin Patil (Red Hat)

[3] https://www.slideshare.net/zakariaelktaoui/how-to-make-a-simple-make-file
Introduction to Makefile
by Zakaria El ktaoui, Consultant SAP SuccessFactors chez Value Pass Consulting

[4] https://medium.com/better-programming/about-var-run-docker-sock-3bfd276e12fd
Docker Tips : about /var/run/docker.sock

[5] https://getintodevops.com/blog/the-simple-way-to-run-docker-in-docker-for-ci
The simple way to run Docker-in-Docker for CI

[6] http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/

[7] https://scene-si.org/2019/12/04/make-dynamic-makefile-targets/

[8] VIDEO: Makefile Tutorials Mar 7 2017

[9] Makefile Youtube playlist

[10] Gnu make documentation

[11] “Intermediate Project Management with GNU Make”

[12] Wikipedia: Makefile

[13] Using make and writing Makefiles

POSIX standard?


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud services comparisons (across vendors)
  16. Cloud regions (across vendors)
  17. AWS Virtual Private Cloud

  18. Azure Cloud Onramp
  19. Azure Cloud
  20. Azure Cloud Powershell
  21. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  22. Digital Ocean
  23. Cloud Foundry

  24. Packer automation to build Vagrant images
  25. Terraform multi-cloud provisioning automation
  26. Hashicorp Vault and Consul to generate and hold secrets

  27. Powershell Ecosystem
  28. Powershell on MacOS
  29. Powershell Desired System Configuration

  30. Jenkins Server Setup
  31. Jenkins Plug-ins
  32. Jenkins Freestyle jobs
  33. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  34. Docker (Glossary, Ecosystem, Certification)
  35. Make Makefile for Docker
  36. Docker Setup and run Bash shell script
  37. Bash coding
  38. Docker Setup
  39. Dockerize apps
  40. Docker Registry

  41. Maven on MacOSX

  42. Ansible

  43. MySQL Setup

  44. SonarQube static code scan

  45. API Management Microsoft
  46. API Management Amazon

  47. Scenarios for load