Wilson Mar bio photo

Wilson Mar


Calendar YouTube Github


One integrated pipepline, for free

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


This tutorial is a hands-on technically deep but succinct tutorial about how to use GitLab’s cloud-based CI-CD service to build programming code into packages, test it, deploy it on servers, and monitor apps as they run.

Use GitLab’s CI/CD capability instead of standing up another package such as Jenkins because it is integrated with GitLab’s Git repository and web portal.

In 2017 Forrester rated GitLab v9.3 the top CI vendor:

And that’s before AutoDevOps was announced with v11.

Forrester cites “GitLab has a broad market reach, with over 80,000 active instances using the open source GitLab Community edition and over 500 enterprises paying for GitLab CI/CD.”

Among limitations Forrester quotes “Despite best-in-class extensibility, reference customers cited limitations in notifications and alerts, calling out difficulty integrating with Slack, among other platforms.”

Marketing landing pages:

  • https://about.gitlab.com/auto-devops/
  • https://about.gitlab.com/features/gitlab-ci-cd/

They say “you can use it for free on GitLab.com”. CI/CD capabilitis are included with both the open source GitLab Community Edition (CE) and the proprietary (licensed for money) GitLab Enterprise Edition. However, free accounts have a limit of 2,000 CI pipeline minutes per group per month.

AutoDevOps / Complete DevOps

The brand name was introduced with GitHub 11 to highlight all its capabilities.


To avoid issues, GitLab runs CI/CD work on different machines than ones holding GitLab projects (repositories). you should too.

In GitLab Land, a “runner” is equivalent to a “node” worker in Jenkins. Runners run on the Digital Ocean cloud managed by GitLab. GitLab currently recommends standing up a server with at least 2 CPU cores and 4GB of RAM.

Source to build gitlab-runner is at https://gitlab.com/gitlab-org/gitlab-runner

GitLab Account

PROTIP: CI/CD is enabled on individual projects by an admin of that instance..

  1. Login to https://gitlab.com
  2. The landing page should be a list of your projects (repositories), as if you clicked Project, then Your projects.

    Fork a repo

  3. If you don’t already have one, fork one, such as this associated with the Demo: CI/CD with GitLab Mar 14, 2017 by Joshua Lambert, Product Manager at GitLab:


  4. Click to select the project you want to enable for CI-CD.
  5. Scroll down to Settings in the left menu and select CI / CD.

    The screen provides a short description of ther different types of GitLab Runners:

    Use case:Quick startCOE & Mix of servicesIsolated team
    Privacy level:PublicKinda PrivatePrivate
    Scheduling:fair usageFIFOFIFO
    • PROTIP: If you want a quick start, use
      Shared Runners by clicking the green “Enable shared Runners”.

      Shared Runners are available to any project, like a cafeteria.

      For convenience, this is the default for new Runners created.

      Shared Runners select jobs to run from a process queue using a “fair usage” algorithm. It selects where the lowest number of jobs are running.

      A Shared Runner can be made into a Specific Runner (but not the reverse way).

    • PROTIP: If you have complex set of different services running on different Runners, use
      Specific Runners which have been set to “Restrict projects for this Runner”.

      From GitLab CI, go to Admin on the topbar which should take you to the Manage Projects page. Click the name of your project, then choose ‘Runners’ from the right hand menu. On that page should be a list of instructions, with the third instruction detailing the alternative token. For example:

    • PROTIP: If you are in an isolated team who don’t want to share, use
      Group Runners to execute code all projects belonging to a designated group name.

      Group Runners can be managed using the Runners API (below).

      See http://ci.gitlab.home/admin/runners

Shared Runner token

Tokens for shared Runners are like “9538b0ab”.

Tag “gce” stands for “Google Compute Engine”. In 2018 GitLab migrated from Azure to Google Cloud after Microsoft bought GitHub.

PROTIP: Alternately, automate this step by making a Registration API call.

Specific Runner token

To uniquely identify Runners on the public internet, Tokens (of random characters) are generated by the GitLab server so Runners can authenticate themselves.

  1. Highlight the token displayed and copy to your Clipboard:

    Use the following registration token during setup: 


    The type of jobs that a Runner is equipped to run, such as “Rails”, “Node”, etc. are specified by tags.

    PROTIP: Tags should be defined when a Docker image is defined with components (such as Node, etc.)


In order for client machines to communicate CI / CD commands with GitLab, install the gitlab-runner client executable.

The GitLab Runner client is written in the Go language. So it can run on Linux, macOS, Windows, FreeBSD, Docker – any platform for which can build Go binaries.

Auto DevOps automatically build, test, and deploy apps based on a predefined Continuous Integration and Delivery configuration.

  1. Click button “Reset runners registration token”.

    NOTE: The Runner tokens in the list are the first 8 characters of that runner’s full token.

    PROTIP: If you are concerned about compromise, put an entry on your calendar to reset periodically. TODO: Script to automatically reset and update scripts using the token.

See https://docs.gitlab.com/ce/ci/runners/

and https://wiki.samba.org/index.php/CI_using_Your_own_gitlab_runner

Runner Installation


Until this issue to create a brew package is available to manage versions, follow https://docs.gitlab.com/runner/install/osx.html

  1. Use my bash script to install gitlab-runner on your Mac at https://github.com/wilsonmar/DevSecOps/blob/master/gitlab/gitlab-runner-mac.sh

    Highlight and copy this and paste it in your Terminal:

    sh -c “$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/gitlab/gitlab-runner-mac.sh)”

    There are comments in that script.

    PROTIP: The folder /usr/local/bin/ is also where git modules are installed.

  2. Verify version installed:

    gitlab-runner --version
    Version:      11.2.0
    Git revision: 35e8515d
    Git branch:   11-2-stable
    GO version:   go1.8.7
    Built:        2018-08-22T15:58:04+00:00
    OS/Arch:      darwin/amd64
  3. Run command on its own for sub-commands:

    gitlab-runner - a GitLab Runner
    gitlab-runner [global options] command [command options] [arguments...]
    11.2.0 (35e8515d)
    GitLab Inc. <support@gitlab.com>
      exec                  execute a build locally
      list                  List all configured runners
      run                   run multi runner service
      register              register a new runner
      install               install service
      uninstall             uninstall service
      start                 start service
      stop                  stop service
      restart               restart service
      status                get status of a service
      run-single            start single runner
      unregister            unregister specific runner
      verify                verify all registered runners
      artifacts-downloader  download and extract build artifacts (internal)
      artifacts-uploader    create and upload build artifacts (internal)
      cache-archiver        create and upload cache artifacts (internal)
      cache-extractor       download and extract cache artifacts (internal)
      help, h               Shows a list of commands or help for one command
    --debug                      debug mode [$DEBUG]
    --log-level value, -l value  Log level (options: debug, info, warn, error, fatal, panic)
    --cpuprofile value           write cpu profile to file [$CPU_PROFILE]
    --help, -h                   show help
    --version, -v                print the version
  4. To verify its’s location:

    gitlab-runner list
    Listing configured runners ConfigFile=/Users/wilsonmar/.gitlab-runner/config.toml
  5. cd to ~/.gitlab-runner folder created by the installer to view config.toml. It contains:

    concurrent = 1
    check_interval = 0

    TODO: Describe config.toml


  1. Place a file named .gitlab-ci.yml in the root of any project. This is what the gitlab-runner looks for and runs every time a commit is pushed to the GitLab server.

    QUESTION: Is there a hook file?

    Another example is about CI for publishing NPM modules: https://www.exclamationlabs.com/blog/continuous-deployment-to-npm-using-gitlab-ci/

    A colection of such files are at: https://gitlab.com/gitlab-org/gitlab-ci-yml

  2. PROTIP: Programs thar read Yml files can be finicky. So run a linter by pasting into the online form at:


    PROTIP: Better yet, specify a hook to run the linter automatically with every commit.

    The .gitlab-ci.yml file contains a series of tasks.


    All the files and modifications put in or do inside the .gitlab-ci.yml are reverted each time a commit is pushed to the server. This can be avoided by specifying caches.

    Stages in pipeline

  3. Stages defined in the file are defined to specify the order to be attempted:

             - build
             - test
             - staging
             - openMr
             - prod

    PROTIP: Each column in the “pipeline” displayed on GitLab’s dashboard is named for each stage. So keep stage names short. Thus, many abbreviate “production” with “prod”.

    Errors in any stage would stop progress to follow-on stages.

    NOTE: “openMr” is a custom stage. Some use a stage named “review” instead of “test”, etc.

  4. Actions are defined for each stage. For example:

    Build My App:
      stage: build
    - node
    - yarn config set cache-folder .yarn
    - yarn install
     - npm run build

    Notice the sample action name “Build My App” can contain spaces, and is the action executed for the stage “build” defined in the list of stages above it.

    In this example, the yarn utility is being used instead of npm. And it’s creating a cache folder that will contain all the yarn configurations that will not be recreated when each project runs (each time it’s pushed).

  5. These and addtional steps are defined in the sample at


    as described in blog https://hackernoon.com/configuring-gitlab-ci-yml-150a98e9765d

    Another example is https://linuxhint.com/gitlab_runner_gitlab_ci/ which provides code.

    Deploying to AWS?

    See instructions from @autonix at https://stackoverflow.com/a/38672045/1057052

Get Token

  1. Grab the shared-Runner token on the admin/runners page

    sudo gitlab-runner register
    Running in system-mode.                            
    Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):


  2. Supply the URL, as there is no “press return for default”:

    Please enter the gitlab-ci token for this runner:
  3. Enter a description for the Runner, you can change this later in GitLab’s UI:

    Please enter the gitlab-ci description for this runner

    [hostame] my-runner
  4. Enter the tags associated with the Runner, you can change this later in GitLab’s UI:

    Please enter the gitlab-ci tags for this runner (comma separated):

  5. Enter the Runner executor:

    Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell:

Docker executor

  1. If you chose Docker as your executor, you’ll be asked for the default image to be used for projects that do not define one in .gitlab-ci.yml:

    Please enter the Docker image (eg. ruby:2.1):
  2. Be sure Docker.app is installed on your mac


To hold passwords, secret keys, define variables which the runner applies to environments.

?? They can be protected by only exposing them to protected branches or tags.

Pipeline triggers

To force a specific branch or tag to get rebuilt with an API call, define triggers

to impersonate their associated user including their access to projects and their project permissions.

Docker Container Registry

Installing the Docker image enables auto-scaling of additional instances to automatically meet demand. See https://docs.gitlab.com/runner/executors/docker_machine.html


Login to a dedicated machine where the container registry proxy will be running Make sure that Docker Engine is installed on that machine

Optionally create a new container registry:

docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    --restart always \
    --name registry registry:2

Configure Runners

  1. Configure Runner to “Run untagged jobs option”.

    Click Save changes for the changes to take effect

  2. Set the maximum job timeout

  3. Verify all registered Runners:

    gitlab-runner verify

Configure Jobs

GitLab looks for a .gitlab-ci.yml file to specify what jobs do. Its format is described at:


A tutorial is at:


We don’t use a shared Runner because then you have to enable each project explicitly for the Runner to run its jobs.


To trigger your build script, the default is to use shell.

Your Runner should show in the Manager Runners page on the GitLab CI Admin Dashboard (topbar > Admin > Runners). Depending on your Runner type, it may be green, blue, or red.

git submodule update –init ls -la


As of Omnibus GitLab 9.0, Prometheus and its related exporters are enabled by default, to enable easy and in depth monitoring of GitLab. Approximately 200MB of memory will be consumed by these processes, with default settings.

If you would like to disable Prometheus and it’s exporters or read more information about it, check the Prometheus documentation. GitLab Ru

Do it

When a developer finishes a set of changes, they can open a merge request, which is equivalent to a “pull request” in GitHub. This is so other contributors to the repo can make comments.

A pipeline is a collection of jobs split in different stages. Every commit pushed to GitLab generates a pipeline attached to that commit. If multiple commits are pushed together the pipeline will be created for the last one only.


Based on https://gitlab.com/help/api/runners.md which at time of this writing is v4 with endpoint “https://gitlab.example.com/api/v4/runners”

NameCall(s)With a...
Register a new RunnerPOST /runners
Verify authentication for a registered RunnerPOST /runners/verify
List all runnersGET /runners/all
GET /runners/all?scope=online
List owned runnersGET /runners
GET /runners?scope=active
Get runner's detailsGET /runners/:id
Update runner's detailsPUT /runners/:id
Remove a runnerDELETE /runners/:id
List runner's jobsGET /runners/:id/jobs
List project's runnersGET /projects/:id/runners
Enable a runner in projectPOST /projects/:id/runners
Disable a runner from projectDELETE /projects/:id/runners/:runner_id
Delete a registered RunnerDELETE /runners token


This was created based partly on documentation at:

  • https://docs.gitlab.com/ee/ci/

  • https://about.gitlab.com/2018/01/22/a-beginners-guide-to-continuous-integration/

  • https://codereviewvideos.com/course/your-own-private-github/video/gitlab-ci-runner-tutorial

  • https://neoteric.eu/automate-your-work-with-gitlab-ci-cd-tool

Udemy video: GitLab/GitLab CI Course For Developers and DevOps 7/2017 [2 hrs] by Peng Xiao

More on DevSecOps

This is one of a series on DevSecOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators
  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling
  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft
  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering