One integrated pipepline, for free
- AutoDevOps / Complete DevOps
- GitLab Account
- Get Token
- Docker executor
- Pipeline triggers
- Docker Container Registry
- Configure Runners
- Configure Jobs
- Do it
- More on DevSecOps
This tutorial is a hands-on technically deep but succinct tutorial about how to use GitLab’s cloud-based CI-CD service to build programming code into packages, test it, deploy it on servers, and monitor apps as they run.
Use GitLab’s CI/CD capability instead of standing up another package such as Jenkins because it is integrated with GitLab’s Git repository and web portal.
In 2017 Forrester rated GitLab v9.3 the top CI vendor:
And that’s before AutoDevOps was announced with v11.
Forrester cites “GitLab has a broad market reach, with over 80,000 active instances using the open source GitLab Community edition and over 500 enterprises paying for GitLab CI/CD.”
Among limitations Forrester quotes “Despite best-in-class extensibility, reference customers cited limitations in notifications and alerts, calling out difficulty integrating with Slack, among other platforms.”
Marketing landing pages:
They say “you can use it for free on GitLab.com”. CI/CD capabilitis are included with both the open source GitLab Community Edition (CE) and the proprietary (licensed for money) GitLab Enterprise Edition. However, free accounts have a limit of 2,000 CI pipeline minutes per group per month.
AutoDevOps / Complete DevOps
The brand name was introduced with GitHub 11 to highlight all its capabilities.
GitLab’s Pricing page details the features offered for each level of pricing.
- Idea to Production with GitLab 10.2
To avoid issues, GitLab runs CI/CD work on different machines than ones holding GitLab projects (repositories). you should too.
In GitLab Land, a “runner” is equivalent to a “node” worker in Jenkins. Runners run on the Digital Ocean cloud managed by GitLab. GitLab currently recommends standing up a server with at least 2 CPU cores and 4GB of RAM.
Source to build gitlab-runner is at https://gitlab.com/gitlab-org/gitlab-runner
PROTIP: CI/CD is enabled on individual projects by an admin of that instance..
- Login to https://gitlab.com
The landing page should be a list of your projects (repositories), as if you clicked Project, then Your projects.
Fork a repo
If you don’t already have one, fork one, such as this associated with the Demo: CI/CD with GitLab Mar 14, 2017 by Joshua Lambert, Product Manager at GitLab:
- Click to select the project you want to enable for CI-CD.
Scroll down to Settings in the left menu and select CI / CD.
The screen provides a short description of ther different types of GitLab Runners:
Shared Specific Group Use case: Quick start COE & Mix of services Isolated team Privacy level: Public Kinda Private Private Scheduling: fair usage FIFO FIFO
PROTIP: If you want a quick start, use
Shared Runners by clicking the green “Enable shared Runners”.
Shared Runners are available to any project, like a cafeteria.
For convenience, this is the default for new Runners created.
Shared Runners select jobs to run from a process queue using a “fair usage” algorithm. It selects where the lowest number of jobs are running.
A Shared Runner can be made into a Specific Runner (but not the reverse way).
PROTIP: If you have complex set of different services running on different Runners, use
Specific Runners which have been set to “Restrict projects for this Runner”.
From GitLab CI, go to Admin on the topbar which should take you to the Manage Projects page. Click the name of your project, then choose ‘Runners’ from the right hand menu. On that page should be a list of instructions, with the third instruction detailing the alternative token. For example:
PROTIP: If you are in an isolated team who don’t want to share, use
Group Runners to execute code all projects belonging to a designated group name.
Group Runners can be managed using the Runners API (below).
Shared Runner token
Tokens for shared Runners are like “9538b0ab”.
Tag “gce” stands for “Google Compute Engine”. In 2018 GitLab migrated from Azure to Google Cloud after Microsoft bought GitHub.
PROTIP: Alternately, automate this step by making a Registration API call.
Specific Runner token
To uniquely identify Runners on the public internet, Tokens (of random characters) are generated by the GitLab server so Runners can authenticate themselves.
Highlight the token displayed and copy to your Clipboard:
Use the following registration token during setup: 1HEHB_jN-wttCDBQnK2n
The type of jobs that a Runner is equipped to run, such as “Rails”, “Node”, etc. are specified by tags.
PROTIP: Tags should be defined when a Docker image is defined with components (such as Node, etc.)
In order for client machines to communicate CI / CD commands with GitLab, install the gitlab-runner client executable.
The GitLab Runner client is written in the Go language. So it can run on Linux, macOS, Windows, FreeBSD, Docker – any platform for which can build Go binaries.
Auto DevOps automatically build, test, and deploy apps based on a predefined Continuous Integration and Delivery configuration.
Click button “Reset runners registration token”.
NOTE: The Runner tokens in the list are the first 8 characters of that runner’s full token.
PROTIP: If you are concerned about compromise, put an entry on your calendar to reset periodically. TODO: Script to automatically reset and update scripts using the token.
Until this issue to create a brew package is available to manage versions, follow https://docs.gitlab.com/runner/install/osx.html
Use my bash script to install gitlab-runner on your Mac at https://github.com/wilsonmar/DevSecOps/blob/master/gitlab/gitlab-runner-mac.sh
Highlight and copy this and paste it in your Terminal:
sh -c “$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/DevSecOps/master/gitlab/gitlab-runner-mac.sh)”
There are comments in that script.
PROTIP: The folder /usr/local/bin/ is also where git modules are installed.
Verify version installed:
Version: 11.2.0 Git revision: 35e8515d Git branch: 11-2-stable GO version: go1.8.7 Built: 2018-08-22T15:58:04+00:00 OS/Arch: darwin/amd64
Run command on its own for sub-commands:
NAME: gitlab-runner - a GitLab Runner USAGE: gitlab-runner [global options] command [command options] [arguments...] VERSION: 11.2.0 (35e8515d) AUTHOR: GitLab Inc. <email@example.com> COMMANDS: exec execute a build locally list List all configured runners run run multi runner service register register a new runner install install service uninstall uninstall service start start service stop stop service restart restart service status get status of a service run-single start single runner unregister unregister specific runner verify verify all registered runners artifacts-downloader download and extract build artifacts (internal) artifacts-uploader create and upload build artifacts (internal) cache-archiver create and upload cache artifacts (internal) cache-extractor download and extract cache artifacts (internal) help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --debug debug mode [$DEBUG] --log-level value, -l value Log level (options: debug, info, warn, error, fatal, panic) --cpuprofile value write cpu profile to file [$CPU_PROFILE] --help, -h show help --version, -v print the version
To verify its’s location:
Listing configured runners ConfigFile=/Users/wilsonmar/.gitlab-runner/config.toml
cd to ~/.gitlab-runner folder created by the installer to view config.toml. It contains:
concurrent = 1 check_interval = 0
TODO: Describe config.toml
Place a file named .gitlab-ci.yml in the root of any project. This is what the gitlab-runner looks for and runs every time a commit is pushed to the GitLab server.
QUESTION: Is there a hook file?
Another example is about CI for publishing NPM modules: https://www.exclamationlabs.com/blog/continuous-deployment-to-npm-using-gitlab-ci/
A colection of such files are at: https://gitlab.com/gitlab-org/gitlab-ci-yml
PROTIP: Programs thar read Yml files can be finicky. So run a linter by pasting into the online form at:
PROTIP: Better yet, specify a hook to run the linter automatically with every commit.
The .gitlab-ci.yml file contains a series of tasks.
All the files and modifications put in or do inside the .gitlab-ci.yml are reverted each time a commit is pushed to the server. This can be avoided by specifying caches.
Stages in pipeline
Stages defined in the file are defined to specify the order to be attempted:
stages: - build - test - staging - openMr - prod
PROTIP: Each column in the “pipeline” displayed on GitLab’s dashboard is named for each stage. So keep stage names short. Thus, many abbreviate “production” with “prod”.
Errors in any stage would stop progress to follow-on stages.
NOTE: “openMr” is a custom stage. Some use a stage named “review” instead of “test”, etc.
Actions are defined for each stage. For example:
Build My App: stage: build tags: - node before_script: - yarn config set cache-folder .yarn - yarn install script: - npm run build
Notice the sample action name “Build My App” can contain spaces, and is the action executed for the stage “build” defined in the list of stages above it.
In this example, the yarn utility is being used instead of npm. And it’s creating a cache folder that will contain all the yarn configurations that will not be recreated when each project runs (each time it’s pushed).
These and addtional steps are defined in the sample at
as described in blog https://hackernoon.com/configuring-gitlab-ci-yml-150a98e9765d
Another example is https://linuxhint.com/gitlab_runner_gitlab_ci/ which provides code.
Deploying to AWS?
See instructions from @autonix at https://stackoverflow.com/a/38672045/1057052
Grab the shared-Runner token on the admin/runners page
sudo gitlab-runner register
Running in system-mode. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
Supply the URL, as there is no “press return for default”:
Please enter the gitlab-ci token for this runner:
Enter a description for the Runner, you can change this later in GitLab’s UI:
Please enter the gitlab-ci description for this runner
Enter the tags associated with the Runner, you can change this later in GitLab’s UI:
Please enter the gitlab-ci tags for this runner (comma separated):
Enter the Runner executor:
Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell:
If you chose Docker as your executor, you’ll be asked for the default image to be used for projects that do not define one in .gitlab-ci.yml:
Please enter the Docker image (eg. ruby:2.1):
Be sure Docker.app is installed on your mac
To hold passwords, secret keys, define variables which the runner applies to environments.
?? They can be protected by only exposing them to protected branches or tags.
To force a specific branch or tag to get rebuilt with an API call, define triggers
to impersonate their associated user including their access to projects and their project permissions.
Docker Container Registry
Installing the Docker image enables auto-scaling of additional instances to automatically meet demand. See https://docs.gitlab.com/runner/executors/docker_machine.html
Login to a dedicated machine where the container registry proxy will be running Make sure that Docker Engine is installed on that machine
Optionally create a new container registry:
docker run -d -p 6000:5000 \ -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \ --restart always \ --name registry registry:2
Configure Runner to “Run untagged jobs option”.
Click Save changes for the changes to take effect
Set the maximum job timeout
Verify all registered Runners:
GitLab looks for a .gitlab-ci.yml file to specify what jobs do. Its format is described at:
A tutorial is at:
We don’t use a shared Runner because then you have to enable each project explicitly for the Runner to run its jobs.
To trigger your build script, the default is to use shell.
Your Runner should show in the Manager Runners page on the GitLab CI Admin Dashboard (topbar > Admin > Runners). Depending on your Runner type, it may be green, blue, or red.
git submodule update –init ls -la
As of Omnibus GitLab 9.0, Prometheus and its related exporters are enabled by default, to enable easy and in depth monitoring of GitLab. Approximately 200MB of memory will be consumed by these processes, with default settings.
If you would like to disable Prometheus and it’s exporters or read more information about it, check the Prometheus documentation. GitLab Ru
When a developer finishes a set of changes, they can open a merge request, which is equivalent to a “pull request” in GitHub. This is so other contributors to the repo can make comments.
A pipeline is a collection of jobs split in different stages. Every commit pushed to GitLab generates a pipeline attached to that commit. If multiple commits are pushed together the pipeline will be created for the last one only.
Based on https://gitlab.com/help/api/runners.md which at time of this writing is v4 with endpoint “https://gitlab.example.com/api/v4/runners”
|Register a new Runner||POST /runners|
|Verify authentication for a registered Runner||POST /runners/verify|
|List all runners||GET /runners/all|
|List owned runners||GET /runners|
|Get runner's details||GET /runners/:id|
|Update runner's details||PUT /runners/:id|
|Remove a runner||DELETE /runners/:id|
|List runner's jobs||GET /runners/:id/jobs|
|List project's runners||GET /projects/:id/runners|
|Enable a runner in project||POST /projects/:id/runners|
|Disable a runner from project||DELETE /projects/:id/runners/:runner_id|
|Delete a registered Runner||DELETE /runners||token|
This was created based partly on documentation at:
Udemy video: GitLab/GitLab CI Course For Developers and DevOps 7/2017 [2 hrs] by Peng Xiao
More on DevSecOps
This is one of a series on DevSecOps:
- ci-cd (Continuous Integration and Continuous Delivery)
- User Stories for DevOps
- Git and GitHub vs File Archival
- Git Commands and Statuses
- Git Commit, Tag, Push
- Git Utilities
- Data Security GitHub
- GitHub API
- Choices for DevOps Technologies
- Pulumi Infrastructure as Code (IaC)
- Java DevOps Workflow
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- AWS server deployment options
- Cloud services comparisons (across vendors)
- Cloud regions (across vendors)
- Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
- Azure Certifications
- Azure Cloud Powershell
- Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
- Azure Networking
- Azure Storage
- Azure Compute
- Digital Ocean
- Packer automation to build Vagrant images
- Terraform multi-cloud provisioning automation
Hashicorp Vault and Consul to generate and hold secrets
- Powershell Ecosystem
- Powershell on MacOS
- Jenkins Server Setup
- Jenkins Plug-ins
- Jenkins Freestyle jobs
- Docker (Glossary, Ecosystem, Certification)
- Make Makefile for Docker
- Docker Setup and run Bash shell script
- Bash coding
- Docker Setup
- Dockerize apps
- API Management Microsoft
- Scenarios for load
- Chaos Engineering