Run pipelines from within GitHub, for free (instead of Jenkins, CircleCI, etc.)
Overview
- Baseline Production example
- From-scratch Tutorials
- Actions in Jobs triggering Workflows
- Run locally
- Baseline Production example
- Strategy Matrix of variations
- Community
- Sample NPM workflow
- Sample repo for GitHub’s Tutorial
- Create Badge
- AWS in GitHub Actions
- Video classes
- YouTube videos
- Blogs
- Documentation
- More
This article describes a production-worthy baseline professional developers and DevSecOps platform engineers can collaborate on refining over time.
https://github.com/bomonike/gha-baseline
At the bottom of this article is my list of video classes, YouTube videos, blogs, and vendor documentation about learning this topic from scratch.
So this aims to be hands-on and deep, yet succinct.
Here we start with our Baseline code.
NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.
Baseline Production example
- Create a new Git repo (with a README.md).
From-scratch Tutorials
This section summarizes their content.
GitHub added an “Actions” tab to repos (in 2019) to perform Continuous Integration (CI) like Jenkins.
GitHub Actions enables software development teams to configure Infrastructure as Code (IaC) for Continuous Integration for NodeJs and a wide range of programming languages.
When developers can merge and deploy code many times in a single day, they can achieve Agile DevSecOps.
Actions in Jobs triggering Workflows
The “Actions” tab within a repository display Workflows stored within the repo’s .github folder. Notice the leading dot to specify a hidden folder.
Click image to pop-up full-size display.
Within the .github folder is a workflows folder whichcontain declarative yml files. Each “workflow” is a separate yaml file, each an automated process that contain one or more logically related jobs.
Each jobs contains one or more steps – tasks executed through a GitHub Actions YAML config file, such as building source code, run tests, or deploy the code that has been built to some remote server.
Build and run tests jobs can be in the same workflow, with the deployment job into a different workflow.
PROTIP: Within a Workflow file named (for example) “build_and_test.yml”, specify a corresponding name such as:
name: Build and Test
A runner is the remote computer that GitHub Actions uses to execute the jobs. Runners can be local, in AWS. Runners are specified by runs-on: lines such as:
runs-on: ubuntu-latest
In addition to Ubuntu, GitHub provides Microsoft Windows, and macOS runners.
A job is trigged for execution by a GitHub Action when some event occurs. Jobs can be scheduled too. Events are specified by the on: section.
on: push: branches: [main] pull_request: branches: [main]
DEFINITION: Actions are individual steps within a job – commands that can be reused in your config file. You can write your custom actions or use existing ones.
Each step has a hyphen and name: and uses:. For an example running Python:
steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up Python Environment uses: actions/setup-python@v2 with: python-version: '3.x' - name: Install Dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run Tests run: | python manage.py test
The Scripts folder contain programmatic sh (Bash shell) files which carry out actions.
Run locally
You can run GitHub Actions locally on your laptop using github.com/nektos/act.
https://github.com/cplee/github-actions-demo
Baseline Production example
-
At github.com, navigate to the repo you want to add GitHub Actions:
-
Create new file .github/workflows folder path from the root of your repo.
This follows the same convention as .circleci.
Each workflow is defined by a yaml-formatted file.
-
Create a workflow yml file named main.yml
PROTIP: To start, rather than creating your own a yaml-formatted file to define each Workflow configuration. An example (using NodeJs) from https://github.com/cplee/github-actions-demo/blob/master/.github/workflows/main.yml
-
PROTIP: Create in your internet browser a bookmark so you can return to this quickly.
-
Edit the main.yml workflow file:
name: 'baseline-workflow' # **What it does**: Scan Terraform code. Save results on S3 buckets based on credentials from HashiCorp Vault. # **Why we have it**: So secrets are not static in GitHub Actions GUI, needing to be repeated in each Action. # **Who does it impact**: Docs content. on: [push] jobs: test-job: runs-on: ubuntu-latest steps: - name: 🚀 Conditions at start run: echo "Stats at start of job ..." - name: 🫶 Get code uses: actions/checkout@v2 - uses: actions/setup-node@v1 - run: 🎉 npm install - run: npm test - name: 🫶 Conditions at end run: echo "Stats at end of job ..." deploy: needs: test-job ...
</pre>
### Job name & environment
PROTIP: The name value should match the name of the yml file.
Encase the name value in single quotes if there is a space or other special character.
See https://docs.github.com/en/actions/using-jobs/using-environments-for-jobs
Notice indents are two spaces by default.
PROTIP: Add "-job" at the end of job names
### on: triggers
<tt>on: push</tt> defines one of the <a target="_blank" href="https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows">>events that trigger</a> a workflow to start:
* watch (repo starred)
* fork (repo forked)
* issues (opened or deleted)
* issue_commment
* create (branch or tag)
* pull_request (opened or closed)
* push (of a commit)
* workflow_dispatch
<br /><br />
### Runners Pricing
REMEMBER: Each job has its own runner (virtual machine isolated from other jobs)
<tt>runs-on:</tt> defines the <a target="_blank" href="https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners">runner</a> within a GitHub hosted environment. Instead of <tt>ubuntu-latest</tt> a version specification can be specified. Alternately, <a href="#MatrixVariations">several versions</a>.
<tt>with:</tt> configures the runner.
CAUTION: See <a target="_blank" href="https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions">cost implications</a> depending on the platform, number CPU cores, etc.
GiHub pre-installs Clang, Bash, Python, Node, etc. for use on each runner.
### Steps with emojis
PROTIP: Use emoji's to visually differentiate step names.
❤️ Initial greeting<br />
👀 Verify Terraform<br />
🫶 Goodbye<br />
* https://emojipedia.org/
### Sample code
https://github.com/SamGallagher95/best-terraform-cd-article/tree/main/terraform
<a name="Marketplace"></a>
### Actions Marketplace
CAUTION: GitHub currently does not dynamically scan 3rd-party actions for malicious activity.
Among 3rd-party Actions in GitHub's public Marketplace, <a target="_blank" href="https://github.com/marketplace?category=&query=sort%3Apopularity-desc&type=actions&verification=">sorted by number of stars</a>:
* https://github.com/marketplace/actions/super-linter (from GitHub)
* https://github.com/marketplace/actions/trufflehog-oss to scan for leaked secrets
* https://github.com/marketplace/actions/configure-aws-credentials-action-for-github-actions
* https://github.com/marketplace/actions/checkout to a specific version of your GitHub repo
</br /><br />
"Verified creator" only means that GitHub has been able to contact the creator.
<tt>needs: test</tt> enforces a dependency to finish successfully.
### Environment secrets
To create buckets in S3 or other AWS services, sepecify:
<pre>env:
GITHUB_TOKEN: $
AWS_ACCESS_KEY_ID: $
</pre>
- Save the changes with a comment. Click the green “Start commit”.
-
To view the status of workflows, press the Actions tab at the top menu.
PROTIP: To get to the top of the screen to see GitHub’s Tabs, on macOS, press command + up_arrow.
- Wait past “Queued” to click the run at the top of the list.
-
Click a job box with green check icon to see step info.
Set up job and Complete job (“Cleaning up orphan processes”) are added by GitHub.
PAT
-
In GitHub Settings > Developer Settings > Define a PAT (Personal Access Token) for expiration in 30 days.
PROTIP: For note, add a time stamp such as “expires 23-12-31”.
Select scopes repo and workflow
-
Click “Set up a workflow yourself” or select a template containing pre-populated yml files from various people.
PROTIP: You can create and share templates for use by others in your own organization. See https://help.github.com/en/actions/hosting-your-own-runners
-
PROTIP: Protect the master branch so it can’t be inadvertently deleted or broken.
-
PROTIP: Setup required reviews so that any pull requests are double checked by teammates.
Strategy Matrix of variations
The “ubuntu-vers” job in the code here run each possible combination of variables, one for each combination of the version and os.
jobs: ubuntu-vers: strategy: matrix: version: [10, 12, 14] os: [ubuntu-latest]
See https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs
Community
https://github.community/t5/GitHub-Actions/bd-p/actions
Sample NPM workflow
-
Let’s look at a yaml workflow file used by GitHub Actions.
A workflow is a unit of automation from start to finish, including the definition of what triggers the automation, what environment or other aspects should be taken account during the automation, and what should happen as a result of the trigger.
-
See https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions
on: scheduled actions
on:
specifications inside that file define a scheduled time when the workflow is triggered.Alternately, workflows can be triggered by events in or outside GitHub, such as a git push or a scheduled time.
The default trigger is to run on every push to every branch:
on: [push]
This example is triggered upon a push to either the master branch or a release branch:
on: push: branches: - master - release/*
In this example, the workflow is triggered to run the master branch anytime there’s a push or pull request.
on: push: branches: [ master ] pull_request: branches: [ master ]
PROTIP: To set a workflow (using crontab specifications) to run at 2:00 AM UTC every day, 1=Monday to 5=Friday:
on: schedule: - cron: "0 2 * * 1-5"
- Minute 0 to 59, or * (no specific value)
- Hour 0 to 23, or * for any value. All times UTC.
- Day of the month 1 to 31, or * (no specific value)
- Mont 1 to 12, or * (no specific value)
- Day of the wee 0 to 7 (0 and 7 both represent Sunday), or * (no specific value)
jobs: block
Workflows are made of jobs, and the template workflow defines a single job with the identifier build.
jobs: build: name: 'Build' runs-on: ubuntu-latest
Several
jobs:
blocks define different sections of a Workflow.runs-on: job host environment
Every job needs a specific host machine specified by the runs-on: field. This template workflow specifies using the latest version of Ubuntu, a Linux-based operating system.
- ubuntu-latest, ubuntu-18.04, or ubuntu-16.04
- windows-latest or windows-2019
- macos-latest or macos-10.15
The above specify the Runner within a GitHub hosted environment or a self-hosted environment.
Ubuntu contains Docker.
Alternately,
runs-on: $
refers to the “os” alternatives in the strategy section.job strategy: matrix
A Job Matrix is designed to build and test code with different environments and configurations.
strategy: matrix: node-version: [10.x, 12.x] os: [ubuntu-latest, windows-latest, macOS-latest]
PROTIP: The code above defines variable
$
which resolves to “10.x”, or “12.x” when referenced in the set of steps below, which are repeated automatically for each node-version specified.CAUTION: Reference the list of releases for the language you’re using, such as this one for NodeJs.
You can also vary the host operating system environment:
strategy: matrix: node-version: [10, 12, 14] os: [ubuntu-latest, windows-latest, macOS-latest]
The above would generate 3 x 3 = 9 job runs.
PROTIP: Different jobs in the matrix are run simultaneously.
Cost of GitHub Actions jobs
GitHub charges on a “pay as you go” basis two ways: by the minute used by each job and what operating system:
There are limits on the number of concurrent jobs: Enterprise licensees have a limit of 180 jobs, of which 50 are macOS jobs, but only 5 macOS jobs for others. Even free accounts get up to 20 concurrent jobs. 40 for those who pay $4 a month. Each team gets 60 jobs at a time.
PROTIP: A job matrix can generate a maximum of 256 jobs per workflow run. This limit also applies to self-hosted runners.
Steps
Each job is made up of one or more steps. In the sample template:
steps: - uses: actions/checkout@v2 - name: Use Node.js $ uses: actions/setup-node@v1 with: node-version: $ - run: npm ci - run: npm run build --if-present - run: npm test
-
(a dash) precedes each action.Echo
Issue a message by running an echo command:
jobs: build: steps: name: Run one-liner run: echo Hello, world!
Notice no quote characters.
- uses: step in Actions coding
- uses: actions/checkout@v2
actions
defines an action from GitHub’s public Marketplace of Actions.checkout@v2
retrieves the latest (such as v2.1.0) in https://github.com/actions/checkout/releases. The action’s home page is at https://github.com/marketplace/actions/checkoutPROTIP: Monitor when versions are updated. When an upgrade is available, search through GitHub repos to see which ones should be upgraded.
- name: step in Actions coding
Because the Node.js version needs to be specified several times:
- run: step in Actions coding
- run: npm ci - run: npm run build --if-present - run: npm test
npm ci
was introduced in NodeJs 5.1 (2018) in place of “npm install” (or yarn) for faster downloading and installation of package dependencies (based on specifications in the package.json file) into the node_modules folder.BTW the new GitHub Package Registry only supports npm as a client for JavaScript packages (at least for now).
npm run build
runs the build field defined in the scripts field within package.json.BTW npm build no longer exists as of 2019.
--if-present
is an optional flag to avoid exiting with a non-zero exit code when the script is undefined.npm test
executes all tests defined.PROTIP: Consider separate test jobs to separate build from test details.
build and publish
PROTIP: Include where you’re publishing if you’re publishing to the gpr (Google Package Registry) as well as NPM.
jobs: build: ... publish-npm: ... publish-gpr:
Slack notification
-
Post to a Slack channel when a new issue is added on GitHub:
name: Slack Issue on: issues: types: [opened] job: post_slack_message: runs-on: ubuntu-latest steps: - uses: rtCamp/action-slack-notify@2.0.0 - env: SLACK_WEBHOOK: $ SLACK_USERNAME: memyselfandi SLACK_CHANNEL: gh-issues
Clear-text of secrets are input in the Security tab.
env: ci: true
env: ci: true
Sample repo for GitHub’s Tutorial
A sample repo was provided in VIDEO: Continuous integration with GitHub Actions [1:55:24] at GitHub Satellite 2020 on 7 May 2020
- Create and use multiple, customized workflows
- Implement a unit testing framework using GitHub Actions
- Use multiple jobs in a workflow and pass artifacts between jobs
- Configure a repository to work in conjunction with GitHub Actions workflows and your team’s workflow.
curl https://api.github.com/octocat
-
Go to and fork
https://github.com/githubsatelliteworkshops/ci-with-actions
BLAH: The pdf in the link satellite-2020-workshops-ci-with-actions.pdf does not have links enabled.
- @pprmk, Sr. Implementation Engineer
- @dechyper, Solutions Architect
- @iamhughes, Sr. DevOps Engineer
-
Throughout the course, return to the list of course agenda at:
- Click “Start free course”. You may be asked to login GitHub.
- [20:01] Choose either “Public” or “Private”, then “Begin GitHub Actions: Continuous Integration”.
- [20:35] Wait for message “you can start your first step”. Scroll down to notice the other courses.
-
Among the 16 steps:
-
Use a templated workflow Create a pull request with a templated workflow
-
Run a templated workflow Wait for GitHub to run the templated workflow and report back the results
-
Add your first test Add your first test script for CI to pick up
-
Read an Actions log Tell the bot which test is failing so we can fix it
-
Fix the test Edit the file that’s causing the test to fail
-
Share the workflow with the team Merge the pull request containing your first workflow so the entire team can use it
-
Create a custom GitHub Actions workflow Edit the existing workflow with new build targets
-
Target a Windows environment Edit your workflow file to build for Windows environments
-
Use multiple jobs Edit your workflow file to separate build and test jobs
-
Run multiple jobs Wait for the result of multiple jobs in your workflow
-
Upload a job’s build artifacts Use the upload action in your workflow file to save a job’s build artifacts
-
Download a job’s build artifacts Use the download action in your workflow file to access a prior job’s build artifacts
-
Share the improved CI workflow with the team Merge the pull request with the improved workflow
-
Automate the review process Add a new workflow file to automate the team’s review process
-
Use an action to automate pull request reviews Use the community action in your new workflow
-
Create an approval job in your new workflow In your new workflow file, create a new job that’ll use the community action
-
Automate approvals Use the community action to automate part of the review approval process
-
Use branch protections Complete the automated review process by protecting the master branch
-
-
[29:34] Click “Start: Use a templated workflow” for the Issue#1 page on your own repo such as this (but with your name instead):
https://github.com/wilsonmar/github-actions-for-ci/issues/1
-
[30:58] Click Actions tab, click “Set up this workflow” or navigate within the repo’s .github/workflows folder to edit file nodejs.yml (the Actions file).
- Copy “Paste “CI for Node” into your invisible Clipboard.
- [31:29] Click “Start commit” to a new branch.
- [32:02] Commit new file.
- [32:05] Double-click to select all of the suggested name to Paste “CI for Node” insted. Click “Create pull request”.
- [32:22] “Review required” and “Merging is blocked” apprears until …
- [32:44] Click on “Details” or Actions tab to see jobs running. Click on a build.
-
[33:11] Click “Pull Requests” tab to return to “CI for Node”.
Vocabulary is defined by the bot.
- [34:08] Add your first test: Click “Pull requests” tab. Click “Add Jest tests”. Click “Merge pull request”.
-
[34:53] Click “Delete branch”.
Read Actions Log
- [35:06] Click on “next step” (created by the bot).
- [35:42] Navigate to the log output: Click on “Actions” tab.
- [36:22] Click the latest “CI for Node” run (at the top. Click a build. Identify a name of a failing test with red “x”. Expand it by clicking it.
- [36:48] Identify the name “Initialize with two players” and copy it.
-
[37:23] To “Pull requests”. In the Comment paste the name of the failing test. Click “Comment”.
Fix the test
- [37:50] Click “Commit suggestion” of “Update src/game.js”.
- [38:10] Click “Commit Changes”.
-
[38:47] Refresh screen until bot makes “Changes approved”.
Share
-
[39:34] Click “Merge pull request”. “Confirm Merge”. “Delete branch”.
-
[56:11] https://github.com/wilsonmar/github-actions-for-ci/issues/1
Step 7: (Work Session 2) Create a custom GitHub Actions workflow
- [1:06:43] Click “Resume”.
Create Badge
Within Actions tab:
AWS in GitHub Actions
First, preconfigure the IAM IdP in your AWS account (see Assuming a Role for details).
Configure your AWS credentials and region environment variables for use in GitHub Actions, add action https://github.com/aws-actions/configure-aws-credentials
- name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v2 with: role-to-assume: arn:aws:iam::123456789100:role/my-github-actions-role aws-region: us-east-2
the action implements the AWS SDK credential resolution chain and exports environment variables for other Actions to use.
v2 of the action uses the Node 16 runtime by default.
This causes the action to perform an AssumeRoleWithWebIdentity call and return temporary security credentials for use by other actions.
https://www.freecodecamp.org/news/how-to-setup-a-ci-cd-pipeline-with-github-actions-and-aws/
Environment variable exports are detected by both the AWS SDKs and the AWS CLI for AWS API calls.
https://github.com/aws-actions
Alternately, https://www.freecodecamp.org/news/how-to-setup-a-ci-cd-pipeline-with-github-actions-and-aws/ Use the AWS Elastic Beanstalk compute service pulled from AWS S3 buckets uploaded from GitHub.
- Setup an AWS Account
- Get into Elastic Beanstalk environment https://us-west-2.console.aws.amazon.com/elasticbeanstalk/home?region=us-west-2#/welcome
- “Create Application” (formerly “Create a New Environment”).
- Application name: PROTIP: Type your name so it’s unique.
- Application tags
- Platform: Choose Python if you like.
- Platform branch
- Platform version
- Application code: select “Sample application” or “Upload your code”.
- Click “Create application”.
- Grab the application name and the environment name at the upper-left:
Wilson230321-env
http://wilson230321-env.eba-iqusqyih.us-west-2.elasticbeanstalk.com/
What’s Next?
- AWS Elastic Beanstalk overview - What is AWS Elastic Beanstalk?
- AWS Elastic Beanstalk concepts
- Deploying a Django Application to AWS Elastic Beanstalk
- Deploy a Flask Application to AWS Elastic Beanstalk
- Using the Elastic Beanstalk Python platform (Customizing and Configuring a Python Container)
- Working with Logs
The new Elastic Beanstalk environment management console described at:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-console.html
such as:
https://us-west-2.console.aws.amazon.com/elasticbeanstalk/home?region=us-west-2#/environments
Video classes
https://www.udemy.com/course/github-actions-the-complete-guide/ 10.5 hour $16.99 GitHub Actions - The Complete Guide Nov 2022 referencing https://github.com/academind/github-actions-course-resources by Maximilian Schwarzmuller
YouTube videos
https://www.youtube.com/watch?v=T6sW1Dk9B4E What every GitHub user should know about VS Code - GitHub Satellite 2020 24:08
Continuous integration with GitHub Actions presented at GitHub Satellite 2020 7 May, 2020
[2] Advanced GitHub Actions: workflows for production grade CI/CD - GitHub Universe 2019
GitHub Actions Now Supports CI/CD | Getting Started Hong Ly
GitHub Actions (CI/CD Flow) Coding Tech
5 Ways to DevOps-ify your App - Github Actions Tutorial Fireship
Introduction to GitHub Actions [38:46] by BlackMarbleLtd CEO @RichardFennell references https://github.com/rfennell/ActionPlayground/blob/master/src/helloworld.ts (Typescript)
Introducing GitHub Package Registry GitHub
GitHub Actions: Open Source Workflow Automation by Bas Peters DATA MINER
Github Actions | Open CICD Platform by Github by Tech Primers
https://www.youtube.com/watch?v=F3wZTDmHCFA GitHub Actions: How to Set Up a Simple Workflow CodingWithChandler
https://www.youtube.com/watch?v=2Ym94MfScZ4 GitHub Actions CI/CD Workflow for a Laravel Application - Part 1: Introduction Oh See Media
VIDEO: Unlocking the Cloud Operating Model with GitHub Actions by Steve Winton, Senior Partner Engineer, GitHub
https://github.com/actionsdesk
https://dev.to/github/export-github-issues-commit-history-and-more-github-artifact-exporter-2ok6 Export GitHub Issues, Commit History and More | GitHub Artifact Exporter by Davide ‘CoderDave’ Benvegnù
-
VIDEO “GitHub Actions Tutorial - Basic Concepts and CI/CD Pipeline with Docker” by TechWorld with Nana
-
VIDEO: Visual Studio Toolbox at Microsoft:
-
VIDEO: Automatic Deployment With Github Actions Traversy Media
-
https://sanderknape.com/2021/01/go-crazy-github-actions/
-
Github Actions and GitOps in One Hour Video Course by Alfredo Deza and Noah Gift
-
Sample app: https://github.com/bsommardahl/anyhasher
Blogs
https://coletiv.com/blog/how-to-setup-continuous-integration-and-deployment-workflows-for-reactjs-using-github-actions/
Documentation
GitHub Actions Documentation is at https://help.github.com/en/actions
https://help.github.com/en/categories/automating-your-workflow-with-github-actions
https://help.github.com/en/actions/configuring-and-managing-workflows
https://help.github.com/en/actions/language-and-framework-guides
https://help.github.com/en/actions/migrating-to-github-actions
https://www.youtube.com/watch?v=qy_HaIaNbkE Automate your CI/CD workflows with GitHub Actions https://resources.github.com/devops/ci-cd-with-github-actions/ Ray Ploski, Field CTO, HashiCorp Peter McCarron, Sr. Technical Marketing Engineer, LaunchDarkly Kassen Qian, Product Manager, Datadog Vanessa Yan, Staff Product Manager, OctoML
https://www.youtube.com/watch?v=TLB5MY9BBa4&pp=ygUJQ29kZXJEYXZl GitHub Actions Tutorial | From Zero to Hero in 90 minutes (Environments, Secrets, Runners, etc) by CoderDave (David Benvegnu)
More
This is one of a series on Git and GitHub:
- Why Git? (file-based backups vs Git clone)
- Git basics (script)
- Git whoops (correct mistakes)
- Git command shortcuts
- Git interactive merge (imerge)
- Git patch
- Git utilities
- Git hooks
- GitHub data security
- GitHub actions for automation JavaScript
- GitHub REST API
- GitHub GraphQL API
- GitHub PowerShell API Programming
- GitHub GraphQL PowerShell Module