Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Integrated DevSecOps from Thoughtworks

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This article are my notes about GoCD, described at website gocd.org (previously at https://go.cd).

My contribution here is logical sequencing for a deep yet concise presentation.

With “Go” in the name one would think that it’s written in the Go language. But GoCD is actually built using Java and JRuby on Rails. It was re-branded in 2010 before Google released the Go language.

GoCD’s was as open-source in 2014 at https://github.com/gocd/gocd. GoCD’s releases are sematically tagged beginning with a year (19 for 2019).

GoCD is from ThoughtWorks, which makes money by providing commercial (paid) plugins for GoCD, but primarily from consulting services. The company has a 90% employee referral rating on Glassdoor.com. It publishes future-thinking Tech Radar each year and hosts XConf in various cities.

Martin Fowler, Thoughtworks Chief Scientist since 2000 [4] (along with Jim Highsmith and other software visionaries) authored the Agile Manifesto in 2001.

Thoughtworkers Jez Humble, Chris Read, and Dan North presented their ground-breaking “Deployment Production Line” at the Agile conference in 2006. While working on the oft-quoted 2010 book Continuous Delivery with and David Farley, Jez Humble began working alongside a team in Beijing to create the product which later became GoCD. Jez Humble also co-wrote with Gene Kim the best-seller DevOps Handbook.

Why?

GoCD was implemented as a tool to enable teams to achieve high performance such as these benchmark results: gocd-benchmarks-637x316
From BOOK: Accelerate quoted by [3]

GoCD also has a “Failure Rate” metric of the percent of jobs that make it all the way through successful Deployment into production.[6]

GoCD’s (paid) Enterprise plugin collects and displays granular analytics visualizations about build time history across jobs:

gocd-build-metrics-1078x668 https://www.gocd.org/analytics.html

Red dots highlight each point of failure.

To drill-down into the gray area representing waiting time for each stage, look at each job’s Workflow Time Distribution:

gocd-workflow-dist-1113x488 [3]

All this is so when things go wrong, it’s easy to identify both the upstream cause and the downstream effects.

CI/CD Delivery to Production, not just Deploy to Test

The “CD” in GoCD is for “Continuous Delivery” (not just Deployment). As Humble states in his website ContinuousDelivery.com:

“Continuous Delivery is the ability to get changes of all types – including new features, configuration changes, bug fixes and experiments – into production, or into the hands of users, safely and quickly in a sustainable way.” [3]

GoCD achieves CD with an automated pipeline of continuous testing and acceptance of small increments of changes to code always in a deployable state.

It’s competitors include CodeFresh, Jenkins X.

Advantages & Value Proposition

GoCD provides visibility (traceability) over the end-to-end workflow from version control to production, even by teams of thousands of developers making changes through a complex pipelines on a daily basis.

gocd-fan-1233x529[1] https://docs.gocd.org/current

GoCD works with both source programming code and infrastructure as code (configurations leveraging Terraform, Docker, Ansible, etc.)

GoCD manages the metadata rather than jobs needing to pass metadata between jobs, as in Jenkins.

VIDEO: Pipelines as Code with GoCD by Tyler Moody of Sonic at DevOpsOKC Jan 25, 2019 [7:50] Gomatic from Thoughtworks is a Python library to reverse engineer pre-existing pipelines by translating Python code into XML and posts to GoCD.

https://github.com/hadolint/hadolint

https://github.com/tomzo/gocd-yaml-config-plugin

Automate out manual approvals

GoCD implements the vision of a pipeline to production as low-risk, predictable, routine, performed on demand. This is achieved by eliminating delays during “code freezes” for integration, testing, and hardening phases that traditionally followed “dev complete”.

gocd-pipeline-diagram-672x431

[1]

PROTIP: Wisdom from Thoughtworkers is that feature and long-lived branches are an anti-patterns. Commit into master. They recommend that every commit should trigger a build. And broken builds should be fixed immediately. That is also the recommendation from Security experts, who say always “be ready for another Heartbleed vulnerability” which required rebuild of the core operating system. Technical debt block the quick action needed to remediate before hackers take advantage of vulnerabilities.

Internal architecture

Unlike Jenkins, which has pipelines as tasks within each job, GoCD’s primary organization are pipelines.

gocd-main-menu-607x67

  • Pipelines (can be grouped and given role-based permissions)
    • Sequential Stages running several Jobs (in parallel). If a particular stage fails, the whole pipeline job fails.

GoCD’s Value Stream Maps track changes from commit to deploy and delivery:

gocd-map-2014-1920x1080.jpg[1] https://docs.gocd.org/current/navigation/value_stream_map.html

PROTIP: You’ll need to ask your Network Admin. to open up default port 8153 to serve UI/API and 8154 to control GoCD Agents.

Tasks (commands invoking shell scripts working on target servers)

  • Environments (Build, QA, Staging, Prod, etc.)
    • Agents contact (elastic) servers where work is done
      • Resources (tags about capabilities of each agent)

BTW: A ThoughtWorker who created Selenium and ThoughtWorks also created Guage.org for functional acceptance testing.


Install

See https://www.go.cd/getting-started/part-1

https://hub.kubeapps.com/charts/stable/gocd (managed by Bitnami) provides a Helm Chart to make it easy to install and operate GoCD in its entirety on a Kubernetes cluster (like brew on macOS). See “Getting Started with GoCD on Kubernetes”. (a running instance of a chart with a specific config is called a release)

gocd-helm-648x257 [5]

A. Install GoCD as a Kubernetes native application with an officially supported helm chart B. Scale GoCD agents seamlessly with the new ElasticAgent plugin that spins up agents on the fly in response to build workload C. Design Docker-based build workflows as Docker in Docker

  1. install Helm

    brew install kubernetes-helm
    helm version
    # Client: &version.Version{SemVer:"v2.14.3", 
    # GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
    
  2. Navigate to or create a folder to create a repo.
  3. Remove the previous folder “gocd” to begin new.
  4. Create folder “gocd”.
kubectl config current-context
   # "minikube" on macOS or "my-cluster"
helm init
   # $HELM_HOME has been configured at /Users/$($username)/.helm.
kubectl get pods --namespace kube-system --selector=app=helm
   # Error: error installing: Post https://192.168.99.100:8443/apis/extensions/v1beta1/namespaces/kube-system/deployments: 
   # dial tcp 192.168.99.100:8443: i/o timeout

   # tiller-deploy STATUS Running
helm search gocd  # in hub.helm
   # stable/gocd
helm install stable/gocd --name gocd-live-demo --namespace gocd-live-demo
helm list
helm status gocd  # pre-baked
   # From post-install instructions:
echo "GoCD server public IP: http://$(kubectl get ingress gocd-server --namespace=gocd -0 jasonpath='{.status.loadBalancer ingress [0].ip}')"
 
   

Docker images

Docker images for GoCD are at https://hub.docker.com/r/gocd/gocd-server. See https://www.gocd.org/2019/06/25/GoCD-non-root-containers

For your custom app, custom Docker images are created when a build pass tests so that Kubernetes makes use of images to load Staging and Production:

gocd-image-push-839x295 [8]

MacOS

On MacOS, install a Server and Agent components:

NOTE: brew search gocd did not return any hits.

  1. https://www.gocd.org/download/#osx provides buttons to download the server and agent components.

  2. Right-click on “Download Server” for the URL to the latest version, such as:

    https://download.gocd.org/binaries/20.1.0-11114/osx/go-server-20.1.0-11114-osx.zip

    Highlight the version, copy to your Clipboard, and paste it below to download in a shcell script is: *

    GOCD_VERSION="20.1.0-11114"
    wget "https://download.gocd.io/binaries/${GOCD_VERSION}/osx/go-server-${GOCD_VERSION}-osx.zip"
    unzip -a "go-server-${GOCD_VERSION}-osx.zip"
    

    Alternately, to manually install GoCD server on Mac:

  3. Drag the GoCD server application to the Applications folder.
  4. Double-click on the Go Server.app icon to open the launcher.
  5. While the GoCD server is starting up, you’ll see a progress bar in the top left of your screen.
  6. Server will start up.
  7. Once the GoCD server has started, it will open your default browser to the GoCD dashboard page defaults to:

    http://localhost:8153/go

  8. To get back to the GoCD dashboard page when the server is running, click on the link in the About box of the GoCD server.

    To install the GoCD Agent on Mac:

    wget "https://download.gocd.io/binaries/${GOCD_VERSION}/osx/go-agent-${GOCD_VERSION}-osx.zip"
    unzip -a "go-agent-${GOCD_VERSION}-osx.zip"
    

    Alternately:

  9. Double-click the file downloaded from the downloads page to unzip the contents.
  10. Drag the Go Agent.app icon to the Applications folder.
  11. Double-click on the Go Agent.app icon to open the launcher.
  12. The very first time you run the GoCD agent on your machine you will be prompted for the hostname or IP address of your GoCD server. By default it will try connecting to the local machine. Click the OK button to continue.
  13. Once agent is started it should be listed http://localhost:8153/go/agents you have to enable the agent.
  14. If want to change the server url the agent is pointing to, goto ~/Library/Preferences/com.thoughtworks.studios.cruise.agent.properties

    ./run-gocd

Create New Pipeline

  1. Click the “+ New Pipeline” button in the upper-right corner of the screen to create and run your first pipeline in GoCD.

  2. Once you’ve completed your first pipeline, look under the hood at your build.

Push changes

The demo[5] uses a shell script to build the sample Kubernetes Guest Book app:

./push_change.sh

Plugins

GoCD’s proposition is (similar to GitLab) – providing the most common CD scenarios out of the box - while Jenkins aims for general-purpose automation by extensibility with 3rd party plugins. That’s how GoCD avoids the issue of plugins that can be missing features, obsolete, or does not work well with others.

GoCD has a handful of extension points that are interoperable with each other:

  • SCM (Source Control Management) = Material Types (Git, Subversion, Mercurial, Perforce, TFS, Pipeline, Package) [32:42]
  • tasks (Ant, NAnt, Rake)
  • notifications,
  • authentication and authorization,
  • configuration,
  • elastic agents (for parallel execution at scale)

See https://www.gocd.org/plugins/#secrets

See https://www.gocd.org/pipelines-as-code.html

https://extensions-docs.gocd.org/

Social Media

References and Learning Resources

[1] VIDEO: Introductory webinar 7 August, 2014 by Ken Mugrage

[2] Martin Fowler – Continuous Delivery Jan 31, 2015 [17:07]

[3] VIDEO: Actionable Continuous Delivery Metrics Nov 27, 2018 by Suzie Prince

[4] ThoughtWorks on Wikipedia

[5] Webinar: Continuous Delivery with Docker, Kubernetes, and GoCD May 7, 2018 by Sheroy Marker (@sheroymarker) and Ken Mugrage

[6] Continuous delivery workflows on modern infrastructure - Run GoCD on Kubernetes Sheroy Marker

[7] Remediation Strategy for Continuous Delivery of Microservices by Sheroy Marker (@sheroymarker)

[8] Continuous Delivery with Docker and Kubernetes Aug 20, 2018 [10:36] by Ken Mugrage

Kief Morris, Principal Cloud Technologist

Setup pipeline for CICD Simple Workflow

VIDEO: CI-CD Using Git-Tags Apr 21, 2019


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering