Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Integrated DevSecOps from Thoughtworks

Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This article are my notes about GoCD, described at website gocd.org (previously at https://go.cd).

My contribution is a logical sequence of content for a deep yet concise presentation.

With “Go” in the name one would think that it’s written in the Go language. But GoCD is actually built using Java and JRuby on Rails. It was re-branded in 2010 before Google released the Go language.

GoCD’s was as open-source in 2014 at https://github.com/gocd/gocd. GoCD’s releases are sematically tagged beginning with a year (19 for 2019).

GoCD is from ThoughtWorks, which makes money by providing commercial (paid) plugins for GoCD, but primarily from consulting services. The company has a 90% employee referral rating on Glassdoor.com. It publishes future-thinking Tech Radar each year and hosts XConf in various cities.

Martin Fowler, Thoughtworks Chief Scientist since 2000 [4] (along with Jim Highsmith and other software visionaries) authored the Agile Manifesto in 2001.

Thoughtworkers Jez Humble, Chris Read, and Dan North presented their ground-breaking “Deployment Production Line” at the Agile conference in 2006. While working on the oft-quoted 2010 book Continuous Delivery with and David Farley, Jez Humble began working alongside a team in Beijing to create the product which later became GoCD. Jez Humble also co-wrote with Gene Kim the best-seller DevOps Handbook.

Why?

GoCD was implemented as a tool to enable teams to achieve high performance such as these benchmark results:

gocd-benchmarks-637x316 From BOOK: Accelerate quoted by [3]

GoCD also has a “Failure Rate” metric of the percent of jobs that make it all the way through successful Deployment into production.[6]

GoCD’s (paid) Enterprise plugin collects and displays granular analytics visualizations about build time history across jobs:

gocd-build-metrics-1078x668 https://www.gocd.org/analytics.html

Red dots highlight each point of failure.

To drill-down into the gray area representing waiting time for each stage, look at each job’s Workflow Time Distribution:

gocd-workflow-dist-1113x488 [3]

All this is so when things go wrong, it’s easy to identify both the upstream cause and the downstream effects.

CI/CD Delivery to Production, not just Deploy to Test

The “CD” in GoCD is for “Continuous Delivery” (not just Deployment). As Humble states in his website ContinuousDelivery.com:

“Continuous Delivery is the ability to get changes of all types – including new features, configuration changes, bug fixes and experiments – into production, or into the hands of users, safely and quickly in a sustainable way.” [3]

GoCD achieves CD with an automated pipeline of continuous testing and acceptance of small increments of changes to code always in a deployable state.

It’s competitors include CodeFresh, Jenkins X.

Advantages & Value Proposition

GoCD provides visibility (traceability) over the end-to-end workflow from version control to production, even by teams of thousands of developers making changes through a complex pipelines on a daily basis.

gocd-fan-1233x529[1] https://docs.gocd.org/current

GoCD works with both source programming code and infrastructure as code (configurations leveraging Terraform, Docker, Ansible, etc.)

GoCD manages the metadata rather than jobs needing to pass metadata between jobs, as in Jenkins.

Automate out manual approvals

GoCD implements the vision of a pipeline to production as low-risk, predictable, routine, performed on demand. This is achieved by eliminating delays during “code freezes” for integration, testing, and hardening phases that traditionally followed “dev complete”.

gocd-pipeline-diagram-672x431

[1]

Wisdom from Thoughtworkers is that feature and long-lived branches are an anti-patterns. Commit into master. They recommend that every commit should trigger a build. And broken builds should be fixed immediately.

Internal architecture

Unlike Jenkins, which has pipelines as tasks within each job, GoCD’s primary organization are pipelines.

gocd-main-menu-607x67

  • Pipelines (can be grouped and given role-based permissions)
    • Sequential Stages running several Jobs (in parallel). If a particular stage fails, the whole pipeline job fails.

GoCD’s Value Stream Maps track changes from commit to deploy and delivery:

gocd-map-2014-1920x1080.jpg[1] https://docs.gocd.org/current/navigation/value_stream_map.html

(Port 8153 is the default.)

Tasks (commands invoking shell scripts working on target servers)

  • Environments (Build, QA, Staging, Prod, etc.)
    • Agents contact (elastic) servers where work is done
      • Resources (tags about capabilities of each agent)

NOTE: A ThoughtWorker create Selenium and ThoughtWorks also created Guage for functional testing.

Install

See https://www.go.cd/getting-started/part-1

https://hub.kubeapps.com/charts/stable/gocd (managed by Bitnami) provides a Helm Chart to make it easy to install and operate GoCD in its entirety on a Kubernetes cluster (like brew on macOS). See “Getting Started with GoCD on Kubernetes”. (a running instance of a chart with a specific config is called a release)

gocd-helm-648x257 [5]

A. Install GoCD as a Kubernetes native application with an officially supported helm chart B. Scale GoCD agents seamlessly with the new ElasticAgent plugin that spins up agents on the fly in response to build workload C. Design Docker-based build workflows as Docker in Docker

  1. install Helm

    brew install kubernetes-helm
    helm version
    # Client: &version.Version{SemVer:"v2.14.3", 
    # GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
    
  2. Navigate to or create a folder to create a repo.
  3. Remove the previous folder “gocd” to begin new.
  4. Create folder “gocd”.
kubectl config current-context
   # "minikube" on macOS or "my-cluster"
helm init
   # $HELM_HOME has been configured at /Users/$($username)/.helm.
kubectl get pods --namespace kube-system --selector=app=helm
   # Error: error installing: Post https://192.168.99.100:8443/apis/extensions/v1beta1/namespaces/kube-system/deployments: 
   # dial tcp 192.168.99.100:8443: i/o timeout

   # tiller-deploy STATUS Running
helm search gocd  # in hub.helm
   # stable/gocd
helm install stable/gocd --name gocd-live-demo --namespace gocd-live-demo
helm list
helm status gocd  # pre-baked
   # From post-install instructions:
echo "GoCD server public IP: http://$(kubectl get ingress gocd-server --namespace=gocd -0 jasonpath='{.status.loadBalancer ingress [0].ip}')"
 
   

Docker images

Docker images for GoCD are at https://hub.docker.com/r/gocd/gocd-server. See https://www.gocd.org/2019/06/25/GoCD-non-root-containers

For your custom app, custom Docker images are created when a build pass tests so that Kubernetes makes use of images to load Staging and Production:

gocd-image-push-839x295 [8]

MacOS

On MacOS, install a Server and Agent components:

To install GoCD server on Mac:

  1. Double-click the file downloaded from https://www.go.cd/download/ and unzip the contents.
  2. Drag the GoCD server application to the Applications folder.
  3. Double-click on the Go Server.app icon to open the launcher.
  4. While the GoCD server is starting up, you’ll see a progress bar in the top left of your screen.
  5. Server will start up.
  6. Once the GoCD server has started, it will open your default browser to the GoCD dashboard page (defaults to: http://localhost:8153/go).
  7. To get back to the GoCD dashboard page when the server is running, click on the link in the About box of the GoCD server.

To install GoCD Agent on Mac:

  1. Double-click the file downloaded from the downloads page to unzip the contents.
  2. Drag the Go Agent.app icon to the Applications folder.
  3. Double-click on the Go Agent.app icon to open the launcher.
  4. The very first time you run the GoCD agent on your machine you will be prompted for the hostname or IP address of your GoCD server. By default it will try connecting to the local machine. Click the OK button to continue.
  5. Once agent is started it should be listed http://localhost:8153/go/agents you have to enable the agent.
  6. If want to change the server url the agent is pointing to, goto ~/Library/Preferences/com.thoughtworks.studios.cruise.agent.properties
./run-gocd

Create New Pipeline:

  1. Click the “+ New Pipeline” button in the upper-right corner of the screen to create and run your first pipeline in GoCD.

  2. Once you’ve completed your first pipeline, look under the hood at your build.

Push changes

The demo[5] uses a shell script to build the sample Kubernetes Guest Book app:

./push_change.sh

Plugins

GoCD’s proposition is (similar to GitLab) – providing the most common CD scenarios out of the box - while Jenkins aims for general-purpose automation by extensibility with 3rd party plugins. That’s how GoCD avoids the issue of plugins that can be missing features, obsolete, or does not work well with others.

GoCD has a handful of extension points that are interoperable with each other:

  • SCM (Source Control Management) = Material Types (Git, Subversion, Mercurial, Perforce, TFS, Pipeline, Package) [32:42]
  • tasks (Ant, NAnt, Rake)
  • notifications,
  • authentication and authorization,
  • configuration,
  • elastic agents (for parallel execution at scale)

See https://www.gocd.org/plugins/#secrets

See https://www.gocd.org/pipelines-as-code.html

https://extensions-docs.gocd.org/

Social Media

References and Learning Resources

[1] VIDEO: Introductory webinar 7 August, 2014 by Ken Mugrage

[2] Martin Fowler – Continuous Delivery Jan 31, 2015 [17:07]

[3] VIDEO: Actionable Continuous Delivery Metrics Nov 27, 2018 by Suzie Prince

[4] ThoughtWorks on Wikipedia

[5] Webinar: Continuous Delivery with Docker, Kubernetes, and GoCD May 7, 2018 by Sheroy Marker (@sheroymarker) and Ken Mugrage

[6] Continuous delivery workflows on modern infrastructure - Run GoCD on Kubernetes Sheroy Marker

[7] Remediation Strategy for Continuous Delivery of Microservices by Sheroy Marker (@sheroymarker)

[8] Continuous Delivery with Docker and Kubernetes Aug 20, 2018 [10:36] by Ken Mugrage

Kief Morris, Principal Cloud Technologist

Setup pipeline for CICD Simple Workflow


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud services comparisons (across vendors)
  16. Cloud regions (across vendors)
  17. AWS Virtual Private Cloud

  18. Azure Cloud Onramp
  19. Azure Cloud
  20. Azure Cloud Powershell
  21. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  22. Digital Ocean
  23. Cloud Foundry

  24. Packer automation to build Vagrant images
  25. Terraform multi-cloud provisioning automation
  26. Hashicorp Vault and Consul to generate and hold secrets

  27. Powershell Ecosystem
  28. Powershell on MacOS
  29. Powershell Desired System Configuration

  30. Jenkins Server Setup
  31. Jenkins Plug-ins
  32. Jenkins Freestyle jobs
  33. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  34. Docker (Glossary, Ecosystem, Certification)
  35. Docker Setup
  36. Dockerize apps

  37. Maven on MacOSX

  38. Ansible

  39. MySQL Setup

  40. SonarQube static code scan

  41. API Management Microsoft
  42. API Management Amazon

  43. Scenarios for load