Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

All the stats that fits on a dashboard

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

There are several approaches to providing visibility of workflow to DevSecOps, to display dashboard the various statistics of a software delivery pipeline.

Organizations can make use of general-purpose visualization tools

There are also purpose-built dashboard software. Hygieia (pronouced hi-GEE-ya), open-sourced by Capital One (the credit card company) is named after the daughter of the Greek god of medicine and personification of hygiene and prevention of illness.

Team Dashboard Widget View

Each team has its own “tactical” dashboard to present detailed information real-time:

hygiea-screenshot-2848x1666
(Click for pop-up full image)

Hygeia provides visual and quick access to detailed process data collected from several sources, so team members don’t have to keep several tabs open:

  • “Build” from individual projects in Jenkins, Bamboo, Hudson, TeamCity, udeploy, xldeploy, Cucumber CI tools

  • “Feature” shows features and items in projects within JIRA, versionone.

  • “Code repo” (commits per day) of a specific branch of a single repo collected from within GitHub, Subversion, BitBucket, GitLab

  • “Quality” shows code coverage and number of defects found in code scans done by SonarQube

  • “Monitor” shows deployments to servers. Red and green dots represent whether the server is up or down

  • Version numbers in a Maven build specification file

  • “Nexus artifacts” from the binary repository

  • ChatOps

  • AWS

Additional collectors specified in docker-compose.yml include artifactory, appdynamics. Emmett?

PROTIP: Again, this visibility should not used for those outside the team to meddle with questions such as “why is this particular one 15”?

QUESTION: Does the dashboard cover these 16 gates in the pipeline (10 Commandments in octal):

DOES16 San Francisco - DevOps at Capital One: Focusing on Pipeline and Measurement IT Revolution by Topol Pal, (Director, Engineering Fellow, Capital One)

  1. Source code version control
  2. Optimum branching strategy [Git and GitHub or GitLab, etc.]
  3. Static analysis [SonarQube]
  4. Code coverage
  5. Vulnerability scan
  6. Open source scan [Black Duck]
  7. Artifact version control [Nexus or Artifactory]
  8. Auto provision
  9. Immutable servers
  10. Integration testing
  11. Performance testing
  12. Build, Deploy, Testing automated for every commit
  13. Automated Change Order
  14. Zero downtime release

Program-level Pipeline Dashboard

hygieia-pgm-shift-left-600x219 *

Added since version 2 is statistical analysis for trends of health and speed. One row for each team (such as “Tetris” in the example).

PROTIP: I recommend against a competitive comparison of numbers achieved by each team because that encourages unintended consequences such as increase in hidden quality issues and technical debt. Each team has different challenges.

Hygieia limits itself to just these environments:

commit > build > DEV > QA > INT > PERF > PROD

Higher Order Metrics

PROTIP: Consider using metrics that reveal “higher order” calculations that can be use to predict future health and be used as the basis for recommendations.

  • Elapsed response time as a function of load?

perf-607x173

Was this pattern of performance predicted from testing?

  • Network latency separate from server response time by having monitoring clients near the server.

  • Difference in response time before and after a change.

dashboard-diff-689x291-28926

  • Cycle time to instantiate a server.

gce-startup-time-640x326

  • Man-Months of backlog in innovations and defect fix effort

  • Man-Months of “Technical Debt”

  • Percent of work unplanned (“error budget”)

  • Percent of development (coding) innovation vs. repetitive work

  • http://www.veracode.com/services/veracode-vs.-on-premise-tools

See

  • http://www.smlcodes.com/tutorials/hygieia-dashboard-tutorial/

Displays of trends over time are important to keep numbers in perspective, both to keep from over reacting to momentary anomalies and from under-reacting to underlying patterns that need to be fixed.

PROTIP: So it’s better to have a set of rotating dashboards (showing trends) than having just a number on a dashboard without context to whether that number is “good” or “bad”.

PROTIP: Have a projection of what was expected at each point in time, especially in the future.

Having an arbitrary target number can be counter-productive unless individual employees have a coherent approach that balances the many conflicting needs.

For example, an insistance on “100% all the time” can lead staff to prioritize caution over innovation.

QUESTION: Should trend information be considered during a Production Readiness Review (PRR)?

Financial and Strategic

Executives and business managers typically focus on financials :

  • Total cost per transaction ratio
  • Total cost as percent of revenue
  • Total revenue per employee

Upper management need to manage over a longer time horizon. So they need to see trends over time, especially those that reflect customer experience (not just internal processes):

  • Availability of the system
  • Productivity of end-users using the system being developed, such as
    purchases, invoices, or other business transactions processed during a peak hour.

  • Customer Net Promoter Score

  • Employee satisfaction
  • Employee turnover rate

Many may balk that the above are not “relevant” to DevOps. But if not, then how important is the justification for going DevOps?

Hygieia Build

http://www.capitalone.io/Hygieia/setup.html references code and automation at
https://github.com/capitalone/Hygieia

  1. In a Google Compute Cloud Console.

    QUESTION: Is there a Terraform template?

  2. Within the Google Cloud Console execute:

    curl -fsSL https://github.com/wilsonmar/hygieia | sh

  3. Automation to instantiate a cluster of servers to establish Hygieia:

    NOTE: Hygieia was written in Java to store data in a MongoDB database.

    The Hygieia API server exposes REST APIs written in Spring Boot and mysema.querydsl.

    The Hygieia core server provides the UI to data collected.

    Open port 9090

  4. Fork https://github.com/capitalone/Hygieia to your own account.

  5. Create a container folder to hold several related repositories.

    git clone https://github.com/ My GitHub Acct /Hygieia

    At the time of writing, this took up 131.1 MB of disk space.

    git clone https://github.com/ My GitHub Acct /Hygieia –depth=1

    At the time of writing, this took up 114.2 MB of disk space.

  6. Download and build via maven using pom.xml file:

    mvn clean install package

    PROTIP: If you enjoy reading the deluge to the console, expand the Terminal width to avoid wrapping.

    The response:

    [INFO] Total time: 08:34 min
    [INFO] Finished at: 2017-03-18T21:11:46-04:00
    [INFO] Final Memory: 108M/1581M
    

    At the time of writing, after install the folder takes 1.23 GB of disk space.

  7. Install MongoDB for the API data store
  8. Run collectors with properties to connect to CI tools
  9. Seteup Dashboard widgets & Visualize

    QUESTION: Can only have one dashboard?

PROTIP: Have computer programs monitor servers and take automatic actions.

More Info

https://gitter.im/capitalone/Hygieia

Videos:

Hygieia configuration

There are several gulpfile.js within $HOME/gits/hygieia/UI/

  • /gulpfile.js
  • /node_modules/gulp-angular-filesort/node_modules/ng-dependencies/node_modules/estraverse/gulpfile.js
  • /node_modules/gulp-angular-filesort/node_modules/ng-dependencies/gulpfile.js
  • /node_modules/gulp-less/examples/gulpfile.js
  • /node_modules/browser-sync/node_modules/dev-ip/gulpfile.js
  • /node_modules/browser-sync/node_modules/browser-sync-client/gulpfile.js
  • /node_modules/gulp-css-globbing/node_modules/vinyl-map/example/gulpfile.js
  • /node_modules/gulp-csso/gulpfile.js

To change the default port, change the gulpfile.js from $HOME/gits/hygieia/UI/

      browserSync.init({
          server: {
              baseDir: hygieia.dist,
              startPath: '/',
              middleware: [proxyMiddleware]
          },
          ghostMode: ghostMode
      });
   

to this:

       browserSync.init({
          port: 9999,
          server: {
              baseDir: hygieia.dist,
              startPath: '/',
              middleware: [proxyMiddleware]
          },
          ghostMode: ghostMode
      });
   

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering