Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Open-source server to display metrics for DevSecOps and others

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This article are my notes about Grafana, marketed at Grafana.com, with documentation at https://grafana.com/docs/grafana/latest

My contribution here is logical sequencing for a deep yet concise hands-on tutorial.

Grafana was as open-source in 2014 at https://github.com/Grafana/Grafana. It is one of the most popular open source projects on GitHub.

Torkel Ödegaard (@Grafana@torkelo), formerly at eBay Stockholm, Sweden, the original developer, spoke Dec. 19, 2014 in San Francisco six months after release.

  • Grafana is written in Golang.
  • [21:53] “I work on this full time, mostly paid by my own savings. But I have sponsors in Squarespace and Dreamhost”.

Install

  1. https://www.grafana.com/download

    Grafana’s releases are sematically tagged.

    There is an Enterprise Edition because the organization makes money by selling its Enterprise edition and cloud services.

    Enterprise edition

    Their Install Guide says “Open Source - Functionally identical to the enterprise version, but you will need to download the enterprise version if you want enterprise features”

    Grafana Enterprise adds enterprise data sources, advanced authentication options (LDAP, SAML), more permission controls, 24x7x365 support, and training from the core Grafana team.

    There is no installer for MacOS Enterprise, so run the Docker image on your Mac.

    Raspberry Pi

    There is an installer for ARM chips (Raspberry Pi). See https://www.hackster.io/search?i=projects&q=grafana

    Releases

  2. Details about each release are at:

    https://github.com/grafana/grafana/releases

  3. Install using Homebrew:

    brew install grafana

    The response at time of writing:

    Downloading https://homebrew.bintray.com/bottles/grafana-6.6.1.mojave.bottle.tar.gz
    ==>
    🍺  /usr/local/Cellar/grafana/6.6.1: 4,417 files, 189.6MB
    
  4. Confirm server version installed:

    grafana-server -v

    The response at time of writing:

    Version 6.6.1 (commit: unknown-dev, branch: master)
  5. To have launchd start grafana now and restart at login:

    sudo brew services start grafana

    Alternately, if you don’t want/need a background service you can just run:

    sudo grafana-server --config=/usr/local/etc/grafana/grafana.ini --homepath /usr/local/share/grafana --packaging=brew cfg:default.paths.logs=/usr/local/var/log/grafana cfg:default.paths.data=/usr/local/var/lib/grafana cfg:default.paths.plugins=/usr/local/var/lib/grafana/plugins
    
    
    
  6. Type in your password and click Allow.

    grafana-accept-790x304

    The response, for example:

    INFO[03-11|17:19:43] Starting Grafana                         logger=server version=6.6.1 commit=unknown-dev branch=master compiled=2020-02-06T10:22:13-0700
    INFO[03-11|17:19:43] Config loaded from                       logger=settings file=/usr/local/share/grafana/conf/defaults.ini
    INFO[03-11|17:19:43] Config loaded from                       logger=settings file=/usr/local/etc/grafana/grafana.ini
    INFO[03-11|17:19:43] Config overridden from command line      logger=settings arg="default.paths.data=/usr/local/var/lib/grafana"
    INFO[03-11|17:19:43] Config overridden from command line      logger=settings arg="default.paths.logs=/usr/local/var/log/grafana"
    INFO[03-11|17:19:43] Config overridden from command line      logger=settings arg="default.paths.plugins=/usr/local/var/lib/grafana/plugins"
    INFO[03-11|17:19:43] Path Home                                logger=settings path=/usr/local/share/grafana
    INFO[03-11|17:19:43] Path Data                                logger=settings path=/usr/local/var/lib/grafana
    INFO[03-11|17:19:43] Path Logs                                logger=settings path=/usr/local/var/log/grafana
    INFO[03-11|17:19:43] Path Plugins                             logger=settings path=/usr/local/var/lib/grafana/plugins
    INFO[03-11|17:19:43] Path Provisioning                        logger=settings path=/usr/local/share/grafana/conf/provisioning
    INFO[03-11|17:19:43] App mode production                      logger=settings
    INFO[03-11|17:19:43] Initializing SqlStore                    logger=server
    INFO[03-11|17:19:43] Connecting to DB                         logger=sqlstore dbtype=sqlite3
    INFO[03-11|17:19:43] Starting DB migration                    logger=migrator
    INFO[03-11|17:19:43] Initializing HTTPServer                  logger=server
    INFO[03-11|17:19:43] Initializing BackendPluginManager        logger=server
    INFO[03-11|17:19:43] Initializing PluginManager               logger=server
    INFO[03-11|17:19:43] Starting plugin search                   logger=plugins
    INFO[03-11|17:19:43] Initializing HooksService                logger=server
    INFO[03-11|17:19:43] Initializing OSSLicensingService         logger=server
    INFO[03-11|17:19:43] Initializing InternalMetricsService      logger=server
    INFO[03-11|17:19:43] Initializing RemoteCache                 logger=server
    INFO[03-11|17:19:43] Initializing QuotaService                logger=server
    INFO[03-11|17:19:43] Initializing ServerLockService           logger=server
    INFO[03-11|17:19:43] Initializing UserAuthTokenService        logger=server
    INFO[03-11|17:19:43] Initializing RenderingService            logger=server
    INFO[03-11|17:19:43] Initializing AlertEngine                 logger=server
    INFO[03-11|17:19:43] Initializing DatasourceCacheService      logger=server
    INFO[03-11|17:19:43] Initializing LoginService                logger=server
    INFO[03-11|17:19:43] Initializing SearchService               logger=server
    INFO[03-11|17:19:43] Initializing TracingService              logger=server
    INFO[03-11|17:19:43] Initializing UsageStatsService           logger=server
    INFO[03-11|17:19:43] Initializing CleanUpService              logger=server
    INFO[03-11|17:19:43] Initializing NotificationService         logger=server
    INFO[03-11|17:19:43] Initializing provisioningServiceImpl     logger=server
    INFO[03-11|17:19:43] Backend rendering via phantomJS          logger=rendering renderer=phantomJS
    WARN[03-11|17:19:43] phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. logger=rendering renderer=phantomJS
    INFO[03-11|17:19:43] Initializing Stream Manager 
    INFO[03-11|17:19:43] HTTP Server Listen                       logger=http.server address=[::]:3000 protocol=http subUrl= socket=
    

    Upgrade Grafana

  7. To upgrade, brew reinstall grafana

Data sources

Grafana has a growing list of data sources it can read from, via plug-ins:

  • CloudWatch
  • Elasticsearch (document database)

  • Graphite
  • Microsoft SQL Server
  • MySQL
  • ProtgreSQL

  • OpenTSDB (time series database)
  • InfluxDB
  • Prometheus

2h 43m PromQL for Prometheus 2.3.1 within a Docker image. He recommends having each project deploy its own Prometheus and Grafana instance rather than a central monitoring stack, sharing the admin burden. need to be highly available.

Datasource Plugin Thrunk (by Sven Nierlein)?

Filters

PROTIP: Define your filter statements before jumping into the Grafana UI.

Panels within Dashboard

Within each dashboard are panels with rows.

To add an additional panel within an established dashboard:

  1. In your profile, get Editor Role.
  2. No need to designate a data source.
  3. Star the dashboard (Mark as favorite) on the top menu.
  4. Set Home Dashboard in Preferences.

  5. Add panel.
  6. Choose visualization.
  7. Graph.

Share (archive) dashboard

  1. Click on the share icon at the top menu.
  2. Click Export tab.
  3. Consider “Export for sharing externally”.
  4. Click “Save to file”. Adjust folder and file name.

    PROTIP: Save the file in GitHub for Infrastructure as Code. Then you won’t need to add a date/time stamp to the file.

  5. Add tab, Add Row
  6. Click on Row gear icon for options.
  7. Update.
  8. Click icon at top. Dummy graph appears.
  9. Change Panel Title.
  10. Select Data Source: “prometheus”.
  11. Add query string
  12. Set Panel Title.

Videos from Grafana.com

Torkel himself presents 12 videos in this YouTube Playlist also listed in the Grafana website:

1/. Building Graphite Queries Sep 15, 2014 [14:02]

A leaf note is a variable that contains data (such as count, dollars, etc.).

{} braces glob individual metrics together.

  1. Templated Graphite Queries Sep 16, 2014 12:55

  2. What’s New in Grafana 2.0 Apr 19, 2015 [13:48]

  3. Installation & Configuration on Ubuntu / Debian 14:36

  4. Installation & Configuration on Redhat / Centos 14:04

  5. Adding data sources, users & organizations 15:17

  6. Beginners guide to building dashboards 10:07

  7. What’s New In Grafana 3.0 19:13

  8. Elasticsearch 24:42

  9. Dashboard Discovery & Sharing 7:40

Templates

Templates enable you to add variables

  1. In Dashboard settings, Features tab, enable Templating checkbox. Close.
  2. Click the gear next to VARIABLES.
  3. Add a variable with a query type.
  4. All option to use a * wildcard in the query.
  5. Click the green > arrow to see values returned.

  6. Also, add an interval type of variable.
  7. Edit the time periods suggested in the Value field.
  8. In the Graph, change the summarize to specify the variable.
  9. VARIABLES: $internval auto [11:10]

  10. Variable “smoothing” with Type custom [11:24]
  11. Change the movingAverage value to $smoothing

SQL

Grafana adds WHERE clause for time, such as $__timeFilter(time).

Plug-ins

grafana-cli plugins install sni-thruk-datasource

Videos

Monitoring Containerized Application Health with Docker
by Elton Stoneman 8 Aug 2018

How To Setup A Grafana Dashboard Step By Step - SQL4automation 21:00

Grafana Project Update & What’s New in v6.0 Mar 11, 2019 [40:31] Grafana Loki for Logs [11:40] In 2013 Grafana was re-written from AngularJs to React.

Setting up Prometheus and Grafana for monitoring your servers Jan 22, 2019 by Eddie Zaneski

How To Setup A Grafana Dashboard Step By Step Nov 5, 2019

Business Dashboards with Grafana and MySQL

Grafana at CERN by Borja Garrido Bear

What we Learned Integrating Grafana with Prometheus

Grafana Loki: Like Prometheus, But for logs by Tom Wilkie (@tom_wilkie), Grafana Labs CNCF [Cloud Native Computing Foundation]

Using Prometheus and Grafana for Monitoring my Power Usage Grafana

Using Postgres, Prometheus and Grafana for Storing, Analyzing and Visualizing Metrics by Erik Nordstrom from TimescaleDB

What we Learned Integrating Grafana with Prometheus Mar 1, 2018 by Dimitri Vanoverbeke from Percona

What Does Kubernetes Look Like? Performance Monitoring and Visualization with Grafana by Jacob Lisi

Automating Grafana Dashboards with Jsonnet Grafana

Weather, Power, and Market Forecasts with Grafana by Max Von Roden & Steffen Knott Mar 23, 2018

Workshop: Writing React Plugins by Terkel

Workshop: Grafana Plugin Showcase, Part 1 Workshop: Grafana Plugin Showcase, Part 2 Grafana

Workshop: Become a contributor - Get started developing Grafana Grafana

Social Media

As of Feb. 14, 2020:

  • Gitter chat
  • 2,522 follow @grafana on Twitter
  • 243,522 follower it on LinkedIn
  • 137,569 people follow its Facebook
  • Get release notices via User group is on Google Groups at
    https://groups.google.com/forum/#!forum/go-cd

  • Subscribe to “The Pipeline” bi-monthly newsletter with continuous delivery news and events.

    ### References about Grafana

    https://www.youtube.com/watch?v=sTP7yzXmdFk&list=PLyJqGMYm0vnO9osZ-EBV6iu2l10muE2A- Free Grafana Complete course for beginners by I.T Security Labs

    https://thenewstack.io/why-you-want-easy-to-setup-grafana-dashboards/

    https://www.youtube.com/watch?v=Xj-TwAZtWHQ ELI5 - What I Learned Teaching Grafana to My Kids Grafana

    https://www.youtube.com/watch?v=4qpI4T6_bUw How To Setup A Grafana Dashboard Step By Step SQL4automation


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering