Wilson Mar bio photo

Wilson Mar


Calendar YouTube Github


Control Kubernetes internals from Golang programs, as a part of the Red Hat Industrial Complex

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


Docker was once the “next big thing”.

Red Hat (now a part of IBM) is now claimed as the “next big thing” in Kubernetes with its Ansible, OpenShift Container Platform, CoreOS, Quay, and Kubernetes Operators.

OLM (Operator Lifecycle Manager)

It has long been said that

“Kubernetes is a platform for building platforms”

Now the Operator Lifecycle Manager (OLM) - powered from https://github.com/operator-framework/operator-lifecycle-manager - is a platform! OLM extends Kubernetes to provide a declarative way to install, manage, and upgrade Operators within a Kubernetes cluster.


  • https://olm.operatorframework.io/ says OLM facilitates installation, upgrade, and role-based access control (RBAC) of operators within a cluster.
  • https://techbloc.net/archives/4372

Operator Hub Registry

https://operatorhub.io/?view=list was launched by Red Hat Feb 2019 as, like Dockerhub (hub.docker.com), a registry (list) of Kubernetes Operators to install. It’s powered by https://github.com/k8s-operatorhub/community-operators

PROTIP: Note that many Operators are for Open Shift, Red Hat’s proprietary flavor of Kubernetes.

Scroll down to filter among Capability Levels:


At time of this writing (January 2023), there are less than 300 vs. thousands in hub.docker.com, terraform.io, etc. QUESTION: What are all the Operators that are needed to achieve full production utility within an enterprise?

The different capability levels are illustrated by this: k8s-operator-2814x1154.jpg

From Helm to Ansible to Golang

The declarative nature of Helm charts (and Terraform) are limited to “Day-1” concerns of installation, configuration, etc. but not “Day-2” concerns of re-configuration, update, backup, failover, restore, etc.


  1. How to make use of an existing Helm chart.
  2. How to quickly bring a “standard” production-quality Kubernetes cluster up with a full set of Operators? In one command?
  3. How can one assure that all operators within Kubernetes all work well with each other?
  4. How to get to Deep Insights and Auto Pilot?
  5. With Operators, how do we see version history of who changed what when? How to analyze logs emitted?

Kubernetes Operators?

Kubernetes Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Kubernetes Operators, as a controller, runs as a Pod on Kubernetes cluster master nodes, in a software loop, continously auditing object states to ensure a desired state, by deploying applications in a standardized and opinionated way.

“With Operators you can stop treating an application as a collection of primitives like Pods, Deployments, Services or ConfigMaps, but instead as a single object that only exposes the knobs that make sense for the application.”

Brandon Philips, CTO of CoreOS blogged on November 3, 2016 wrote:

“An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user. … Examples of complex administrative tasks that an Operator might handle include safe coordination of application upgrades, configuration of backups to offsite storage, service discovery via native Kubernetes APIs, application TLS certificate configuration, and disaster recovery.”

Kubernetes Operators are installed as a Kubernetes deployment kind of yaml (Stateful Sets), which define a new third party application instance type.


  • https://sdk.operatorframework.io/

  1. Install Golang.

    PROTIP: Ironically, https://sdk.operatorframework.io/docs/building-operators/golang/installation/ still specifies use of Docker.

  2. There is a brew formula to install the operator-sdk app:

    brew info operator-sdk
  3. Install the operator-sdk app. On a Mac:

    brew install operator-sdk
    ==> Pouring go--1.19.4.arm64_monterey.bottle.tar.gz
    🍺  /opt/homebrew/Cellar/go/1.19.4: 12,452 files, 629.1MB
    ==> Installing operator-sdk
    ==> Pouring operator-sdk--1.26.0.arm64_monterey.bottle.tar.gz
    ==> Caveats
    zsh completions have been installed to:
    ==> Summary
    🍺  /opt/homebrew/Cellar/operator-sdk/1.26.0: 10 files, 201.1MB
    ==> Running `brew cleanup operator-sdk`...
    Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
    Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
    ==> Caveats
    ==> operator-sdk
    zsh completions have been installed to:

    Alternately, see

    • https://sdk.operatorframework.io/docs/installation/
    • https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/

  4. PROTIP: To save typing toil and time, I’ve defined among my other keyboard aliases the letter “o” so I don’t have to type “operator-sdk” every time:

    alias o="operator-sdk"
  5. Run aliases.sh, then view the operator-sdk command menu using the alias:

    CLI tool for building Kubernetes extensions and tools.
      operator-sdk [flags]
      operator-sdk [command]
    The first step is to initialize your project:
     operator-sdk init [--plugins= [--project-version=]]
    <PLUGIN KEYS> is a comma-separated list of plugin keys from the following table
    and <PROJECT VERSION> a supported project version for these plugins.
                                    Plugin keys | Supported project versions
            ansible.sdk.operatorframework.io/v1 |                          3
               declarative.go.kubebuilder.io/v1 |                       2, 3
        deploy-image.go.kubebuilder.io/v1-alpha |                          3
                           go.kubebuilder.io/v2 |                       2, 3
                           go.kubebuilder.io/v3 |                          3
                     go.kubebuilder.io/v4-alpha |                          3
                grafana.kubebuilder.io/v1-alpha |                          3
               helm.sdk.operatorframework.io/v1 |                          3
     hybrid.helm.sdk.operatorframework.io/v1-alpha |                          3
            quarkus.javaoperatorsdk.io/v1-alpha |                          3
    For more specific help for the init command of a certain plugins and project version
    configuration please run:
     operator-sdk init --help --plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]
    Default plugin keys: "go.kubebuilder.io/v3"
    Default project version: "3"
    Available Commands:
      alpha            Alpha-stage subcommands
      bundle           Manage operator bundle metadata
      cleanup          Clean up an Operator deployed with the 'run' subcommand
      completion       Load completions for the specified shell
      create           Scaffold a Kubernetes API or webhook
      edit             Update the project configuration
      generate         Invokes a specific generator
      help             Help about any command
      init             Initialize a new project
      olm              Manage the Operator Lifecycle Manager installation in your cluster
      pkgman-to-bundle Migrates packagemanifests to bundles
      run              Run an Operator in a variety of environments
      scorecard        Runs scorecard
      version          Print the operator-sdk version
      -h, --help                     help for operator-sdk
       --plugins strings          plugin keys to be used for this subcommand execution
       --project-version string   project version (default "3")
       --verbose                  Enable verbose logging
    Use "operator-sdk [command] --help" for more information about a command.
  6. Get the version (with Kubernetes meta) from https://github.com/operator-framework/operator-sdk/releases

    operator-sdk version
    operator-sdk version: "v1.26.0", commit: "cbeec475e4612e19f1047ff7014342afe93f60d2", kubernetes version: "v1.25.0", go version: "go1.19.4", GOOS: "darwin", GOARCH: "arm64"
  7. To get just the operator-sdk version number (such as v1.26.0):

    operator-sdk version | awk '{print $3}' | tr -d '",' 

    awk ‘{print $3}’ yields “v1.26.0”,

    tr -d ‘”,’ removes the double-quotes and comma

    Modify existing Operator

    PROTIP: Deepak Singh Dhami recommends modifying one before creating your own.

    https://operatorhub.io/operator/vault at https://github.com/banzaicloud/bank-vaults/tree/main/operator by Nándor István Krácser at Banzai Cloud (acquired by Cisco). The yaml sample as part of the Banzai Cloud Pipeline

    apiVersion: vault.banzaicloud.com/v1alpha1
    kind: Vault
      name: vault
      size: 1
      image: 'vault:1.0.0'
      bankVaultsImage: 'banzaicloud/bank-vaults:latest'
     prometheus.io/scrape: 'true'
     prometheus.io/port: '9102'
      serviceAccount: vault
      serviceType: ClusterIP
     annotations: null
     spec: {}
       secretNamespace: default
         path: /vault/file
         address: ''
         tls_cert_file: /vault/tls/server.crt
         tls_key_file: /vault/tls/server.key
       statsd_address: 'localhost:9125'
     ui: true
       - name: allow_secrets
         rules: >-
           path "secret/*" { capabilities = ["create" "read" "update" "delete"
           "list"] }
       - type: kubernetes
           - name: default
             bound_service_account_names: default
             bound_service_account_namespaces: default
             policies: allow_secrets
             ttl: 1h
       - path: secret
         type: kv
         description: General secrets.
           version: 2
       - type: kv
         path: secret/data/accounts/aws
             AWS_ACCESS_KEY_ID: secretId
             AWS_SECRET_ACCESS_KEY: s3cr3t
     - name: VAULT_LOG_LEVEL
       value: debug

    CAUTION: Using vault:1.0.0?

New Go Operator



  1. Activate Go module support before using the SDK:

    export GO111MODULE=on
  2. To create a new Operator of your very own:

    operator-sdk init --domain example.com --repo github.com/example/memcached-operator

    –domain is the prefix of the API group custom resources will be created in.

    The command above generates a go.mod file.


    • https://github.com/alexcpn/go_operator_2022 is referenced by
    • https://medium.com/techlogs/go-kubernetes-operator-963461e528c5 by Alex Punnen who recommands
    • https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/
    • https://medium.com/techlogs/go-kubernetes-operator-963461e528c5
    • https://book.kubebuilder.io/quick-start.html
    • https://book.kubebuilder.io/cronjob-tutorial/cronjob-tutorial.html
    • Read till Section 1.7 https://book.kubebuilder.io/cronjob-tutorial/controller-implementation.html

    Unit Testing

  3. Static tests consist of unit, formatting, and doc link tests.


  4. Docs only

    make test-static

  5. Code

    make test-all


  • https://itnext.io/analyzing-value-of-operator-framework-for-kubernetes-community-5a65abc259ec

  • https://docs.tungsten.io/en/latest/tungsten-fabric-operator/working-with-kubernetes-operators-101.html


  • https://learning.oreilly.com/library/view/kubernetes-in-production/9781800202450/
  • https://www.amazon.com/Kubernetes-Operators-Automating-Container-Orchestration/dp/1492048046
  • https://www.youtube.com/watch?v=i9V4oCa5f9I
  • https://www.youtube.com/watch?v=ha3LjlD6g7g “Kubernetes Operator simply explained in 10 mins” by Nana
  • https://k21academy.com/docker-kubernetes/kubernetes-operator/
  • https://cloudark.medium.com/why-to-write-kubernetes-operators-9b1e32a24814 from 2018
  • https://www.linux.com/topic/cloud/demystifying-kubernetes-operators-operator-sdk-part-1/
  • https://shipit.dev/posts/k8s-operators-with-python-part-1.html Creating CRDs
  • https://shipit.dev/posts/k8s-operators-with-python-part-2.html Implementing Controller
  • https://www.cncf.io/blog/2022/06/15/kubernetes-operators-what-are-they-some-examples/
  • https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
  • https://www.weave.works/blog/creating-custom-kubernetes-operators
  • https://cloud.redhat.com/blog/build-your-kubernetes-operator-with-the-right-tool
  • CNCF Operator white paper

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators
  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling
  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft
  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering