K8s Operators

Control Kubernetes internals from Golang programs, as a part of the Red Hat Industrial Complex

Overview

OLM (Operator Lifecycle Manager)
Operator Hub Registry
- From Helm to Ansible to Golang
Kubernetes Operators?
operator-sdk
- Modify existing Operator
New Go Operator
- Unit Testing
Utilities
Resources
More on DevOps

Docker was once the “next big thing”.

Red Hat (now a part of IBM) is now claimed as the “next big thing” in Kubernetes with its Ansible, OpenShift Container Platform, CoreOS, Quay, and Kubernetes Operators.

OLM (Operator Lifecycle Manager)

It has long been said that

“Kubernetes is a platform for building platforms”

Now the Operator Lifecycle Manager (OLM) - powered from https://github.com/operator-framework/operator-lifecycle-manager - is a platform! OLM extends Kubernetes to provide a declarative way to install, manage, and upgrade Operators within a Kubernetes cluster.

References:

https://olm.operatorframework.io/ says OLM facilitates installation, upgrade, and role-based access control (RBAC) of operators within a cluster.
https://techbloc.net/archives/4372

Operator Hub Registry

https://operatorhub.io/?view=list was launched by Red Hat Feb 2019 as, like Dockerhub (hub.docker.com), a registry (list) of Kubernetes Operators to install. It’s powered by https://github.com/k8s-operatorhub/community-operators

PROTIP: Note that many Operators are for Open Shift, Red Hat’s proprietary flavor of Kubernetes.

Scroll down to filter among Capability Levels:

k8s-operator-registry-capabilities-384x370

At time of this writing (January 2023), there are less than 300 vs. thousands in hub.docker.com, terraform.io, etc. QUESTION: What are all the Operators that are needed to achieve full production utility within an enterprise?

The different capability levels are illustrated by this:

From Helm to Ansible to Golang

The declarative nature of Helm charts (and Terraform) are limited to “Day-1” concerns of installation, configuration, etc. but not “Day-2” concerns of re-configuration, update, backup, failover, restore, etc.

QUESTIONS:

How to make use of an existing Helm chart.
How to quickly bring a “standard” production-quality Kubernetes cluster up with a full set of Operators? In one command?
How can one assure that all operators within Kubernetes all work well with each other?
How to get to Deep Insights and Auto Pilot?
With Operators, how do we see version history of who changed what when? How to analyze logs emitted?

Kubernetes Operators?

Kubernetes Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Kubernetes Operators, as a controller, runs as a Pod on Kubernetes cluster master nodes, in a software loop, continously auditing object states to ensure a desired state, by deploying applications in a standardized and opinionated way.

“With Operators you can stop treating an application as a collection of primitives like Pods, Deployments, Services or ConfigMaps, but instead as a single object that only exposes the knobs that make sense for the application.”

Brandon Philips, CTO of CoreOS blogged on November 3, 2016 wrote:

“An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user. … Examples of complex administrative tasks that an Operator might handle include safe coordination of application upgrades, configuration of backups to offsite storage, service discovery via native Kubernetes APIs, application TLS certificate configuration, and disaster recovery.”

Kubernetes Operators are installed as a Kubernetes deployment kind of yaml (Stateful Sets), which define a new third party application instance type.

operator-sdk

https://sdk.operatorframework.io/

Install Golang.

PROTIP: Ironically, https://sdk.operatorframework.io/docs/building-operators/golang/installation/ still specifies use of Docker.
There is a brew formula to install the operator-sdk app:
```
brew info operator-sdk
```

Install the operator-sdk app. On a Mac:

brew install operator-sdk

==> Pouring go--1.19.4.arm64_monterey.bottle.tar.gz
🍺  /opt/homebrew/Cellar/go/1.19.4: 12,452 files, 629.1MB
==> Installing operator-sdk
==> Pouring operator-sdk--1.26.0.arm64_monterey.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/operator-sdk/1.26.0: 10 files, 201.1MB
==> Running `brew cleanup operator-sdk`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Caveats
==> operator-sdk
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions

Alternately, see

https://sdk.operatorframework.io/docs/installation/
https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/

PROTIP: To save typing toil and time, I’ve defined among my other keyboard aliases the letter “o” so I don’t have to type “operator-sdk” every time:
```
alias o="operator-sdk"
```

Run aliases.sh, then view the operator-sdk command menu using the alias:

CLI tool for building Kubernetes extensions and tools.
 
Usage:
  operator-sdk [flags]
  operator-sdk [command]
 
Examples:
The first step is to initialize your project:
 operator-sdk init [--plugins= [--project-version=]]
 
<PLUGIN KEYS> is a comma-separated list of plugin keys from the following table
and <PROJECT VERSION> a supported project version for these plugins.

                                Plugin keys | Supported project versions
-----------------------------------------------+----------------------------
        ansible.sdk.operatorframework.io/v1 |                          3
           declarative.go.kubebuilder.io/v1 |                       2, 3
    deploy-image.go.kubebuilder.io/v1-alpha |                          3
                       go.kubebuilder.io/v2 |                       2, 3
                       go.kubebuilder.io/v3 |                          3
                 go.kubebuilder.io/v4-alpha |                          3
            grafana.kubebuilder.io/v1-alpha |                          3
           helm.sdk.operatorframework.io/v1 |                          3
 hybrid.helm.sdk.operatorframework.io/v1-alpha |                          3
        quarkus.javaoperatorsdk.io/v1-alpha |                          3
 
For more specific help for the init command of a certain plugins and project version
configuration please run:
 operator-sdk init --help --plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]
 
Default plugin keys: "go.kubebuilder.io/v3"
Default project version: "3"
 
 
Available Commands:
  alpha            Alpha-stage subcommands
  bundle           Manage operator bundle metadata
  cleanup          Clean up an Operator deployed with the 'run' subcommand
  completion       Load completions for the specified shell
  create           Scaffold a Kubernetes API or webhook
  edit             Update the project configuration
  generate         Invokes a specific generator
  help             Help about any command
  init             Initialize a new project
  olm              Manage the Operator Lifecycle Manager installation in your cluster
  pkgman-to-bundle Migrates packagemanifests to bundles
  run              Run an Operator in a variety of environments
  scorecard        Runs scorecard
  version          Print the operator-sdk version
 
Flags:
  -h, --help                     help for operator-sdk
   --plugins strings          plugin keys to be used for this subcommand execution
   --project-version string   project version (default "3")
   --verbose                  Enable verbose logging
 
Use "operator-sdk [command] --help" for more information about a command.
</pre>

Get the version (with Kubernetes meta) from https://github.com/operator-framework/operator-sdk/releases

operator-sdk version

operator-sdk version: "v1.26.0", commit: "cbeec475e4612e19f1047ff7014342afe93f60d2", kubernetes version: "v1.25.0", go version: "go1.19.4", GOOS: "darwin", GOARCH: "arm64"

To get just the operator-sdk version number (such as v1.26.0):

operator-sdk version | awk '{print $3}' | tr -d '",'

awk ‘{print $3}’ yields “v1.26.0”,

tr -d ‘”,’ removes the double-quotes and comma

Modify existing Operator

PROTIP: Deepak Singh Dhami recommends modifying one before creating your own.

https://operatorhub.io/operator/vault at https://github.com/banzaicloud/bank-vaults/tree/main/operator by Nándor István Krácser at Banzai Cloud (acquired by Cisco). The yaml sample as part of the Banzai Cloud Pipeline

apiVersion: vault.banzaicloud.com/v1alpha1
kind: Vault
metadata:
  name: vault
spec:
  size: 1
  image: 'vault:1.0.0'
  bankVaultsImage: 'banzaicloud/bank-vaults:latest'
  annotations:
 prometheus.io/scrape: 'true'
 prometheus.io/port: '9102'
  serviceAccount: vault
  serviceType: ClusterIP
  ingress:
 annotations: null
 spec: {}
  unsealConfig:
 kubernetes:
   secretNamespace: default
  config:
 storage:
   file:
     path: /vault/file
 listener:
   tcp:
     address: '0.0.0.0:8200'
     tls_cert_file: /vault/tls/server.crt
     tls_key_file: /vault/tls/server.key
 telemetry:
   statsd_address: 'localhost:9125'
 ui: true
  externalConfig:
 policies:
   - name: allow_secrets
     rules: >-
       path "secret/*" { capabilities = ["create" "read" "update" "delete"
       "list"] }
 auth:
   - type: kubernetes
     roles:
       - name: default
         bound_service_account_names: default
         bound_service_account_namespaces: default
         policies: allow_secrets
         ttl: 1h
 secrets:
   - path: secret
     type: kv
     description: General secrets.
     options:
       version: 2
 startupSecrets:
   - type: kv
     path: secret/data/accounts/aws
     data:
       data:
         AWS_ACCESS_KEY_ID: secretId
         AWS_SECRET_ACCESS_KEY: s3cr3t
  vaultEnvsConfig:
 - name: VAULT_LOG_LEVEL
   value: debug

CAUTION: Using vault:1.0.0?

New Go Operator

https://sdk.operatorframework.io/build/

https://devenes.medium.com/discovering-the-power-of-go-for-kubernetes-5-open-source-tools-to-try-d8b5eb328075

Activate Go module support before using the SDK:
```
export GO111MODULE=on
```
To create a new Operator of your very own:
```
operator-sdk init --domain example.com --repo github.com/example/memcached-operator
```
–domain is the prefix of the API group custom resources will be created in.

The command above generates a go.mod file.

References:
- https://github.com/alexcpn/go_operator_2022 is referenced by
- https://medium.com/techlogs/go-kubernetes-operator-963461e528c5 by Alex Punnen who recommands
- https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/
- https://medium.com/techlogs/go-kubernetes-operator-963461e528c5
- https://book.kubebuilder.io/quick-start.html
- https://book.kubebuilder.io/cronjob-tutorial/cronjob-tutorial.html
- Read till Section 1.7 https://book.kubebuilder.io/cronjob-tutorial/controller-implementation.html
Unit Testing
Static tests consist of unit, formatting, and doc link tests.

https://sdk.operatorframework.io/docs/contribution-guidelines/testing/
Docs only

make test-static
Code

make test-all

Utilities

https://itnext.io/analyzing-value-of-operator-framework-for-kubernetes-community-5a65abc259ec
https://docs.tungsten.io/en/latest/tungsten-fabric-operator/working-with-kubernetes-operators-101.html

Resources

https://learning.oreilly.com/library/view/kubernetes-in-production/9781800202450/
https://www.amazon.com/Kubernetes-Operators-Automating-Container-Orchestration/dp/1492048046
https://www.youtube.com/watch?v=i9V4oCa5f9I
https://www.youtube.com/watch?v=ha3LjlD6g7g “Kubernetes Operator simply explained in 10 mins” by Nana
https://k21academy.com/docker-kubernetes/kubernetes-operator/
https://cloudark.medium.com/why-to-write-kubernetes-operators-9b1e32a24814 from 2018
https://www.linux.com/topic/cloud/demystifying-kubernetes-operators-operator-sdk-part-1/
https://shipit.dev/posts/k8s-operators-with-python-part-1.html Creating CRDs
https://shipit.dev/posts/k8s-operators-with-python-part-2.html Implementing Controller
https://www.cncf.io/blog/2022/06/15/kubernetes-operators-what-are-they-some-examples/
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
https://www.weave.works/blog/creating-custom-kubernetes-operators
https://cloud.redhat.com/blog/build-your-kubernetes-operator-with-the-right-tool
CNCF Operator white paper

More on DevOps

This is one of a series on DevOps:

Wilson Mar