- Open Sourced
- OpenShift support
- Kubernetes vernacular cartoon
- Terminology background
- Options for deployment
- kubectl CLI client install
- Sample micro-service apps
- pod.yml manifests
- Kubelet Daemonset.yaml
- Learning resources
- Make your own K8s
This page is being actively worked on, so please pardon the “dust”.
This is a tutorial presented as a hands-on workshop with commentary along the way.
The word “Kubernetes” is a registered trademark of the Linux Foundation, which maintains the website https://kubernetes.io and source code at
- v1.0 was committed on July 2015 within GitHub
- v1.6 was led by a CoreOS developer.
- v1.7 was led by Google.
- v1.8 is led by a Microsoft employee (Jaice Singer DuMars) after Microsoft joined the CNCF July 2017.
This blog and podcast about origins note that the Kubernetes logo has 7 sides because its initial developers were Star Trek fans: The predecessor to Kub was called Borg. A key Borg character is called “7 of 9”.
Anyway, its Google heritage means Kubernetes is about scaling for a lot of traffic with redundancies to achieve high availability (HA).
If you want to pay for Kubernetes, https://www.redhat.com/en/technologies/cloud-computing/openshift Red Hat® OpenShift is a container application platform that brings Docker and Kubernetes to the enterprise.
- Twitter: @kubernetesio
- Google+ Group: Kubernetes
https://groups.google.com/forum/#!forum/kubernetes-dev for contributors to the Kubernetes project to discuss design and implementation issues.
- https://stackoverflow.com/search?q=k8s+or+kubernetes for developers
- https://serverfault.com/search?q=k8s+or+kubernetes for sysadmins.
https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&disable_polymer=true Kubernetes Google Community video chats
- https://cloud.google.com/support/docs/issue-trackers to report bugs
Other orchestration systems for Docker containers:
https://codefresh.io/kubernetes-guides/kubernetes-cloud-aws-vs-gcp-vs-azure/ Kubernetes in the Cloud: AWS vs. GCP vs. Azure
Amazon Elastic Container Service for Kubernetes (Amazon EKS) was introduced December 2017 to work with AWS Elastic Load Balancing, IAM authentication, Amazon VPC isolation, AWS PrivateLink access, and AWS CloudTrail logging. EKS runs three Kubernetes masters across three Availability Zones in order to ensure high availability. EKS automatically provides a “highly available and scalable Kubernetes service” EKS automatically detects and replaces unhealthy masters, and provides automated version upgrades and patching for the masters. so you don’t have to choose appropriate instance types.
Kubernetes vernacular cartoon
Kubernetes automates the deployment (creation) of Dockerized apps running as containers within pods arranged in clusters of nodes (previously called minions).
Kubernetes is called an “orchestrator” of various micro-service apps built from Docker images in a binary repository (such as Nexus or Artifactory). The images are built by Docker reading Dockerfiles in a configuration source repository such as GitHub.
BTW, instead of Docker, Kubernetes also works with rkt (pronounced “rocket”). But this tutorial focuses on Docker.
Deployments manage Pods.
Every Pod has a unique IP. There is one IP Address per Pod. In other words, containers within a Pod share a network namespace.
Every container has its own unique port number within its pod’s IP.
Containers also share attached data volumes available within each Pod.
System administrators control the Master node UI in the cloud or write scripts that invoke kubectl command-line client program that controls the Kubernetes Master node.
The program talks to the Kubernetes API Server.
The Server gets Scheduler ???
The Server obtains from Controller Manager ???
The API Server and Scheduler stores data in an ETCD Cluster.
Kubernetes instantiates and then manages the state of containers.
Kubernestes can move
The Kube Proxy communicates only with Pod admin. whereas Kubelets communicate with individual pods as well.
HAProxy VRRP (Virtual Router Redundancy Protocol) http://searchnetworking.techtarget.com/definition/VRRP automatically assigns available Internet Protocol routers to participating hosts.
cAdvisor https://github.com/google/cadvisor (Container Advisor) collects, aggregates, processes, and exports information about running containers in order to provide container admins an understanding of the resource usage and performance characteristics of their running containers.
https://github.com/coreos/flannel flannel configures a IPv4 “layer 3” network fabric designed for Kubernetes. between multiple nodes in a cluster.
CoreOS’s Tectonic sets up flannel in the Kubernetes clusters it creates using the open source Tectonic Installer to drive the setup process.
flannel is also widely used outside of kubernetes. When deployed outside of kubernetes, etcd is always used as the datastore.
- Infrastructure as code
- Manage containers
- Naming and discovery
- Mounting storage systems
- Balancing loads
- Rolling updates
- Distributing secrets/config
- Checking application health
- Monitoring resources
- Accessing and ingesting logs
- Replicating application instances
- Horizontal autoscaling
- Debugging applications
Kubernetes is often abbreviated as “k8s”, with 8 replacing the number of characters between k and s.
Kubernetes is the ancient Greek word for “helmsman” (people who pilot cargo ships). Thus the nautical references and why k8s experts are called “captain” and why associated products have nautical themes, such as “helm”.
Kops for AWS (at https://github.com/kubernetes/kops) enables multi-master, multi-AZ cluster setup and management of multiple instance groups. See “How Qubit built its production ready Kubernetes (k8s) environments”
Options for deployment
There are several ways to obtain a running instance of Kubernetes, listed from easiest to most difficult:
A) “Kubernetes Engine” is a container management SaaS product running within the Google Compute Platform (GCP) on top of Google Compute Engine providing machines. It was previously called Google Container Engine (GKE).
But one can run k8s containers in other clouds or within private data centers.
B) Pull an image from Docker Hub within a Google Compute or AWS cloud instance.
CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.
Minikube goes beyond Docker For Mac (DFM) and Docker for Windows (DFW) and includes a node and a Master when it spins up a local environment.
CAUTION: At time of writing, https://github.com/kubernetes/minikube has 257 issues and 20 pending Pull Requests.
Install on a Mac:
brew install docker-machine-driver-xhyve
brew install minikube -y
BLAH: On Nov 9, 2017 I got an error message doing the above.
On Mac: minikube start –vm-driver=xhyve On Windows: minikube start –vm-driver=hyperv
kubectl CLI client install
Running outside Kubernetes servers is
kubectl (kube + ctl),
the CLI tool for k8s. It’s automatically installed within Google cloud instances.
Install on a Mac:
brew install kubectl -y
🍺 /usr/local/Cellar/kubernetes-cli/1.8.3: 108 files, 50.5MB
kubectl version --client
Kubectl 1.8 scale is now the preferred way to control graceful delete.
Kubectl 1.8 rollout and rollback now support stateful sets ???
Sample micro-service apps
I have built a GitHub repository containing example apps and utilities that process them at:
The remainder of this tutorial refers to files in this repository.
The repo is based on work from others, especially Kelsy Hightower, the Google Developer Advocate.
- https://github.com/kelseyhightower/app - an example 12-Factor application.
- https://hub.docker.com/r/kelseyhightower/monolith - Monolith includes auth and hello services.
- https://hub.docker.com/r/kelseyhightower/auth - Auth microservice. Generates JWT tokens for authenticated users.
- https://hub.docker.com/r/kelseyhightower/hello - Hello microservice. Greets authenticated users.
- https://hub.docker.com/r/ngnix - Frontend to the auth and hello services.
These sample apps are manipulated by https://github.com/kelseyhightower/craft-kubernetes-workshop
- Create a Node.js server
- Create a Docker container image
- Create a container cluster
- Create a Kubernetes pod
Scale up your services
- Provision a complete Kubernetes cluster using Kubernetes Engine.
- Deploy and manage Docker containers using kubectl.
- Break an application into microservices using Kubernetes’ Deployments and Services.
### Infrastructure as code
Use an internet browser to view
The script downloads a repository forked from googlecodelabs: https://github.com/wilsonmar/orchestrate-with-kubernetes/tree/master/kubernetes
This repository contains several kinds of .yaml files, which can also have the extension .yml. Kubernetes also recognizes .json files, but YAML files are easier to work with.
The files are call “Manifests” because they declare the desired state.
Open an internet browser tab to view it.
reverse proxy to front-end
The web service consists of a front-end and a proxy served by the NGINX web server configured using two files in the
These are explained in detail at https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-as-a-web-server-and-reverse-proxy-for-apache-on-one-ubuntu-14-04-droplet
SSL keys referenced are installed from the
- ca-key.pem - Certificate Authority’s private key
- ca.pem - Certificate Authority’s public key
- cert.pem - public key
- key.pem - private key
- healthy-monolith.yaml configures “livenessProbe” (in folder healthz) and “readinessProbe” (in folder readiness) on port 81
kubectl config current-context
How Kubernetes Engine works
Pods are defined by a manifest file read by the apiserver which deploys nodes.
Nodes were previously called “minions” who do work. Apparently Google namers forgot about the existance of NodeJS.
Pods abstact the network and storage of containers. So each pod can hold one or more containers, all of which share the same IP address, hostname, namespaces, and other resources.
Pods go into “succeeded” state after being run because pods have short lifespans – deleted and recreated as necessary.
Each container has a different port. Containers within the same pod are on the same localhost.
Pods are replicated across several Compute Engine nodes.
The replication controller automatically adds or removes pods to comply with the specified number of pod replicas declared are running across nodes. This makes GKE “self healing” to provide high availability and reliability with “autoscaling” up and down based on demand.
In this diagram:
From the https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
List all pods, including in the system namespace:
kubectl get nodes --all-namespaces
Included are pods named:
- api server (aka “master”) accepts kubectl commands
- etcd (cluster store) for HA (High Availability) in control pane
- controller to watch for changes and maintain desired state
- dns (domain name server)
- proxy load balances across all pods in a service
- scheduler watches api server for new pods to assign work to new pods
List app pods in the default namespace:
kubectl get nodes
For more details:
kubectl describe pods
apiVersion: v1 kind: Pod metadata: name: cadvisor spec: containers: - name: cadvisor image: google/cadvisor:v0.22.0 volumeMounts: - name: rootfs mountPath: /rootfs readOnly: true - name: var-run mountPath: /var/run readOnly: false - name: sys mountPath: /sys readOnly: true - name: docker mountPath: /var/lib/docker readOnly: true ports: - name: http containerPort: 8080 protocol: TCP args: - --profiling - --housekeeping_interval=1s volumes: - name: rootfs hostPath: path: / - name: var-run hostPath: path: /var/run - name: sys hostPath: path: /sys - name: docker hostPath: path: /var/lib/docker
rc.yml (Replication Controller) defines the number of replicas and
apiVersion: v1 kind: ReplicationController metadata: name: cadvisor spec: replicas: 5 selector: app hello template: metadata: labels: app: hello-world spec: containers: - name: hello image: account/image:latest ports: containerPort: 8080
kubectl apply -f rc.yml
The response expected:
replicationcontroller "hello" configured
List, in wide format, the number of replicated nodes:
kubectl get rc -o wide
DESIRED, CURRENT, READY
Get more detail:
kubectl describe rc
svc.yml defines the services:
apiVersion: v1 kind: Service metadata: name: hello-svc labels: app: hello-world spec: type: NodePort ports: - port: 8080 protocol: TCP selector: app: hello-world
PROTIP: The selector should match the pods.xml.
kubectl create -f svc.yml
The response expected:
service "hello-svc" created
kubectl get svc
kubectl describe svc hello-svc
List end points addresses:
kubectl describe ep hello-svc
deploy.yml defines the deploy:
apiVersion: v1beta kind: Deployment
Deployment wraps around replica sets, a newer version of doing rolling-update on Replication Controller. Old replica sets can revert roll-back by just changing the deploy.yml file.
Kubelets instantiate pods – each a set of containers running under a single IP address, the fundamental units nodes.
A Kubelet agent program is installed on each server to watch the apiserver and register each node with the cluster.
PROTIP: Use a DaemonSet when running clustered Kubernetes with static pods to run a pod on every node. Static pods are managed directly by the kubelet daemon on a specific node, without the API server observing it.
A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.
Deleting a DaemonSet will clean up the Pods it created. Some typical uses of a DaemonSet are:
- running a cluster storage daemon, such as glusterd, ceph, on each node.
- running a logs collection daemon on every node, such as fluentd or logstash.
- running a node monitoring daemon on every node, such as Prometheus Node Exporter, collectd, Datadog agent, New Relic agent, or Ganglia gmond.
Start kubelet daemon:
kubelet --pod-manifest-path=the directory
This periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there.
Note: Kubelet ignores files starting with a dot when scanning the specified directory.
PROTIP: By default, Kubelets exposes endpoints on port 10255.
Containers can be Docker or rkt (pluggable)
/spec, /healthz reports status.
The container engine pulls images and stopping/starting containers.
etcd provides a reliable way to notify the rest of the cluster about configuration changes for shared configuration and service discovery.
Data stored by Kubernetes in etcd includes jobs being scheduled, created and deployed, pod/service details and state, namespaces and replication information, etc.
etcd provides a REST API for CRUD operations as well as an interface to register watchers on specific nodes.
etcd is a simple, distributed, consistent key-value store.
The Controller Network Interface is installed using basic cbr0 using the bridge and host-local CNI plugins.
The CNI plugin is selected by passing Kubelet the command-line option:
Nigel Poulton (@NigelPoulton, nigelpoulton.com)
Getting Started with Kubernetes 22 May 2017
Book “Docker for Sysadmins”.
Make your own K8s
Kelsey Hightower, in https://github.com/kelseyhightower/kubernetes-the-hard-way, shows the steps of how to create Compute Engine yourself:
- Cloud infrastructure firewall and load balancer provisioning
- setup a CA and TLS cert gen.
- setup TLS client bootstrap and RBAC authentication
- bootstrap a HA etcd cluster
- bootstrap a HA Kubernetes Control Pane
- Bootstrap Kubernetes Workers
- Config K8 client for remote access
- Manage container network routes
- Deploy clustesr DNS add-on
http://post.oreilly.com/rd/9z1z07qqefah7igu9ftil9os3s666asu5a6lfhh1eko Kubernetes adventures on Azure, Part 1 (Linux cluster) Having read several books on Kubernetes, Ivan Fioravanti, writing for Hackernoon, says it’s time to start adventuring in the magical world of Kubernetes for real! And he does so using Microsoft Azure. Enjoy the step-by-step account of his escapade (part 1).
https://www.pelo.tech/blog/running-kubernetes Kubernetes with Gcloud and Terraform by Adron Hall April 5, 2017
http://blog.adron.me/articles/setting-up-gcp-container-cluster/ Setting up a GCP Container Cluster - Part I by Adron Hall and published on January, 31 of 2017.
For GKE we disable all legacy authentication, enable RBAC (Role Based Access Control), and enable IAM authentication.