Wilson Mar bio photo

Wilson Mar

Hello. Join me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

Container engine


This page is being actively worked on, so please pardon the “dust”.

This is a tutorial presented as a hands-on workshop with commentary along the way.

Open Sourced

Kubernetes was created inside Google (using the Golang programming language) and used for over a decade before being open-sourced in 2014 to the CNCF (Cloud Native Computing Foundation).

The word “Kubernetes” is a registered trademark of the Linux Foundation, which maintains the website https://kubernetes.io and source code at


  • v1.0 was committed on July 2015 within GitHub
  • v1.6 was led by a CoreOS developer.
  • v1.7 was led by Google.
  • v1.8 is led by a Microsoft employee (Jaice Singer DuMars) after Microsoft joined the CNCF July 2017.

kubernetes-logo-125x134-15499.png This blog and podcast about origins note that the Kubernetes logo has 7 sides because its initial developers were Star Trek fans: The predecessor to Kub was called Borg. A key Borg character is called “7 of 9”.

Anyway, its Google heritage means Kubernetes is about scaling for a lot of traffic with redundancies to achieve high availability (HA).

OpenShift support

If you want to pay for Kubernetes, https://www.redhat.com/en/technologies/cloud-computing/openshift Red Hat® OpenShift is a container application platform that brings Docker and Kubernetes to the enterprise.


  • Twitter: @kubernetesio
  • https://slack.k8s.io
  • Google+ Group: Kubernetes
  • https://groups.google.com/forum/#!forum/kubernetes-dev for contributors to the Kubernetes project to discuss design and implementation issues.

  • https://stackoverflow.com/search?q=k8s+or+kubernetes for developers
  • https://serverfault.com/search?q=k8s+or+kubernetes for sysadmins.
  • https://groups.google.com/forum/#!forum/kubernetes-sig-scale
  • https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&disable_polymer=true Kubernetes Google Community video chats

  • https://cloud.google.com/support/docs/issue-trackers to report bugs


  • #KubeCon


See https://kubernetes.io/docs/setup/pick-right-solution/

Other orchestration systems for Docker containers:

  • Docker Swarm

  • Mesos from Apache, which runs other containers in addition to Docker. K8SM is a Mesos Framework developed for Apache Mesos to use Google’s Kubernetes. Installation.

https://codefresh.io/kubernetes-guides/kubernetes-cloud-aws-vs-gcp-vs-azure/ Kubernetes in the Cloud: AWS vs. GCP vs. Azure

Amazon Elastic Container Service for Kubernetes (Amazon EKS) was introduced December 2017 to work with AWS Elastic Load Balancing, IAM authentication, Amazon VPC isolation, AWS PrivateLink access, and AWS CloudTrail logging. EKS runs three Kubernetes masters across three Availability Zones in order to ensure high availability. EKS automatically provides a “highly available and scalable Kubernetes service” EKS automatically detects and replaces unhealthy masters, and provides automated version upgrades and patching for the masters. so you don’t have to choose appropriate instance types.

Kubernetes vernacular cartoon

Kubernetes automates the deployment (creation) of Dockerized apps running as containers within pods arranged in clusters of nodes (previously called minions).


Click on the diagram for an animation of how the various aspects of Kubernetes fit together: k8s-arch-ruo91-797x451-104467

Kubernetes is called an “orchestrator” of various micro-service apps built from Docker images in a binary repository (such as Nexus or Artifactory). The images are built by Docker reading Dockerfiles in a configuration source repository such as GitHub.

BTW, instead of Docker, Kubernetes also works with rkt (pronounced “rocket”). But this tutorial focuses on Docker.

Deployments manage Pods.

Every Pod has a unique IP. There is one IP Address per Pod. In other words, containers within a Pod share a network namespace.

Every container has its own unique port number within its pod’s IP.

Containers also share attached data volumes available within each Pod.

Kubelet agents

System administrators control the Master node UI in the cloud or write scripts that invoke kubectl command-line client program that controls the Kubernetes Master node.

The program talks to the Kubernetes API Server.

The Server gets Scheduler ???

The Server obtains from Controller Manager ???

The API Server and Scheduler stores data in an ETCD Cluster.

Kubernetes instantiates and then manages the state of containers.

Kubernestes can move

The Kube Proxy communicates only with Pod admin. whereas Kubelets communicate with individual pods as well.

HAProxy VRRP (Virtual Router Redundancy Protocol) http://searchnetworking.techtarget.com/definition/VRRP automatically assigns available Internet Protocol routers to participating hosts.

  1. communicate
  2. over
  3. Master
  4. ETCD


cAdvisor https://github.com/google/cadvisor (Container Advisor) collects, aggregates, processes, and exports information about running containers in order to provide container admins an understanding of the resource usage and performance characteristics of their running containers.


Yongbok Kim (who writes in Korean) posted (on Jan 24, 2016) a master map of how all the pieces relate to each other:
Click on the diagram to pop-up a full-sized diagram: k8s_details-ruo91-350x448.jpg

https://github.com/coreos/flannel flannel configures a IPv4 “layer 3” network fabric designed for Kubernetes. between multiple nodes in a cluster.

CoreOS’s Tectonic sets up flannel in the Kubernetes clusters it creates using the open source Tectonic Installer to drive the setup process.

flannel is also widely used outside of kubernetes. When deployed outside of kubernetes, etcd is always used as the datastore.

  • Infrastructure as code
  • Manage containers
  • Naming and discovery
  • Mounting storage systems
  • Balancing loads
  • Rolling updates
  • Distributing secrets/config
  • Checking application health
  • Monitoring resources
  • Accessing and ingesting logs
  • Replicating application instances
  • Horizontal autoscaling
  • Debugging applications

Terminology background

Kubernetes is often abbreviated as “k8s”, with 8 replacing the number of characters between k and s.

Kubernetes is the ancient Greek word for “helmsman” (people who pilot cargo ships). Thus the nautical references and why k8s experts are called “captain” and why associated products have nautical themes, such as “helm”.

Kops for AWS (at https://github.com/kubernetes/kops) enables multi-master, multi-AZ cluster setup and management of multiple instance groups. See “How Qubit built its production ready Kubernetes (k8s) environments”

Options for deployment

There are several ways to obtain a running instance of Kubernetes, listed from easiest to most difficult:

A) “Kubernetes Engine” is a container management SaaS product running within the Google Compute Platform (GCP) on top of Google Compute Engine providing machines. It was previously called Google Container Engine (GKE).

But one can run k8s containers in other clouds or within private data centers.

A) Minikube spins up a local environment on your laptop.

B) Pull an image from Docker Hub within a Google Compute or AWS cloud instance.

C) Use a Dockerfile to build your own Docker image containing Kubernetes.

D) Use Homebrew to install Kubernetes natively on you Mac.

E) Download installer to install locally.

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.


Minikube goes beyond Docker For Mac (DFM) and Docker for Windows (DFW) and includes a node and a Master when it spins up a local environment.

CAUTION: At time of writing, https://github.com/kubernetes/minikube has 257 issues and 20 pending Pull Requests.

  1. Install on a Mac:

    brew install docker-machine-driver-xhyve
    brew install minikube -y

    BLAH: On Nov 9, 2017 I got an error message doing the above.

minikube version

On Mac: minikube start –vm-driver=xhyve On Windows: minikube start –vm-driver=hyperv

minikube dashboard

minikube stop

minikube delete

kubectl CLI client install

Running outside Kubernetes servers is kubectl (kube + ctl), the CLI tool for k8s. It’s automatically installed within Google cloud instances.

  1. Install on a Mac:

    brew install kubectl -y
    🍺  /usr/local/Cellar/kubernetes-cli/1.8.3: 108 files, 50.5MB
  2. Verify

    kubectl version --client

Kubectl 1.8 scale is now the preferred way to control graceful delete.

Kubectl 1.8 rollout and rollback now support stateful sets ???

Sample micro-service apps

I have built a GitHub repository containing example apps and utilities that process them at:


The remainder of this tutorial refers to files in this repository.

The repo is based on work from others, especially Kelsy Hightower, the Google Developer Advocate.

  • https://github.com/kelseyhightower/app - an example 12-Factor application.
  • https://hub.docker.com/r/kelseyhightower/monolith - Monolith includes auth and hello services.
  • https://hub.docker.com/r/kelseyhightower/auth - Auth microservice. Generates JWT tokens for authenticated users.
  • https://hub.docker.com/r/kelseyhightower/hello - Hello microservice. Greets authenticated users.
  • https://hub.docker.com/r/ngnix - Frontend to the auth and hello services.

These sample apps are manipulated by https://github.com/kelseyhightower/craft-kubernetes-workshop

  1. Install
  2. Create a Node.js server
  3. Create a Docker container image
  4. Create a container cluster
  5. Create a Kubernetes pod
  6. Scale up your services

  7. Provision a complete Kubernetes cluster using Kubernetes Engine.
  8. Deploy and manage Docker containers using kubectl.
  9. Break an application into microservices using Kubernetes’ Deployments and Services.

This “Kubernetes” folder contains scripts to implement what was described in the “Orchestrating the Cloud with Kubernetes” hands-on lab which is part of the “Kubernetes in the Google Cloud” quest.

### Infrastructure as code

  1. Use an internet browser to view


    The script downloads a repository forked from googlecodelabs: https://github.com/wilsonmar/orchestrate-with-kubernetes/tree/master/kubernetes


    This repository contains several kinds of .yaml files, which can also have the extension .yml. Kubernetes also recognizes .json files, but YAML files are easier to work with.

    The files are call “Manifests” because they declare the desired state.

  2. Open an internet browser tab to view it.

    reverse proxy to front-end

    The web service consists of a front-end and a proxy served by the NGINX web server configured using two files in the nginx folder:

    • frontend.conf
    • proxy.conf

    These are explained in detail at https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-as-a-web-server-and-reverse-proxy-for-apache-on-one-ubuntu-14-04-droplet

    SSL keys

    SSL keys referenced are installed from the tls folder:

    • ca-key.pem - Certificate Authority’s private key
    • ca.pem - Certificate Authority’s public key
    • cert.pem - public key
    • key.pem - private key


    • auth.yaml
    • frontend.yaml
    • hello-green.yaml
    • hello-canary.yaml
    • hello.yaml


    • healthy-monolith.yaml configures “livenessProbe” (in folder healthz) and “readinessProbe” (in folder readiness) on port 81
    • monolith.yaml
    • secure-monolith.yaml


    • auth.yaml
    • frontend.yaml
    • hello-blue.yaml
    • hello-green.yaml
    • hello.yaml
    • monolith.yaml


    Current context

    kubectl config current-context

How Kubernetes Engine works



Pods are defined by a manifest file read by the apiserver which deploys nodes.

Nodes were previously called “minions” who do work. Apparently Google namers forgot about the existance of NodeJS.

Pods abstact the network and storage of containers. So each pod can hold one or more containers, all of which share the same IP address, hostname, namespaces, and other resources.

Pods go into “succeeded” state after being run because pods have short lifespans – deleted and recreated as necessary.

Each container has a different port. Containers within the same pod are on the same localhost.

Pods are replicated across several Compute Engine nodes.

The replication controller automatically adds or removes pods to comply with the specified number of pod replicas declared are running across nodes. This makes GKE “self healing” to provide high availability and reliability with “autoscaling” up and down based on demand.

In this diagram:

From the https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/

  1. List all pods, including in the system namespace:

    kubectl get nodes --all-namespaces

    Included are pods named:

    • api server (aka “master”) accepts kubectl commands
    • etcd (cluster store) for HA (High Availability) in control pane
    • controller to watch for changes and maintain desired state
    • dns (domain name server)
    • proxy load balances across all pods in a service
    • scheduler watches api server for new pods to assign work to new pods

  2. List app pods in the default namespace:

    kubectl get nodes
  3. For more details:

    kubectl describe pods

pod.yml manifests

An example (cadvisor):

apiVersion: v1
kind: Pod
  name:   cadvisor
    - name: cadvisor
      image: google/cadvisor:v0.22.0
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: false
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: http
          containerPort: 8080
          protocol: TCP
        - --profiling
        - --housekeeping_interval=1s
    - name: rootfs
        path: /
    - name: var-run
        path: /var/run
    - name: sys
        path: /sys
    - name: docker
path: /var/lib/docker

Replication rc.yml

The rc.yml (Replication Controller) defines the number of replicas and

apiVersion: v1
kind: ReplicationController
  name:   cadvisor
  replicas: 5
     app hello
        app: hello-world
    - name: hello
      image: account/image:latest
        containerPort: 8080
  1. Apply replication:

    kubectl apply -f rc.yml

    The response expected:

    replicationcontroller "hello" configured
  2. List, in wide format, the number of replicated nodes:

    kubectl get rc -o wide
  3. Get more detail:

    kubectl describe rc

Service rc.yml

The svc.yml defines the services:

apiVersion: v1
kind: Service
  name: hello-svc
      app: hello-world
  type: NodePort
  - port: 8080
    protocol: TCP
    app: hello-world

PROTIP: The selector should match the pods.xml.

  1. Create services:

    kubectl create -f svc.yml

    The response expected:

    service "hello-svc" created
  2. List:

    kubectl get svc
  3. List details:

    kubectl describe svc hello-svc
  4. List end points addresses:

    kubectl describe ep hello-svc

Deploy yml

The deploy.yml defines the deploy:

apiVersion: v1beta
kind: Deployment

Deployment wraps around replica sets, a newer version of doing rolling-update on Replication Controller. Old replica sets can revert roll-back by just changing the deploy.yml file.

Kubelet Daemonset.yaml

Kubelets instantiate pods – each a set of containers running under a single IP address, the fundamental units nodes.

A Kubelet agent program is installed on each server to watch the apiserver and register each node with the cluster.

PROTIP: Use a DaemonSet when running clustered Kubernetes with static pods to run a pod on every node. Static pods are managed directly by the kubelet daemon on a specific node, without the API server observing it.

  • https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.

Deleting a DaemonSet will clean up the Pods it created. Some typical uses of a DaemonSet are:

  • running a cluster storage daemon, such as glusterd, ceph, on each node.
  • running a logs collection daemon on every node, such as fluentd or logstash.
  • running a node monitoring daemon on every node, such as Prometheus Node Exporter, collectd, Datadog agent, New Relic agent, or Ganglia gmond.
  1. Start kubelet daemon:

    kubelet --pod-manifest-path=the directory 

    This periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there.

    Note: Kubelet ignores files starting with a dot when scanning the specified directory.

    PROTIP: By default, Kubelets exposes endpoints on port 10255.

    Containers can be Docker or rkt (pluggable)

    /spec, /healthz reports status.

The container engine pulls images and stopping/starting containers.

  • https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

etcd storage

etcd provides a reliable way to notify the rest of the cluster about configuration changes for shared configuration and service discovery.

Data stored by Kubernetes in etcd includes jobs being scheduled, created and deployed, pod/service details and state, namespaces and replication information, etc.

etcd provides a REST API for CRUD operations as well as an interface to register watchers on specific nodes.

etcd is a simple, distributed, consistent key-value store.


The Controller Network Interface is installed using basic cbr0 using the bridge and host-local CNI plugins.


The CNI plugin is selected by passing Kubelet the command-line option:


Learning resources

Nigel Poulton (@NigelPoulton, nigelpoulton.com)

Make your own K8s

Kelsey Hightower, in https://github.com/kelseyhightower/kubernetes-the-hard-way, shows the steps of how to create Compute Engine yourself:

  • Cloud infrastructure firewall and load balancer provisioning
  • setup a CA and TLS cert gen.
  • setup TLS client bootstrap and RBAC authentication
  • bootstrap a HA etcd cluster
  • bootstrap a HA Kubernetes Control Pane
  • Bootstrap Kubernetes Workers
  • Config K8 client for remote access
  • Manage container network routes
  • Deploy clustesr DNS add-on

http://post.oreilly.com/rd/9z1z07qqefah7igu9ftil9os3s666asu5a6lfhh1eko Kubernetes adventures on Azure, Part 1 (Linux cluster) Having read several books on Kubernetes, Ivan Fioravanti, writing for Hackernoon, says it’s time to start adventuring in the magical world of Kubernetes for real! And he does so using Microsoft Azure. Enjoy the step-by-step account of his escapade (part 1).




This diagram k8s-services-flow-847x644-100409


https://www.pelo.tech/blog/running-kubernetes Kubernetes with Gcloud and Terraform by Adron Hall April 5, 2017

http://blog.adron.me/articles/setting-up-gcp-container-cluster/ Setting up a GCP Container Cluster - Part I by Adron Hall and published on January, 31 of 2017.


For GKE we disable all legacy authentication, enable RBAC (Role Based Access Control), and enable IAM authentication.