Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Instagram Youtube

Github Stackoverflow Pinterest

Container engine for every cloud


This is a hands-on introduction with insightful commentary carefully sequenced to make complex material easier to understand quickly. This is not a “demo”, but an immersive step-by-step “deep dive” tutorial aimed to make you productive.

Why Kubernetes?

Kubernetes is called “container orchestration” software because it automates the deployment, scaling and management of containerized applications*.

“Containerized” microservice apps are dockerized into images pulled from DockerHub or private security-vetted images in Docker Enterprise, Quay.io, or an organization’s own binary repository setup using Nexus or Artifactory. Kubernetes also works with rkt (pronounced “rocket”) containers. But this tutorial focuses on Docker.

Each Kubernetes node has a different IP address.


Kubernetes automates resilience into containers by abstacting the network and storage of virtual containers in replaceable “pods”. Each pod can hold one or more Docker container hosts.

Within a pod, each container has a different port number.

Containers within the same pod share the same IP address, hostname, Linux namespaces, cgroups, storage, and other resources.

Kubernetes replicates Pods (the same set of containers in each) across several worker nodes (VM or physical machines).


A private ClusterIP is accessible by nodes only within the same cluster.

Services listen on the same nodePort (TCP 30000 - 32767 defined by --service-node-port-range).


The diagram above is referenced throughout this tutorial, particularly in the Details section below. It is by Yongbok Kim who presents animations on his website.

Communications with outside service network callers occur through a single Virtual IP address (VIP) going through a kube-proxy pod within each node. The Kube-proxy load balances traffic to deployments, which are load-balanced sets of pods within each node. Kube-proxy IPVS Mode is native to the Linux kernel. CBR0 (Custom Bridge zero) forwards the eth0, which rewrites the destination IP to a pod behind the Service3:18 into chapter 6 Big Picture

PROTIP: Kubernetes recently added auto-scaling based on metrics API measurement of demand. Before that, Kubernetes manages the instantiating, starting, stopping, updating, and deleting of a pre-defined number of pod replicas based on declarations in *.yaml files or interactive commands.

The number of pods replicated is based on deployment yaml files. Service yaml files specify what ports are used in deployments.

This Architectural Diagram pdf:  k8s-linuxacademy-arch-912x415-32433.jpg is described in the Linux Academy’s CKA course of 05:34:43 hours of videos by Chad Miller (@OpenChad).

Open Sourced = Free

kubernetes-logo-125x134-15499.png Kubernetes was created inside Google (using the Golang programming language). Logos for each Google service are 6 sided hexagons. This blog and podcast note that the Kubernetes logo has 7 sides because its initial developers were Star Trek fans: The predecessor to Kubernetes was called the Borg. A key Borg character is called “7 of 9”.

Kubernetes was used inside Google for over a decade before being open-sourced in 2014 to the Cloud Native Computing Foundation (cncf.io) collective.

Kubernetes is often abbreviated as “k8s”, with 8 replacing the number of characters between k and s. Thus, https://k8s.io redirects you to https://kubernetes.io, the home page for the software. PROTIP: The word “Kubernetes” is a registered trademark of the Linux Foundation, which maintains the website https://kubernetes.io and source code at https://github.com/kubernetes/kubernetes

  • v1.0 was committed on July 2015 within GitHub
  • v1.6 was led by a CoreOS developer.
  • v1.7 was led by a Googler.
  • v1.8 is led by a Microsoft employee (Jaice Singer DuMars) after Microsoft joined the CNCF July 2017.

Its Google heritage means Kubernetes is about scaling for a lot of traffic with redundancies to achieve high availability (HA).

Certification in Kubernetes

On November 8, 2016 CNCF announced their 3-hour task-based Certified Kubernetes Administrator (CKA) and 2-hour Kubernetes Application Developer (CKAD) exams. Each is $300, which includes one free retake. To compare the domain focus for each exam:

19% Core Concepts
12% Installation, Configuration & Validation
12% Security
11% Networking
11% Cluster Maintenance
10% Troubleshooting
08% Application Lifecycle Management
07% Storage
05% Scheduling
05% Logging / Monitoring
20% Pod Design
18% Configuration
18% Observability
13% Services & Networking
13% Core Concepts
10% Multi-Container Pods
08% State Persistence

CNCF is part of the Linux Foundation, so…

  1. Get an account (Linux Foundation credentials ) at https://identity.linuxfoundation.org.

    It’s a non-profit organization, thus the “.org”.

  2. Login to https://linuxfoundation.org and join as a member for a $100 discount toward certifications.
  3. Go to https://training.linuxfoundation.org/linux-courses/system-administration-training/kubernetes-fundamentals and pay for the $300 exam or for $199 more take their class.

  4. Use your Linux Foundation credentials to sign-in at examslocal.com, and select either or both of two exams from CNCF:

    • Linux Foundation : Certified Kubernetes Administrator (CKA) - English
    • Linux Foundation : Certified Kubernetes Application Developer (CKAD) - English

  5. Click “Or Sign In With” tab and select “Sign in for exams powered by the Linux Foundation”.
  6. Log in using your preferred account.
  7. Click “Handbook link” to download it.
  8. Select the date, then click OK.
  9. Setup your home computer to take the exam at home using the Chrome extension from “Innovative Exams”, which uses your laptop camera and microphone watching you use a virtual Ubuntu machine.
  10. Take the 180 minute (2 hour) exam.

No multiple choice questions.

PROTIP: The Linux Foundation exam focuses on “pure” commands only and excludes add-ons such as OpenStack.

“How I passed the CKA (Certified Kubernetes Administrator) Exam” by Krystian Nowaczyk (who maintains https://github.com/ramitsurana/awesome-kubernetes) provides a list of resources he used.

Free Playpen

play-with-k8s.com gives you a 4-hour playpen.

If you want to pay for Kubernetes support, Red Hat® OpenShift, at https://www.redhat.com/en/technologies/cloud-computing/openshift, enables Docker and Kubernetes for the enterprise by adding external host names and role-based security.

One can run k8s containers in other clouds or within private data centers using OpenStack from RedHat.

Being open-source has enabled Kubernetes to flourish on several clouds*



Other orchestration systems for Docker containers:

  • Docker Swarm incorporated Rancher from Rancher Labs.

  • Mesosphere DC/OS (Data Center Operating System) runs Apache Mesos to abstract CPU, memory, storage to provide an API to program a multi-cloud multi-tenant data center (at Twitter, Yelp, Ebay, Azure, Apple, etc.) as if it’s a single pool of resources. Kubernetes can run on top of it, but the DC/OS has premium (licensed) enterprise features. So it’s not for you if you never want to pay for anything.

    Mesos from Apache, which runs other containers in addition to Docker. K8SM is a Mesos Framework developed for Apache Mesos to use Google’s Kubernetes. Installation.

    See Container Orchestration Wars (2017) at the Velocity Conf 19 Jun 2017 by Karl Isenberg (@karlfi) of Mesosphere

  • Hashicorp Nomad.


A Kublet agent program is automatically installed when a node is created. Each kubelet is called the “control pane” that runs nodes under its control.

Kublet constantly compares the status of pods against what is declared in yaml files, and starts or deletes pods as necessary to meet the request.

Restarting Kublet itself depends on the operating system (Monit on Debian or systemctl on systemd-based systems).

Master node

Nodes are joined to the master node using the kubeadm join program and command.

The master node itself is crated by the kubeadm init command which establishes folders and invokes the Kubernetes API server. That command is installed along with the kubectl package. There is a command with the same name used to obtain the version. The kubectl get nodes command lists basic information about each node. The describe command provides more detailed information.

API Server

The kubectl client communicates using REST API calls to an API Server which handles authentication and authorization.

kubectl get apiservices

API’s were initially monolithic but has since been split up into:

  • core “” to handle pod & svc & ep (endpoint)
  • apps to handle deploy, sts, ds
  • authorization to handle role, rb
  • storage to handle pv (persistent volume) and pvc, sc (storage classes)


The API Server puts nodes in “pending” state when it sends requests to bring them up and down to the Scheduler to do so only when there are enough resources available. The scheduler can operate according to a schedule. But whether it does or not are defined in rules obayed by the Scheduler about nodes called “Taints”. Rules obayed by the Scheduler about pods are called “Tolerances”. Such details are reaveled using the kubectl describe nodes command.

etcd storage

The API Server and Scheduler persists their configuration and status information in a ETCD cluster (from CoreOS).

Kubernetes data stored in etcd includes jobs being scheduled, created and deployed, pod/service details and state, namespaces, and replication details.

It’s called a cluster because, for resiliancy, etcd replicates data across nodes. This is why there is a minimum of two worker nodes per cluster. ???


The Node controller assigns a CIDR block to newly registered nodes, then continually monitors node health. When necessary, it taints unhealthy nodes and gracefully evicts unhealthy pods. The default timeout is 40 seconds.

Load balancing among nodes (hosts within a cloud) are handled by third-party port forwarding via Ingress controllers. See Ingress definitions.

An “Ingress” is a collection of rules that allow inbound connections to reach the cluster services.

In Kubernetes the Ingress Controller could be a NGINX container providing reverse proxy capabilities, and the Ingress Resource defines the connection rules.


OpenShift’s Router is instead a HAProxy container (taking the place of NGINX).

This diagram illustrates what OpenShift adds: kubernetes-openshift-502x375-107638

“The primary grouping concept in Kubernetes is the namespace. Namespaces are also a way to divide cluster resources between multiple uses. That being said, there is no security between namespaces in Kubernetes; if you are a “user” in a Kubernetes cluster, you can see all the different namespaces and the resources defined in them.” – from the book: OpenShift for Developers, A Guide for Impatient Beginners by Grant Shipley and Graham Dumpleton.


Projects in OpenShift provide “walls” between namespaces, ensuring that users or applications can only see and access what they are allowed to. OpenShift projects wrap a namespace by adding security annotations which control access to that namespace. Access is controlled through an authentication and authorization model based on users and groups.

Plug-in Network

PROTIP: Kubernetes uses third-party services to handle load balancing and port forwarding through ingress objects managed by an ingress controller.

CNI (Container Network Interface)

Other CNI vendors include Calico, Cilium, Contiv, Weavenet.

HA Proxy cluster

For network resiliency, HA Proxy cluster distributes traffic among nodes.


To collect resource usage and performance characteristics of running containers, many install a pod containing Google’s Container Advisor (cAdvisor). It aggregates and exports telemetry to an InfluxDB database for visualization using Grafana.

Google’s Heapster is also be used to send metrics to Google’s cloud monitoring console.

Helm charts

The name Kubernetes is the ancient Greek word for people who pilot cargo ships – “helmsman” in English. Thus the nautical references and why Kubernetes experts are called “captain” and why associated products have nautical themes, such as “Helm”.

A Helm chart can be used to quickly create an OpenFaaS (Serverless) cluster on your laptop.

git clone https://github.com/openfaas/faas-netes && cd faas-netes
   kubectl apply -f ./namespaces.yml 
   kubectl apply -f ./yaml_armhf

Deploy a scalable web application to Kubernetes using Helm


  • Infrastructure as code
  • Manage containers
  • Naming and discovery
  • Mounting storage systems
  • Balancing loads
  • Rolling updates
  • Distributing secrets/config
  • Checking application health
  • Monitoring resources
  • Accessing and ingesting logs
  • Replicating application instances
  • Horizontal autoscaling
  • Debugging applications

Containers are declared by yaml such as this to run Alphine Linux Docker container:

apiVersion: v1
kind: Pod
  name: alpine
  namespace: default
  - name: alpine
    image: alpine
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

Kubernetes is written in the Go language, so it can run on Windows, Linux, and MacOS (the need to install a JVM).

Raspberry Pi

Read how the legendary Scott Hanselman built Kubernetes on 6 Raspberry Pi nodes, each with a 32GB SD card to a 1GB RAM ARM chip (like on smartphones).

Hansel talked with Alex Ellis (@alexellisuk) keeps his instructions with shell file updated for running on the Pis to install OpenFaaS.

CNCF Ambassador Chris Short developed the rak8s (pronounced rackets) library to make use of Ansible.


  • https://blog.hypriot.com/getting-started-with-docker-on-your-arm-device/
  • https://blog.sicara.com/build-own-cloud-kubernetes-raspberry-pi-9e5a98741b49

Architecture diagram

Yongbok Kim (who writes in Korean) posted (on Jan 24, 2016) a master map of how all the pieces relate to each other:
Click on the diagram to pop-up a full-sized diagram: k8s_details-ruo91-350x448.jpg

BTW What are now called “nodes” were previously called minions. Apparently Google namers forgot about the existance of NodeJs, which refers to nodes differently.


End-to-end tests by those who develop Kubernetes are coded in Ginko and Gomega (because Kubernets is written in Go).

The Kubtest suite builds, stages, extracts, and brings up the cluster. After testing, it dumps logs and tears down the test rig.


  • Twitter: @kubernetesio
  • https://slack.k8s.io
  • Google+ Group: Kubernetes
  • https://groups.google.com/forum/#!forum/kubernetes-announce for announcements
  • https://groups.google.com/forum/#!forum/kubernetes-dev for contributors to the Kubernetes project to discuss design and implementation issues.
  • https://stackoverflow.com/search?q=k8s+or+kubernetes for developers
  • https://serverfault.com/search?q=k8s+or+kubernetes for sysadmins.
  • https://groups.google.com/forum/#!forum/kubernetes-sig-scale
  • https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&disable_polymer=true Kubernetes Google Community video chats

  • https://cloud.google.com/support/docs/issue-trackers to report bugs

  • KubeCon.io Conferences (#KubeConio)

Installation options

There are several ways to obtain a running instance of Kubernetes.

Rancher is a deployment tool for Kubernetes that also provides networking and load balancing support. Rancher initially created it’s own framework (called Cattle) to coordinate Docker containers across multiple hosts, at a time when Docker was limited to running on a single host. Now Rancher’s networking provides a consistent solution across a variety of platforms, especially on bare metal or standard (non cloud) virtual servers. In addition to Kubernetes, Rancher enables users to deploy a choice of Cattle, Docker Swarm, Apache Mesos upstream project for DCOS (Data Center Operating System). Rancher eventually become part of Docker Swarm.

Within https://github.com/kubernetes/kops KOPS

Minikube offline

B) Minikube spins up a local environment on your laptop.

NOTE: Ubuntu on LXD offers a 9-instance Kubernetes cluster on localhost.

PROTIP: CAUTION your laptop going to sleep may ruin minikube.

Server install

C) install Kubernetes natively on CentOS.

D) Pull an image from Docker Hub within a Google Compute or AWS cloud instance.

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.

See https://kubernetes.io/docs/setup/pick-right-solution


Minikube goes beyond Docker For Mac (DFM) and Docker for Windows (DFW) and includes a node and a Master when it spins up in a local environment (such as your laptop).

CAUTION: At time of writing, https://github.com/kubernetes/minikube has 257 issues and 20 pending Pull Requests.

  1. Install on a Mac Docker:

    brew install docker-machine-driver-xhyve
  2. Install on a Mac Minikube:

    brew install minikube -y
  3. Verify if its command works by getting the version:

    minikube version
  4. Show the current context:

    kubectl config current-context

    The response on minikube is “minikube”.

  5. Start the service:

    On Mac:

    minikube start --vm-driver=xhyve

    On Windows:

    minikube start --vm-driver=hyperv
  6. Dashboard

    minikube dashboard
  7. Stop the service:

    minikube stop
  8. Recover space:

    minikube delete

    Kubectl 1.8 scale is now the preferred way to control graceful delete.

    Kubectl 1.8 rollout and rollback now support stateful sets ???

    kubectl CLI client install

    Kubernetes administrators use the kubectl (kube + ctl) the CLI tool running outside Kubernetes servers to control them. It’s automatically installed within Google cloud instances, but on Macs clients:

  9. Install on a Mac:

    brew install kubectl -y
    🍺  /usr/local/Cellar/kubernetes-cli/1.8.3: 108 files, 50.5MB
  10. Verify

    kubectl version --client

    A sample response:

    Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
    1. Check the status of the job using the kubectl describe command.

    2. When a job is complete, view its results:

    kubectl logs counter

    The API Server routes several kinds of yaml declaration files: Pod, Deployment of pods, Service, Job, Configmap.

    API primatives ???



cat < /etc/yum.repos.d/kubernetes.repo
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg


cat <  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1


  1. On Ubuntu, install:

    apt install -y docker.io
  2. To make sure Docker and Kublet are using the same systemd driver:

    cat <<EOF >/etc/docker/daemon.json
      "exec-opts": ["native.cgroupdriver=systemd"]
  3. Install the keys:

    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
  4. sources:

    cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    <deb http://apt.kubernetes.io/ kubernetes-xenial main
  5. To download new sources:

    apt update
  6. To download the programs:

    apt install -y kubelet kubeadm kubectl


This section further explains the architecture diagram above.

This sequence of commands:

  1. Select “CloudNativeKubernetes” sandboxes.
  2. Select the first instance as the “Kube Master”.
  3. Login that server (user/123456).
  4. Change the password as prompted on the Ubuntu 16.04.3 server.

    Deploy Kubernetes master node

  5. Use this command to deploy the master node which controls the other nodes. So it’s deployed first which invokes the API Server

    sudo kubeadm init --pod-network-cidr=


    The address is the default for Flannel.

    Flow diagram


    The diagram above is by Walter Liu

    Flannel for Minikube

    When using Minikube locally, a CNI () is needed. So setup Flannel from CoreOS using the open source Tectonic Installer (@TectonicStack). It configures a IPv4 “layer 3” network fabric designed for Kubernetes.

    The response suggests several commands:

  6. Create your .kube folder:

    mkdir -p $HOME/.kube
  7. Copy in a configuration file:

    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  8. Give ownership of “501:20”:

    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  9. Make use of CNI:

    sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube.flannel.yml

    The response:

    clusterrole "flannel" created
    clusterrolebinding "flannel" created
    serviceaccount "flannel" created
    configmap "kube-flannel.cfg" created
    daemonset "kube-flannel.ds" created

    configmaps in cfg files are used to define environment variables.

  10. List pods created:

    kubectl get pods --all-namespaces -o wide

    Specifying wide output adds the IP address column

    Included are pods named:

    • api server (aka “master”) accepts kubectl commands
    • etcd (cluster store) for HA (High Availability) in control pane
    • controller to watch for changes and maintain desired state
    • dns (domain name server)
    • proxy load balances across all pods in a service
    • scheduler watches api server for new pods to assign work to new pods

    System administrators control the Master node UI in the cloud or write scripts that invoke kubectl command-line client program that controls the Kubernetes Master node.

    Proxy networking

    The Kube Proxy communicates only with Pod admin. whereas Kubelets communicate with individual pods as well.

    Each node has a Flannel and a proxy.

    The Server obtains from Controller Manager ???

  11. Switch to the webpage of servers to Login to the next server.
  12. Be root with sudo -i and provide the password.
  13. Join the node to the master by pasting in the command captured earlier, as root:

    kubeadm join --token ... --discovery-token-ca-cert-hash sha256:...

    Note the above is one long command. So you may need to use a text editor.

    Deployments manage Pods.

    Every Pod has a unique IP. There is one IP Address per Pod. In other words, containers within a Pod share a network namespace.

    Every container has its own unique port number within its pod’s IP.


  14. Switch to the webpage of servers to Login to the 3rd server.
  15. Again Join the node to the master by pasting in the command captured earlier:
  16. Get the list of nodes instantiated:

    kubectl get nodes
  17. To get list of events sorted by timestamp:

    kubectl get events --sort-by='.metadata.creationTimestamp'
  18. Create the initial log file so that Docker mounts a file instead of a directory:

    touch /var/log/kube-appserver.log
  19. Create in each node a folder:

    mkdir /srv/kubernetes
  20. Get a utility to generate TLS certificates:

    brew install easyrsa
  21. Run it:

    ./easyrsa init-pki

    Master IP address

  22. Run it:

    echo $MASTER_IP
  23. Run it:

    ./easyrsa --batch "--req-cn=${MASTER_IP}@`date +%s`* build-ca nopass


    To register watchers on specific nodes.??? Kubernetes supports TLS certifications for encryption over the line.

    REST API CRUD operations are used For authorization, Kubernetes supports Role Base Access Control (RBAC), (ABAC), and Webhook. Admission ???

  24. Put in that folder (in each node):

    • basic_auth.csv user and password
    • ca.crt - the certificate authority certificate from pki folder
    • known_tokens.csv kublets use to talk to the apiserver
    • kubecfg.crt - client cert public key
    • kubecfg.key - client cert private key
    • server.cert - server cert public key from issued folder
    • server.key - server cert private key

  25. Copy from API server to each master node:

    cp kube-apiserver.yaml  /etc/kubernetes/manifests/

    The kublet compares its contents to make it so, uses the manifests folder to create kube-apiserver instances.

  26. For details about each pod:

    kubectl describe pods


    Deploy service

  27. To deploy a service:

    kubectl expose deployment *deployment-name* [options]

Container Storage Interface (CSI)

### Volumes

Containers also share attached data volumes available within each Pod.

Kubelet agents

HAProxy VRRP (Virtual Router Redundancy Protocol) http://searchnetworking.techtarget.com/definition/VRRP automatically assigns available Internet Protocol routers to participating hosts.

A Persistent Volume (PV) is a provisioned block of storage for use by the cluster.

A Persistent Volume Claim (PVC) is a request for that storage by a user, and once granted, is used as a “claim check” for

Recycling policies are Retain (keep the contents) and Recycle (Scrub the contents).



  1. To drain a node out of service temporarily for maintenance:

    kubectl drain node3.mylabserver.com --ignore-daemonsets


    daemonsets (ds)

    Usually for system services or other pods that need to physically reside on every node in the cluster, such as for network services. They can also be deployed only to certain nodes using labels and node selectors.

  2. To return to service:

    kubectl uncordon node3.mylabserver.com

Sample micro-service apps

The repo is based on work from others, especially Kelsy Hightower, the Google Developer Advocate.

  • https://github.com/kelseyhightower/app - an example 12-Factor application.
  • https://hub.docker.com/r/kelseyhightower/monolith - Monolith includes auth and hello services.
  • https://hub.docker.com/r/kelseyhightower/auth - Auth microservice. Generates JWT tokens for authenticated users.
  • https://hub.docker.com/r/kelseyhightower/hello - Hello microservice. Greets authenticated users.
  • https://hub.docker.com/r/ngnix - Frontend to the auth and hello services.

These sample apps are manipulated by https://github.com/kelseyhightower/craft-kubernetes-workshop

  1. Install
  2. Create a Node.js server
  3. Create a Docker container image
  4. Create a container cluster
  5. Create a Kubernetes pod
  6. Scale up your services

  7. Provision a complete Kubernetes cluster using Kubernetes Engine.
  8. Deploy and manage Docker containers using kubectl.
  9. Break an application into microservices using Kubernetes’ Deployments and Services.

This “Kubernetes” folder contains scripts to implement what was described in the “Orchestrating the Cloud with Kubernetes” hands-on lab which is part of the “Kubernetes in the Google Cloud” quest.

Bob Reselman’s 3-day hands-on classes on Kubernetes makes use of bash scripts and sample app at https://github.com/reselbob/CoolWithKube

Infrastructure as code

  1. Use an internet browser to view


    The script downloads a repository forked from googlecodelabs: https://github.com/wilsonmar/orchestrate-with-kubernetes/tree/master/kubernetes


    This repository contains several kinds of .yaml files, which can also have the extension .yml. Kubernetes also recognizes .json files, but YAML files are easier to work with.

    The files are call “Manifests” because they declare the desired state.

  2. Open an internet browser tab to view it.

    reverse proxy to front-end

    The web service consists of a front-end and a proxy served by the NGINX web server configured using two files in the nginx folder:

    • frontend.conf
    • proxy.conf

    These are explained in detail at https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-as-a-web-server-and-reverse-proxy-for-apache-on-one-ubuntu-14-04-droplet

    SSL keys

    SSL keys referenced are installed from the tls folder:

    • ca-key.pem - Certificate Authority’s private key
    • ca.pem - Certificate Authority’s public key
    • cert.pem - public key
    • key.pem - private key

Kind yaml files

The kinds of yaml files:

### Deployments

  • auth.yaml
  • frontend.yaml
  • hello-green.yaml
  • hello-canary.yaml
  • hello.yaml

### pods

  • healthy-monolith.yaml configures “livenessProbe” (in folder healthz) and “readinessProbe” (in folder readiness) on port 81
  • monolith.yaml
  • secure-monolith.yaml

### services samples

  • auth.yaml
  • frontend.yaml
  • hello-blue.yaml
  • hello-green.yaml
  • hello.yaml
  • monolith.yaml


How Google Kubernetes Engine works



PROTIP: For GKE we disable all legacy authentication, enable RBAC (Role Based Access Control), and enable IAM authentication.

Pods are defined by a manifest file read by the apiserver which deploys nodes.

Pods go into “succeeded” state after being run because pods have short lifespans – deleted and recreated as necessary.

The replication controller automatically adds or removes pods to comply with the specified number of pod replicas declared are running across nodes. This makes GKE “self healing” to provide high availability and reliability with “autoscaling” up and down based on demand.

In this diagram:

From the https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/

  1. List all pods, including in the system namespace:

    kubectl get nodes --all-namespaces

pod.yml manifests

An example (cadvisor):

apiVersion: v1
kind: Pod
  name:   cadvisor
    - name: cadvisor
      image: google/cadvisor:v0.22.0
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: false
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: http
          containerPort: 8080
          protocol: TCP
        - --profiling
        - --housekeeping_interval=1s
    - name: rootfs
        path: /
    - name: var-run
        path: /var/run
    - name: sys
        path: /sys
    - name: docker
path: /var/lib/docker

Replication rc.yml

The rc.yml (Replication Controller) defines the number of replicas and

apiVersion: v1
kind: ReplicationController
  name: cadvisor
  replicas: 5
     app hello
        app: hello-world
    - name: hello
      image: account/image:latest
        containerPort: 8080
  1. Apply replication:

    kubectl apply -f rc.yml

    The response expected:

    replicationcontroller "hello" configured
  2. List, in wide format, the number of replicated nodes:

    kubectl get rc -o wide
  3. Get more detail:

    kubectl describe rc

Service rc.yml

The svc.yml defines the services:

apiVersion: v1
kind: Service
  name: hello-svc
      app: hello-world
  type: NodePort
  - port: 8080
    protocol: TCP
    app: hello-world

PROTIP: The selector should match the pods.xml.

One type of service is load balancer within a cloud:

apiVersion: v1
kind: Service
  name: la-lb-service
    app: la-lb
  - port: 80
    protocol: TCP
    targetPort: 9376
  type: LoadBalancer
  1. To create services:

    kubectl create -f svc.yml

    The response expected:

    service "hello-svc" created
  2. List:

    kubectl get svc
  3. List details:

    kubectl describe svc hello-svc
  4. List end points addresses:

    kubectl describe ep hello-svc

OpenShift routes to services

Services can be referenced by external clients using a host name such as “hello-svc.mycorp.com” by using OpenShift Enterprise, which uses “routes” that defines the rules the HAProxy applies to incoming connections.

Routes are deployed by an OpenShift Enterprise administrator as routers to nodes in an OpenShift Enterprise cluster. To clarify, the default Router in Openshift is an actual HAProxy container providing reverse proxy capabilities.

Deploy yml

The deploy.yml defines the deploy:

apiVersion: apps/v1beta2
kind: Deployment
  name: nginx-deployment
  replicas: 2
      app: nginx
        app: nginx
    - name: nginx
      image: nginx:1.7.9
      - containerPort: 80
        protocol: TCP
      net: gigabit

Deployment wraps around replica sets, a newer version of doing rolling-update on Replication Controller. Old replica sets can revert roll-back by just changing the deploy.yml file.

PROTIP: Don’t run apt-upgrade within containers, which breaks the image-container relationship controls.

  1. Retrieve the yaml for a deployment:

    kubectl get deployment nginx-deployment -o yaml

    Notice the “RollingUpdateStrategy: 25% max unavilable, 25% max surge”.

  2. Begin rollout of a new desired version from the command line:

    kubectl set image deployment/nginx-deployment nginx=nginx:1.8

    Alternately, edit the yaml file to nginx:1.9.1 and:

    kubectl apply -f nginx-deployment.yaml
  3. View Rollout a new desired version:

    kubectl rollout status deployment/nginx-deployment
  4. Describe the yaml for a deployment:

    kubectl describe deployment nginx-deployment

    kubectl get deployments 

    kubectl get deployments 
  7. List the history:

    kubectl rollout history deployment/nginx-deployment --revision=3
  8. Backout the revision:

    kubectl rollout undo deployment/nginx-deployment --to-revision=2

Security Context

The security.yml defines a secrurity context pod:

apiVersion: v1
kind: Pod
  name: security-context.pod
    runAsUser: 1000
    fsGroup: 2000
  - name: sam-vol
    emptyDir: {}
  - name: sample-container
    image: gcr.io/google-samples/node-hello:1.0
    - name: sam-vol
      mountPath: /data/demo
      allowPrivilegeEscalation: false
  1. Create the pod:

    kubectl create -f security.yaml

    This can take several minutes.

  2. Enter the security context:

    kubectl exec -it security-context-pod -- sh
  3. See the users:

    ps aux
  4. See that the group is “2000” as specified:

    cd /data && ls -al
  5. Exit the security context:

  6. Delete the security context:

    kubectl delete -f security.yaml

Kubelet Daemonset.yaml

Kubelets instantiate pods – each a set of containers running under a single IP address, the fundamental units nodes.

A Kubelet agent program is installed on each server to watch the apiserver and register each node with the cluster.

PROTIP: Use a DaemonSet when running clustered Kubernetes with static pods to run a pod on every node. Static pods are managed directly by the kubelet daemon on a specific node, without the API server observing it.

  • https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.

Deleting a DaemonSet will clean up the Pods it created. Some typical uses of a DaemonSet are:

  • running a cluster storage daemon, such as glusterd, ceph, on each node.
  • running a logs collection daemon on every node, such as fluentd or logstash.
  • running a node monitoring daemon on every node, such as Prometheus Node Exporter, collectd, Datadog agent, New Relic agent, or Ganglia gmond.
  1. Start kubelet daemon:

    kubelet --pod-manifest-path=the directory 

    This periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there.

    Note: Kubelet ignores files starting with a dot when scanning the specified directory.

    PROTIP: By default, Kubelets exposes endpoints on port 10255.

    Containers can be Docker or rkt (pluggable)

    /spec, /healthz reports status.

The container engine pulls images and stopping/starting containers.

  • https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

CNI Plugins

The Controller Network Interface (CNI) is installed using basic cbr0 using the bridge and host-local CNI plugins.

The CNI plugin is selected by passing Kubelet the command-line option:


See https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/

  • Flannel
  • Cisco ACI
  • Cilium
  • Contiv
  • Contrail
  • NSX-T
  • OpenVswitch
  • Project Calico
  • Weave Net

Learning resources

Nigel Poulton (@NigelPoulton, nigelpoulton.com), Docker Captain:

Make your own K8s

Kelsey Hightower, in https://github.com/kelseyhightower/kubernetes-the-hard-way, shows the steps of how to create Compute Engine yourself:

  • Cloud infrastructure firewall and load balancer provisioning
  • setup a CA and TLS cert gen.
  • setup TLS client bootstrap and RBAC authentication
  • bootstrap a HA etcd cluster
  • bootstrap a HA Kubernetes Control Pane
  • Bootstrap Kubernetes Workers
  • Config K8 client for remote access
  • Manage container network routes
  • Deploy clustesr DNS add-on

O’Reilly book Kubernetes adventures on Azure, Part 1 (Linux cluster) Having read several books on Kubernetes, Ivan Fioravanti, writing for Hackernoon, says it’s time to start adventuring in the magical world of Kubernetes for real! And he does so using Microsoft Azure. Enjoy the step-by-step account of his escapade (part 1).



The 8 labs covering 8 hours of the Kubernetes in the Google Cloud Qwiklab quest


https://github.com/kubeflow/kubeflow makes deployment of Kubernetes for Machine Learning (TensorFlow) using Kafka


by Adron Hall:

Julia Evans

  • https://jvns.ca/categories/kubernetes/













Kubernetes Dominates in IT Job Searches

Learning, Video and Live

Kubernetes for Beginners by Siraj Jan 8, 2019 [11:04]

Kubernetes Deconstructed Dec 15, 2017 [33:14] by Carson Anderson of DOMO (@carsonoid)

Solutions Engineering Hangout: Terraform for Instant K8s Clusters on AWS EKS by HashiCorp

Introduction to Microservices, Docker, and Kubernetes by James Quigley

Kubernetes in Docker for Mac April 17, 2018 by Guillaume Rose, Guillaume Tardif

YOUTUBE: What is Kubernetes? Jun 18, 2018 by Jason Rahm

Video courses

Kubernetes for Machine Learning

This article talks about Jupyter notebooks correctness and functionality being dependent on their environment, called “training serving skew”. To get around that, use the Binder service which takes Jupyter notebooks within a Git repository to build a container image, then launches the image in a Kubernetes cluster with an exposed route accessible from the public internet.

OpenShift’s Source-to-image (S2I) and Graham Dumpleton’s OpenShift S2I builder builds artifacts from source and injects them into docker images.

It’s used by Seldon-Core to scale Machine Learning environments. There are Seldon-Core Examples

Seldon-Core is used by Kubeflow makes deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. It provides templates and custom resources to deploy TensorFlow and other machine learning libraries and tools on Kubernetes. Included in Kubeflow is JupyterHub to create and manage multi-user interactive Jupyter notebooks. It began as TensorFlow Extended at Google.

https://github.com/kubernetes-incubator is a collection of repositories such as the spartakus Anonymous Usage Collector, metrics-server, external-dns which configures external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services, and kube-aws which is a command-line tool to declaratively manage Kubernetes clusters on AWS.

https://radanalytics.io Oshinko empowers intelligent app developement on the OpenShift platform deploying and managing Apache Spark clusters It has a spark cluster management app (oshinko-webui)



Vicky Tanya Seno at Santa Monica College is preparing a course of Kubernetes on ACloud.guru

Kubstack https://twitter.com/pst418


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud regions
  16. AWS Virtual Private Cloud
  17. Azure Cloud Onramp
  18. Azure Cloud
  19. Azure Cloud Powershell
  20. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  21. Digital Ocean
  22. Cloud Foundry

  23. Packer automation to build Vagrant images
  24. Terraform multi-cloud provisioning automation

  25. Powershell Ecosystem
  26. Powershell on MacOS
  27. Powershell Desired System Configuration

  28. Jenkins Server Setup
  29. Jenkins Plug-ins
  30. Jenkins Freestyle jobs
  31. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  32. Dockerize apps
  33. Docker Setup
  34. Docker Build

  35. Maven on MacOSX

  36. Ansible

  37. MySQL Setup

  38. SonarQube static code scan

  39. API Management Microsoft
  40. API Management Amazon

  41. Scenarios for load