Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Container engine in clouds

Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This is a hands-on introduction with insightful commentary carefully sequenced to make complex material easier to understand quickly. This is not a “demo”, but an immersive step-by-step “deep dive” tutorial aimed to make you productive.

Why Kubernetes?

Kubernetes is called “container orchestration” software because it automates the deployment, scaling and management of containerized applications[Wikipedia].

“Containerized” microservice apps are dockerized into images pulled from DockerHub or private security-vetted images in Docker Enterprise, Quay.io, or an organization’s own binary repository setup using Nexus or Artifactory. Kubernetes also works with rkt (pronounced “rocket”) containers. IBM Red Hat has their “containerd” technology. But this tutorial focuses on Docker containers.

Kubernetes automates resilience by abstacting the network and storage shared by ephemeral replaceable pods which the Kubernetes Controller replicates to increase capacity.

In the illustration below, each pod (each a different color) encapsulates one or more (Docker) container hosts (operating processes, each shown as a circle):

k8s-container-sets-479x364.jpg

Containers within the same pod share the same IP address, hostname, Linux namespaces, cgroups, storage Volumes, and other resources. Within a pod, each container has a different port number and can communicate with other containers in the same pod.

The Service Mesh Istio architecture has an “Envoy proxy” in each pod to facilitate the communictions and retry logic from the business logic containers in its pod.

Kubernetes replicates Pods (the same set of containers in each) across several worker Nodes (VM or physical machines).

Each set of pods are within a node. Kubernetes assigns each node with a different external IP address.

kubernetes-structure-502x205-12351.png

Nodes are managed together as a namespace. Kubernetes can manage several namespaces running in each cluster.

“The primary grouping concept in Kubernetes is the namespace. Namespaces are also a way to divide cluster resources between multiple uses. That being said, there is no security between namespaces in Kubernetes; if you are a “user” in a Kubernetes cluster, you can see all the different namespaces and the resources defined in them.” – from the book: OpenShift for Developers, A Guide for Impatient Beginners by Grant Shipley and Graham Dumpleton.

OpenShift project wall namespaces

Projects in OpenShift provide “walls” between namespaces, ensuring that users or applications can only see and access what they are allowed to. OpenShift projects wrap a namespace by adding security annotations which control access to that namespace. Access is controlled through an authentication and authorization model based on users and groups.

This diagram illustrates what OpenShift adds: kubernetes-openshift-502x375-107638

OpenShift’s Router is instead a HAProxy container (taking the place of NGINX).

k8s-openshift-projects-461x277-64498

Cluster networking

A private ClusterIP is accessible by nodes only within the same cluster.

Services listen on the same nodePort (TCP 30000 - 32767 defined by --service-node-port-range).

k8s-arch-ruo91-797x451-104467

The diagram above is referenced throughout this tutorial, particularly in the Details section below. It is by Yongbok Kim who presents animations on his website.

Communications with outside service network callers occur through a single Virtual IP address (VIP) going through a kube-proxy pod within each node. The Kube-proxy load balances traffic to deployments, which are load-balanced sets of pods within each node. Kube-proxy IPVS Mode is native to the Linux kernel. CBR0 (Custom Bridge zero) forwards the eth0, which rewrites the destination IP to a pod behind the Service3:18 into chapter 6 Big Picture

Kubernetes manages the instantiating, starting, stopping, updating, and deleting of a pre-defined number of pod replicas based on declarations in *.yaml files or interactive commands.

The number of pods replicated is based on deployment yaml files. Service yaml files specify what ports are used in deployments.

In 2019 Kubernetes added auto-scaling based on metrics API measurement of demand.

This Architectural Diagram pdf:  k8s-linuxacademy-arch-912x415-32433.jpg is described in the Linux Academy’s CKA course of 05:34:43 hours of videos by Chad Miller (@OpenChad).

Read Docs and tutorials from Kubernetes.io.

Open Sourced and collaborative

Kubernetes was created inside Google (using the Golang programming language).

Kubernetes was used inside Google for over a decade before being open-sourced in 2014 to the Cloud Native Computing Foundation (cncf.io) collective.

kubernetes-logo-125x134-15499.png This blog and podcast revealed that the predecessor to Kubernetes was called “the Borg” becuase initial developers were fans of Star Trek Next Generation. In the TV series, the Borg subsumes all civilizations it encounters into its “collective”. The logo for Kubernetes inside the 6 sided hexagons representing each Google service has 7 sides. This is because a beloved character in the TV series, played by the curvacious Jeri Ryan, is a converted Borg called “7 of 9”.

Its Google heritage means Kubernetes is about scaling for a lot of traffic with redundancies to achieve high availability (HA).

Kubernetes is often abbreviated as k8s, with 8 replacing the number of characters between k and s. Thus, https://k8s.io redirects you to https://kubernetes.io, the home page for the software.

Although the word “kubernetes” is a Greek word meaning “captain”, it has become a registered trademark of the Linux Foundation, which maintains the website https://kubernetes.io and source code at https://github.com/kubernetes/kubernetes

  • v1.0 was committed on July 2015 within GitHub, and released on July 21, 2015
  • v1.6 was led by a CoreOS developer
  • v1.7 was led by a Googler
  • v1.8 is led by a Microsoft employee (Jaice Singer DuMars) after Microsoft joined the CNCF July 2017

Certification in Kubernetes

On November 8, 2016 CNCF announced their 3-hour task-based Certified Kubernetes Administrator (CKA) and 2-hour Kubernetes Application Developer (CKAD) exams. Each is $300, which includes one free retake. To compare the domain focus for each exam:

CK Admin. CK App. Dev.
19% Core Concepts
12% Installation, Configuration & Validation
12% Security
11% Networking
11% Cluster Maintenance
10% Troubleshooting
08% Application Lifecycle Management
07% Storage
05% Scheduling
05% Logging / Monitoring
13% Core Concepts
18% Configuration
20% Pod Design
18% Observability
13% Services & Networking
10% Multi-Container Pods
08% State Persistence

The exam has no multiple choice questions.

CNCF is part of the Linux Foundation, so…

  1. Get an account (Linux Foundation credentials ) at https://identity.linuxfoundation.org.

    It’s a non-profit organization, thus the “.org”.

  2. Login to https://linuxfoundation.org and join as a member for a $100 discount toward certifications.
  3. Go to https://training.linuxfoundation.org/linux-courses/system-administration-training/kubernetes-fundamentals and pay for the $300 exam or for $199 more take their class.

  4. Use your Linux Foundation credentials to sign-in at examslocal.com, and select either or both of two exams from CNCF:

    • Linux Foundation : Certified Kubernetes Administrator (CKA) - English
    • Linux Foundation : Certified Kubernetes Application Developer (CKAD) - English

  5. Click “Or Sign In With” tab and select “Sign in for exams powered by the Linux Foundation”.
  6. Log in using your preferred account.
  7. Click “Handbook link” to download it.
  8. Select the date, then click OK.
  9. Setup your home computer to take the exam at home using the Chrome extension from “Innovative Exams”, which uses your laptop camera and microphone watching you use a virtual Ubuntu machine.
  10. Take the 180 minute (2 hour) exam.

PROTIP: The Linux Foundation exam focuses on “pure” Kubernetes commands only and excludes add-ons such as OpenStack.

Resources:

Multi-cloud

Being open-source has enabled Kubernetes to flourish on several clouds*

  • Google Kubernetes Engine (GKE) is a container management SaaS product. GKE runs within the Google Compute Platform (GCP) on top of Google Compute Engine providing machines. GKE in GCP integration covers networking and VPC, monitoring, logging, and CI/CD.

    k8s-gcp-738x314-14535

    A search for “Kubernetes” within the GCP Console yields:

    k8s-gcp-search-656x866-37655

  • Amazon ECS (Elastic Container Service for Kubernetes) is “supercharged” by the
    Amazon EKS (Elastic Kubernetes Service), which provides deeper integration into AWS infrastructure (than ECS) for better reliability (at higher cost). Amazon said it runs upstream K8s, not a fork (such as AWS ELasticSearch), so it should be portable to other clouds and on-premises.

    ECS is free since Amazon charges for the underlying EC2 instances and related resources for each task ECS runs.

    But each EKS cluster costs an additional $144 USD per month (20 cents per hour in the lowest cost us-east-1 region), for EKS to administer a “Control Plane” across Availability Zones.

    The diagram (from cloudnaut) illustrates the differences between ECS vs. EKS clusters.

    eks-ecs-load-balacing-960x720-32943.png

    ECS uses an Application Load Balancer (ALB) to distribute load servicing clients. When EKS was introduced December 2017, it supported only Classic Load Balancer (CLB), with beta support for Application Load Balancer (ALB) or Network Load Balancer (NLB).

    Within the cluster, distribution among pods can be random or based on the round robin algorithm.

    EKS incurs additional cross-AZ network traffic charges because, to ensure high availability, EKS runs within each node a proxy to distribute traffic in and out of pods across three Kubernetes masters across three Availability Zones. So this additional processing may also require larger instance types, which EKS automatically selects.

    Instance type selection is an important consideration because AWS limits the number of IP Addresses per network interface based on instance size, from 2 to a max of 15. Not all AWS EC2 instance types are equipped with the Elastic Network Interface (ENI) that ECS and EKS need to virtually redistribute load among pods. Both ECS and EKS detects and automatically replaces unhealthy masters, provide version upgrades, and automated patching for masters. A secondary private IPv4 network interface is used so that in the event of an instance failure, that interface and/or secondary private IPv4 address can be transferred to a hot standby instance by EKS.

    eks-ecs-vpc-eni-960x720-31322

    While ECS assigns separate ENI to each ECS task (a group of containers), EKS attaches multiple ENIs per instance, with multiple private IP addresses assigned to each ENI. Since EKS shares network interfaces among pods, a different Security Group cannot be specified to restric a specific pod.

    Moreover, network interfaces, multiple private IPv4 addresses, and IPv6 addresses are only available for instances running under a isolated VPC (Virtual Private Cloud) and perhaps with AWS PrivateLink access. So EKS requires AWS VPC. For best isolation (rather than sharing), create a different VPC and Security Group for each cluster.

    Both ECS and EKS is accessed from its ECS CLI console and supports ECS API commands and Docker Compose. AWS CloudTrail logging.

    Also, EKS leverage IAM authentication, but did not provide out-of-the-box support Task IAM Roles (pods) used to grant access to AWS resources like ECS (AmazonEKSClusterPolicy and AmazonEKSServicePolicy).

    For example, to allow containers to access S3, DynamoDB, SQS, or SES at runtime.

    Behind the scenes, Amazon used Hashicorp Packer config. scripts to make EKS-optimized AMIs run on Amazon Linux 2. The machines are preconfigured with Docker, kubelet, and the AWS/Heptio AMI Authenticator DaemonSet, plus a EC2 User Data bootstrap script that automatically join an EKS cluster. AMIs that have GPU support are also generated for users who have defined a AWS Marketplace Subscription.

    See the EKS Manifest diagram explained by Mark Richman (@mrichman) in his video class, with code at https://github.com/linuxacademy/eks-deep-dive-2019.

  • AWS Fargate Launch Type provides for horizontal scaling on Amazon’s own fleet of EC2 clusters. It’s informally called the “AWS Container Manager”.

    Fargate supports “awsvpc” network mode natively so that tasks running on the same instance share that’s instance’s ENI.

    “Once you do get your cluster running, there’s nothing to worry about except monitoring performance and, as demand changes, adjusting the scale of your service.” – David Clinton*

    This totalcloud.io article compares ECS, EKS, and Fargate.

    A concern with Fargate is its time to load.

  • Microsoft’s Azure Kubernetes Service (AKS)

  • Digital Ocean - see https://blog.digitalocean.com/introducing-digitalocean-kubernetes

  • KUBE2GO, OpenShift Dedicated, OpenShift Online,

  • Kops for AWS (at https://github.com/kubernetes/kops) is open-source to enable multi-master, multi-AZ cluster setup and management of multiple instance groups. Admins must stand up the masters, unlike in ECS/EKS. See “How Qubit built its production ready Kubernetes (k8s) environments”

  • https://github.com/kubernetes-sigs/kubespray

  • Rancher Kubernetes Engine (RKE) simplifies cluster administration (on EC2, Azure, GCE, Digital Ocean, EKS, AKS, GKE, vSphere or bare metal) - (provisiong, authentication, RBAC, Policy, Security, monitoring, Capacity scaling, Cost control). Its catalog is based on Helm.

Alternatives / Competitors

  • Docker Swarm incorporated Rancher from Rancher Labs (#RancherK8s).

    See Creating an Amazon EC2 Cluster using Rancher

  • Mesosphere DC/OS (Data Center Operating System) runs Apache Mesos to abstract CPU, memory, storage to provide an API to program a multi-cloud multi-tenant data center (at Twitter, Yelp, Ebay, Azure, Apple, etc.) as if it’s a single pool of resources. Kubernetes can run on top of it, but the DC/OS has premium (licensed) enterprise features. So it’s not for you if you never want to pay for anything.

    Mesos from Apache, which runs other containers in addition to Docker. K8SM is a Mesos Framework developed for Apache Mesos to use Google’s Kubernetes. Installation.

    See Container Orchestration Wars (2017) at the Velocity Conf 19 Jun 2017 by Karl Isenberg (@karlfi) of Mesosphere

  • Hashicorp Nomad.

  • Red Hat (which IBM bought in 2018) offers its OpenShift to enable Docker and Kubernetes for the enterprise by adding external host names (projects) that add role-based security around namespaces. See:

    https://www.redhat.com/en/technologies/cloud-computing/openshift,

    One can run k8s containers in other clouds or within private data centers using OpenStack from RedHat.

    • Note that IBM (which bought Red Hat in 2018) has been pushing “containerd”, its replacement for Docker.

Other orchestration systems managing Docker containers:

  • Kubernetes by Google
  • Centos
  • Atomic
  • Consul, Terraform
  • Serf
  • Cloudify
  • Helios

Add-ons to Kubernetes


Kublet

A Kublet agent program is automatically installed when a node is created. Each kubelet manages what is called the “control pane” which allocates IP addresses and runs nodes under its control.

Kublet constantly compares the status of pods against what is declared in yaml files, and starts or deletes pods as necessary to meet the request.

Restarting Kublet itself depends on the operating system (monit on Debian or systemctl on systemd-based systems).

Master node

Nodes are joined to the master node using the kubeadm join program and command.

The master node itself is crated by the kubeadm init command which establishes folders and invokes the Kubernetes API server. That command is installed along with the kubectl package (pronounced “cube cuddle”). There is a command with the same name used to obtain the version.

The kubectl get nodes command lists basic information about each node.

The describe command provides more detailed information.

API Server

The kubectl client communicates using REST API calls to an API Server which handles authentication and authorization.

kubectl get apiservices

API’s were initially monolithic but has since been split up into:

  • core “” to handle pod & svc & ep (endpoint)
  • apps to handle deploy, sts, ds
  • authorization to handle role, rb
  • storage to handle pv (persistent volume) and pvc, sc (storage classes)

Scheduler

The API Server puts nodes in “pending” state when it sends requests to bring them up and down to the Scheduler to do so only when there are enough resources available. The scheduler can operate according to a schedule. But whether it does or not are defined in rules (called “Taints”) obeyed by the Scheduler about nodes. Rules obeyed by the Scheduler about pods are called “Tolerances”. Such details are reaveled using the kubectl describe nodes command.

etcd storage

The API Server and Scheduler persists their configuration and status information in a ETCD cluster (from CoreOS).

Kubernetes data stored in etcd includes jobs being scheduled, created and deployed, pod/service details and state, namespaces, and replication details.

It’s called a cluster because, for resiliancy, etcd replicates data across nodes. This is why there is a minimum of two worker nodes per cluster.

eksctl

  1. See https://eksctl.io about installing the eksctl CLI tool for creating clusters on EKS. It is written and supported (via Slack) by GitOps vendor weave.works in Go, and uses CloudFormation.

  2. To create a EKS cluster:

    eksctl create cluster

HA Proxy cluster

For network resiliency, HA Proxy cluster distributes traffic among nodes.

Node Controllers and Ingress

The Node controller assigns a CIDR block to newly registered nodes, then continually monitors node health. When necessary, it taints unhealthy nodes and gracefully evicts unhealthy pods. The default timeout is 40 seconds.

Load balancing among nodes (hosts within a cloud) are handled by third-party port forwarding via Ingress controllers. See Ingress definitions.

An “Ingress” is a collection of rules that allow inbound connections to reach the cluster services.

Ingress Resource defines the connection rules.

In Kubernetes the Ingress Controller could be a NGINX container providing reverse proxy capabilities.

Plug-in Network

PROTIP: Kubernetes uses third-party services to handle load balancing and port forwarding through ingress objects managed by an ingress controller.

CNI (Container Network Interface)

Flannel.

Other CNI vendors include Calico, Cilium, Contiv, Weavenet.

cAdvisor

To collect resource usage and performance characteristics of running containers, many install a pod containing Google’s Container Advisor (cAdvisor). It aggregates and exports telemetry to an InfluxDB database for visualization using Grafana.

Google’s Heapster is also be used to send metrics to Google’s cloud monitoring console.


Helm charts

The name Kubernetes is the ancient Greek word for people who pilot cargo ships – “helmsman” in English. Thus the nautical references and why Kubernetes experts are called “captain” and why associated products have nautical themes, such as “Helm”.

A Helm chart can be used to quickly create an OpenFaaS (Serverless) cluster on your laptop.

git clone https://github.com/openfaas/faas-netes && cd faas-netes
   kubectl apply -f ./namespaces.yml 
   kubectl apply -f ./yaml_armhf
   

Deploy a scalable web application to Kubernetes using Helm

Topics

  • Infrastructure as code
  • Manage containers
  • Naming and discovery
  • Mounting storage systems
  • Balancing loads
  • Rolling updates
  • Distributing secrets/config
  • Checking application health
  • Monitoring resources
  • Accessing and ingesting logs
  • Replicating application instances
  • Horizontal autoscaling
  • Debugging applications

Containers are declared by yaml such as this to run Alphine Linux Docker container:

apiVersion: v1
kind: Pod
metadata:
  name: alpine
  namespace: default
spec:
  containers:
  - name: alpine
    image: alpine
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
   

Kubernetes is written in the Go language, so it can run on Windows, Linux, and MacOS (the need to install a JVM).

Raspberry Pi

Read how the legendary Scott Hanselman built Kubernetes on 6 Raspberry Pi nodes, each with a 32GB SD card to a 1GB RAM ARM chip (like on smartphones).

Hansel talked with Alex Ellis (@alexellisuk) keeps his instructions with shell file updated for running on the Pis to install OpenFaaS.

CNCF Ambassador Chris Short developed the rak8s (pronounced rackets) library to make use of Ansible.

Others:

  • https://blog.hypriot.com/getting-started-with-docker-on-your-arm-device/
  • https://blog.sicara.com/build-own-cloud-kubernetes-raspberry-pi-9e5a98741b49

Architecture diagram

Yongbok Kim (who writes in Korean) posted (on Jan 24, 2016) a master map of how all the pieces relate to each other:
Click on the diagram to pop-up a full-sized diagram: k8s_details-ruo91-350x448.jpg

BTW What are now called “nodes” were previously called minions. Apparently Google namers forgot about the existance of NodeJs, which refers to nodes differently.

Testing

End-to-end tests by those who develop Kubernetes are coded in Ginko and Gomega (because Kubernets is written in Go).

The Kubtest suite builds, stages, extracts, and brings up the cluster. After testing, it dumps logs and tears down the test rig.

Social

  • Twitter: @kubernetesio
  • https://slack.k8s.io
  • Google+ Group: Kubernetes
  • https://groups.google.com/forum/#!forum/kubernetes-announce for announcements
  • https://groups.google.com/forum/#!forum/kubernetes-dev for contributors to the Kubernetes project to discuss design and implementation issues.
  • https://stackoverflow.com/search?q=k8s+or+kubernetes for developers
  • https://serverfault.com/search?q=k8s+or+kubernetes for sysadmins.
  • https://groups.google.com/forum/#!forum/kubernetes-sig-scale
  • https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ&disable_polymer=true Kubernetes Google Community video chats

  • https://cloud.google.com/support/docs/issue-trackers to report bugs

  • KubeCon.io Conferences (#KubeConio)

Installation options

There are several ways to obtain a running instance of Kubernetes.

Rancher is a deployment tool for Kubernetes that also provides networking and load balancing support. Rancher initially created it’s own framework (called Cattle) to coordinate Docker containers across multiple hosts, at a time when Docker was limited to running on a single host. Now Rancher’s networking provides a consistent solution across a variety of platforms, especially on bare metal or standard (non cloud) virtual servers. In addition to Kubernetes, Rancher enables users to deploy a choice of Cattle, Docker Swarm, Apache Mesos upstream project for DCOS (Data Center Operating System). Rancher eventually become part of Docker Swarm.

Within https://github.com/kubernetes/kops KOPS

Minikube offline

B) Minikube spins up a local environment on your laptop.

NOTE: Ubuntu on LXD offers a 9-instance Kubernetes cluster on localhost.

PROTIP: CAUTION your laptop going to sleep may ruin minikube.

Server install

C) install Kubernetes natively on CentOS.

D) Pull an image from Docker Hub within a Google Compute or AWS cloud instance.

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.

See https://kubernetes.io/docs/setup/pick-right-solution

Minikube

Minikube goes beyond Docker For Mac (DFM) and Docker for Windows (DFW) and includes a node and a Master when it spins up in a local environment (such as your laptop).

CAUTION: At time of writing, https://github.com/kubernetes/minikube has 257 issues and 20 pending Pull Requests.

  1. Install on a Mac Docker:

    
    brew install docker-machine-driver-xhyve
    
  2. Install on a Mac Minikube:

    
    brew install minikube -y
    
  3. Verify if its command works by getting the version:

    minikube version
  4. Show the current context:

    
    kubectl config current-context
    

    The response on minikube is “minikube”.

  5. Start the service:

    On Mac:

    minikube start --vm-driver=xhyve
    

    On Windows:

    minikube start --vm-driver=hyperv
    
  6. Dashboard

    minikube dashboard
  7. Stop the service:

    minikube stop
  8. Recover space:

    minikube delete

    Kubectl 1.8 scale is now the preferred way to control graceful delete.

    Kubectl 1.8 rollout and rollback now support stateful sets ???

    kubectl CLI client install

    Kubernetes administrators use the kubectl (kube + ctl) the CLI tool running outside Kubernetes servers to control them. It’s automatically installed within Google cloud instances, but on Macs clients:

  9. Install on a Mac:

    
    brew install kubectl -y
    
    🍺  /usr/local/Cellar/kubernetes-cli/1.8.3: 108 files, 50.5MB
    
  10. Verify

    
    kubectl version --client
    

    A sample response:

    Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
    
    1. Check the status of the job using the kubectl describe command.

    2. When a job is complete, view its results:

    kubectl logs counter

    The API Server routes several kinds of yaml declaration files: Pod, Deployment of pods, Service, Job, Configmap.

    API primatives ???

https://plugins.jetbrains.com/plugin/10485-kubernetes

CentOS

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
   

Also:

cat <  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
   

Ubuntu

  1. On Ubuntu, install:

    apt install -y docker.io
  2. To make sure Docker and Kublet are using the same systemd driver:

    cat <<EOF >/etc/docker/daemon.json
    {
      "exec-opts": ["native.cgroupdriver=systemd"]
    }
    EOF
  3. Install the keys:

    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
  4. sources:

    cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
    <deb http://apt.kubernetes.io/ kubernetes-xenial main
    <EOF
  5. To download new sources:

    apt update
  6. To download the programs:

    apt install -y kubelet kubeadm kubectl

Details

This section further explains the architecture diagram above.

This sequence of commands:

  1. Select “CloudNativeKubernetes” sandboxes.
  2. Select the first instance as the “Kube Master”.
  3. Login that server (user/123456).
  4. Change the password as prompted on the Ubuntu 16.04.3 server.

    Deploy Kubernetes master node

  5. Use this command to deploy the master node which controls the other nodes. So it’s deployed first which invokes the API Server

    sudo kubeadm init --pod-network-cidr=10.244.0.0/16

    kubernetes-nodes-363x120-20150

    The address is the default for Flannel.

    Flow diagram

    k8s-services-flow-847x644-100409

    The diagram above is by Walter Liu

    Flannel for Minikube

    When using Minikube locally, a CNI () is needed. So setup Flannel from CoreOS using the open source Tectonic Installer (@TectonicStack). It configures a IPv4 “layer 3” network fabric designed for Kubernetes.

    The response suggests several commands:

  6. Create your .kube folder:

    mkdir -p $HOME/.kube
  7. Copy in a configuration file:

    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  8. Give ownership of “501:20”:

    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  9. Make use of CNI:

    sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube.flannel.yml

    The response:

    clusterrole "flannel" created
    clusterrolebinding "flannel" created
    serviceaccount "flannel" created
    configmap "kube-flannel.cfg" created
    daemonset "kube-flannel.ds" created
    

    configmaps in cfg files are used to define environment variables.

  10. List pods created:

    kubectl get pods --all-namespaces -o wide

    Specifying wide output adds the IP address column

    Included are pods named:

    • api server (aka “master”) accepts kubectl commands
    • etcd (cluster store) for HA (High Availability) in control pane
    • controller to watch for changes and maintain desired state
    • dns (domain name server)
    • proxy load balances across all pods in a service
    • scheduler watches api server for new pods to assign work to new pods

    System administrators control the Master node UI in the cloud or write scripts that invoke kubectl command-line client program that controls the Kubernetes Master node.

    Proxy networking

    The Kube Proxy communicates only with Pod admin. whereas Kubelets communicate with individual pods as well.

    Each node has a Flannel and a proxy.

    The Server obtains from Controller Manager ???

  11. Switch to the webpage of servers to Login to the next server.
  12. Be root with sudo -i and provide the password.
  13. Join the node to the master by pasting in the command captured earlier, as root:

    kubeadm join --token ... 172.31.21.55:6443 --discovery-token-ca-cert-hash sha256:...

    Note the above is one long command. So you may need to use a text editor.

    Deployments manage Pods.

    Every Pod has a unique IP. There is one IP Address per Pod. In other words, containers within a Pod share a network namespace.

    Every container has its own unique port number within its pod’s IP.

    kubernetes-ports-381x155-19677

  14. Switch to the webpage of servers to Login to the 3rd server.
  15. Again Join the node to the master by pasting in the command captured earlier:
  16. Get the list of nodes instantiated:

    kubectl get nodes
  17. To get list of events sorted by timestamp:

    kubectl get events --sort-by='.metadata.creationTimestamp'
  18. Create the initial log file so that Docker mounts a file instead of a directory:

    touch /var/log/kube-appserver.log
    
  19. Create in each node a folder:

    mkdir /srv/kubernetes
    
  20. Get a utility to generate TLS certificates:

    brew install easyrsa
    
  21. Run it:

    ./easyrsa init-pki
    

    Master IP address

  22. Run it:

    MASTER_IP=172.31.38.152
    echo $MASTER_IP
    
  23. Run it:

    ./easyrsa --batch "--req-cn=${MASTER_IP}@`date +%s`* build-ca nopass
    

    Watchers

    To register watchers on specific nodes.??? Kubernetes supports TLS certifications for encryption over the line.

    REST API CRUD operations are used For authorization, Kubernetes supports Role Base Access Control (RBAC), (ABAC), and Webhook. Admission ???

  24. Put in that folder (in each node):

    • basic_auth.csv user and password
    • ca.crt - the certificate authority certificate from pki folder
    • known_tokens.csv kublets use to talk to the apiserver
    • kubecfg.crt - client cert public key
    • kubecfg.key - client cert private key
    • server.cert - server cert public key from issued folder
    • server.key - server cert private key

  25. Copy from API server to each master node:

    
    cp kube-apiserver.yaml  /etc/kubernetes/manifests/
    

    The kublet compares its contents to make it so, uses the manifests folder to create kube-apiserver instances.

  26. For details about each pod:

    
    kubectl describe pods
    

    Expose

    Deploy service

  27. To deploy a service:

    kubectl expose deployment *deployment-name* [options]

Container Storage Interface (CSI)

### Volumes

Containers also share attached data volumes available within each Pod.

Kubelet agents

HAProxy VRRP (Virtual Router Redundancy Protocol) http://searchnetworking.techtarget.com/definition/VRRP automatically assigns available Internet Protocol routers to participating hosts.

A Persistent Volume (PV) is a provisioned block of storage for use by the cluster.

A Persistent Volume Claim (PVC) is a request for that storage by a user, and once granted, is used as a “claim check” for

Recycling policies are Retain (keep the contents) and Recycle (Scrub the contents).

configmap

Activities

  1. To drain a node out of service temporarily for maintenance:

    kubectl drain node3.mylabserver.com --ignore-daemonsets

    DaemonSets

    daemonsets (ds)

    Usually for system services or other pods that need to physically reside on every node in the cluster, such as for network services. They can also be deployed only to certain nodes using labels and node selectors.

  2. To return to service:

    kubectl uncordon node3.mylabserver.com

Sample micro-service apps

The repo is based on work from others, especially Kelsy Hightower, the Google Developer Advocate.

  • https://github.com/kelseyhightower/app - an example 12-Factor application.
  • https://hub.docker.com/r/kelseyhightower/monolith - Monolith includes auth and hello services.
  • https://hub.docker.com/r/kelseyhightower/auth - Auth microservice. Generates JWT tokens for authenticated users.
  • https://hub.docker.com/r/kelseyhightower/hello - Hello microservice. Greets authenticated users.
  • https://hub.docker.com/r/ngnix - Frontend to the auth and hello services.

These sample apps are manipulated by https://github.com/kelseyhightower/craft-kubernetes-workshop

  1. Install
  2. Create a Node.js server
  3. Create a Docker container image
  4. Create a container cluster
  5. Create a Kubernetes pod
  6. Scale up your services

  7. Provision a complete Kubernetes cluster using Kubernetes Engine.
  8. Deploy and manage Docker containers using kubectl.
  9. Break an application into microservices using Kubernetes’ Deployments and Services.

This “Kubernetes” folder contains scripts to implement what was described in the “Orchestrating the Cloud with Kubernetes” hands-on lab which is part of the “Kubernetes in the Google Cloud” quest.

Bob Reselman’s 3-day hands-on classes on Kubernetes makes use of bash scripts and sample app at https://github.com/reselbob/CoolWithKube

Infrastructure as code

  1. Use an internet browser to view

    https://github.com/wilsonmar/DevSecOps/blob/master/Kubernetes/k8s-gcp-hello.sh

    The script downloads a repository forked from googlecodelabs: https://github.com/wilsonmar/orchestrate-with-kubernetes/tree/master/kubernetes

    Declarative

    This repository contains several kinds of .yaml files, which can also have the extension .yml. Kubernetes also recognizes .json files, but YAML files are easier to work with.

    The files are call “Manifests” because they declare the desired state.

  2. Open an internet browser tab to view it.

    reverse proxy to front-end

    The web service consists of a front-end and a proxy served by the NGINX web server configured using two files in the nginx folder:

    • frontend.conf
    • proxy.conf

    These are explained in detail at https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-as-a-web-server-and-reverse-proxy-for-apache-on-one-ubuntu-14-04-droplet

    SSL keys

    SSL keys referenced are installed from the tls folder:

    • ca-key.pem - Certificate Authority’s private key
    • ca.pem - Certificate Authority’s public key
    • cert.pem - public key
    • key.pem - private key

Kind yaml files

The kinds of yaml files:

### Deployments

  • auth.yaml
  • frontend.yaml
  • hello-green.yaml
  • hello-canary.yaml
  • hello.yaml

### pods

  • healthy-monolith.yaml configures “livenessProbe” (in folder healthz) and “readinessProbe” (in folder readiness) on port 81
  • monolith.yaml
  • secure-monolith.yaml

### services samples

  • auth.yaml
  • frontend.yaml
  • hello-blue.yaml
  • hello-green.yaml
  • hello.yaml
  • monolith.yaml

Label

How Google Kubernetes Engine works

kubernetes-pods-599x298-35069

https://google-run.qwiklab.com/focuses/639?parent=catalog

PROTIP: For GKE we disable all legacy authentication, enable RBAC (Role Based Access Control), and enable IAM authentication.

Pods are defined by a manifest file read by the apiserver which deploys nodes.

Pods go into “succeeded” state after being run because pods have short lifespans – deleted and recreated as necessary.

The replication controller automatically adds or removes pods to comply with the specified number of pod replicas declared are running across nodes. This makes GKE “self healing” to provide high availability and reliability with “autoscaling” up and down based on demand.

In this diagram:

From the https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/

  1. List all pods, including in the system namespace:

    
    kubectl get nodes --all-namespaces
    

pod.yml manifests

An example (cadvisor):

apiVersion: v1
kind: Pod
metadata:
  name:   cadvisor
spec:
  containers:
    - name: cadvisor
      image: google/cadvisor:v0.22.0
      volumeMounts:
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: false
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
      ports:
        - name: http
          containerPort: 8080
          protocol: TCP
      args:
        - --profiling
        - --housekeeping_interval=1s
  volumes:
    - name: rootfs
      hostPath:
        path: /
    - name: var-run
      hostPath:
        path: /var/run
    - name: sys
      hostPath:
        path: /sys
    - name: docker
      hostPath:
path: /var/lib/docker
   

Replication rc.yml

The rc.yml (Replication Controller) defines the number of replicas and

apiVersion: v1
kind: ReplicationController
metadata:
  name: cadvisor
spec:
  replicas: 5
  selector:
     app hello
  template:
    metadata:
      labels:
        app: hello-world
  spec:
    containers:
    - name: hello
      image: account/image:latest
      ports:
        containerPort: 8080
   
  1. Apply replication:

    
    kubectl apply -f rc.yml
    

    The response expected:

    replicationcontroller "hello" configured
    
  2. List, in wide format, the number of replicated nodes:

    
    kubectl get rc -o wide
    
    DESIRED, CURRENT, READY
    
  3. Get more detail:

    
    kubectl describe rc
    

Service rc.yml

The svc.yml defines the services:

apiVersion: v1
kind: Service
metadata:
  name: hello-svc
    labels:
      app: hello-world
spec:
  type: NodePort
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: hello-world
   

PROTIP: The selector should match the pods.xml.

One type of service is load balancer within a cloud:

apiVersion: v1
kind: Service
metadata:
  name: la-lb-service
spec:
  selector:
    app: la-lb
  ports:
  - port: 80
    protocol: TCP
    targetPort: 9376
  type: LoadBalancer
  clusterIP: 10.0.171.223
  loadBalancerIP: 78.12.23.17
   
  1. To create services:

    
    kubectl create -f svc.yml
    

    The response expected:

    service "hello-svc" created
    
  2. List:

    
    kubectl get svc
    
  3. List details:

    
    kubectl describe svc hello-svc
    
  4. List end points addresses:

    
    kubectl describe ep hello-svc
    

OpenShift routes to services

Services can be referenced by external clients using a host name such as “hello-svc.mycorp.com” by using OpenShift Enterprise, which uses “routes” that defines the rules the HAProxy applies to incoming connections.

Routes are deployed by an OpenShift Enterprise administrator as routers to nodes in an OpenShift Enterprise cluster. To clarify, the default Router in Openshift is an actual HAProxy container providing reverse proxy capabilities.

Deploy yml

The deploy.yml defines the deploy:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
  spec:
    containers:
    - name: nginx
      image: nginx:1.7.9
      ports:
      - containerPort: 80
        protocol: TCP
    nodeSelector:
      net: gigabit
   

Deployment wraps around replica sets, a newer version of doing rolling-update on Replication Controller. Old replica sets can revert roll-back by just changing the deploy.yml file.

PROTIP: Don’t run apt-upgrade within containers, which breaks the image-container relationship controls.

  1. Retrieve the yaml for a deployment:

    kubectl get deployment nginx-deployment -o yaml

    Notice the “RollingUpdateStrategy: 25% max unavilable, 25% max surge”.

  2. Begin rollout of a new desired version from the command line:

    kubectl set image deployment/nginx-deployment nginx=nginx:1.8

    Alternately, edit the yaml file to nginx:1.9.1 and:

    kubectl apply -f nginx-deployment.yaml
  3. View Rollout a new desired version:

    kubectl rollout status deployment/nginx-deployment
  4. Describe the yaml for a deployment:

    kubectl describe deployment nginx-deployment
  5. List the DESIRED, CURRENT, UP-TO-DATE, AVAILABLE:

    kubectl get deployments 
  6. List the DESIRED, CURRENT, UP-TO-DATE, AVAILABLE:

    kubectl get deployments 
  7. List the history:

    kubectl rollout history deployment/nginx-deployment --revision=3
  8. Backout the revision:

    kubectl rollout undo deployment/nginx-deployment --to-revision=2

Security Context

The security.yml defines a secrurity context pod:

apiVersion: v1
kind: Pod
metadata:
  name: security-context.pod
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 2000
  volumess:
  - name: sam-vol
    emptyDir: {}
  containers:
  - name: sample-container
    image: gcr.io/google-samples/node-hello:1.0
    volumeMounts:
    - name: sam-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false
   
  1. Create the pod:

    kubectl create -f security.yaml

    This can take several minutes.

  2. Enter the security context:

    kubectl exec -it security-context-pod -- sh
  3. See the users:

    ps aux
  4. See that the group is “2000” as specified:

    cd /data && ls -al
  5. Exit the security context:

    exit
  6. Delete the security context:

    kubectl delete -f security.yaml

Kubelet Daemonset.yaml

Kubelets instantiate pods – each a set of containers running under a single IP address, the fundamental units nodes.

A Kubelet agent program is installed on each server to watch the apiserver and register each node with the cluster.

PROTIP: Use a DaemonSet when running clustered Kubernetes with static pods to run a pod on every node. Static pods are managed directly by the kubelet daemon on a specific node, without the API server observing it.

  • https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.

Deleting a DaemonSet will clean up the Pods it created. Some typical uses of a DaemonSet are:

  • running a cluster storage daemon, such as glusterd, ceph, on each node.
  • running a logs collection daemon on every node, such as fluentd or logstash.
  • running a node monitoring daemon on every node, such as Prometheus Node Exporter, collectd, Datadog agent, New Relic agent, or Ganglia gmond.
  1. Start kubelet daemon:

    
    kubelet --pod-manifest-path=the directory 
    

    This periodically scans the directory and creates/deletes static pods as yaml/json files appear/disappear there.

    Note: Kubelet ignores files starting with a dot when scanning the specified directory.

    PROTIP: By default, Kubelets exposes endpoints on port 10255.

    Containers can be Docker or rkt (pluggable)

    /spec, /healthz reports status.

The container engine pulls images and stopping/starting containers.

  • https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

CNI Plugins

The Controller Network Interface (CNI) is installed using basic cbr0 using the bridge and host-local CNI plugins.

The CNI plugin is selected by passing Kubelet the command-line option:

   --network-plugin=cni 
   

See https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/

  • Flannel
  • Cisco ACI
  • Cilium
  • Contiv
  • Contrail
  • NSX-T
  • OpenVswitch
  • Project Calico
  • Weave Net

Learning resources

Nigel Poulton (@NigelPoulton, nigelpoulton.com), Docker Captain:

Make your own K8s

Kelsey Hightower, in https://github.com/kelseyhightower/kubernetes-the-hard-way, shows the steps of how to create Compute Engine yourself:

  • Cloud infrastructure firewall and load balancer provisioning
  • setup a CA and TLS cert gen.
  • setup TLS client bootstrap and RBAC authentication
  • bootstrap a HA etcd cluster
  • bootstrap a HA Kubernetes Control Pane
  • Bootstrap Kubernetes Workers
  • Config K8 client for remote access
  • Manage container network routes
  • Deploy clustesr DNS add-on

O’Reilly book Kubernetes adventures on Azure, Part 1 (Linux cluster) Having read several books on Kubernetes, Ivan Fioravanti, writing for Hackernoon, says it’s time to start adventuring in the magical world of Kubernetes for real! And he does so using Microsoft Azure. Enjoy the step-by-step account of his escapade (part 1).

Qwiklab

https://run.qwiklab.com/searches/lab?keywords=Build%20a%20Slack%20Bot%20with%20Node.js%20on%20Kubernetes&utm_source=endlab&utm_medium=email&utm_campaign=nextlab

The 8 labs covering 8 hours of the Kubernetes in the Google Cloud Qwiklab quest

Kubeflow

https://github.com/kubeflow/kubeflow makes deployment of Kubernetes for Machine Learning (TensorFlow) using Kafka

AWS K8s Cluster Auto-scaler

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md provides deep-dive notes and code.

References

by Adron Hall:

Julia Evans

  • https://jvns.ca/categories/kubernetes/

Drone.io

http://www.nkode.io/2016/10/18/valuable-container-platform-links-kubernetes.html

https://medium.com/@ApsOps/an-illustrated-guide-to-kubernetes-networking-part-1-d1ede3322727

https://cloud.google.com/solutions/heterogeneous-deployment-patterns-with-kubernetes

https://cloud.google.com/solutions/devops/

https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.html

https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html

https://devops.college/the-journey-from-monolith-to-docker-to-kubernetes-part-1-f5dbd730f620

https://github.com/ramitsurana/awesome-kubernetes

Social

https://kubernetes.io/community

Jobs

Kubernetes Dominates in IT Job Searches

Learning, Video and Live

Kubernetes for Beginners by Siraj Jan 8, 2019 [11:04]

Kubernetes Deconstructed Dec 15, 2017 [33:14] by Carson Anderson of DOMO (@carsonoid)

Solutions Engineering Hangout: Terraform for Instant K8s Clusters on AWS EKS by HashiCorp

Introduction to Microservices, Docker, and Kubernetes by James Quigley

Kubernetes in Docker for Mac April 17, 2018 by Guillaume Rose, Guillaume Tardif

YOUTUBE: What is Kubernetes? Jun 18, 2018 by Jason Rahm

Video courses

Kubernetes for Machine Learning

This article talks about Jupyter notebooks correctness and functionality being dependent on their environment, called “training serving skew”. To get around that, use the Binder service which takes Jupyter notebooks within a Git repository to build a container image, then launches the image in a Kubernetes cluster with an exposed route accessible from the public internet.

OpenShift’s Source-to-image (S2I) and Graham Dumpleton’s OpenShift S2I builder builds artifacts from source and injects them into docker images.

It’s used by Seldon-Core to scale Machine Learning environments. There are Seldon-Core Examples

Seldon-Core is used by Kubeflow makes deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. It provides templates and custom resources to deploy TensorFlow and other machine learning libraries and tools on Kubernetes. Included in Kubeflow is JupyterHub to create and manage multi-user interactive Jupyter notebooks. It began as TensorFlow Extended at Google.

https://github.com/kubernetes-incubator is a collection of repositories such as the spartakus Anonymous Usage Collector, metrics-server, external-dns which configures external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services, and kube-aws which is a command-line tool to declaratively manage Kubernetes clusters on AWS.

https://radanalytics.io Oshinko empowers intelligent app developement on the OpenShift platform deploying and managing Apache Spark clusters It has a spark cluster management app (oshinko-webui)

Resources

https://github.com/hjacobs/kubernetes-failure-stories

Vicky Tanya Seno at Santa Monica College is preparing a course of Kubernetes on ACloud.guru

Kubstack @pst418

GCP PODCAST: Kubernetes and Google Container Engine hosts Francesc Campoy Flores and Mark Mandel interview Brian Dorsey, Developer Advocate, Google Cloud Platform. Comments at r/gcppodcast

https://medium.com/@writetomiglani/how-to-ace-the-certified-kubernetes-administrator-exam-in-7-days-e4603ac40746

Free Playpen

play-with-k8s.com gives you a 4-hour playpen. But that site is gone.

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud regions
  16. AWS Virtual Private Cloud
  17. Azure Cloud Onramp
  18. Azure Cloud
  19. Azure Cloud Powershell
  20. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  21. Digital Ocean
  22. Cloud Foundry

  23. Packer automation to build Vagrant images
  24. Terraform multi-cloud provisioning automation

  25. Powershell Ecosystem
  26. Powershell on MacOS
  27. Powershell Desired System Configuration

  28. Jenkins Server Setup
  29. Jenkins Plug-ins
  30. Jenkins Freestyle jobs
  31. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  32. Dockerize apps
  33. Docker Setup
  34. Docker Build

  35. Maven on MacOSX

  36. Ansible

  37. MySQL Setup

  38. SonarQube static code scan

  39. API Management Microsoft
  40. API Management Amazon

  41. Scenarios for load