Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

How to keep secrets secret, but still shared and refreshed.

Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Here is a hands-on tutorial about how to install and use Hashicorp’s Vault (vaultproject.io) to securely access secret keys and Hashicorp Consul to store key/value pairs. Installation is from scratch on a cloud environment using Docker and docker-compose.

The unique contribution of this article is an attempt to provide a deep yet concise approach, done by using automation which are then explained. This course assumes participants bring a Mac or Windows laptop and have prior experience with Linux CLI commands.

PROTIP: Where we want to end up is having the system handle secrets. For example, on my desktop I have 1Password, which can I either click “fill” or copy and paste a password without knowing what it is. Great to have when I’m sharing my screen.

At the end of this tutorial, you should be able to:

What are secrets?

A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more.

Vault provides high-level policy management, secret leasing, audit logging, and automatic revocation.

Vault forces a mandatory lease contract with clients. All secrets read from Vault have an associated lease to enable key usage auditing, perform key rolling, and ensure automatic revocation. Vault provides multiple revocation mechanisms to give operators a clear “break glass” procedure after a potential compromise.

Vault from Hashicorp provides a unified interface to secrets while providing tight access control plus recording a detailed audit log.

Vault is open-sourced at https://github.com/hashicorp/vault with a marketing home page at https://vaultproject.io. It can be deployed to practically any environment, and does not require any special hardware (such as physical HSMs (Hardware Security Modules).

VIDEO: Introduction to HashiCorp Vault Mar 23, 2018 by Armon Dadgar, Hashicorp’s CTO, is a whiteboard talk about avoiding “secret sprawl” living in clear text with empheral (temporary) passwords and cryptographic offload to a central service: hashicorp-vault-dadgar-927x522-94211

Alternatives

Alternatives to secret management include:

  • Environment variables in a clear-text file loaded into operating system variables, such as:

    docker run -e VARNAME=mysecret ...

    “The twelve-factor app stores config in environment variables”.

    However, this is NOT cool anymore because the value of variables (secrets) can end up in logs. All processes have access to secrets (no RBAC).

    And this mechanism makes periodic key rotation manually cumbersome.

  • Docker Secrets was NOT designed for unlicensed (free) standalone containers*, but for Enterprise licensed (paid) Docker Swarm services in commands such as:

    docker service create --secret db_pass --name secret-test alpine bash

    db_pass is a file (with .txt extension) encrypted by a command such as:

    echo "mysecret" | docker secret create db_pass -
     docker secret ls

    Secrets are stored in Docker’s logging infra within its “Raft” distributed leader consensus mechanism shared with Swarm managers. So encryption needs to be locked in Swarm.

    Secrets can be added to a running service, but key rotation requires container restart.

    When the service is created (or updated), the secret is mounted onto the container in the /run/secrets directory which custom program can retrieve*

    def get_secret(secret_name):
      try:
          with open('/run/secrets/{0}'.format(secret_name), 'r') as secret_file:
              return secret_file.read()
      except IOError:
          return None
     
    database_password = get_secret('db_pass')
     
  • Kubernetes secrets are stored in its etcd process.

    --experimental-encryption-provider-config

    https://github.com/Boostport/kubernetes-vault

  • Cloud-base KMS (Key Management Service) such as from Amazon

  • The Aqua utility provides secrets management to orchestrators so that:

    docker run -it --rm -e SECRET={dev-vault.secret/password} \
     --name ubuntu ubuntu /bin/bash
    docker inspect ubuntu -f ""

    returns:

    ["SECRET={dev.vault-secret/password}","PATH=/usr/local/sbin:..."]

Requirements for secret keeping

Coverage of what features a secrets service should have:

  • Installed in sealed mode
  • RBAC (Role-based Access Control) so each user has only the rights for his/her specific role. This has to be enabled in Kubernetes:

    --authorization-mode=RBAC
  • Limit access to designated containers
  • Encrypted transmission with Mutual authentication (MTLS)
  • Audit logging

  • Change value of an existing secret (key rotation) without rebooting. This is the strong point with Vault.

  • Revocation

Learning Resources

https://learn.hashicorp.com/vault

Katacode’s “Store Secrets using Hashicorp Vault” provides a web-based interactive bash terminal.

Install Vault server

There are several ways to obtain a running instance of Hashicorp Vault, listed from easiest to most difficult:

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.

A. Vault cloud service

  • Azure Vault

B. Use Homebrew to install Vault natively on you Mac.

C. Pull an image from Docker Hub

D. Download from Hashicorp to install locally.

E. Use a Dockerfile to build your own Docker image. if you’re not using vault frequently, and want to get the latest when you do.


Cloud service

Vault is an open source tool that can be deployed to any environment. It is well suited for cloud environments where HSMs are either not available or are cost prohibitive.

  1. Create within your internal cloud, Google Cloud, Amazon EC2, Microsoft Azure, etc. a VM instance of an Ubuntu server. 4 GB RAM and 10 GB drive is the minimum.

    A sample command to create a Google Cloud instance:

    gcloud beta compute --project "${THIS_PROJECT_NAME}" instances create "${THIS_INSTANCE_NAME}" --zone "us-central1-f" --machine-type "n1-standard-1" --subnet "default" --maintenance-policy "MIGRATE" --service-account "{$GCP_ACCT}@developer.gserviceaccount.com" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --min-cpu-platform "Automatic" --tags "http","https","web","http-server","https-server" --image "ubuntu-1604-xenial-v20171026a" --image-project "ubuntu-os-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "${THIS_INSTANCE_NAME}"
    

    If you’re going to be using Vault a lot on your Mac, install using Homebrew:

  2. There are several packages with the name “vault”:

    brew search vault

    Note there are several:

    ==> Formulae
    ssh-vault   vault   vault-cli   vaulted
     
    ==> Casks
    aws-vault   gmvault
    
  3. Verify the source:

    brew info vault

    You should see:

    vault: stable 1.2.1 (bottled), HEAD
    Secures, stores, and tightly controls access to secrets
    https://vaultproject.io/
    Not installed
    From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/vault.rb
    
  4. Install:

    brew install vault
    ==> Downloading https://homebrew.bintray.com/bottles/vault-1.2.1.mojave.bottle.t
    ==> Downloading from https://akamai.bintray.com/4a/4afd77a02f34fbfc3f910d99bbb09
    ######################################################################## 100.0%
    ==> Pouring vault-1.2.1.mojave.bottle.tar.gz
    🍺  /usr/local/Cellar/vault/1.2.1: 6 files, 117.2MB
    

The great thing with Homebrew is you can upgrane and uninstall easily.


Build image using Dockerfile

Create Vault within a Docker image from scratch:

  1. Install Git

    
    apt-get update && apt-get install -y \
      git
    
  2. Use Git to obtain the Dockerfile :

    
    git clone https://github.com/wilsonmar/vault.git --depth=1 
    cd vault
    
  3. Create a docker image locally:

    sudo docker build -f Dockerfile -t demo:vault . 
    

    This would run Maven, and a test job.

  4. Run the Dockerfile at:

    https://raw.githubusercontent.com/wilsonmar/Vault/master/Dockerfile

    Its contains:

    FROM ubuntu:16.04
    RUN apt-get update
    RUN apt-get update && apt-get install -y \
      default-jre \
      default-jdk \
      git \
      maven 
     
    RUN mvn -version
    RUN git clone https://github.com/hashicorp/vault???.git --depth=1
    

    The above provides commands to install Vault within a blank Docker container.

    Vault-jvm/examples/sample-app is a simple sample app, which is replaced with a real app in the real world.

Use Docker image

On a Linux server instance’s Terminal CLI:

  1. Add Docker’s public GPG key :

    
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    

    OK is the expected response.

  2. View the Linux version code referenced in a later command:

    
    lsb_release -cs
    

    This returns stretch for Debinan and xenial for Ubuntu.

  3. Install Docker for Ubuntu (not Debian):

    sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
    
  4. Update repository:

    sudo apt-get update
    
  5. List policies:

    apt-cache policy docker-ce
    

    The response:

    docker-ce:
      Installed: (none)
      Candidate: 17.09.0~ce-0~ubuntu
      Version table:
      17.09.0~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.06.2~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.06.1~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.06.0~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.03.2~ce-0~ubuntu-xenial 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.03.1~ce-0~ubuntu-xenial 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.03.0~ce-0~ubuntu-xenial 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
    
  6. Install Docker Community Edition:

    sudo apt-get install -y docker-ce
    

    Sample response:

    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following additional packages will be installed:
      aufs-tools cgroupfs-mount libltdl7
    Suggested packages:
      mountall
    The following NEW packages will be installed:
      aufs-tools cgroupfs-mount docker-ce libltdl7
    0 upgraded, 4 newly installed, 0 to remove and 17 not upgraded.
    Need to get 21.2 MB of archives.
    After this operation, 100 MB of additional disk space will be used.
    Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
    Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4,970 B]
    Get:3 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB]
    Get:4 https://download.docker.com/linux/ubuntu xenial/stable amd64 docker-ce amd64 17.09.0~ce-0~ubuntu [21.0 MB]
    Fetched 21.2 MB in 0s (22.7 MB/s)     
    Selecting previously unselected package aufs-tools.
    (Reading database ... 66551 files and directories currently installed.)
    Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ...
    Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
    Selecting previously unselected package cgroupfs-mount.
    Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...
    Unpacking cgroupfs-mount (1.2) ...
    Selecting previously unselected package libltdl7:amd64.
    Preparing to unpack .../libltdl7_2.4.6-0.1_amd64.deb ...
    Unpacking libltdl7:amd64 (2.4.6-0.1) ...
    Selecting previously unselected package docker-ce.
    Preparing to unpack .../docker-ce_17.09.0~ce-0~ubuntu_amd64.deb ...
    Unpacking docker-ce (17.09.0~ce-0~ubuntu) ...
    Processing triggers for libc-bin (2.23-0ubuntu9) ...
    Processing triggers for man-db (2.7.5-1) ...
    Processing triggers for ureadahead (0.100.0-19) ...
    Processing triggers for systemd (229-4ubuntu20) ...
    Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
    Setting up cgroupfs-mount (1.2) ...
    Setting up libltdl7:amd64 (2.4.6-0.1) ...
    Setting up docker-ce (17.09.0~ce-0~ubuntu) ...
    Processing triggers for libc-bin (2.23-0ubuntu9) ...
    Processing triggers for systemd (229-4ubuntu20) ...
    Processing triggers for ureadahead (0.100.0-19) ...
    
  7. List Docker container status:

    sudo systemctl status docker
    

    The response:

    ● docker.service - Docker Application Container Engine
    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Active: active (running) since Sat 2017-11-04 22:00:35 UTC; 1min 28s ago
      Docs: https://docs.docker.com
     Main PID: 13524 (dockerd)
    CGroup: /system.slice/docker.service
            ├─13524 /usr/bin/dockerd -H fd://
            └─13544 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.552925012Z" level=warning msg="Your kernel does not support swap me
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.553123462Z" level=warning msg="Your kernel does not support cgroup 
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.553267498Z" level=warning msg="Your kernel does not support cgroup 
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.554662024Z" level=info msg="Loading containers: start."
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.973517284Z" level=info msg="Default bridge (docker0) is assigned wi
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.019418706Z" level=info msg="Loading containers: done."
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.029599857Z" level=info msg="Docker daemon" commit=afdb6d4 graphdriv
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.029962340Z" level=info msg="Daemon has completed initialization"
    Nov 04 22:00:35 vault-1 systemd[1]: Started Docker Application Container Engine.
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.054191848Z" level=info msg="API listen on /var/run/docker.sock"
    log files:
    
  8. Verify Docker version in case you need to troubleshoot:

    docker --version
    

    The response:

    Docker version 17.09.0-ce, build afdb6d4
    
  9. Download the Docker image maintained by Hashicorp:

    docker pull vault
    </pre>
    
    ### Alternate Docker images
    
    
    https://hub.docker.com/r/sjourdan/vault
    has Hashicorp Vault on a minimal Alpine Linux box
    
    
    https://hub.docker.com/r/kintoandar/hashicorp-vault
    has Hashicorp Vault on a tiny busybox
    
    
  10. Set environment variables so IP addresses used for the redirect and cluster addresses in Vault’s configuration is the address of the named interface inside the container (e.g. eth0):

    VAULT_REDIRECT_INTERFACE 
    VAULT_CLUSTER_INTERFACE 
    
  11. Run the image using the file storage backend at path /vault/file, with a default secret lease duration of one week and a maximum of (720h/24) 30 days:

    docker run --cap-add=IPC_LOCK -e 'VAULT_LOCAL_CONFIG={"backend": {"file": {"path": "/vault/file"}}, "default_lease_ttl": "168h", "max_lease_ttl": "720h"}' vault server
    

    --cap-add=IPC_LOCK: locks memory, which prevents it from being swapped to disk (and thus exposing keys).

    See https://www.vaultproject.io/docs/config/index.html

    NOTE: At startup, the server reads .hcl and .json configuration files from the /vault/config folder. Information passed into VAULT_LOCAL_CONFIG is written into local.json in this directory and read as part of reading the directory for configuration files.

  12. Start consul container with web ui on default port 8500:

    docker run -p 8400:8400 -p 8500:8500 -p 8600:53/udp \
     --hostname consul \
     --name consul progrium/consul \
     -server -bootstrap -ui-dir /ui
    

Binary install

  1. Hashicorp’s steps for installing Vault are at https://vaultproject.io/docs/install.

  2. Installers for a large number of operating systems are downloaded from Hashicorp’s website:

    https://www.vaultproject.io/downloads.html

    • vault_0.7.3_darwin_amd64.zip for Mac 64 expands to a vault app of 59.6 MB.

  3. Verify the SHA256 hash.
  4. On a Mac, drag and drop the vault app file to your root Applications folder.
  5. Set the PATH to Vault.
  6. Double-click on the vault app.

    If you get an error that the binary could not be found, then your PATH environment variable was not setup properly.

    This automated script should install vault at version 0.1.2 into folder:

    /opt/vault_0.1.2

    (the current version for you will likely be different that 0.1.2).

    The installer configures itself by default to listen on localhost port 8200, and registers it as a service called vault-server.

To uninstall, move that folder to trash.

NOTE: Also found vault in chefdk/embedded/lib/ruby/gems/2.5.0/gems/train-1.5.6/lib/train/transports/clients/azure/vault.rb

Verify install

No matter how it was installed:

  1. Open a new Terminal window to Verify:

    vault status
    

    The response:

    Key               Value
    ---                    -----
    Recovery Seal Type     shamir
    Initialized            true
    Sealed                 false
    Total Recovery Shares  5
    Threshold              3
    Version                1.0.2
    ...
    

    Journaling

    Show that secrets are not displayed when using Azure Keyvault:

    sudo journalctl -u 

    Restart Vault

    sudo systemctl restart vault
    

    Start Server

  2. Start the Dev Server per https://www.vaultproject.io/intro/getting-started/dev-server.html

    vault server -dev
    

    PROTIP: This is the command put in a server start-up script.

    Alternately, specific a configuration file in the current folder:

    vault server -config=config-file.hcl
    
  3. Open the web page URL:

    http://127.0.0.1:8200/vault/init
  4. Re-start:

    sudo systemectl restart vault

Unsealing

When a Vault server is started, it starts in a sealed state.

No operations are possible with a Vault that is sealed.

Unsealing is the process of constructing the master key needed to read the decryption key used to decrypt the data.

[3:36] Vault splits the master key into 5 to 10 chars for that many different trustsed people to see a different portion. This is so that all those same people would provide their portion when the master is needed again. CAUTION: The master key should not be stored anywhere.

Alternately, sealing can be done by auto-unseal by using a cloud key from Azure Key Vault, such as this example stanza:

seal "azurekeyvault" {
      tenant_id     = "12345678-1234-1234-1234-1234567890"
      client_id     = "12345678-1234-1234-1234-1234567890"
      client_secret = "DDOU..."
      vault_name    = "hc-vault"
      key_name      = "vault_key"
   }
   
./vault_ unseal af29615803fc23334c3a93f8ad58353b587f50eb0399d23a6950721cbae94948
   

The response confirms:

Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
   

shamir refers to Shamir’s secret sharing algorithm defined at: https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

Higher Key Threshold values would require more key holders to perform unseal with their parts of the key. This provides an additional level of security for accessing data.

Re-Sealing

In case of an emergency, such as:

  • If a secret stored in Vault is leaked - a new secret should be generated and replaced in Vault, with a key rotation following.
  • If vault user credentials are leaked - the user credentials should be revoked and a key rotation should be performed.
  • If vault unseal keys are leaked - a rekey should be performed.

Vault should be sealed immediately to prevent any actions or requests to be performed against the Vault server:

vault seal
   

This buys time to investigate the cause of the issue and to find an appropriate solution.

Principals

Generate principal

  1. Using Azure to obtain a lease:

    vault read azure/cred/reader-role

    Notice “lease_duration”.

    The lease can be renewed by running the command again.

    PROTIP: This should be in a script that incorporates other revocations when someone leaves an organization.

Revoke a lease

To revoke a lease on Azure:

vault lease revoke -prefix azure/creds/reader-role

Configure Vault

VIDEO: HashiCorp Vault on Azure [13:24] by Yoko Hyakuna.

https://github.com/Voronenko/hashi_vault_utils provides command scripts and commentary.

A sample config-file.hcl file:

ui = true
   disable_mlock = true
 
   # use the file backend
   storage "file {
      path = "data"
   }
   listener "tcp" {
      address = "0.0.0.0:8200"
      tls_disable = 1
   }
   

VIDEO: How does Vault encrypt data?

Authorization

To continue working with Vault:

  1. Identify yourself by providing the initial root token using the auth command, such as:

    ./vault_ auth 98df443c-65ee-d843-7f4b-9af8c426128a
    

    The response:

    Successfully authenticated! The policies that are associated
    with this token are listed below:
        
    root
    

    The Access Control policy named “root” policy gives “superuser” level access to everything in Vault.

    As we plan to store secrets for multiple projects, we should be able to clearly separate access to secrets that belong to different projects. And this is where policies do their job.

    Policies in Vault are formatted with HCL, a human-readable configuration format. It is also JSON-compatible. An example policy is shown below:

    path "secret/project/name" {
      policy = "read"
    }
    

Install Consul server

Using Hashicorp’s Consul as a backend to Vault provides the durable storage of encrypted data at rest necessary for fault tolerance, availability, and scalability.

Hashicorp’s Nomad ???


Jenkins plug-in

https://github.com/jenkinsci/hashicorp-vault-plugin is a Jenkins plug-in to establish a build wrapper to populate environment variables from secrets stored in HashiCorp’s Vault. It uses the “App Role” authentication backend which Hashicorp explicitly recommends for machine-to-machine authentication.

The plug-in allows use of a GitHub/GitLab personal access token Github Access Token (https://github.com/blog/1509-personal-api-tokens)

Alternately, a Vault Token - either configured directly in Jenkins or read from an arbitrary file on the Jenkins Machine.

An example in Java is with Java

??? Vault Token Credential, just that the token is read from a file on your Jenkins Machine. You can use this in combination with a script that periodically refreshes your token.

See https://github.com/amarruedo/hashicorp-vault-jenkins

### GitHub Token


   vault auth -method=github token=GITHUB_ACCESS_TOKEN
   

Upon success, a Vault token will be stored at $HOME/.vault-token.

vault list secret/path/to/bucket
   

This uses the token at $HOME/.vault-token if it exists.

See http://chairnerd.seatgeek.com/practical-vault-usage/

https://www.vaultproject.io/intro/getting-started/deploy.html

Handling secrets in CLI

  1. As a demonstration, store the secret value “Pa$$word321” named “donttell”:

    vault write secret/donttell value=Pa$$word321 excited=yes
    

    secret/ is necessary.

    Because commands are stored in shell history, it’s preferred to use files when handling secrets.

  2. Retrieve the secret just added:

    vault read secret/donttell
    

    The response, for example:

    Key                 Value
    ---                 -----
    refresh_interval    768h0m0s
    excited             yes
    value               Pa$$word321
    
  3. Output a secret into a JSON file:

    vault read -format=json secret/donttel
    
    {
     "request_id": "68315073-6658-e3ff-2da7-67939fb91bbd",
     "lease_id": "",
     "lease_duration": 2764800,
     "renewable": false,
     "data": {
         "excited": "yes",
         "value": "Pa$$word321"
     },
     "warnings": null
    }   
  4. Delete a secret:

    vault delete secret/donttel
    
    Success! Deleted 'secret/donttel' if it existed.
    

Rekey

Vault’s rekey command allows for the recreation of unseal keys as well as changing the number of key shares and key threshold. This is useful for adding or removing Vault admins.

Vault’s rotate command is used to change the encryption key used by Vault. This does not require anything other than a root token. Vault will continue to stay online and responsive during a rotate operation.

Store and access secrets within a program

Use libraries for:

  • Python
  • C#
  • Java
  • Node JavaScript

Store Secrets using Hashicorp Vault

The hands-on Katakoda lab Store Secrets using Hashicorp Vault makes use of a vault.hcl file:

backend "consul" {
  address = "consul:8500"
  advertise_addr = "consul:8300"
  scheme = "http"
}
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}
disable_mlock = true

It specifies Consul as the backend to store secrets. Consul runs in HA mode. scheme = “http” should be set to scheme = “https” (use TLS) in production. 0.0.0.0 binds Vault to listen on all IP addresses.

  1. The vault.hcl file is processed by:

    docker create -v /config --name config busybox; docker cp vault.hcl config:/config/;bc973810b4bb77788b37d269b669ba9559a001c5dab7da557c887f7de024d2f0
  2. Launch a single Consul agent:

    docker run -d --name consul \
      -p 8500:8500 \
      consul:v0.6.4 \
      agent -dev -client=0.0.0.0 \
      9b21b47f350931081232d4730341c1221bc086d5bb581bdf06992a334a0c51bf
    

    In production, we’d want to have a cluster of 3 or 5 agents as a single node can lead to data loss.

  3. Launch a single vault-dev container:

    docker run -d --name vault-dev \
    --link consul:consul \
    -p 8200:8200 \
    --volumes-from config \
    cgswong/vault:0.5.3 server -config=/config/vault.hcl
    71f518bb3165313a1e8e8d809e44b0a251dd7c138c5f045d634bae34439d1af7
    

    PROTIP: Volumes are used to hold data.

  4. Create an alias “vault” to proxy commands to vault to the Docker container.

    alias vault='docker exec -it vault-dev vault "$@"'
    export VAULT_ADDR=http://127.0.0.1:8200
    
  5. Initialise the vault so keys go into file keys.txt:

    vault init -address=${VAULT_ADDR} > keys.txt
    cat keys.txt
    

References

“How to Install Vault” on CodeMentor

github.com/dandb/jenkins-provining

VIDEO by Damien Roche at Dun & Bradstreet on 30 April 2017

Kubernetes: Up & Integrated — Secrets & Configuration by Tristan Colgate-McFarlane vault-qubit-895x759-56525

https://www.joyent.com/blog/secrets-management-in-the-autopilotpattern Vault provides encryption at rest for secrets, encrypted communication of those secrets to clients, and role-based access control and auditability for secrets. And it does so while allowing for high-availability configuration with a straightforward single-binary deployment. See the Vault documentation for details on their security and threat model. https://www.vaultproject.io/docs/internals/security.html

Vault uses Shamir’s Secret Sharing to control access to the “first secret” that we use as the root of all other secrets. A master key is generated automatically and broken into multiple shards. A configurable threshold of k shards is required to unseal a Vault with n shards in total.


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Cloud services comparisons (across vendors)
  16. Cloud regions (across vendors)
  17. AWS Virtual Private Cloud

  18. Azure Cloud Onramp
  19. Azure Cloud
  20. Azure Cloud Powershell
  21. Bash Windows using Microsoft’s WSL (Windows Subystem for Linux)

  22. Digital Ocean
  23. Cloud Foundry

  24. Packer automation to build Vagrant images
  25. Terraform multi-cloud provisioning automation
  26. Hashicorp Vault and Consul to generate and hold secrets

  27. Powershell Ecosystem
  28. Powershell on MacOS
  29. Powershell Desired System Configuration

  30. Jenkins Server Setup
  31. Jenkins Plug-ins
  32. Jenkins Freestyle jobs
  33. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  34. Docker (Glossary, Ecosystem, Certification)
  35. Docker Setup
  36. Dockerize apps

  37. Maven on MacOSX

  38. Ansible

  39. MySQL Setup

  40. SonarQube static code scan

  41. API Management Microsoft
  42. API Management Amazon

  43. Scenarios for load