Wilson Mar bio photo

Wilson Mar


Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

How to keep secrets secret, but still shared and refreshed.

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


Here is a hands-on tutorial about how to install and use Hashicorp’s Vault (vaultproject.io) to securely access secret keys and Hashicorp Consul to store key/value pairs. Installation is from scratch on a cloud environment using Docker and docker-compose.

Vault provides “Encryption as a Service” to an enterprise in order to centralize secrets across the organization. Making secret handling easy (automated) enables replacement of long-lived secrets with dynamically generated asymetric X.509 certificates that have a controlled lease period.

The unique contribution of this article is to provide a deep yet concise approach, done by using automation which are then explained. This course assumes participants bring a Mac or Windows laptop and have prior experience with Linux CLI commands.

PROTIP: Where we want to end up is having the system handle secrets. For example, on my desktop I have 1Password, which can I either click “fill” or copy and paste a password without knowing what it is. Great to have when I’m sharing my screen.

At the end of this tutorial, you should be able to:

Install Server binaries

Precompiled Vault binaries are available at https://releases.hashicorp.com/vault

PROTIP: Enterprise and free versions have different binaries. Paid Enterprise editions include Read Replicas and Replication for DR, plus MFA, Sentinel, and HSM Auto-Unseal with FIPS 140-2 & Seal Wrap. A system service file is needed for prod instances.

PROTIP: Vault has a single program file for server and client.

Skill Certification

In 2020 Hashicorp offers a $70 for 1 hour certification exam for Vault.

1 Compare authentication methods

  • Describe authentication methods
  • Choose an authentication method based on use case
  • Differentiate human vs. system auth methods

2 Create Vault policies

  • Illustrate the value of Vault policy
  • Describe Vault policy syntax: path
  • Describe Vault policy syntax: capabilities
  • Craft a Vault policy based on requirements

3 Assess Vault tokens

  • Describe Vault token
  • Differentiate between service and batch tokens. Choose one based on use-case
  • Describe root token uses and lifecycle
  • Define token accessors
  • Explain time-to-live
  • Explain orphaned tokens
  • Create tokens based on need

4 Manage Vault leases

  • Explain the purpose of a lease ID
  • Renew leases
  • Revoke leases

5 Compare and configure Vault secrets engines

  • Choose a secret method based on use case
  • Contrast dynamic secrets vs. static secrets and their use cases
  • Define transit engine
  • Define secrets engines

6 Utilize Vault CLI

  • Authenticate to Vault
  • Configure authentication methods
  • Configure Vault policies
  • Access Vault secrets
  • Enable Secret engines
  • Configure environment variables

7 Utilize Vault UI

  • Authenticate to Vault
  • Configure authentication methods
  • Configure Vault policies
  • Access Vault secrets
  • Enable Secret engines

8 Be aware of the Vault API

  • Authenticate to Vault via Curl
  • Access Vault secrets via Curl

9 Explain Vault architecture

  • Describe the encryption of data stored by Vault
  • Describe cluster strategy
  • Describe storage backends
  • Describe the Vault agent
  • Describe secrets caching
  • Be aware of identities and groups
  • Describe Shamir secret sharing and unsealing
  • Be aware of replication
  • Describe seal/unseal
  • Explain response wrapping
  • Explain the value of short-lived, dynamically generated secrets

10 Explain encryption as a service * Configure transit secret engine

  • Encrypt and decrypt secrets
  • Rotate the encryption key

What are secrets?

A secret is any “clear text” that you want to tightly control access to, such as API keys, passwords, certificates, and more.

Vault provides high-level policy management, secret leasing, audit logging, and automatic revocation.

Vault forces a mandatory lease contract with clients. All secrets read from Vault have an associated lease to enable key usage auditing, perform key rolling, and ensure automatic revocation. Vault provides multiple revocation mechanisms to give operators a clear “break glass” procedure after a potential compromise.

Vault from Hashicorp provides a unified interface to secrets while providing tight access control plus recording a detailed audit log.

Vault is open-sourced at https://github.com/hashicorp/vault with a marketing home page at https://vaultproject.io. It can be deployed to practically any environment, and does not require any special hardware (such as physical HSMs (Hardware Security Modules).

VIDEO: Introduction to HashiCorp Vault Mar 23, 2018 by Armon Dadgar, Hashicorp’s CTO, is a whiteboard talk about avoiding “secret sprawl” living in clear text with empheral (temporary) passwords and cryptographic offload to a central service: hashicorp-vault-dadgar-927x522-94211

It’s really dangerous to keep in GitHub plain-text secrets such as API Keys, etc. This is even if secrets are ecrypted (using GPG) because old versions hidden in history can be decrypted using old keys.


Alternatives to Hashicorp Vault include Vormetrix, Red-Hat Keycloak




Only the storage backend (which durably stores encrypted data) and the HTTP API are outside the barrier that is sealed and unsealed.

When the Vault server is started, it must be provided with a storage backend so that data is available across restarts. The HTTP API similarly must be started by the Vault server on start so that clients can interact with it.



Within App Programming Code

The “12 Factor App” advocates for app programming code to obtain secret data from environment variables rather than hard-coding them in code stored within GitHub.

Populating environment variables with clear-text secrets would occur outside the app, in the run-time environment. Seveal utilities have been created for that.

Daytona client from Cruise

https://github.com/cruise-automation/daytona is written in Golang to be a “lighter, alternative, implementation” Vault client CLI services and containers. It automates authentication, fetching of secrets, and token renewal to Kubernetes, AWS, and GCP. Daytona is performant becuase it pre-fetches secrets upon launch and store them either in environment variables, as JSON in a specified file, or as singular or plural secrets in a specified file.

Basic Course

When given 30-day access to the Vault Basics course, its lessons are for running in dev mode:

  • The Vault CLI - Run the Vault Command Line Interface (CLI).
  • Your First Secret - Run a Vault dev server and write your first secret.


  • The Vault API - Use the Vault HTTP API

    curl http://localhost:8200/v1/sys/health | jq
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{
    "initialized": true,
    "sealed": false,
    "standby": false,
    "performance_standby": false,
    "replication_performance_mode": "disabled",
    "replication_dr_mode": "disabled",
    "server_time_utc": 1591126861,
    "version": "1.2.3",
    "cluster_name": "vault-cluster-2a4c0e97",
    "cluster_id": "0b74ccb6-8cee-83b8-faa6-dc7355481e4b"
    100   294  100   294    0     0  49000      0 --:--:-- --:--:-- --:--:-- 58800
    curl --header "X-Vault-Token: root" http://localhost:8200/v1/secret/data/my-first-secret | jq
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    100   289  100   289    0     0  32111      0 --:--:-- --:--:-- --:--:-- 32111
    "request_id": "1fbb67f5-04a2-5db1-06b4-8210a6959565",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
      "data": {
        "age": "100"
      "metadata": {
        "created_time": "2020-06-02T19:36:39.21280375Z",
        "deletion_time": "",
        "destroyed": false,
        "version": 1
    "wrap_info": null,
    "warnings": null,
    "auth": null
  • Run a Production Server - Configure, run, initialize, and unseal a production mode Vault server.

    Production servers are configured by a vault-config.hcl file (in folder /vault/config) read by server init command

    vault server -config=/vault/config/vault-config.hcl
     vault operator init -key-shares=1 -key-threshold=1

    NOTICE that vault command parameters have a single dash, not a double-dash.


    Save the unseal key response and an initial root token to set the “VAULT_TOKEN” environment variable, using the initial root token that the “init” command returned:

    export VAULT_TOKEN="$root_token"

You next need to unseal your Vault server, providing the unseal key that the “init” command returned: vault operator unseal This will return the status of the server which should show that “Initialized” is “true” and that “Sealed` is “false”.

To check the status of your Vault server at any time, you can run the “vault status” command. If it shows that “Sealed” is “true”, re-run the “vault operator unseal” command.

Finally, log into the Vault UI with your root token. If you have problems, double-check that you ran all of the above commands.

  • Use the KV V2 Secrets Engine - Enable and use an instance of the KV v2 Secrets engine (the default when running in dev mode):

    vault secrets enable -version=2 kv


    vault secrets enable kv-v2
  • Use the Userpass Auth Method - Enable and use the userpass authentication method.

  • Use Vault Policies - Use Vault policies to grant different users access to different secrets. “Least privilege” correspond to HTTP verbs:

    path "secret/dev/team-1/*" {
    capabilities = ["create", "read", "list", "update", "delete"]

NOTE: Labs timeout every 2 hours.

Using Envconsul with GitHub

envconsul from HashiCorp, at https://github.com/hashicorp/envconsul populates values in environment variables referenced within programming code (12-factor applications which get their configuration via the environment).

Envconsul is launched as a subprocess (daemon) which retrieves secrets using REST API calls of KV (Key Value) pairs in Vault/Consul based on “configuration files” specified in the HashiCorp Configuration Language.

It works on many major operating systems with no runtime requirements. On MacOS:

brew install envconsul
   envconsul -v
v0.9.2 ()

For the full list of command-line options:

envconsul -h

Envconsul is also available via a Docker container for scheduled environments.

Secrets are requested based on a specification of secrets to be fetched from Hashicorp Vault based on a configuration file. A sample of its contents is this, which requests the api-key field of the secret at secret/production/third-party:


Credentials authorizing retrieval requests are defined …

Using Vaultenv with GitHub

https://github.com/channable/vaultenv populates values in OS environment variables referenced within programming code by making a syscall from the exec family. Vaultenv replaces its own process with your app. After your service has started, vaultenv is not running anymore.

If secrets in Vault change, Vaultenv does not automatically restart services. By comparison, envconsul from HashiCorp (also describe here), daemonizes and spawns child processes to manage the lifecycle of the process it provides secrets.

Vaultenv retrieves secrets using REST API calls of KV (Key Value) pairs based on “behavior configuration files” specified in the following files traveling with the programming code:

  • $CWD/.env (as popularized by Ruby gems)
  • /etc/vaultenv.conf
  • $HOME/.config/vaultenv/vaultenv.conf

Within its configuration file, secrets are requested based on a specification of secrets to be fetched from Hashicorp Vault, such as this requesting the api-key field of the secret at secret/production/third-party.


The utility is written in the Haskell language under a 3-clause BSD license and releases run on Linux (has not been tested on any other platform, such as macOS).

Alternatives to secret management

  • Environment variables in a clear-text file loaded into operating system variables, such as:

    docker run -e VARNAME=mysecret ...

    “The twelve-factor app stores config in environment variables”.

    However, this is NOT cool anymore because the value of variables (secrets) can end up in logs. All processes have access to secrets (no RBAC).

    And this mechanism makes periodic key rotation manually cumbersome.

  • Docker Secrets was NOT designed for unlicensed (free) standalone containers*, but for Enterprise licensed (paid) Docker Swarm services in commands such as:

    docker service create --secret db_pass --name secret-test alpine bash

    db_pass is a file (with .txt extension) encrypted by a command such as:

    echo "mysecret" | docker secret create db_pass -
     docker secret ls

    Secrets are stored in Docker’s logging infra within its “Raft” distributed leader consensus mechanism shared with Swarm managers. So encryption needs to be locked in Swarm.

    Secrets can be added to a running service, but key rotation requires container restart.

    When the service is created (or updated), the secret is mounted onto the container in the /run/secrets directory which custom program can retrieve*

    def get_secret(secret_name):
          with open('/run/secrets/{0}'.format(secret_name), 'r') as secret_file:
              return secret_file.read()
      except IOError:
          return None
    database_password = get_secret('db_pass')
  • Kubernetes secrets are stored in its etcd process.



  • Cloud-base KMS (Key Management Service) such as from Amazon

  • The Aqua utility provides secrets management to orchestrators so that:

    docker run -it --rm -e SECRET={dev-vault.secret/password} \
     --name ubuntu ubuntu /bin/bash
    docker inspect ubuntu -f ""



Requirements for secret keeping

Coverage of what features a secrets service should have:

  • Installed in sealed mode

  • RBAC (Role-based Access Control) so each user has only the rights for his/her specific role. This has to be enabled in Kubernetes:

  • Limit access to designated containers
  • Encrypted transmission with Mutual authentication (MTLS)
  • Audit logging

  • Change value of an existing secret (key rotation) without rebooting. This is the strong point with Vault.

  • Revocation

Learning Resources


Katacode’s “Store Secrets using Hashicorp Vault” provides a web-based interactive bash terminal.

https://www.vaultproject.io/docs/internals/security/ Security Model

Install Vault server

There are several ways to obtain a running instance of Hashicorp Vault, listed from easiest to most difficult:

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.

See https://github.com/hashicorp/vault-guides and https://devopstales.github.io/linux/hashicorp-vault/

A. Vault cloud service

  • Azure Vault (https://jpvelasco.com/test-driving-the-azure-key-vault-client-samples/)

B. Use Homebrew to install Vault natively on you Mac.

C. Pull an image from Docker Hub

D. Download from Hashicorp to install locally.

E. Use a Dockerfile to build your own Docker image. if you’re not using vault frequently, and want to get the latest when you do.

Cloud service

Vault is an open source tool that can be deployed to any environment. It is well suited for cloud environments where HSMs are either not available or are cost prohibitive.

  1. Create within your internal cloud, Google Cloud, Amazon EC2, Microsoft Azure, etc. a VM instance of an Ubuntu server. 4 GB RAM and 10 GB drive is the minimum.

    A sample command to create a Google Cloud instance:

    gcloud beta compute --project "${THIS_PROJECT_NAME}" instances create "${THIS_INSTANCE_NAME}" --zone "us-central1-f" --machine-type "n1-standard-1" --subnet "default" --maintenance-policy "MIGRATE" --service-account "{$GCP_ACCT}@developer.gserviceaccount.com" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --min-cpu-platform "Automatic" --tags "http","https","web","http-server","https-server" --image "ubuntu-1604-xenial-v20171026a" --image-project "ubuntu-os-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "${THIS_INSTANCE_NAME}"

    If you’re going to be using Vault a lot on your Mac, install using Homebrew:

  2. There are several packages with the name “vault”:

    brew search vault

    Note there are several:

    ==> Formulae
    infosec/core/govaultenv ✔   ssh-vault  vault ✔ vault-cli  vaulted
    ==> Casks
    aws-vault                                                            btcpayserver-vault                                                   gmvault
    If you meant "vault" specifically:
    It was migrated from homebrew/cask to homebrew/core.
  3. Verify the source:

    brew info vault

    You should see:

    vault: stable 1.3.2 (bottled), HEAD
    Secures, stores, and tightly controls access to secrets
    /usr/local/Cellar/vault/1.3.0 (6 files, 124.2MB) *
      Built from source on 2019-11-18 at 05:05:44
    From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/vault.rb
    ==> Dependencies
    Build: go@1.12 ✘, gox ✔
    ==> Options
     Install HEAD version
    ==> Analytics
    install: 7,836 (30 days), 24,016 (90 days), 90,804 (365 days)
    install-on-request: 7,461 (30 days), 22,993 (90 days), 86,629 (365 days)
    build-error: 0 (30 days)
  4. Install pre-requisite Go language:xxx

    brew install vault
  5. Install:

    brew install vault
    => Downloading https://homebrew.bintray.com/bottles/vault-1.3.2.mojave.bottle.tar.gz
    ==> Pouring vault-1.3.2.mojave.bottle.tar.gz
    🍺  /usr/local/Cellar/vault/1.3.2: 6 files, 124.2MB
  6. The great thing with Homebrew is you can upgrade and uninstall easily.

    brew upgrade vault
  7. Verify version

    vault --version

    Vault v1.3.2 was the response at time of writing.

  8. Define the version for use in commands within ~/.bash_profile:

    export VAULT_VERSION="1.3.2"
  9. See menu of commands by running the command without parameters:

    Usage: vault <command> [args]
    Common commands:
     read        Read data and retrieves secrets
     write       Write data, configuration, and secrets
     delete      Delete secrets and configuration
     list        List data or secrets
     login       Authenticate locally
     agent       Start a Vault agent
     server      Start a Vault server
     status      Print seal and HA status
     unwrap      Unwrap a wrapped secret
    Other commands:
     audit          Interact with audit devices
     auth           Interact with auth methods
     debug          Runs the debug command
     kv             Interact with Vault's Key-Value storage
     lease          Interact with leases
     namespace      Interact with namespaces
     operator       Perform operator-specific tasks
     path-help      Retrieve API help for paths
     plugin         Interact with Vault plugins and catalog
     policy         Interact with policies
     print          Prints runtime configurations
     secrets        Interact with secrets engines
     ssh            Initiate an SSH session
     token          Interact with tokens

    Vault commands are described here online.

  10. Install auto completions: https://www.vaultproject.io/docs/commands/#autocompletion

    vault -autocomplete-install

    No message is returned, but it adds to the bottom of your ~/.bash_profile file:

    complete -C /usr/local/bin/vault vault
  11. Restart your terminal

    exec $SHELL
  12. Use autocomplete by typing vault k then press tab to complete.

    vault kv

    Vault kv store commands

    PROTIP: https://www.vaultproject.io/docs/commands/index.html

  13. Add a key:

    vault kv put hello/api username=john
  14. List keys and values:

    vault kv list hello
  15. Retrieve a keys and values:

    vault kv get hello/api 
  16. Delete a key’s metadata:

    vault kv metadata delete hello/api 
  17. Delete a key:

    vault kv delete hello/api 

    Vault secret engine commands

  18. Enable the AWS secrets engine:

    vault secrets enable aws
    Success! Enabled the aws secrets engine at: aws/

    See https://www.vaultproject.io/docs/secrets/kv/kv-v2/

  19. Enable for writing the root account within the AWS secrets engine in the CLI:

    vault write aws/config/root \
     access_key=1234567890abcdefg \
     secret_key=... \


To use Consul as the storage backend, download and install it on each node in the cluster, along with these different stanzas:

storage "consul" {
   address = ""
   path = "vault/"
listener "tcp" {
   address = ""
   cluster_address = "0.0.0.:8201"
   tls_cert_file = "/etc/certs/"
   tls_cert_key = "/etc/certs/vaultkey"
seal "awskms" {
   region = "us-east-1"
   kms_key_id = "f3459282-439a-b233-e210-3487b77c7e2"
api_addr = ""
ui = true
cluster_name = "my_cluster"
log_level = "info"


Build Docker image using Dockerfile

Create Vault within a Docker image from scratch:


  1. Install Git in the Linux server:

    apt-get update && apt-get install -y \
  2. Use Git to obtain the Dockerfile based on the Spring Cloud Vault sample app

    git clone https://github.com/???/vault.git --depth=1 
    cd vault
  3. Create a docker image locally:

    sudo docker build -f Dockerfile -t demo:vault . 

    This would run Maven, and a test job.

    If you get a message: “unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /Users/…/projects/vault/Dockerfile: no such file or directory

  4. Run the Dockerfile at:


    Its contains:

    FROM ubuntu:16.04
    RUN apt-get update
    RUN apt-get update && apt-get install -y \
      default-jre \
      default-jdk \
      git \
    RUN mvn -version
    RUN git clone https://github.com/hashicorp/vault???.git --depth=1

    The above provides commands to install Vault within a blank Docker container.

    Vault-jvm/examples/sample-app is a simple sample app, which is replaced with a real app in the real world.

Use Docker image

On a Linux server instance’s Terminal CLI:

  1. Add Docker’s public GPG key :

    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

    OK is the expected response.

  2. View the Linux version code referenced in a later command:

    lsb_release -cs

    This returns stretch for Debinan and xenial for Ubuntu.

  3. Install Docker for Ubuntu (not Debian):

    sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
  4. Update repository:

    sudo apt-get update
  5. List policies:

    apt-cache policy docker-ce

    The response:

      Installed: (none)
      Candidate: 17.09.0~ce-0~ubuntu
      Version table:
      17.09.0~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.06.2~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.06.1~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.06.0~ce-0~ubuntu 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.03.2~ce-0~ubuntu-xenial 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.03.1~ce-0~ubuntu-xenial 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
      17.03.0~ce-0~ubuntu-xenial 500
         500 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
  6. Install Docker Community Edition:

    sudo apt-get install -y docker-ce

    Sample response:

    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following additional packages will be installed:
      aufs-tools cgroupfs-mount libltdl7
    Suggested packages:
    The following NEW packages will be installed:
      aufs-tools cgroupfs-mount docker-ce libltdl7
    0 upgraded, 4 newly installed, 0 to remove and 17 not upgraded.
    Need to get 21.2 MB of archives.
    After this operation, 100 MB of additional disk space will be used.
    Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
    Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4,970 B]
    Get:3 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB]
    Get:4 https://download.docker.com/linux/ubuntu xenial/stable amd64 docker-ce amd64 17.09.0~ce-0~ubuntu [21.0 MB]
    Fetched 21.2 MB in 0s (22.7 MB/s)     
    Selecting previously unselected package aufs-tools.
    (Reading database ... 66551 files and directories currently installed.)
    Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ...
    Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
    Selecting previously unselected package cgroupfs-mount.
    Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...
    Unpacking cgroupfs-mount (1.2) ...
    Selecting previously unselected package libltdl7:amd64.
    Preparing to unpack .../libltdl7_2.4.6-0.1_amd64.deb ...
    Unpacking libltdl7:amd64 (2.4.6-0.1) ...
    Selecting previously unselected package docker-ce.
    Preparing to unpack .../docker-ce_17.09.0~ce-0~ubuntu_amd64.deb ...
    Unpacking docker-ce (17.09.0~ce-0~ubuntu) ...
    Processing triggers for libc-bin (2.23-0ubuntu9) ...
    Processing triggers for man-db (2.7.5-1) ...
    Processing triggers for ureadahead (0.100.0-19) ...
    Processing triggers for systemd (229-4ubuntu20) ...
    Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
    Setting up cgroupfs-mount (1.2) ...
    Setting up libltdl7:amd64 (2.4.6-0.1) ...
    Setting up docker-ce (17.09.0~ce-0~ubuntu) ...
    Processing triggers for libc-bin (2.23-0ubuntu9) ...
    Processing triggers for systemd (229-4ubuntu20) ...
    Processing triggers for ureadahead (0.100.0-19) ...
  7. List Docker container status:

    sudo systemctl status docker

    The response:

    ● docker.service - Docker Application Container Engine
    Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Active: active (running) since Sat 2017-11-04 22:00:35 UTC; 1min 28s ago
      Docs: https://docs.docker.com
     Main PID: 13524 (dockerd)
    CGroup: /system.slice/docker.service
            ├─13524 /usr/bin/dockerd -H fd://
            └─13544 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.552925012Z" level=warning msg="Your kernel does not support swap me
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.553123462Z" level=warning msg="Your kernel does not support cgroup 
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.553267498Z" level=warning msg="Your kernel does not support cgroup 
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.554662024Z" level=info msg="Loading containers: start."
    Nov 04 22:00:34 vault-1 dockerd[13524]: time="2017-11-04T22:00:34.973517284Z" level=info msg="Default bridge (docker0) is assigned wi
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.019418706Z" level=info msg="Loading containers: done."
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.029599857Z" level=info msg="Docker daemon" commit=afdb6d4 graphdriv
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.029962340Z" level=info msg="Daemon has completed initialization"
    Nov 04 22:00:35 vault-1 systemd[1]: Started Docker Application Container Engine.
    Nov 04 22:00:35 vault-1 dockerd[13524]: time="2017-11-04T22:00:35.054191848Z" level=info msg="API listen on /var/run/docker.sock"
    log files:
  8. Verify Docker version in case you need to troubleshoot:

    docker --version

    The response:

    Docker version 17.09.0-ce, build afdb6d4
  9. Download the Docker image maintained by Hashicorp:

    docker pull vault
    ### Alternate Docker images
    has Hashicorp Vault on a minimal Alpine Linux box
    has Hashicorp Vault on a tiny busybox
  10. Set environment variables so IP addresses used for the redirect and cluster addresses in Vault’s configuration is the address of the named interface inside the container (e.g. eth0):

  11. Run the image using the file storage backend at path /vault/file, with a default secret lease duration of one week and a maximum of (720h/24) 30 days:

    docker run --cap-add=IPC_LOCK -e 'VAULT_LOCAL_CONFIG={"backend": {"file": {"path": "/vault/file"}}, "default_lease_ttl": "168h", "max_lease_ttl": "720h"}' vault server

    --cap-add=IPC_LOCK: locks memory, which prevents it from being swapped to disk (and thus exposing keys).

    See https://www.vaultproject.io/docs/config/index.html

    NOTE: At startup, the server reads .hcl and .json configuration files from the /vault/config folder. Information passed into VAULT_LOCAL_CONFIG is written into local.json in this directory and read as part of reading the directory for configuration files.

  12. Start consul container with web ui on default port 8500:

    docker run -p 8400:8400 -p 8500:8500 -p 8600:53/udp \
     --hostname consul \
     --name consul progrium/consul \
     -server -bootstrap -ui-dir /ui

Binary install

  1. Hashicorp’s steps for installing Vault are at https://vaultproject.io/docs/install.

  2. Installers for a large number of operating systems are downloaded from Hashicorp’s website:


    • vault_0.7.3_darwin_amd64.zip for Mac 64 expands to a vault app of 59.6 MB.

  3. Verify the SHA256 hash.
  4. On a Mac, drag and drop the vault app file to your root Applications folder.
  5. Set the PATH to Vault.
  6. Double-click on the vault app.

    If you get an error that the binary could not be found, then your PATH environment variable was not setup properly.

    This automated script should install vault at version 0.1.2 into folder:


    (the current version for you will likely be different that 0.1.2).

    The installer configures itself by default to listen on localhost port 8200, and registers it as a service called vault-server.

To uninstall, move that folder to trash.

NOTE: Also found vault in chefdk/embedded/lib/ruby/gems/2.5.0/gems/train-1.5.6/lib/train/transports/clients/azure/vault.rb

Verify install

No matter how it was installed:

  1. Open a new Terminal window to Verify:

    vault status

    The response:

    Key               Value
    ---                    -----
    Recovery Seal Type     shamir
    Initialized            true
    Sealed                 false
    Total Recovery Shares  5
    Threshold              3
    Version                1.0.2


    Show that secrets are not displayed when using Azure Keyvault:

    sudo journalctl -u 

    Restart Vault

    sudo systemctl restart vault

    Start Server

  2. Start the Dev Server per https://www.vaultproject.io/intro/getting-started/dev-server.html

    vault server -dev

    PROTIP: This is the command put in a server start-up script.

    Alternately, specific a configuration file in the current folder:

    vault server -config=config-file.hcl
  3. Open the web page URL:
  4. Re-start:

    sudo systemectl restart vault


When a Vault server is started, it starts in a sealed state.

No operations are possible with a Vault that is sealed.

Unsealing is the process of constructing the master key needed to read encryption key to encrypt data and decryption key used to decrypt data.

PROTIP: Decryption keys are stored with data, in a form encrypted by a master key.

[3:36] Vault splits the master key into 5 to 10 chars for that many different trusted people to see a different portion. This is so that all those same people would provide their portion when the master is needed again. CAUTION: The master key should not be stored anywhere but in memory.

Alternately, sealing can be done by auto-unseal by using a cloud key from Azure Key Vault, such as this example stanza:

seal "azurekeyvault" {
      tenant_id     = "12345678-1234-1234-1234-1234567890"
      client_id     = "12345678-1234-1234-1234-1234567890"
      client_secret = "DDOU..."
      vault_name    = "hc-vault"
      key_name      = "vault_key"
./vault_ unseal af29615803fc23334c3a93f8ad58353b587f50eb0399d23a6950721cbae94948

The response confirms:

Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0

Shamir refers to the Shamir secret sharing algorithm defined at: https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

Higher Key Threshold values would require more key holders to perform unseal with their parts of the key. This provides an additional level of security for accessing data.


In case of an emergency, such as:

  • If a secret stored in Vault is leaked - a new secret should be generated and replaced in Vault, with a key rotation following.
  • If vault user credentials are leaked - the user credentials should be revoked and a key rotation should be performed.
  • If vault unseal keys are leaked - a rekey should be performed.

Vault should be sealed immediately to prevent any actions or requests to be performed against the Vault server:

vault seal

This buys time to investigate the cause of the issue and to find an appropriate solution.


Generate principal

  1. Using Azure to obtain a lease:

    vault read azure/cred/reader-role

    Notice “lease_duration”.

    The lease can be renewed by running the command again.

    PROTIP: This should be in a script that incorporates other revocations when someone leaves an organization.

Revoke a lease

To revoke a lease on Azure:

vault lease revoke -prefix azure/creds/reader-role

Vault on AWS

There are several options for hosting Vault.

https://aws.amazon.com/quickstart/architecture/vault describes “A unified interface to manage and encrypt secrets on the AWS Cloud”.

hashicorp-vault-on-aws-architecture c47a3bf846dc964bb4464471a764b26f1b0d9639

Install Vault within AWS EKS cluster

Hashicorp announced a Helm chart to setup Vault in Kubernetes



/Users/wilson_mar/Library/Python/3.7/lib/python/site-packages/ansible/modules/cloud/amazon To the templates we would need to add monitoring/Observability, SIEM, etc.

There is a Consul provider helm chart

Configure Vault

VIDEO: HashiCorp Vault on Azure [13:24] by Yoko Hyakuna.

https://github.com/Voronenko/hashi_vault_utils provides command scripts and commentary.

A sample config-file.hcl file:

ui = true
   disable_mlock = true
   # use the file backend
   storage "file {
      path = "data"
   listener "tcp" {
      address = ""
      tls_disable = 1

VIDEO: How does Vault encrypt data?


To continue working with Vault:

  1. Identify yourself by providing the initial root token using the auth command, such as:

    ./vault_ auth 98df443c-65ee-d843-7f4b-9af8c426128a

    The response:

    Successfully authenticated! The policies that are associated
    with this token are listed below:

    The Access Control policy named “root” policy gives “superuser” level access to everything in Vault.

    As we plan to store secrets for multiple projects, we should be able to clearly separate access to secrets that belong to different projects. And this is where policies do their job.

    Policies in Vault are formatted with HCL, a human-readable configuration format. It is also JSON-compatible. An example policy is shown below:

    path "secret/project/name" {
      policy = "read"

Install Consul server

Using Hashicorp’s Consul as a backend to Vault provides the durable storage of encrypted data at rest necessary for fault tolerance, availability, and scalability.

Hashicorp’s Nomad ???

Jenkins plug-in

https://github.com/jenkinsci/hashicorp-vault-plugin is a Jenkins plug-in to establish a build wrapper to populate environment variables from secrets stored in HashiCorp’s Vault. It uses the “App Role” authentication backend which Hashicorp explicitly recommends for machine-to-machine authentication.

The plug-in allows use of a GitHub/GitLab personal access token Github Access Token (https://github.com/blog/1509-personal-api-tokens)

Alternately, a Vault Token - either configured directly in Jenkins or read from an arbitrary file on the Jenkins Machine.

An example in Java is with Java

??? Vault Token Credential, just that the token is read from a file on your Jenkins Machine. You can use this in combination with a script that periodically refreshes your token.

See https://github.com/amarruedo/hashicorp-vault-jenkins

### GitHub Token

   vault auth -method=github token=GITHUB_ACCESS_TOKEN

Upon success, a Vault token will be stored at $HOME/.vault-token.

vault list secret/path/to/bucket

This uses the token at $HOME/.vault-token if it exists.

See http://chairnerd.seatgeek.com/practical-vault-usage/


Handling secrets in CLI

  1. As a demonstration, store the secret value “Pa$$word321” named “donttell”:

    vault write secret/donttell value=Pa$$word321 excited=yes

    secret/ is necessary.

    Because commands are stored in shell history, it’s preferred to use files when handling secrets.

  2. Retrieve the secret just added:

    vault read secrets/apps/web/username
    vault read secrets/apps/portal/username
    vault read secrets/common/api_key
    vault read secret/donttell

    The response, for example:

    Key                 Value
    ---                 -----
    refresh_interval    768h0m0s
    excited             yes
    value               Pa$$word321
  3. Output a secret into a JSON file:

    vault read -format=json secret/donttel
     "request_id": "68315073-6658-e3ff-2da7-67939fb91bbd",
     "lease_id": "",
     "lease_duration": 2764800,
     "renewable": false,
     "data": {
         "excited": "yes",
         "value": "Pa$$word321"
     "warnings": null
  4. Delete a secret:

    vault delete secret/donttel
    Success! Deleted 'secret/donttel' if it existed.


Vault’s rekey command allows for the recreation of unseal keys as well as changing the number of key shares and key threshold. This is useful for adding or removing Vault admins.

Vault’s rotate command is used to change the encryption key used by Vault. This does not require anything other than a root token. Vault will continue to stay online and responsive during a rotate operation.

Store and access secrets within a program

Use libraries for:

  • Python
  • C#
  • Java
  • Node JavaScript
  • Golang

Several Vault clients have been written.



Vault CLI Katakoda hands-on lab

The hands-on Katakoda lab Store Secrets using Hashicorp Vault makes use of a vault.hcl file:

backend "consul" {
  address = "consul:8500"
  advertise_addr = "consul:8300"
  scheme = "http"
listener "tcp" {
  address = ""
  tls_disable = 1
disable_mlock = true

It specifies Consul as the backend to store secrets. Consul runs in HA mode. scheme = “http” should be set to scheme = “https” (use TLS) in production. binds Vault to listen on all IP addresses.

  1. The vault.hcl file is processed by:

    docker create -v /config --name config busybox; docker cp vault.hcl config:/config/;bc973810b4bb77788b37d269b669ba9559a001c5dab7da557c887f7de024d2f0
  2. Launch a single Consul agent:

    docker run -d --name consul \
      -p 8500:8500 \
      consul:v0.6.4 \
      agent -dev -client= \

    In production, we’d want to have a cluster of 3 or 5 agents as a single node can lead to data loss.

  3. Launch a single vault-dev container:

    docker run -d --name vault-dev \
    --link consul:consul \
    -p 8200:8200 \
    --volumes-from config \
    cgswong/vault:0.5.3 server -config=/config/vault.hcl

    PROTIP: Volumes are used to hold data.

  4. Create an alias “vault” to proxy commands to vault to the Docker container.

    alias vault='docker exec -it vault-dev vault "$@"'
    export VAULT_ADDR=
  5. Initialise the vault so keys go into file keys.txt:

    vault init -address=${VAULT_ADDR} > keys.txt
    cat keys.txt

CA for SSH

Vault can serve as a Root or Intermediate Certificate Authority.


“How to Install Vault” on CodeMentor


VIDEO by Damien Roche at Dun & Bradstreet on 30 April 2017

Kubernetes: Up & Integrated — Secrets & Configuration by Tristan Colgate-McFarlane vault-qubit-895x759-56525

https://www.joyent.com/blog/secrets-management-in-the-autopilotpattern Vault provides encryption at rest for secrets, encrypted communication of those secrets to clients, and role-based access control and auditability for secrets. And it does so while allowing for high-availability configuration with a straightforward single-binary deployment. See the Vault documentation for details on their security and threat model. – See https://www.vaultproject.io/docs/internals/security.html

Vault uses Shamir’s Secret Sharing to control access to the “first secret” that we use as the root of all other secrets. A master key is generated automatically and broken into multiple shards. A configurable threshold of k shards is required to unseal a Vault with n shards in total.

namic Credentials and Encryption as a data service, and “Policy as Code” vs “Secrets as Code.”

VIDEO COURSE: Getting Started with HashiCorp Vault by Bryan Krausen (@btkrausen)





  1. See whether “Type” of secrets engines “database” are enabled:

    vault secrets list

  2. Enable the Database secrets engine on the Vault server.

    vault secrets enable -path=lob_a/workshop/database database

    The expected response include “Success! Enabled the database secrets engine at: lob_a/workshop/database/

Vault’s Database secrets engine dynamically generates credentials (username and password) for many databases.

In this challenge, you will configure the database secrets engine you enabled in the previous challenge on the path lob_a/workshop/database to work with the local instance of the MySQL database. We use a specific path rather than the default “database” to illustrate that multiple instances of the database secrets engine could be configured for different lines of business that might each have multiple databases.

  1. Configure the Database Secrets Engine on the Vault server.

All secrets engines must be configured before they can be used.

We first need to configure the database secrets engine to use the MySQL database plugin and valid connection information. We are configuring a database connection called “wsmysqldatabase” that is allowed to use two roles that we will create below.

vault write lob_a/workshop/database/config/wsmysqldatabase \
  plugin_name=mysql-database-plugin \
  connection_url=":@tcp(localhost:3306)/" \
  allowed_roles="workshop-app","workshop-app-long" \
  username="hashicorp" \

This will not return anything if successful.

Note that the username and password are templated in the “connection_url” string, getting their values from the “username” and “password” fields. We do this so that reading the path “lob_a/workshop/database/config/wsmysqldatabase” will not show them.

To test this, try running this command:

vault read lob_a/workshop/database/config/wsmysqldatabase

ey Value — —– allowed_roles [workshop-app workshop-app-long] connection_details map[connection_url::@tcp(localhost:3306)/ username:hashicorp] plugin_name mysql-database-plugin root_credentials_rotate_statements []

You will not see the username and password.

We used the initial MySQL username “hashicorp” and password “Password123” above. Validate that you can login to the MySQL server with this command:

mysql -u hashicorp -pPassword123 You should be given a mysql> prompt.

Logout of the MySQL server by typing \q at the mysql> prompt. This should return you to the root@vault-mysql-server:~# prompt.

We can make the configuration of the database secrets engine even more secure by rotating the root credentials (actually just the password) that we passed into the configuration. We do this by running this command:

vault write -force lob_a/workshop/database/rotate-root/wsmysqldatabase This should return “Success! Data written to: lob_a/workshop/database/rotate-root/wsmysqldatabase”.

Now, if you try to login to the MySQL server with the same command given above, it should fail and give you the message “ERROR 1045 (28000): Access denied for user ‘hashicorp’@’localhost’ (using password: YES)”. Please verify that:

mysql -u hashicorp -pPassword123 Note: You should not use the actual root user of the MySQL database (despite the reference to “root credentials”); instead, create a separate user with sufficient privileges to create users and to change its own password.

Now, you should create the first of the two roles we will be using, “workshop-app-long”, which generates credentials with an initial lease of 1 hour that can be renewed for up to 24 hours.

vault write lob_a/workshop/database/roles/workshop-app-long
creation_statements=”CREATE USER ‘’@’%’ IDENTIFIED BY ‘‘;GRANT ALL ON my_app.* TO ‘’@’%’;”
max_ttl=”24h” This should return “Success! Data written to: lob_a/workshop/database/roles/workshop-app-long”.

And then create the second role, “workshop-app” which has shorter default and max leases of 3 minutes and 6 minutes. (These are intentionally set long enough so that you can use the credentials generated for the role to connect to the database but also see them expire in the next challenge.)

vault write lob_a/workshop/database/roles/workshop-app
creation_statements=”CREATE USER ‘’@’%’ IDENTIFIED BY ‘‘;GRANT ALL ON my_app.* TO ‘’@’%’;”
max_ttl=”6m” This should return “Success! Data written to: lob_a/workshop/database/roles/workshop-app”.

The database secrets engine is now configured to talk to the MySQL server and is allowed to create users with two different roles. In the next challenge, you’ll generate credentials (username and password) for these roles.

  • Generate and use dynamic database credentials for the MySQL database.

  • Renew and revoke database credentials for the MySQL database.

https://www.vaultproject.io/docs/secrets/databases/mysql-maria/ https://www.vaultproject.io/docs/secrets/databases/#usage https://www.vaultproject.io/api/secret/databases/#generate-credentials

Generate dynamic credentials for a MySQL database from Vault.


Alternative: Environment variables

https://www.youtube.com/watch?v=IolxqkL7cD8 Hiding passwords in environment variables on Windows

import os
db_user = os.environ.get('DB_USER')
db_password = os.environ.get('DB_PASS')

Alternative: Google Secret Manager



Alternative: JupyterLab Credential Store


Alternative: python-dotenv

Vicki Boykis blogged about the alternatives, which includes this for Jupyter notebook coders:

%load_ext dotenv
import os

“dotenv” is from python-dotenv at

It retrieves an .env file created to define your project’s secret environment variables, using the package’s command line tool) at

That .env file name is specified in the .gitignore so it is ignored when pushing to github.

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering

More on Security

This is one of a series on Security:

  1. Git Signing
  2. Hashicorp Vault

  3. WebGoat known insecure PHP app and vulnerability scanners
  4. Test for OWASP using ZAP on the Broken Web App

  5. Encrypt all the things

  6. AWS Security (certification exam)
  7. AWS IAM (Identity and Access Management)

  8. Cyber Security
  9. Security certifications