Wilson Mar bio photo

Wilson Mar


Calendar YouTube Github


Secure SSH accessing servers in AWS and other clouds: implement Zero-Trust with granular control of least-privilege just-in-time access using HashiCorp Boundary

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


The world is rushing to HashiCorp’s Boundary due to the need to install Zero-Trust Networking, especially within the US federal government in repsonse to the White House Executive Order in 2020.

Zero-Trust is about shifting from obsolete assumptions, processes, and tools. For example:

“What were we thinking creating a system that leaves long-term SSH keys around to be stolen?”

Since its announcement on October 2020, HashiCorp’s Boundary is revolutionizing how we connect remotely into servers (such as PostgreSQL, MySQL, etc.) within AWS, Azure, Google, IBM, and other clouds.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Why? The problem addressed

HashiCorp Boundary beats vpn and bastion hosts 1466x610.jpg

No more Bastion Host Jump Box

The now “traditional” (passe security-wise) is to go through a Linux SSH bastion host, with a command such as:

    ssh -L

“What were we thinking when we create Bastion hosts – essentially paying for an additonal server for hackers to access?

Bastion host, users are not differentiated because the host provides “all or nothing” access.

And SSH takes some work to setup securely - users would need to mess with another set of secrets to a specific IP address.

No more VPN tunnels

VPNs improves on Bastion hosts in that VPNs controls access by each user’s IP address. VPNs are difficult to install by each user. And they’re expensive, introduce bandwidth limits, and don’t scale well. Multiple VPNs are difficult to setup, and even more costly.

That’s why Boundary manages the “identity” of each user.

Solution: Identity-based

Boundary is called “Software-Defined Perimeter (SDP)” software because it provides secure access into a private network’s targets (app server endpoints) you want to access.

VIDEO:: HashiCorp’s Boundary authenticates & authorizes based on the identity name of a server rather than its IP address and port number. (via OIDC protocol) with a 3rd-party IdP (trusted identity provider such as Okta, GitHub, Auth0, AWS, Azure, GCP, etc.).

Boundary automates service discovery as workloads are created or changed dynamically.

Additionally, Boundary provides Just-In-Time network access. A user may be allowed access to a server for only, say, 5-minute session the same day. Credentials provided by Boundary are temporary, so it can’t be lost like a static secret. And if lost, damage is limited.

Boundary maintains a Dynamic Host Catalog of hosts and targets (in a Postgres database).

In other words, HashiCorp’s Boundary is an intelligent proxy.

Different ways to run Boundary Server

There are different ways:

SaaS Boundary on SaaS HCP

Boundary on HCP (SaaS service managed by HashiCorp) went (free) beta preview July 2022. HCP Pricing: HCP Boundary is FREE for the first 50 sessions each month, using less than 1GB of data transfer per session. Beyond that, you are charged $.50 per session and $.50/per additional GB of session transfer data.

  1. Create a HCP account at


  2. On the left HCP menu, click Services: Boundary.
  3. Click the blue “Deploy Boundary”.

    HCP Boundary init

  4. Define a Boundary name according to a convention such as:


  5. Define an Administrator account name (according to a convention)


  6. Define a strong password (more than 20 characters)
  7. Click “Deploy”.

  8. Copy and save the Admin UI, such as:


  9. Copy the Cluster URL, such as:


    This is the address registered users would provide to gain access.

Install for local usage

TODO: I’m working on a shell file that does the following with one command.

  1. In a browser specify URL:


    Notice that HashiCorp supports a wide range of modern and legacy operating systems:
    macOS, Windows, Linux, FreeBSD, NetBSD, OpenBSD, Solaris

    PROTIP: The Boundary executable contains functionality to be used as a client, worker, server, in Dev mode

    Boundary consists of two server components: Controllers, which serve the API and coordinate session requests; and Workers, which perform the actual session handling. A normal Boundary installation will consist of one or more Controllers paired with one or more Workers. A single Boundary binary can act in either of these two modes.

    HashiCorp has created executables for both CLI and <a href=”#GUI app.

    CLI executable install

  2. On macOS, HashiCorp doesn’t provide a “brew install boundary”, so:

    brew upgrade
    brew tap hashicorp/tap
    brew install hashicorp/tap/boundary

    NOTE: The installer recognizes whether you have a M1/M2 ARM or an Intel machine.

  3. Install autocompletion to $HOME/.bash_profile and $HOME/.zshrc so this only needs to be done once:

    boundary config autocomplete install

    There is no response to that unless an error occures, such as running it more than once.

  4. The above would enable you to press tab once or twice after typing:

    accounts              credentials           managed-groups
    auth-methods          database              roles
    auth-tokens           dev                   scopes
    authenticate          groups                server
    config                host-catalogs         sessions
    connect               host-sets             targets
    credential-libraries  hosts                 users
    credential-stores     logout                workers
  5. Get the program’s version:

    boundary version


    boundary -v
    Version information:
      Git Revision:        02e410af7a2606ae242b8637d8a02754f0a5f43e
      Version Number:      0.11.2

    PROTIP: CLI code to get just the version (“0.11.2”) is:

    BOUNDARY_VERSION=$( boundary -v | grep "Version" | awk '{print $3}' )

    NOTE: sed -n ‘4p’ is a (counter-intuitive) alternative to grep.

  6. View the latest and previous releases and its SHA for the git commit creating a release at:


    CLI Shortcuts

  7. Alternately, if you prefer less typing, for quicker invocation of “boundary”, set an alias to use “bdy” instead

    alias bdy="boundary"
  8. That would enable you to spend less time typing:

    bdy version

    Command List

  9. Use the alias to get a list of commands (and functionality from Boundary):

    bdy --help

    A menu of concepts in alphabetical order:

    Usage: boundary <command> [args]
     accounts                  Manage Boundary accounts
     auth-methods              Manage Boundary auth methods
     auth-tokens               Manage Boundary auth tokens
     authenticate              Authenticate the Boundary command-line client
     config                    Manage resources related to Boundary's local configuration
     connect                   Connect to a target through a Boundary worker
     credential-libraries      Manage Boundary credential librarys
     credential-stores         Manage Boundary credential stores
     credentials               Manage Boundary credentials
     database                  Manage Boundary's database
     dev                       Start a Boundary dev environment
     groups                    Manage Boundary groups
     host-catalogs             Manage Boundary host catalogs
     host-sets                 Manage Boundary host sets
     hosts                     Manage Boundary hosts
     logout                    Delete the current token within Boundary and forget it locally
     managed-groups            Manage Boundary managed groups
     roles                     Manage Boundary roles
     scopes                    Manage Boundary scopes
     server                    Start a Boundary server
     sessions                  Manage Boundary sessions
     targets                   Manage Boundary targets
     users                     Manage Boundary users
     workers                   Manage Boundary workers

    Documentation online

    Clicking the question mark icon brings you to

Connection Protocol Wrappers

Boundary comes with built-in wrappers for popular layer 7 connection protocols, such as:

  • ssh: defaults to the local SSH client (ssh)
  • postgres: defaults to the official Postgres CLI client (psql)
  • rdp: defaults to the built-in Windows RDP client (mstsc)
  • http: defaults to curl
  • kube: defaults to kubectl

### Postgres database in Docker

  1. Install Docker Desktop and have it running on its default port.

  2. Make use of the Docker container image in Docker Hub (https://hub.docker.com/_/postgres)

    docker run --name postgres-db -e POSTGRES_PASSWORD=docker -p 5432:5432 -d postgres

    This prevents this error message when you do boundary dev :

    Error creating dev database container: unable to start dev database with dialect postgres: could not start resource: : dial unix /var/run/docker.sock: connect: connection refused

    Boundary dev mode controller

  3. Instantiate a Boundary controller-mode process locally on a Terminal session using default parameter values:

    boundary dev

    Sample reponse:

    ==> Boundary server configuration:
         [Controller] AEAD Key Bytes: vL7W9ben+hLr5vbwGl1H+9Sr1Psp38bc
           [Recovery] AEAD Key Bytes: gmHrDl6zSd8NtAe5+IhHBUbX9GZD86sV
        [Worker-Auth] AEAD Key Bytes: JpmOdRcp7RHwyVXOYb7r+dRQHsn/V9KU
                [Recovery] AEAD Type: aes-gcm
                    [Root] AEAD Type: aes-gcm
     [Worker-Auth-Storage] AEAD Type: aes-gcm
             [Worker-Auth] AEAD Type: aes-gcm
                                 Cgo: disabled
      Controller Public Cluster Addr:
              Dev Database Container: sleepy_wiles
                    Dev Database Url: postgres://postgres:password@localhost:55000/boundary?sslmode=disable
          Generated Admin Login Name: admin
            Generated Admin Password: password
           Generated Host Catalog Id: hcst_1234567890
                   Generated Host Id: hst_1234567890
               Generated Host Set Id: hsst_1234567890
       Generated Oidc Auth Method Id: amoidc_1234567890
              Generated Org Scope Id: o_1234567890
      Generated Password Auth Method Id: ampw_1234567890
          Generated Project Scope Id: p_1234567890
                 Generated Target Id: ttcp_1234567890
      Generated Unprivileged Login Name: user
     Generated Unprivileged Password: password
                          Listener 1: tcp (addr: "", cors_allowed_headers: "[]", cors_allowed_origins: "[*]", cors_enabled: "true", max_request_duration: "1m30s", purpose: "api")
                          Listener 2: tcp (addr: "", max_request_duration: "1m30s", purpose: "cluster")
                          Listener 3: tcp (addr: "", max_request_duration: "1m30s", purpose: "ops")
                          Listener 4: tcp (addr: "", max_request_duration: "1m30s", purpose: "proxy")
                           Log Level: info
                               Mlock: supported: false, enabled: false
                             Version: Boundary v0.11.0
                         Version Sha: 1d42091e81ca11353376ce116275890e3ae67f6b
          Worker Auth Current Key Id: auction-acutely-shawl-bonanza-semifinal-portal-worry-bodacious
            Worker Auth Storage Path: /var/folders/rq/rvb3xv916b976fm4sszjym400000gq/T/nodeenrollment3782666038
            Worker Public Proxy Addr:
    ==> Boundary server started! Log data will stream in below:

    WARNING: Notice in the output above that Boundary in dev mode by default uses an (insecure) login name ‘admin’ and password ‘password’.

    VIDEO: Alternately, specify parameters:

    boundary dev \
    -api-listen-address= \
    -cluster-listen-address= \
    -proxy-listen-address= \

    Alternately, run in background with the &:

    boundary dev -database-url=postgres://postgres:postgres@boundary-database:5432/postgres?sslmode=disable \
    -cluster-listen-address= \
    -api-listen-address= &
  4. Create a new Terminal window if you want to make any more CLI commands.

    BOUNDARY_ADDR URL with port

  5. Notice in the output above the Listener IP address, which defines the Boundary Controller URL address, with its standard port:

    export BOUNDARY_ADDR=""

    If instead the Boundary Controller is on a server:

    export BOUNDARY_ADDR=""


    PROTIP: Craft and use a shell script so you don’t have to remember this.

  6. Get a reminder:

    boundary authenticate help
    Usage: boundary authenticate [sub command] [options] [args]
      This command authenticates the Boundary commandline client using a specified auth
      method. Examples:
     Authenticate with a password auth method:
       $ boundary authenticate password -auth-method-id ampw_1234567890 -login-name foo
     Authenticate with an OIDC auth method:
       $ boundary authenticate oidc -auth-method-id amoidc_1234567890
      Please see the auth method subcommand help for detailed usage information.
     oidc        Invoke the OIDC auth method to authenticate with Boundary
     password    Invoke the password auth method to authenticate with Boundary
  7. Authenticate!

    boundary authenticate password \
      -auth-method-id=ampw_1234567890 \
      -login-name=admin \
      -password=password \

    If you see this error message:

    Password flag must be used with env:// or file:// syntax or left empty for an interactive prompt

    The expected response is something like this:

          Authentication information:
         Account ID:      apw_BPPNtEX82N
         Auth Method ID:  ampw_1234567890
         Expiration Time: Wed, 14 Oct 2020 18:30:40 PDT
         User ID:         u_1234567890

Boundary.app GUI

  1. To install the Desktop client, click the .dmg (64-bit) on macOS.

    Drag the Boundary.app icon and drop on the app folder at:

  2. If you access it often, drag the icon and drop it among others.

  3. Invoke the app by double-clicking or

  4. Type the URI to the Boundary server:

    HashiCorp Boundary.app GUI Landing

  5. Login Authentication

    HashiCorp Boundary Auth

    Boundary automates and standardizes the workflow for on-boarding and off-boarding hosts and targets.

    It provides one-click deployment.

  6. Select Roles.

  7. Select Administration and then click the Principals tab.

    Notice that admin user is listed. User, group, and project are a type of principal which can be assigned to roles.

  8. Click on the Grants tab to view the permissions allowed on this role. Grants represent strings of actions on resources:

    id=&LT;resource_id>; action=&LT;actions>

    The grant for Administration role indicates that all actions (actions=) on all resources (id=;type=*) are permitted.

  9. Return to the Roles list and select Login and Default Grants role.

  10. Click the Grants to view its permissions

    A role can have multiple grants defined. Those grants are deleted when the role is deleted. A grant is also deleted if its associated resource is deleted.

  11. Select Projects and then Generated project scope.

    Notice that you can see Sessions, Targets and Host Catalogs.

  12. Select Host Catalogs.

  13. Select Generated host catalog.

  14. Click on the Host Sets tab and then Generated host set to view its details.

  15. Click on the Hosts tab to view attached hosts.

    Currently, Generated host with ID, hst_1234567890 is the only host attached to this host set. From the Manage menu, you can add or delete hosts from the host set.

  16. Select Generated host. Its Address is set to localhost.

  17. Select Targets from the left-pane.

  18. Select Generated target. The generated target allows TCP connection, and its ID is ttcp_1234567890.

    Using the Manage menu, you can add additional host sets to the target, or delete this target.

Self-Hosted Local server

  1. To install Boundary on different platforms (on a single cloud region), navigate to a folder associated with the account where you’ll create cloned repositories:

    export PROJDIR="$HOME/github-wilsonmar"
  2. Because there are many other branches (taking up space), clone so that only the master branch is downloaded:

    git clone git@github.com:hashicorp/boundary.git --depth 1
    cd boundary
  3. Clone the dev:

    git clone git@github.com:hashicorp/boundary-reference-architecture.git --depth 1
    cd boundary-reference-architecture/deployment


    There is automation for aws, azure, docker (compose), docker_cts, gcp (Google Cloud Platform), Kubernetes. Most examples use Terraform for provisioning and configuring Boundary.

    boundary auth-methods list

Configuring Boundary Cluster

<a name=scopes”></a>

Boundary Scope

Scopes are the foundational part of Boundary. They allow users to partition resources and assign ownership of resources to principals. modeled as a container.

There are three type of scopes within Boundary: Global, Org, and Project. There is only one global scope, which is the entry point for initial administration/setup and to manage the org scopes. Under the global scope, you can create multiple org scopes.

A scope can contain child scopes, forming a tree.


Org scopes are used to hold IAM-related resources and project scopes.

Explore resources in the Generated org scope using Boundary’s Admin Console.

### Organization scope

Each organization is a top-level container (scope) whichowns zero to many projects and zero to many authentication methods. An organization inherits from scope, allowing it to own zero to many groups, roles, policies, targets, host catalogs, or credential stores.

  1. Specify different orgs (organizations) such as “Engineering”, etc.

    HashiCorp Boundary.app GUI menu

    Projects are child scope within an organization.

  2. Specify project, each with its own scope ID (to organize targets and host catalogs):

    • DevOps - such as Jenkins (instead of SaaS GitHub Actions).

    • Dev - the environment used by developers to unit test functionality. Often scrubbed or obfuscated data for security purposes.

    • Test - an environment also known as pre-prod used to ensure merging with production systems can be performed successfully. May have full volume or production data to ensure functionality.

    • Staging - an environment also known as pre-prod used to ensure merging with production systems can be performed successfully. May have full volume or production data to ensure functionality.

    • Production – the production or live environment where all live data and systems inter-operate. Requires implementation of all security controls.

    • Support - contains CSM (Customer Support Management) system to track communications with customers


  1. To migrate a throw-away instance:

    export BOUNDARY_DB_CONFIG="/etc/boundary/controller.hcl"
    boundary database init -config $"{BOUNDARY_DB_CONFIG}" \
       -skip-auth-method-creation \
       -skip-scopes-creation \



    Alternately, to migrate a long-running instance, specify those 3 skips in the controller.hcl file:

    export BOUNDARY_DB_CONFIG="/etc/boundary/controller.hcl"
    boundary database init -config $"{BOUNDARY_DB_CONFIG}"

  2. Define a host catalog (a collection of hosts and host sets) for:

    Within the DevOps project:

    • CICD

    Within each app project:

    • Application Stack
    • Monitoring

    Inside the Boundary catalog are “host-sets” that are a collection of hosts which are identical except for access requirements. This allows a random host to be selected automatically so that it’s less fragile than specifying a specific host name.

    • App Servers
    • Log Servers


    Group DevOps
    Devs Testers Prod
    SOC Log
    Build Server Yes Yes - - - -
    SSH (all) Yes - - - - -
    Staging App Yes Yes - - - -
    Staging Logs Yes Yes Yes - - -
    Staging App Admin Yes Yes - - - -
    Prod. App Admin - - - - Yes -
    Prod. Logs Yes Yes - - - -
    Prod. Log Delete - - - - - Yes

    Hosts are specific Boundary server instances, created under a host set.

  3. Assign names using a convention such as “app-server_0” and “log-server_0” (numbers starting from zero?).

    TODO: Assign IP address???

    Boundary targets are the server which we seek to remote into.

  4. In each app project, specify a Boundary target as a logical collection of host sets which may be used to initiate sessions.
    • Production Application Admin
    • Production SSH
    • Production Logs

    Parameters for each target set:

    • Name
    • Description
    • Maximum Duration (in seconds)
    • Maximum Connections (specify 2 in case one gets stuck. -1 means infinite)
    • Default Port

  5. Optionally, in the Internal project, specify a target such as “Ticketing” for the ticketing app server.



    grant scope to a particular project.

### KMS

-recover-config controller.hcl is specified in most every boundary command TODO:

Within AWS:

# Root KMS configuration block: this is the root key for Boundary
# Use a production KMS such as AWS KMS in production installs
    kms "awskms" {
      purpose = "root"
      region = "us-east-1"
      kms_key_id = "..."
# Worker authorization KMS
# This key is the same key used in the worker configuration  
    kms "awskms" {
      purpose = "worker-auth"
      region = "us-east-1"
      kms_key_id = "..."
# Recovery KMS block: configures the recovery key for Boundary
    kms "awskms" {
      purpose = "recovery"
      region = "us-east-1"
      kms_key_id = "..."

PROTIP: Remove the Recovery KMS block if recovery is not needed (such as on dev clusters).

config keys are also optional.

### Users

  1. List users:

    boundary users list -scope-id global \
    -recover-config controller.hcl
  2. Add

    boundary users add-accounts -id $"{BOUNDARY_THIS_USER_ID}" \
    -account $"{BOUNDARY_THIS_ACCOUNT_ID}" \
    -recovery-config controller.hcl

PROTIP: Creating a role also involves creating grants and principals:

  1. Login admin role:

    boundary roles create -grant-scope-id global -scope-id global \
    -name Administration \
    -recovery-config controller.hcl
  2. add-grants to do everything for Administration :

    boundary roles add-grants -id $"{BOUNDARY_THIS_GRANT_ID}" \
    -grant "id=*;type=*;actions=*" \
    -recovery-config controller.hcl
  3. add-principal for Administration:

   BOUNDARY_ORG_ID="???" \
   boundary roles add-principals -id $"{BOUNDARY_PRINCIPAL_ID}" \
   -principal $"{BOUNDARY_ORG_ID}" \
   -recovery-config controller.hcl

### For Users

  1. Login globally:

    boundary roles create -grant-scope-id global -scope-id global \
    -name "Login and Default Grants" \
    -recovery-config controller.hcl
  2. add-grants templated for account.id:

    boundary roles add-grants -id $"{BOUNDARY_THIS_GRANT_ID}" \
    -grant "id=*;type-scope;actions=list,no-op" \
    -grant "id=;actions=read,change-password" \
    -grant "id=*;type=auth-token;actions=list,read:self,delete:self" \
    -recovery-config controller.hcl
  3. add-principals

    boundary roles add-principals -id $"{BOUNDARY_PRINCIPAL_ID}" \
    -principal u_anon \
    -recovery-config controller.hcl

Use with Okta IdP


Within “$PROJDIR/boundary-reference-architecture/deployment/gcp/gcp/templates” are template (.tpl) files:

  • boundary.hcl.tpl
  • controller.hcl.tpl
  • worker.hcl.tpl

On Linux, the worker service is specified in:

  1. Specify

    boundary auth-methods create password -scope-id global \
    -recovery-config controller.hcl

    Alternately, specify the specific Method ID when creating passwords:

    boundary accounts create password -auth-method-id $"{BOUNDARY_THIS_METHOD_ID}" \
    -login-name admin \
    -recovery-config controller.hcl

Human-to-machine Access via Boundary

Hands-on interactive lab environment, HashiCorp Learn: https://learn.hashicorp.com/

Machine Auth & Auth using Vault

Machine-to-machine Access using Consul

Which app can talk with each service?



VIDEO: HashiConf Digital Keynote - Boundary by CTO Armon Dadgar explains Boundary vs. Traditional Access approaches