Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

How to keep secrets out of GitHub

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Conditions enabling leak of secrets from GitHub and other source repositories:

  1. Your GitHub account password can be stolen
  2. Hackers are doing “Dorking” scans looking for secrets in GitHub
  3. You may forget to add .gitignore or remove local secrets
  4. Secrets remain in prior commit history
  5. Static passwords can be cracked eventually
  6. You can lose secrets when your laptop crashes or is lost
  7. PAT (Personal Access Tokens) are static, subject to theft
  8. SSH keys to access GitHub are static, subject to theft

Recommendations offered in this article:

  1. Setup 2FA with an authenticator to physically confirm GitHub access
  2. Automate scans to look for secrets locally before/after pushing to GitHub
  3. Refer to secrets as environment variables in your code
  4. Remove secrets in prior commit history on GitHub
  5. Rotate keys to encrypted files
  6. Save secret keys in the cloud
  7. Rotate Personal Access Tokens every month/quarter/year
  8. Rotate SSH keys automatically daily as certificates generated by a client referencing a CA


PROBLEM 1. Your GitHub account password can be stolen

Someone watching over your shoulder can see your password being typed in.

A key-logger program (“spyware”) installed on your laptop can capture what you type on your keyboard.

SOLUTION 1. Enable 2FA with an Authenticator app

Most enterprise GitHub instances route login to their GitHub organization automatically through their Duo or other multi-factor authentication process on each user’s own smart mobile phone.

If you use an employer-provided smartphone, your employer GitHub account would be protected via SSO through Duo, Okta, etc. But your personal GitHub account is still exposed.

One-time 2FA setup

PROTIP: To protect GitHub accounts opened using a Gmail or other private email address (not a corporate email), we recommended that you enable GitHub’s 2FA (Two-factor Authentication).

There are many tutorials on how to do this on YouTube.

  1. Use an internet browser (Chrome) to go to:

    https://github.com/settings/security

    You can reach this by clicking the avatar picture on the top right on the black band for the drop-down menu to select “Settings”, then in the left sidebar, click Account security.

  2. Click the green “Enable two-factor authentication”.

    2fa-enable-300x123

    Alternately, an “Enabled” button appears if you’re already enabled. In that case, click “Edit” to “Authenticator App” under “Two-factor methods”.

    PROTIP: 2FA requires you to obtain a TOTP (Time-based One-time Password) number in addition to your password.

  3. Provide your password again.
  4. On the Two-factor authentication page, click the green “Set up using an app” button for a list of recovery codes.

    2fa-recovery-codes-528x718

    CAUTION: Recovery codes are the only way to access your account again if you lose your 2FA device. GitHub Support will not be able to restore access to your account, forcing you to lose all repos and history created under that account.

  5. PROTIP: Print and save your recovery codes in a safe place. You get a list of recovery codes because each can only be used once instead of your static Password to get back into your account.

    1. Click Copy, then paste in a file within 1Password or other password-protected location. This enables you to copy and paste later.
    2. Click Print to save a hard copy of your recovery codes
    3. Click Download to save your recovery codes in clear text on your device (not recommended)

  6. Click “Next” for a QR code for your Authenticator app to read.

  7. If you already have Duo, open that app. Alternately, go to the Store app on your smartphone and install one:

  8. In your chosen authenticator app, click “+” for the camera to appear.
  9. Hold your camera to position the QR code to take up the screen to scan it.
  10. After scanning, the last item on the app displays a six-digit code that you can enter on GitHub.

    PROTIP: Ignore the space between the numbers. Enter just the 6 numbers.

  11. Click “Enable”.
  12. Click your avatar and select “Sign out”.
  13. Sign in again.

    Responding to a 2FA challenge

  14. When you see a “Two-factor authentication” challenge on a web page:

    2fa-code-entry-238x300

  15. On your smart phone, open your authentication app (Duo, etc.).
  16. Click the arrow on the right edge of the “GITHUB” entry listing your account name, so that you see six numbers.

  17. On your laptop’s browser, type those numbers in the Authentication Code field entry.
  18. Success means you see your landing page at “github.com”.

References:


PROBLEM 2. Hackers are doing “Dorking” scans looking for secrets in GitHub

A “dork” is American slang for someone who is both academically inclined and also silly and clumsy.*

Rogue “dorking” scanners are looking through every public repository, every commit made, every day, looking for secrets.

Examples below have triggers in bold:

In a PowerShell file to define a new VM in Azure:

$username = 'demoadmin'
$password = ConvertTo-Secure-String 'pa$$word1234' -AsPlainText -Force
$WindowsCred = New-Object System.Management.Automation.PSCredential ($username, $password)
New-AzVM -ResourceGroupName 'psdemo-rg' `
   -Name 'psdemo-win-az' `
   -Image 'Win2019Datacenter' `
   -Credential $WindowsCred `
   -OpenPorts 3389

In a CLI shell file to define a new VM in Azure:

az vm create --resource-group "$myRG" --name "$myVMName" \
   --image "$myWinOSImage" \
   --admin-username "$adminUsername" \
   --admin-password "$adminPassword"

References:

  • http://www.securityweek.com/github-search-makes-easy-discovery-encryption-keys-passwords-source-code
  • http://www.itworld.com/article/2921135/security/add-github-dorking-to-list-of-security-concerns.html

SOLUTION 2. Automate “Dorking” scans of you code

There are two places scanning should occur: locally on laptops and in the GitHub cloud.

GitHub itself has a “GitHub Advanced Security (GHAS)” offering for Enterprise users. Its advantage is that it scans without any developer effort.

It’s a “premium” service ($36.69 per committer with 90 days free) because regular expressions needs to be written to detect secrets applicable to each circumstance, such as AWS credentials, etc.

Coverage of tools is important, so are several competing 3rd-party utilities which detect secrets in GitHub repositories:

  • Nightfall with SSO

On the laptop locally, it takes effort to install Git hooks to fire upon git commit commands.

The options:

git-secrets at github.com/awslabs/git-secrets is from AWS Labs, so does not scan secrets for other clouds. However, there is less concern about malware in the utility. Procedures to install it is detailed at git-secret.io.

secret-bridge by Duo sends Slack messages (PROTIP: Slack messages can have significantly better response than emails). It automates execution of several secret detection tools based on repository events.

github-data-security-secret-bridge-2648x77g

GitHub Advanced Security, a licensed SaaS service from GitHub itself, sends emails about secrets found within Enterprise GitHub instances. Its advantage is that its simple analysis reports are available along with code, which no other vendors can provide.

AWS Macie (described at docs.aws.amazon.com/macie) is a SaaS cloud service from Amazon that uses machine learning and pattern matching to discover and protect your sensitive data in AWS S3 buckets, at scale. Macie’s alerts, or findings, can be searched and filtered in the AWS Management Console and sent to Amazon EventBridge (CloudWatch Events), for integration with existing workflow or event management systems, or to be used in combination to take automated remediation actions using AWS Step Functions. It can be invoked from commands in AWS CLI.

Git Guardian (at gitguardian.com) is an automated monitoring cloud utility to detect API keys and other credentials and secrets exposed in source code on public SaaS or private (internal/on-prem) GitHub. Free on public repos. Its documentation is published at https://docs.gitguardian.com/internal-repositories-monitoring/home.

GuardRails.io (@guardrailsio) on GitHub’s Marketplace uses proprietary code. It produces reports and provides login security, which are important to keep knowledge of internal vulnerabilities isolated to only those who need to know. Its free plan scans only PRs and retains data for a day. It claims use of machine learning to automatically identifies the programming languages and frameworks for each file and the appropriate security tools.

If you would rather not have others look through your code, consider programs you run locally:

GitLeaks is a post-commit utility that runs locally or as a local GitHub Action (Gitleak’s author, Zachary Rice, currently works at GitLab). Written in Go, it is open-sourced at https://github.com/zricethezav/gitleaks/blob/master/config/default.go which are RegEx. Please contribute a search string if you know of potential strings leak (such as for accessing Azure). It scans git repos (or files) for known patterns of strings using regex and Shannon entropy. Josphat Mutai’s blog describes it’s cool features.

“GittyLeaks” open-source code.

“Detect Secrets” open-source code.

TruffleHog is open-source code.

VIDEO: course Securing the Application Lifecycle in Microsoft Azure</> by Reza Salehi describes use of Microsoft DevLabs created CredScan (Credential Scanner) for use within licensed Visual Studio IDE clients and within the Azure cloud run as a task in Microsoft Security Code Analysis Extension.


PROBLEM 3. You may forget to add .gitignore or remove local secrets

Some devs write code (in shell scripts, etc.) to read “.env” files containing secrets.

When a repository is created on Github.com, you have an option of creating a .gitignore file to travel with content in the repository.

Folder and file names specified in that file Git ignores when pushing up to GitHub.

So some devs over-depend on this mechanism and save secrets.

The problem with this approach is we sometimes forget to add that line in .gitignore before pushing to GitHub.

The other problem with this approach is what happens when your laptop crashes?

Your passwords and encryption keys can be lost forever if they are not backed up.

Ignore change

  1. Tell Git to ignore changes to a file in the future by constructing this command by replacing file with your own file’s name:

    git update-index --assume-unchanged  file
    

    However, this works only on a single branch.

    On a change of branch, Git detects changes in the config file, and you’ll have to either undo them, or check them in.

  2. To track changes again, construct this command by replacing file with your own file’s name:

    git update-index --no-assume-unchanged file
    

References:

  • http://www.codeproject.com/Articles/602146/Keeping-sensitive-config-settings-secret-with-Azur

  • http://gitready.com/intermediate/2009/02/18/temporarily-ignoring-files.html

Another option is to use pre and post-commit hooks to automatically add/remove secret config values when checking in and out, using a Python program.

SOLUTION 3. Refer to secrets as environment variables in your code

Among the “12 Factor App” 12factor.net) which summarizes the basic design principles of modern web apps. In Factor 3: Config:

Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not.

A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

Examples of code to retrieve an environment variable into the program:

Python coding

Python programs reference process.env.SECRET_PASS.

PHP coding

PHP programs use getenv('SECRET_PASS');.

C# coding

C# programs use:

System.Environment.GetEnvironmentVariable("SECRET_PASS", _ EnvironmentVariableTarget.Process)

JavaScript (NodeJs) coding

JavaScript is restricted from accessing operating system environment variables by Internet browser sandboxing. So setup a REST API endpoint on a server you control to call with your app’s JavaScript. For example, to get MailChimp to send email from your web app, create a MailChimp API on your server using this Wordpress middleware API which stores your credentials securely on the server, and makes the real API call on your request. It then sends back the data, optionally filtering out any data you don’t want exposed publicly first. Test that the Same Origin Policy prevents rogue “leaching” sites from calling your API endpoint (cross-site request forgery (CSRF), one of the OWASP Top 10 Vulnerabilities) by adding X-CSRF-CHECK: 1 onto your application’s AJAX calls and having the backend check for $_SERVER[‘HTTP_X_CSRF_CHECK’] or generate and exchange a custom session key:

  1. Client embeds client_key in request for API library.
  2. Server determines host that has access to the API, if any.
  3. Server picks “salt” for a session key and sends it to the client with the library [or as part of another pre-auth exchange].
  4. Client calculates a session_key using hash(document.location.host + session_salt).
  5. Client uses session_key + client_key for an API call.
  6. Server validates the call by looking up the client_key’s host and “salt” in the session, computing the hash, and comparing to the provided client_key.
  7. Server validates SameSite Cookie Attribute.

See Security in Gatsby

  1. Some GitHub contains sample values in a file

    CAUTION: The problem is that some simply save the .env file in the same folder, which is then subject to being pushed to a GitHub repository.

Azure PowerShell to Login unattended

  • On Windows:

    (Get-Credential).password | ConvertFrom-SecureString | set-content “$HOME\AzurePassword.txt”
  • On MacOS/Linux home folder:

    (Get-Credential).password | ConvertFrom-SecureString | set-content “~/AzurePassword.txt”
  • Both:

    $UserName = “JohnDoe@evilcorp.com”
    $Password = Get-Content $file | ConvertTo-SecureString
    $credential = New-Object System.Management.Automation.PsCredential($UserName, $Password)
    # Login to the Azure console:
    Login-AzAccount -Credential $credential 
     

An extention of this concept is to reference a file name that is actually reached via a symlink to a folder outside of the Git repository, such as on the user’s $HOME folder.

  1. On a Mac, this sample command is used to create a symlink. For example:

    ln -s ~/.aws/credentials  credentials
    ln -s ~/.aws/config  config
    
  2. On Windows, to create a “Shortcut” file, right-click the folder icon you want to make a shortcut of, and select “Create shortcut” from the right-click menu that appears.

    Alternatively, you can right-click and hover your mouse over the “Send to” option and select “Desktop”.

GitLab

By contrast, https://docs.gitlab.com/ee/ci/secrets/index.html

Sync from Dropbox

This on-line tool generates a direct link from a share link into Dropbox, Google Drive, and Microsoft OneDrive.

Certificate from file

In Bash, an export command is used to bring in the public key generated by ssh-gen into the user’s home hidden .ssh folder:

export RSA_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)
   

But PowerShell’s equivalent reads certificate files created using putty-gen or Mysysgit (the Git client for Windows):

$RSA_PUBLIC_KEY = Get-Content "~/.ssh/id_rsa.pub"
   # echo "RSA_PUBLIC_KEY=$RSA_PUBLIC_KEY"
   

Among the many variations:

$RSA_PUBLIC_KEY = [IO.File]::ReadAllText("~/.ssh/id_rsa.pub").split("`n")
   

The split method using the back-tick adds a trailing empty line at the bottom of the file.

In shell file: Secrets file to Hash Table

On my Mac I used a text editor to create a text file containing these (fake) secrets:

GITHUB_PASSWORD = '234sdsdvs32'
GITHUB_TOKEN = '1234567890123456789012345678901234567890'
   

Notice there are no dollar signs in front of the key names.

PROTIP: I prefre to not store the user name along with its password.

The file is stored in a .secrets file (no extension) in my Mac user home folder, so they can be invoked as a Bash script:

source ~/.secrets

NOTE: The dot command is equivalent to the source command.

BLAH: The Bash “source” function is not recognized by PowerShell and variables need to have dollar signs. So rather than creating a password file containing:

$GITHUB_PASSWORD = '234sdsdvs32'
$GITHUB_TOKEN = '1234567890123456789012345678901234567890'
   

I can also create a .ps1 file which defines a hashtable (a collection of key/value pairs, also called “associative arrays”):

[ordered]@{Key1=Value1;Key2=Value2}
   

However, I prefer to read the text file previously read by Bash so I end up with a hash table named $SECRETS in PowerShell, from which it retireves a specific property:


   $SECRETS = Get-Content "$home/.secrets" | ConvertFrom-StringData
   # don't echo $SECRETS.GITHUB_PASSWORD
   # don't echo $SECRETS.GITHUB_TOKEN
   

Automatic Encryption

github-data-security-git-v04-640x304-60868.jpg

Automation scripts running on the desktop often need to provide passwords to various web services. It’s inconvenient to type the credentials in every time the script runs, especially when it’s run overnight on a schedule. Some make the credentials available in a secrets file on their laptop, with contents in clear text so they can be changed.

When git add, commit, and push commands are issued to a folder initialized for git, files specified in .gitignore are blocked from being uploaded to GitHub or other online repository.

The problem with this approach is that if the local secrets file is ever deleted, or the whole laptop is destroyed or stolen, the secrets are gone too.

What we want to consider here is an encrypted secrets file resting, encrypted, within the GitHub cloud and brought down locally by a git fetch or pull. This means that changes would be versioned. But what the changes are would not be evident due to the encrytion.

Mechanisms for encryption and decryption is provided by a utility GitHub repository installed on Mac laptops using Homebrew from:

The repository from sobolevn in Moscow, Russia, who specializes in Elixir. His library provides for initalization of a .gitsecrets folder to hold public keys created using the GPG utility. Its “tell” program emails the private keys it creates so it’s off the machine.

Encrypted files do not need to be automatically decrypted into clear text file until secrets need to be edited to change the behavior desired in shell scripts when they are run.

The secreate file can be encrypted automatically on git commit when a git hook program recognizes the need for encryption so the file can be safely pushed into GitHub again.

If the script has code to decrypt the secret files itself based on the public key generated, the clear text file can be removed locally after editing. There is then no need for the clear text file to be referenced.

When someone is out - just delete their public key, re-encrypt the files, and they won’t be able to decrypt secrets anymore.

PROTIP: This is not a totally secure approach for extremely sensitive production data because, any encryption can be hacked given enough time using on supercomputers now commonly available to hackers. This is dealt with in the next section.


PROBLEM 4. Secrets remain in prior commit history

PROTIP: If a file is deleted using git rm and a commit is made, a vestige of that data still exist in the repository’s history (.git folder).

SOLUTION 4. Remove secrets in prior commit history on GitHub

See https://microsoft.github.io/AzureTipsAndTricks/blog/tip115.html

Git has a git-filter-branch command which rebuilds a repo one commit at a time without the offending content. The Git Real 2 course covers this.

  1. Make commits and push so there is nothing in your local staging area.

  2. Zip up the repo so you have a fall-back.

  3. Make a copy of the “poodles” repo locally as backup:

    git clone poodles burning-poodles

  4. Rebuild the repo one commit at a time after applying the shell script rm function to remove the secrets.txt file from –all commit files: (change secrets.txt to your file’s name)

    git filter-branch --tree-filter 'rm -f secrets.txt -- --all'

    Note .gitignore rules are not applied here.

    This is a very I/O intensive operation and will take a long time on larger repos.

  5. There are options that change other information:

    --env-filter rewrites author/committer name/email/time environment variables

    --msg-filter rewrites commit message text.

  6. Remove (prune) commits which are now empty becuase the offending file they reference have been removed:

    git filter-branch -f --prune-empty

  7. Notify all those who may have forked or cloned or downloaded the repo.

BFG Java Utility

Utility program BFG Repo-Cleaner (bfg.jar) is faster due to it being written in Scala (a varient of Java). This webpage explains commands such as replacing such as this to find known passwords and replace them with ***REMOVED***.

   java -jar bfg.jar --replace-text passwords.txt  my-repo.git
   

PROBLEM 5. Static passwords and keys can be cracked

Some devs make use of programs that encrypt files stored in GitHub.

Storing encrypted data in GitHub

“Aside from an initial unlock command that needs to be used after cloning the repository, git-crypt encryption and decryption operations happen transparently. I find this workflow to be superior to git-secret and BlackBox.” says blackbox, git-secret bash script, and git-crypt.

git-gpg stores encrypted git repositories on third-party / potentially insecure servers, but stores all changes to source files as compressible textual deltas (a key reason for using git in the first place). The repository is encrypted remotely but the local version has no encrypted blobs inside.

Other benefits include architectural simplicity and low footprint: it consists of a single Python script added to your executable path.

CAUTION: Their achilles heel is that old versions of keys are stored in Git history.

Cracking

CAUTION: The problem is that recent advances in computing hardware enable passwords and keys to be cracked.

Thieves can now direct thousands of computers to guess encryption keys quickly.

SOLUTION 5. Rotate keys to encrypted files

The down-side of storing encrypted files is that GitHub, being designed to work with text, can no longer compare and version encrypted files.

As binary files, entire files are versioned using Git LFS (Large File Store) capabilities.

But diff of individual lines within encrypted files are not currently conviently available.

LFS for binary files

NOTE: The Git smudge filter is what converts the LFS pointer stored in Git with the actual large file from the LFS server. If your local repository does not have the LFS object, the smudge filter will have to download it. Network issues can affect downloading and thus smudge filter operation.


PROBLEM 6. You can lose secrets when your laptop crashes or is lost

Some devs code their programs to read “.env” files in a folder outside Git repository folder, such as the user’s root foler.

But mechanisms to backup those secrets can be problematic.

Many enterprises block USB drives from being plugged in (to avoid STUXNET vulnerability).

The application 1Password takes a compromise approach of allowing password crypts to be transferred among multiple local devices, but not over a public network.

SOLUTION 6. Save secret keys in the cloud

Cloud vendors have the money to hire top professionals figuring out how to keep data safe for a lot of others.

If you do still store secrets in .env files, store a backup of them in a cloud vendor’s key store:

GitHub Secrets

In 2020 GitHub introduced the Secrets page within the Settings tab of each repository for use with GitHub Actions pipelines.

  1. BLAH: The “Manage organization secrets” to define secrets for the whole organization (which can include other repos) is there for convenience.

    github-secrets-drop-708x644

  2. Typically, “New repository secret” would be clicked to define secrets applicable to only the current repository.

    NOTE: “YOUR_SECRET_NAME” is in all caps because that’s the convention for naming static variables programs/pipeline code use to obtain secret values.

    Secrets are environment variables that are encrypted.

    PROTIP: In the name add a word for the type of secret, such as “API_ACCESS_TOKEN”, etc.

    Environment secrets

    Repository secrets

  3. Type plain text secrets in the form so GitHub can hide the secret for reference within GitHub Actions pipelines.

    Secrets entered are encrypted before they reach GitHub because GitHub uses a “libsodium sealed box” which encrypts each message using an ephemeral key pair (using a secret part destroyed right after encryption processing). So the sender cannot decrypt its own message later. The recipient decrypts messages using its private key.*

Secrets are not passed to workflows that are triggered by a pull request from a fork.

CAUTION: Anyone with collaborator access to this repository can use these secrets for Actions. So some prefer to store their secrets elsewhere, such as Hashicorp Vault or cloud key store.


Some utilities (such as SOP) reference your AWS credentials stored in folder $HOME/.aws to authenticate against KMS so you can encrypt and decrypt without a password.

VIDEO: Keeping secrets in your infrastructure pipeline at GitHub Universe 2020 [deck]

Rosemary Wang (<a target=”_blank” href=”https://linkedin.com/in/rosemarywang””>@rosemarywang</a>), Developer Advocate, HashiCorp code

Offline capable?

CAUTION: Programming reference to a cloud key store may slow progress to those who work offline, and need a cache of secrets on their laptop.

AWS CLI Configuration

  1. AWS provides a command to define your credentials to access its cloud:

    aws configure
    

    BLAH: Although AWS in 2015 enabled users to switch roles in the Console, switching roles in the CLI is not available at time of writing.

  2. List the location on a Mac or Linux machine:

    ls ~/.aws
    
  3. List the location on Windows:

    dir %UserProfile%\.aws
    

    Insidie the .aws folder is a credentials file containing, for example:

    [default]
    aws_access_key_id = ABCDEFGVSYNHR5G2VNGQ
    aws_secret_access_key = 123456nVqH3AWz5pGQcZ/+JDHB4dBM2BDNtzUsnK
    
    [user2]
    aws_access_key_id=AKIAI44QH8DHBEXAMPLE
    aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
    
  4. On a Mac, store:

    export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY'
    export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
    
    • https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/
  5. Insidie the .aws folder is a config file containing, for example:

    [default]
    region=us-west-2
    output=json
    
    [profile e1]
    region=us-east-1
    output=text
    

    PROTIP: Define profile names with the region.

    aws s3 ls \-\-profile default
    

    See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

  6. Configuration includes enabling auto completion for AWS CLI, edit file /etc/bashrc to add:

    complete -C aws_completer aws
    

    KMS utilities on AWS

    Zemanta’s https://github.com/Zemanta/py-secretcrypt and https://github.com/Zemanta/go-secretcrypt keeps secrets encrypted with Amazon KMS (Key Management Service) in repos, which are decrypted on the fly by the application. Access control is managed through AWS KMS key policies, with EC2 instances running the applications having permissions to decrypt the secrets.

    https://github.com/fugue/credstash uses AWS KMS for key wrapping and master-key storage, and DynamoDB for credential storage and sharing. Works in several flavors of Linux, in a variety of programming languages.

Google Cloud secret keeping

After setting up Google Cloud CLI, adapt this shell script to establish a key, encrypt, and decrpt:

KEY_NAME="talk-key"
KEYRING_FILE="my-sample-keyring"
CLEAR_TEXT_FILE_PATH="/tmp/the-secret-of-cloud.txt"
 
gcloud kms keys create "${KEY_NAME}" --location global --keyring "${KEYRING_FILE}" --purpose encryption
echo "clear text contents to be encrypted" > "${CLEAR_TEXT_FILE_PATH}"
 
gcloud kms encrypt --location global --keyring "${KEYRING_FILE}" --key "${KEY_NAME}" \
   --plaintext-file "${CLEAR_TEXT_FILE_PATH}"
cat "${CLEAR_TEXT_FILE_PATH}".encrypted
 
gcloud kms decrypt --location global --keyring "${KEYRING_FILE}" --key "${KEY_NAME}" \
   --ciphertext-file \
   --plaintext-file=-
   

SOPS decrypt to RAM for editing

To avoid forgetting to re-encrypt files decrypted on your laptop, https://github.com/mozilla/sops (SOP = Secrets OPerationS), written by Mozilla (the alternative browser) to decrypt files in RAM where you can edit and the utility re-encrypts files automatically before saving them to disk.

An example using GCP KMS:

CLEAR_TEXT_FILE_PATH="/tmp/myfile.enc.yaml"
sops --encrypt --gcp-kms "${KEY_NAME}"   "${CLEAR_TEXT_FILE_PATH}" > "${CLEAR_TEXT_FILE_PATH}"
sops --decrypt "${CLEAR_TEXT_FILE_PATH}"
sops "${CLEAR_TEXT_FILE_PATH}"  # to edit then save using default vim editor
sops --decrypt "${CLEAR_TEXT_FILE_PATH}"
   

Note that sops automatically decrypts to use Git diff commands to identify between two files differences in specific lines.

References about this topic:

Azure Key Vault

References:

  • Azure Key Vault
  • http://www.codeproject.com/Articles/602146/”>Keeping-sensitive-config-settings-secret-with-Azur”>Keeping sensitive config settings secrete with Azure</a>

PROBLEM 7. PAT (Personal Access Tokens) are static, subject to theft

Github allows for several ways to authenticate and authorize access to repositories. In addition to passwords, Github enables users to create a Personal Access Token (PAT) to authenticate instead of a password.

The concern is that any static key is subject to cracking and theft. Anyeone who knows a password/PAT can access all data of that user on GitHub, include deletion.

SOLUTION 7. Rotate Personal Access Tokens every month/quarter/year

PROTIP: In your Calendar (such as Outlook), create an appointment for each Private Access Token created.


PROBLEM 8. SSH keys to access GitHub are static, subject to theft

  1. When at a repository on GitHub, in the Code section, you want to clone a repository to your laptop, click the green ”Code” button:

    github-code-button-252x94

  2. You are presented with a choice of “HTTP”, “SSH”, and (new) “GitHub CLI”:

    github-ssh-menu-764x554

    If you choose “Download ZIP”, it would include only the most recent version of files and won’t include the history of changes (“commits”) made.

    Underlined in red is SSH (Secure Shell), used for a more secure connection than HTTPS. The SSH protocol is used extensively by Linux operating systems to secure transmission between servers and laptop clients. (Linus Torvolds, who created Linux, also created Git)

    Instead of passwords, using SSH means that we won’t have to input a password each time a Git command is issued.

    TMI: The public key file and private key file generated use what is called “asymmetric cryptography” algorithms. Instead of exchanging a single mutually known (“symmetric”) password, smart mathematics is used such that the public key is manually pasted in GitHub GUI to be used to decrypt data which was encrypted using the other part of the key pair.

    However, to make user of SSH involves each user (developer) generating SSH keys locally on a laptop, obtaining the contents of the public key, and pasting that in a GitHub GUI. Explained below:

  3. Open a Terminal window.

  4. Navigate to the (hidden) folder which programs go to (by default) to find SSH key files. On a Mac:

    cd $HOME/.ssh

    If the folder doesn’t exist, create it.

    SSH makes use of a pair of files generated together by the ssh-keygen utility program used by Linux users.

  5. PROTIP: To avoid typing mistakes, construct commands to define variables substituted with your information:

    export GITHUB_ORG="gmail_acct"
    export MY_EMAIL_ADDRESS="john-doe@gmail.com"
    

    {GITHUB_ORG} is a variable which defines the file name and also the folder where you store repositories under your account.

  6. Create a pair of SSH key files by copying the line below and pasting in your Terminal:

    ssh-keygen -t rsa -f "${GITHUB_ORG}" -C "${MY_EMAIL_ADDRESS}" -N ""

    -N “” specifies that no Passphrase will be requested when the key is used. Otherwise, you’ll have to type your password in when executing every git command.

    NOTE: If -f “${GITHUB_ORG}” is omitted, you will be prompted for it. Pressing Enter at the prompt will result in a default name of “id_rsa”.

    The response is something like this:

    Generating public/private rsa key pair.
    Your identification has been saved in mck_acct.
    Your public key has been saved in mck_acct.pub.
    The key fingerprint is:
    SHA256:o3um68DCgu29uREm2Di6tgGrSCTUaHZdhOS4Aj6nHnc wilson_mar@NYC-192850-C02Z70CMLVDT
    The key's randomart image is:
    +---[RSA 3072]----+
    |    ..oo         |
    |  o +..          |
    |.= + o           |
    |*+. .            |
    |*=ooo   S        |
    |*==+ . . .       |
    |==+.+E.          |
    |==+o.+ .o        |
    |=oo =+==         |
    +----[SHA256]-----+
    
  7. Make sure the ssh agent program used to register key identities is running:

    eval "$(ssh-agent -s)"

    A sample response is:
    Agent pid 21631

  8. Register the newly created key identities:

    ssh-add "$HOME/.ssh/${GITHUB_ORG}"

    Example response on a laptop configured with “johndoe” as the machine user name:

    Identity added: /Users/johndoe/.ssh/gmail_acct (john-doe@gmail.com)
  9. Copy the contents of the public key file to your machine’s Clipboard. On a Mac:

    pbcopy < "$HOME/.ssh/${GITHUB_ORG}.pub"

    On Windows:

    clip < "$HOME/.ssh/${GITHUB_ORG}.pub"

    Add the newly created public key to your GitHub account

  10. Switch to an internet browser.

  11. Construct your GitHub.com by substituting the example with your GitHub.com account name, for example:

    https://github.com/wilson-mar
  12. Login using your password and 2FA challenge.

  13. Click on the drop-down arrow icon beside the icon (avatar) at the top right of GitHub’s navigation bar.

  14. Select “Settings” in the drop-down.

  15. Click on “SSH and GPG keys” link on the navigation menu at the left.

  16. Click on the green “New SSH key” button by the top right.

  17. Click inside the “Title” field and type your machine and file name you are about to paste. Example:

    Windows 10 laptop file id_rsa.pub

  18. Click inside the “Key” data entry field which says “Begins with”…

  19. To paste from Clipboard: On macOS, press command+V. On Windows, press ctrl+V.

    “ssh-rsa” should be the first line, followed by something humans cannot read.

  20. Click “Add SSH key”.

  21. Enter your GitHub password again when prompted to confirm.

  22. You should receive an email with subject:

    [EXT][GitHub] A new public key was added to your account

    Make use of SSH protocol

  23. Switch back to view your repository view on GitHub.com at the Code pop-up as shown above.

    Notice that “git@github.com:username” appears instead of “https://github.com/username”.

References about this topics:

  • https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
  • A 10-minute video course at https://mckinsey.udemy.com/course/github-ultimate/learn/lecture/4731854#overview


The above steps are rather cumbersome for the user and can be dangerous for security because the SSH keys generated remain stored locally and possibly over a long period of time.

Someone else can copy an SSH key and access your account on a rogue computer.

SOLUTION 8. Rotate SSH keys automatically daily as certificates generated by a client referencing a CA

A more modern way that automatically generates SSH keys every day, each user of GitHub doesn’t have ssh-keygen to run and copying and pasting to a GitHub form.

The solution makes use of an new extension to the SSH protocol implemented by GitHub for its GitHub Enterprise Cloud license users (not individual users).

References:

The SSH protocol extension involves an additional SSH certificate file added to Git requests.

The SSH certificate file is created by what is called a policy “wrapper” program because it wraps policies around the public key, such as limiting the life span of the key wrapped to 24 hours.

The Policy Wrapper program is a Hashicorp Vault server maintained by an enterprise.

To ensure the authenticity of certificates, GitHub references another public key – the Certificate Authority public key generated on the Vault server and pasted into GitHub by the administrator.

When GitHub receives and unwraps the request, it enforces the policy.

Having the 24 hour key rotation policy in place means new certificate are created every day, by what we call a “key rotation” program/script running on laptops. It makes an API call to the Vault server and receives the SSH certificate file.

Here is a static flowchart of the process described above:

github-data-ssh-keyrotation-2806x1656

Steps to make this happen include:

  1. As the Organization’s Owner, enable SSH Certificate processing for GitHub organization.

  2. Create a Vault API CA (Certificate Authority) service.

  3. Enroll users to the Vault API service.

  4. Perform penetration tests of the Vault API server.

  5. Ensure the Vault API service has the capacity needed (SSH load testing using PureLoad or https://github.com/shazow/ssh-hammer).

  6. Create/Test a “key rotation” program which calls the Vault CA API to obtain a SSH certificate to send to GitHub:

  7. Install “key rotation” client program (with associated dependencies) on all laptops.

    The program would also need to create/edit .git/config files.

  8. Require SSH Certificate processing for all access to GitHub organization.

  9. Each user within an Enterprise GitHub Organization, enable “Single sign-on organizations” in the SSH keys section, click “Enable SSO”, then “Authorize”.

  10. To troubleshoot:

    ssh -Tv git@github.com

    or if it’s a GitHub Organization, for example:

    ssh -Tv g-12345678@github.com
  11. Performance testing SSH connections using Neoload’s Terminal add-on


Local Diagram

Here is a draft diagram describing how the various techniques above working together:

As we write functions within application source files, we put them within a Git folder, and commit changes (which Git stores in its .git folder containing history).

The private API keys and crypto certificates from Certificate Authorities we collectively call secrets for accessing web services can be conveninently just copied into the Git folder.

When files are pushed up to GitHub or other repository, .gitignore settings should prevent the certificate from being uploaded and thus risk exposure.

PROTIP: Many say it’s NOT a good idea to keep secrets such as passwords and other private data in a GitHub repository. Murphy’s Law applies here too.

PROTIP: Organizations should do their own scans to find issues before others do.

CAUTION: But even after data is removed from the current repository, like the Padora’s Box legend, whatever was exposed can nevertheless live on in any forks, clones, or zip files others have taken of the repository.

Private file in them can be referenced in profile scripts that load files and environment variables within memory accessible by application programs.


Resources

https://dev.to/himadriganguly/configure-ssh-server-with-key-based-and-two-factor-authentication-3oc2

https://github.com/mozilla/sops sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

More

This is one of a series on Git and GitHub:

  1. Git and GitHub videos

  2. Why Git? (file-based backups vs Git clone)
  3. Git Markdown text

  4. Git basics (script)
  5. Git whoops (correct mistakes)
  6. Git messages (in commits)

  7. Git command shortcuts
  8. Git custom commands

  9. Git-client based workflows

  10. Git HEAD (Commitish references)

  11. Git interactive merge (imerge)
  12. Git patch
  13. Git rebase

  14. Git utilities
  15. Git-signing

  16. Git hooks
  17. GitHub data security
  18. TFS vs GitHub

  19. GitHub actions for automation JavaScript
  20. GitHub REST API
  21. GitHub GraphQL API
  22. GitHub PowerShell API Programming
  23. GitHub GraphQL PowerShell Module