How to keep secrets out of GitHub
In our individual machines,
we use the
ssh-keygen utility to generate key pairs.
The public key we copy into each server
so we can
with the private side of the pair (instead of a password).
PROTIP: For those who only want to create credential once, one approach is to store credentials in a cloud drive (such as Dropbox, Box, Google Drive, or Microsoft OneDrive). Credentials there can be downloaded along with SSH scripts to simplify execution.
Some prefer to use an encrypted USB Solid State Drive for sole physical posession. But if that’s lost or stolen, security can be compromised.
The application 1Password takes a compromise approach of allowing password crypts to be transferred among multiple local devices, but not over a public network.
As we write functions within application source files, we put them within a Git folder, and commit changes into .git history.
The private API keys and crypto certificates from Certificate Authorities we collectively call secrets for accessing web services can be conveninently just copied into the Git folder.
When files are pushed up to GitHub or other repository, .gitignore settings should prevent the certificate from being uploaded and thus risk exposure.
PROTIP: Many say it’s NOT a good idea to keep secrets such as passwords and other private data in a GitHub repository. Murphy’s Law applies here too.
Rogue “dorking” scanners are looking through every public repository, looking for secrets.
PROTIP: Organizations should do their own scans to find issues before others do.
If private information is found, we can use the BFG utility to remove it.
CAUTION: But even after data is removed from the current repository, like the Padora’s Box legend, whatever was exposed can nevertheless live on in any forks, clones, or zip files others have taken of the repository.
Private file in them can be referenced in profile scripts that load files and environment variables within memory accessible by application programs.
There are utilities (called “dorking”) that scan through all GitHub repos looking for exposed keys.
Get it out of there!
What if you found out that your private data has been exposed in a GitHub repo?
PROTIP: If a file is deleted using
git rm and a commit is made,
a vestige of that data still exist in the repository’s history (.git folder).
Utility program BFG Repo-Cleaner (bfg.jar) is faster due to it being written in Scala (a varient of Java). This webpage explains commands such as replacing such as this to find known passwords and replace them with ***REMOVED***.
java -jar bfg.jar --replace-text passwords.txt my-repo.git
Git is designed such that every file and folder is represented only once (and given a unique SHA-1 hash-id).
Git has a
git-filter-branch command which
rebuilds a repo one commit at a time without the offending
content. The Git Real 2 course covers this.
Make commits and push so there is nothing in your local staging area.
Zip up the repo so you have a fall-back.
Make a copy of the repo as backup:
git clone poodles burning-poodles
Rebuild the repo one commit at a time after applying the shell script rm function to remove the secrets.txt file from –all commit files: (change secrets.txt to your file’s name)
git filter-branch --tree-filter 'rm -f secrets.txt -- --all'
Note .gitignore rules are not applied here.
This is a very I/O intensive operation and will take a long time on larger repos.
There are options that change other information:
--env-filterrewrites author/committer name/email/time environment variables
--msg-filterrewrites commit message text.
Remove (prune) commits which are now empty becuase the offending file they reference have been removed:
git filter-branch -f --prune-empty
Notify all those who may have forked or cloned or downloaded the repo.
You can tell Git to ignore changes to a file in the future:
git update-index --assume-unchanged file
However, this works only on a single branch. On a change of branch, Git detects changes in the config file, and you’ll have to either undo them, or check them in.
To track changes again:
git update-index --no-assume-unchanged file
Another option is to use pre and post-commit hooks to automatically add/remove secret config values when checking in and out, using a Python program.
Certificate from file
In Bash, an export command is used to bring in the public key generated by ssh-gen into the user’s home hidden .ssh folder:
export RSA_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)
But PowerShell’s equivalent reads certificate files created using putty-gen or Mysysgit (the Git client for Windows):
$RSA_PUBLIC_KEY = Get-Content "~/.ssh/id_rsa.pub" # echo "RSA_PUBLIC_KEY=$RSA_PUBLIC_KEY"
$RSA_PUBLIC_KEY = [IO.File]::ReadAllText("~/.ssh/id_rsa.pub").split("`n")
The split method using the back-tick adds a trailing empty line at the bottom of the file.
Secrets file to Hash Table
On my Mac I used a text editor to create a text file containing these (fake) secrets:
GITHUB_PASSWORD = '234sdsdvs32' GITHUB_TOKEN = '1234567890123456789012345678901234567890'
Notice there are no dollar signs in front of the key names.
PROTIP: I prefre to not store the user name along with its password.
The file is stored in a .secrets file (no extension) in my Mac user home folder, so they can be invoked as a Bash script:
NOTE: The dot command is equivalent to the source command.
BLAH: The Bash “source” function is not recognized by PowerShell and variables need to have dollar signs. So rather than creating a password file containing:
$GITHUB_PASSWORD = '234sdsdvs32' $GITHUB_TOKEN = '1234567890123456789012345678901234567890'
I can also create a .ps1 file which defines a hashtable (a collection of key/value pairs, also called “associative arrays”):
However, I prefer to read the text file previously read by Bash so I end up with a hash table named $SECRETS in PowerShell, from which it retireves a specific property:
$SECRETS = Get-Content "$home/.secrets" | ConvertFrom-StringData # don't echo $SECRETS.GITHUB_PASSWORD # don't echo $SECRETS.GITHUB_TOKEN
AWS CLI Configuration
Although AWS in 2015 enabled users to switch roles in the Console, switching roles in the CLI is not yet availble.
AWS provides a command to define admin access:
List the location on a Mac or Linux machine:
List the location on Windows:
Insidie the .aws folder is a credentials file containing, for example:
[default] aws_access_key_id = ABCDEFGVSYNHR5G2VNGQ aws_secret_access_key = 123456nVqH3AWz5pGQcZ/+JDHB4dBM2BDNtzUsnK [user2] aws_access_key_id=AKIAI44QH8DHBEXAMPLE aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
On a Mac, store:
export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY' export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
Insidie the .aws folder is a config file containing, for example:
[default] region=us-west-2 output=json [profile e1] region=us-east-1 output=text
PROTIP: Define profile names with the region.
aws s3 ls \-\-profile default
Configuration includes enabling auto completion for AWS CLI, edit file /etc/bashrc to add:
complete -C aws_completer aws
On a Mac, this sample command is used to create a symlink. For example:
ln -s ~/.aws/credentials credentials ln -s ~/.aws/config config
On Windows, a “Shortcut” is created to a file.
Sync from Dropbox
This on-line tool generates a direct link from a share link into Dropbox, Google Drive, and Microsoft OneDrive.
To insert secret key in a Mac’s .bash_profile script that the operating system executes upon boot-up, on can:
But this is not as secure as k
Programming to retrieve an environment variable into the program:
Python programs reference
PHP programs use
C# programs use
System.Environment.GetEnvironmentVariable("SECRET_PASS", _<br />EnvironmentVariableTarget.Process).
Introduction to Keybase 2014-11-26 social network and a crypto keyserver.
More on DevOps
This is one of a series on DevOps:
- Choices for DevOps Technologies
- Java DevOps Workflow
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- Digital Ocean
- Cloud regions
- AWS Virtual Private Cloud
- Git and GitHub vs File Archival
- Git Commands and Statuses
- Data Security GitHub
- Git Commit, Tag, Push
- Git Utilities
- Jenkins Server Setup
- Jenkins Plug-ins
- Jenkins Freestyle jobs
- Dockerize apps
- Docker Setup
- Powershell Ecosystem
- Powershell on MacOS
- API Management Microsoft
- Scenarios for load