How to keep secrets out of GitHub
In our individual machines,
we use the
ssh-keygen utility to generate key pairs.
The public key we copy into each server
so we can
with the private side of the pair (instead of a password).
PROTIP: For those who only want to create credential once, one approach is to store credentials in a cloud drive (such as Dropbox, Box, Google Drive, or Microsoft OneDrive). Credentials there can be downloaded along with SSH scripts to simplify execution.
Some prefer to use an encrypted USB Solid State Drive for sole physical posession. But if that’s lost or stolen, security can be compromised.
The application 1Password takes a compromise approach of allowing password crypts to be transferred among multiple local devices, but not over a public network.
As we write functions within application source files, we put them within a Git folder, and commit changes into .git history.
The private API keys and crypto certificates from Certificate Authorities we collectively call secrets for accessing web services can be conveninently just copied into the Git folder.
When files are pushed up to GitHub or other repository, .gitignore settings should prevent the certificate from being uploaded and thus risk exposure.
PROTIP: Many say it’s NOT a good idea to keep secrets such as passwords and other private data in a GitHub repository. Murphy’s Law applies here too.
Rogue “dorking” scanners are looking through every public repository, looking for secrets.
PROTIP: Organizations should do their own scans to find issues before others do.
If private information is found, we can use the BFG utility to remove it.
CAUTION: But even after data is removed from the current repository, like the Padora’s Box legend, whatever was exposed can nevertheless live on in any forks, clones, or zip files others have taken of the repository.
Private file in them can be referenced in profile scripts that load files and environment variables within memory accessible by application programs.
There are utilities (called “dorking”) that scan through all GitHub repos looking for exposed keys.
Get it out of there!
What if you found out that your private data has been exposed in a GitHub repo?
PROTIP: If a file is deleted using
git rm and a commit is made,
a vestige of that data still exist in the repository’s history (.git folder).
Utility program BFG Repo-Cleaner (bfg.jar) is faster due to it being written in Scala (a varient of Java). This webpage explains commands such as replacing such as this to find known passwords and replace them with ***REMOVED***.
java -jar bfg.jar --replace-text passwords.txt my-repo.git
Git is designed such that every file and folder is represented only once (and given a unique SHA-1 hash-id).
Git has a
git-filter-branch command which
rebuilds a repo one commit at a time without the offending
content. The Git Real 2 course covers this.
Make commits and push so there is nothing in your local staging area.
Zip up the repo so you have a fall-back.
Make a copy of the repo as backup:
git clone poodles burning-poodles
Rebuild the repo one commit at a time after applying the shell script rm function to remove the secrets.txt file from –all commit files: (change secrets.txt to your file’s name)
git filter-branch --tree-filter 'rm -f secrets.txt -- --all'
Note .gitignore rules are not applied here.
This is a very I/O intensive operation and will take a long time on larger repos.
There are options that change other information:
--env-filterrewrites author/committer name/email/time environment variables
--msg-filterrewrites commit message text.
Remove (prune) commits which are now empty becuase the offending file they reference have been removed:
git filter-branch -f --prune-empty
Notify all those who may have forked or cloned or downloaded the repo.
You can tell Git to ignore changes to a file in the future:
git update-index --assume-unchanged file
However, this works only on a single branch. On a change of branch, Git detects changes in the config file, and you’ll have to either undo them, or check them in.
To track changes again:
git update-index --no-assume-unchanged file
Another option is to use pre and post-commit hooks to automatically add/remove secret config values when checking in and out, using a Python program.
Certificate from file
In Bash, an export command is used to bring in the public key generated by ssh-gen into the user’s home hidden .ssh folder:
export RSA_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)
But PowerShell’s equivalent reads certificate files created using putty-gen or Mysysgit (the Git client for Windows):
$RSA_PUBLIC_KEY = Get-Content "~/.ssh/id_rsa.pub" # echo "RSA_PUBLIC_KEY=$RSA_PUBLIC_KEY"
$RSA_PUBLIC_KEY = [IO.File]::ReadAllText("~/.ssh/id_rsa.pub").split("`n")
The split method using the back-tick adds a trailing empty line at the bottom of the file.
Secrets file to Hash Table
On my Mac I used a text editor to create a text file containing these (fake) secrets:
GITHUB_PASSWORD = '234sdsdvs32' GITHUB_TOKEN = '1234567890123456789012345678901234567890'
Notice there are no dollar signs in front of the key names.
PROTIP: I prefre to not store the user name along with its password.
The file is stored in a .secrets file (no extension) in my Mac user home folder, so they can be invoked as a Bash script:
NOTE: The dot command is equivalent to the source command.
BLAH: The Bash “source” function is not recognized by PowerShell and variables need to have dollar signs. So rather than creating a password file containing:
$GITHUB_PASSWORD = '234sdsdvs32' $GITHUB_TOKEN = '1234567890123456789012345678901234567890'
I can also create a .ps1 file which defines a hashtable (a collection of key/value pairs, also called “associative arrays”):
However, I prefer to read the text file previously read by Bash so I end up with a hash table named $SECRETS in PowerShell, from which it retireves a specific property:
$SECRETS = Get-Content "$home/.secrets" | ConvertFrom-StringData # don't echo $SECRETS.GITHUB_PASSWORD # don't echo $SECRETS.GITHUB_TOKEN
AWS CLI Configuration
Although AWS in 2015 enabled users to switch roles in the Console, switching roles in the CLI is not yet availble.
AWS provides a command to define admin access:
List the location on a Mac or Linux machine:
List the location on Windows:
Insidie the .aws folder is a credentials file containing, for example:
[default] aws_access_key_id = ABCDEFGVSYNHR5G2VNGQ aws_secret_access_key = 123456nVqH3AWz5pGQcZ/+JDHB4dBM2BDNtzUsnK [user2] aws_access_key_id=AKIAI44QH8DHBEXAMPLE aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
On a Mac, store:
export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY' export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
Insidie the .aws folder is a config file containing, for example:
[default] region=us-west-2 output=json [profile e1] region=us-east-1 output=text
PROTIP: Define profile names with the region.
aws s3 ls \-\-profile default
Configuration includes enabling auto completion for AWS CLI, edit file /etc/bashrc to add:
complete -C aws_completer aws
On a Mac, this sample command is used to create a symlink. For example:
ln -s ~/.aws/credentials credentials ln -s ~/.aws/config config
On Windows, a “Shortcut” is created to a file.
Sync from Dropbox
This on-line tool generates a direct link from a share link into Dropbox, Google Drive, and Microsoft OneDrive.
To insert secret key in a Mac’s .bash_profile script that the operating system executes upon boot-up, on can:
But this is not as secure as k
Programming to retrieve an environment variable into the program:
Python programs reference
PHP programs use
C# programs use
System.Environment.GetEnvironmentVariable("SECRET_PASS", _<br />EnvironmentVariableTarget.Process).
Automation scripts running on the desktop often need to provide passwords to various web services. It’s inconvenient to type the credentials in every time the script runs, especially when it’s run overnight on a schedule. Some make the credentials available in a secrets file on their laptop, with contents in clear text so they can be changed.
When git add, commit, and push commands are issued to a folder initialized for git, files specified in .gitignore are blocked from being uploaded to GitHub or other online repository.
The problem with this approach is that if the local secrets file is ever deleted, or the whole laptop is destroyed or stolen, the secrets are gone too.
What we want to consider here is an encrypted secrets file resting, encrypted, within the GitHub cloud and brought down locally by a git fetch or pull. This means that changes would be versioned. But what the changes are would not be evident due to the encrytion.
Mechanisms for encryption and decryption is provided by a utility GitHub repository installed on Mac laptops using Homebrew from:
The repository from sobolevn in Moscow, Russia, who specializes in Elixir. His library provides for initalization of a .gitsecrets folder to hold public keys created using the GPG utility. Its “tell” program emails the private keys it creates so it’s off the machine.
Encrypted files do not need to be automatically decrypted into clear text file until secrets need to be edited to change the behavior desired in shell scripts when they are run.
The secreate file can be encrypted automatically on git commit when a git hook program recognizes the need for encryption so the file can be safely pushed into GitHub again.
If the script has code to decrypt the secret files itself based on the public key generated, the clear text file can be removed locally after editing. There is then no need for the clear text file to be referenced.
When someone is out - just delete their public key, re-encrypt the files, and they won’t be able to decrypt secrets anymore.
PROTIP: This is not a totally secure approach for extremely sensitive production data because, any encryption can be hacked given enough time using on supercomputers now commonly available to hackers.
This blog provides an alternative using the openssl utility.
NOTE: The Git smudge filter is what converts the LFS pointer stored in Git with the actual large file from the LFS server. If your local repository does not have the LFS object, the smudge filter will have to download it. Network issues can affect downloading and thus smudge filter operation.
Introduction to Keybase 2014-11-26 social network and a crypto keyserver.
This is one of a series on Git and GitHub:
- Why Git? (file-based backups vs Git clone)
- Git Markdown text
- Git-client based workflows
- Git whoops (correct mistakes)
- Git rebase
- Git interactive merge (imerge)
- Git HEAD (Commitish references)
- Git custom commands
- Git utilities
- TFS vs GitHub
- GitHub REST API
- GitHub GraphQL API
- GitHub PowerShell API Programming
- GitHub GraphQL PowerShell Module