Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

How to keep secrets out of GitHub


Overview

In our individual machines, we use the ssh-keygen utility to generate key pairs. The public key we copy into each server so we can SSH with the private side of the pair (instead of a password).

PROTIP: For those who only want to create credential once, one approach is to store credentials in a cloud drive (such as Dropbox, Box, Google Drive, or Microsoft OneDrive). Credentials there can be downloaded along with SSH scripts to simplify execution.

Some prefer to use an encrypted USB Solid State Drive for sole physical posession. But if that’s lost or stolen, security can be compromised.

The application 1Password takes a compromise approach of allowing password crypts to be transferred among multiple local devices, but not over a public network.

Git

As we write functions within application source files, we put them within a Git folder, and commit changes into .git history.

The private API keys and crypto certificates from Certificate Authorities we collectively call secrets for accessing web services can be conveninently just copied into the Git folder.

When files are pushed up to GitHub or other repository, .gitignore settings should prevent the certificate from being uploaded and thus risk exposure.

PROTIP: Many say it’s NOT a good idea to keep secrets such as passwords and other private data in a GitHub repository. Murphy’s Law applies here too.

Rogue “dorking” scanners are looking through every public repository, looking for secrets.

PROTIP: Organizations should do their own scans to find issues before others do.

If private information is found, we can use the BFG utility to remove it.

CAUTION: But even after data is removed from the current repository, like the Padora’s Box legend, whatever was exposed can nevertheless live on in any forks, clones, or zip files others have taken of the repository.

A more secure approach is to define a configuration script that establishes a symlink to reference secret files in folders outside of the Git repository.

Private file in them can be referenced in profile scripts that load files and environment variables within memory accessible by application programs.

Dorking scans

There are utilities (called “dorking”) that scan through all GitHub repos looking for exposed keys.

  • http://www.securityweek.com/github-search-makes-easy-discovery-encryption-keys-passwords-source-code

  • http://www.itworld.com/article/2921135/security/add-github-dorking-to-list-of-security-concerns.html

Get it out of there!

What if you found out that your private data has been exposed in a GitHub repo?

PROTIP: If a file is deleted using git rm and a commit is made, a vestige of that data still exist in the repository’s history (.git folder).

Utility program BFG Repo-Cleaner (bfg.jar) is faster due to it being written in Scala (a varient of Java). This webpage explains commands such as replacing such as this to find known passwords and replace them with ***REMOVED***.

   java -jar bfg.jar --replace-text passwords.txt  my-repo.git
   

Git is designed such that every file and folder is represented only once (and given a unique SHA-1 hash-id).

Git has a git-filter-branch command which rebuilds a repo one commit at a time without the offending content. The Git Real 2 course covers this.

  1. Make commits and push so there is nothing in your local staging area.

  2. Zip up the repo so you have a fall-back.

  3. Make a copy of the repo as backup:

    git clone poodles burning-poodles

  4. Rebuild the repo one commit at a time after applying the shell script rm function to remove the secrets.txt file from –all commit files: (change secrets.txt to your file’s name)

    git filter-branch --tree-filter 'rm -f secrets.txt -- --all'

    Note .gitignore rules are not applied here.

    This is a very I/O intensive operation and will take a long time on larger repos.

  5. There are options that change other information:

    --env-filter rewrites author/committer name/email/time environment variables

    --msg-filter rewrites commit message text.

  6. Remove (prune) commits which are now empty becuase the offending file they reference have been removed:

    git filter-branch -f --prune-empty

  7. Notify all those who may have forked or cloned or downloaded the repo.

Config Script

You can tell Git to ignore changes to a file in the future:


   git update-index --assume-unchanged  file
   

However, this works only on a single branch. On a change of branch, Git detects changes in the config file, and you’ll have to either undo them, or check them in.

To track changes again:


   git update-index --no-assume-unchanged file
   
  • http://www.codeproject.com/Articles/602146/Keeping-sensitive-config-settings-secret-with-Azur

  • http://gitready.com/intermediate/2009/02/18/temporarily-ignoring-files.html

Another option is to use pre and post-commit hooks to automatically add/remove secret config values when checking in and out, using a Python program.

Certificate from file

In Bash, an export command is used to bring in the public key generated by ssh-gen into the user’s home hidden .ssh folder:

   export RSA_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)
   

But PowerShell’s equivalent reads certificate files created using putty-gen or Mysysgit (the Git client for Windows):


   $RSA_PUBLIC_KEY = Get-Content "~/.ssh/id_rsa.pub"
   # echo "RSA_PUBLIC_KEY=$RSA_PUBLIC_KEY"
   

Among the many variations:

   $RSA_PUBLIC_KEY = [IO.File]::ReadAllText("~/.ssh/id_rsa.pub").split("`n")
   

The split method using the back-tick adds a trailing empty line at the bottom of the file.

Secrets file to Hash Table

On my Mac I used a text editor to create a text file containing these (fake) secrets:

GITHUB_PASSWORD = '234sdsdvs32'
GITHUB_TOKEN = '1234567890123456789012345678901234567890'
   

Notice there are no dollar signs in front of the key names.

PROTIP: I prefre to not store the user name along with its password.

The file is stored in a .secrets file (no extension) in my Mac user home folder, so they can be invoked as a Bash script:

source ~/.secrets

NOTE: The dot command is equivalent to the source command.

BLAH: The Bash “source” function is not recognized by PowerShell and variables need to have dollar signs. So rather than creating a password file containing:

$GITHUB_PASSWORD = '234sdsdvs32'
$GITHUB_TOKEN = '1234567890123456789012345678901234567890'
   

I can also create a .ps1 file which defines a hashtable (a collection of key/value pairs, also called “associative arrays”):

   [ordered]@{Key1=Value1;Key2=Value2}
   

However, I prefer to read the text file previously read by Bash so I end up with a hash table named $SECRETS in PowerShell, from which it retireves a specific property:


   $SECRETS = Get-Content "$home/.secrets" | ConvertFrom-StringData
   # don't echo $SECRETS.GITHUB_PASSWORD
   # don't echo $SECRETS.GITHUB_TOKEN
   

AWS CLI Configuration

Although AWS in 2015 enabled users to switch roles in the Console, switching roles in the CLI is not yet availble.

AWS provides a command to define admin access:

   aws configure
   

List the location on a Mac or Linux machine:

   ls ~/.aws
   

List the location on Windows:

   dir %UserProfile%\.aws
   

Insidie the .aws folder is a credentials file containing, for example:

   [default]
   aws_access_key_id = ABCDEFGVSYNHR5G2VNGQ
   aws_secret_access_key = 123456nVqH3AWz5pGQcZ/+JDHB4dBM2BDNtzUsnK

   [user2]
   aws_access_key_id=AKIAI44QH8DHBEXAMPLE
   aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
   

On a Mac, store:

   export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY'
   export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
   
  • https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/

Insidie the .aws folder is a config file containing, for example:

   [default]
   region=us-west-2
   output=json

   [profile e1]
   region=us-east-1
   output=text
   

PROTIP: Define profile names with the region.

   aws s3 ls \-\-profile default
   

See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Configuration includes enabling auto completion for AWS CLI, edit file /etc/bashrc to add:

   complete -C aws_completer aws
   

On a Mac, this sample command is used to create a symlink. For example:

   ln -s ~/.aws/credentials  credentials
   ln -s ~/.aws/config  config
   

On Windows, a “Shortcut” is created to a file.

Sync from Dropbox

This on-line tool generates a direct link from a share link into Dropbox, Google Drive, and Microsoft OneDrive.

Enviornment variables

To insert secret key in a Mac’s .bash_profile script that the operating system executes upon boot-up, on can:

   echo "export SECRET_PASS=12345678910" >> ~/app-root/data/.bash_profile

But this is not as secure as k

Programming to retrieve an environment variable into the program:

  • Python programs reference process.env.SECRET_PASS.

  • PHP programs use getenv('SECRET_PASS');.

  • C# programs use System.Environment.GetEnvironmentVariable("SECRET_PASS", _<br />EnvironmentVariableTarget.Process).

NOTE: Internet browser sandboxing restricts JavaScript from accessing operating system environment variables.

Resources

github.com/keybase

More

This is one of a series on Git and GitHub:

  1. Git and GitHub videos

  2. Why Git? (file-based backups vs Git clone)
  3. Git-client based workflows
  4. Git whoops (correct mistakes)
  5. Git rebase
  6. Git commits with a Tag and Signature
  7. Git HEAD (Commitish references)
  8. Git commands and statuses

  9. Git custom commands
  10. Git utilities
  11. Git hooks

  12. GitHub data security

  13. TFS vs GitHub
  14. GitHub REST API
  15. GitHub GraphQL API
  16. GitHub PowerShell API Programming
  17. GitHub GraphQL PowerShell Module