Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Instagram Youtube

Github Stackoverflow Pinterest

How to keep secrets out of GitHub


Overview

In our individual machines, we use the ssh-keygen utility to generate key pairs. The public key we copy into each server so we can SSH with the private side of the pair (instead of a password).

PROTIP: For those who only want to create credential once, one approach is to store credentials in a cloud drive (such as Dropbox, Box, Google Drive, or Microsoft OneDrive). Credentials there can be downloaded along with SSH scripts to simplify execution.

Some prefer to use an encrypted USB Solid State Drive for sole physical posession. But if that’s lost or stolen, security can be compromised.

The application 1Password takes a compromise approach of allowing password crypts to be transferred among multiple local devices, but not over a public network.

Git

As we write functions within application source files, we put them within a Git folder, and commit changes into .git history.

The private API keys and crypto certificates from Certificate Authorities we collectively call secrets for accessing web services can be conveninently just copied into the Git folder.

When files are pushed up to GitHub or other repository, .gitignore settings should prevent the certificate from being uploaded and thus risk exposure.

PROTIP: Many say it’s NOT a good idea to keep secrets such as passwords and other private data in a GitHub repository. Murphy’s Law applies here too.

Rogue “dorking” scanners are looking through every public repository, looking for secrets.

PROTIP: Organizations should do their own scans to find issues before others do.

If private information is found, we can use the BFG utility to remove it.

CAUTION: But even after data is removed from the current repository, like the Padora’s Box legend, whatever was exposed can nevertheless live on in any forks, clones, or zip files others have taken of the repository.

A more secure approach is to define a configuration script that establishes a symlink to reference secret files in folders outside of the Git repository.

Private file in them can be referenced in profile scripts that load files and environment variables within memory accessible by application programs.

Dorking scans

There are utilities (called “dorking”) that scan through all GitHub repos looking for exposed keys.

  • http://www.securityweek.com/github-search-makes-easy-discovery-encryption-keys-passwords-source-code

  • http://www.itworld.com/article/2921135/security/add-github-dorking-to-list-of-security-concerns.html

Get it out of there!

What if you found out that your private data has been exposed in a GitHub repo?

PROTIP: If a file is deleted using git rm and a commit is made, a vestige of that data still exist in the repository’s history (.git folder).

Utility program BFG Repo-Cleaner (bfg.jar) is faster due to it being written in Scala (a varient of Java). This webpage explains commands such as replacing such as this to find known passwords and replace them with ***REMOVED***.

   java -jar bfg.jar --replace-text passwords.txt  my-repo.git
   

Git is designed such that every file and folder is represented only once (and given a unique SHA-1 hash-id).

Git has a git-filter-branch command which rebuilds a repo one commit at a time without the offending content. The Git Real 2 course covers this.

  1. Make commits and push so there is nothing in your local staging area.

  2. Zip up the repo so you have a fall-back.

  3. Make a copy of the repo as backup:

    git clone poodles burning-poodles

  4. Rebuild the repo one commit at a time after applying the shell script rm function to remove the secrets.txt file from –all commit files: (change secrets.txt to your file’s name)

    git filter-branch --tree-filter 'rm -f secrets.txt -- --all'

    Note .gitignore rules are not applied here.

    This is a very I/O intensive operation and will take a long time on larger repos.

  5. There are options that change other information:

    --env-filter rewrites author/committer name/email/time environment variables

    --msg-filter rewrites commit message text.

  6. Remove (prune) commits which are now empty becuase the offending file they reference have been removed:

    git filter-branch -f --prune-empty

  7. Notify all those who may have forked or cloned or downloaded the repo.

Config Script

You can tell Git to ignore changes to a file in the future:


   git update-index --assume-unchanged  file
   

However, this works only on a single branch. On a change of branch, Git detects changes in the config file, and you’ll have to either undo them, or check them in.

To track changes again:


   git update-index --no-assume-unchanged file
   
  • http://www.codeproject.com/Articles/602146/Keeping-sensitive-config-settings-secret-with-Azur

  • http://gitready.com/intermediate/2009/02/18/temporarily-ignoring-files.html

Another option is to use pre and post-commit hooks to automatically add/remove secret config values when checking in and out, using a Python program.

Certificate from file

In Bash, an export command is used to bring in the public key generated by ssh-gen into the user’s home hidden .ssh folder:

   export RSA_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)
   

But PowerShell’s equivalent reads certificate files created using putty-gen or Mysysgit (the Git client for Windows):


   $RSA_PUBLIC_KEY = Get-Content "~/.ssh/id_rsa.pub"
   # echo "RSA_PUBLIC_KEY=$RSA_PUBLIC_KEY"
   

Among the many variations:

   $RSA_PUBLIC_KEY = [IO.File]::ReadAllText("~/.ssh/id_rsa.pub").split("`n")
   

The split method using the back-tick adds a trailing empty line at the bottom of the file.

Secrets file to Hash Table

On my Mac I used a text editor to create a text file containing these (fake) secrets:

GITHUB_PASSWORD = '234sdsdvs32'
GITHUB_TOKEN = '1234567890123456789012345678901234567890'
   

Notice there are no dollar signs in front of the key names.

PROTIP: I prefre to not store the user name along with its password.

The file is stored in a .secrets file (no extension) in my Mac user home folder, so they can be invoked as a Bash script:

source ~/.secrets

NOTE: The dot command is equivalent to the source command.

BLAH: The Bash “source” function is not recognized by PowerShell and variables need to have dollar signs. So rather than creating a password file containing:

$GITHUB_PASSWORD = '234sdsdvs32'
$GITHUB_TOKEN = '1234567890123456789012345678901234567890'
   

I can also create a .ps1 file which defines a hashtable (a collection of key/value pairs, also called “associative arrays”):

   [ordered]@{Key1=Value1;Key2=Value2}
   

However, I prefer to read the text file previously read by Bash so I end up with a hash table named $SECRETS in PowerShell, from which it retireves a specific property:


   $SECRETS = Get-Content "$home/.secrets" | ConvertFrom-StringData
   # don't echo $SECRETS.GITHUB_PASSWORD
   # don't echo $SECRETS.GITHUB_TOKEN
   

AWS CLI Configuration

Although AWS in 2015 enabled users to switch roles in the Console, switching roles in the CLI is not yet availble.

AWS provides a command to define admin access:

   aws configure
   

List the location on a Mac or Linux machine:

   ls ~/.aws
   

List the location on Windows:

   dir %UserProfile%\.aws
   

Insidie the .aws folder is a credentials file containing, for example:

   [default]
   aws_access_key_id = ABCDEFGVSYNHR5G2VNGQ
   aws_secret_access_key = 123456nVqH3AWz5pGQcZ/+JDHB4dBM2BDNtzUsnK

   [user2]
   aws_access_key_id=AKIAI44QH8DHBEXAMPLE
   aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
   

On a Mac, store:

   export AWS_ACCESS_KEY_ID='YOUR_AWS_API_KEY'
   export AWS_SECRET_ACCESS_KEY='YOUR_AWS_API_SECRET_KEY'
   
  • https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/

Insidie the .aws folder is a config file containing, for example:

   [default]
   region=us-west-2
   output=json

   [profile e1]
   region=us-east-1
   output=text
   

PROTIP: Define profile names with the region.

   aws s3 ls \-\-profile default
   

See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Configuration includes enabling auto completion for AWS CLI, edit file /etc/bashrc to add:

   complete -C aws_completer aws
   

On a Mac, this sample command is used to create a symlink. For example:

   ln -s ~/.aws/credentials  credentials
   ln -s ~/.aws/config  config
   

On Windows, a “Shortcut” is created to a file.

Sync from Dropbox

This on-line tool generates a direct link from a share link into Dropbox, Google Drive, and Microsoft OneDrive.

Enviornment variables

To insert secret key in a Mac’s .bash_profile script that the operating system executes upon boot-up, on can:

   echo "export SECRET_PASS=12345678910" >> ~/app-root/data/.bash_profile

But this is not as secure as k

Programming to retrieve an environment variable into the program:

  • Python programs reference process.env.SECRET_PASS.

  • PHP programs use getenv('SECRET_PASS');.

  • C# programs use System.Environment.GetEnvironmentVariable("SECRET_PASS", _<br />EnvironmentVariableTarget.Process).

NOTE: Internet browser sandboxing restricts JavaScript from accessing operating system environment variables.

Automatic Encryption

github-data-security-git-v04-640x304-60868.jpg

Automation scripts running on the desktop often need to provide passwords to various web services. It’s inconvenient to type the credentials in every time the script runs, especially when it’s run overnight on a schedule. Some make the credentials available in a secrets file on their laptop, with contents in clear text so they can be changed.

When git add, commit, and push commands are issued to a folder initialized for git, files specified in .gitignore are blocked from being uploaded to GitHub or other online repository.

The problem with this approach is that if the local secrets file is ever deleted, or the whole laptop is destroyed or stolen, the secrets are gone too.

What we want to consider here is an encrypted secrets file resting, encrypted, within the GitHub cloud and brought down locally by a git fetch or pull. This means that changes would be versioned. But what the changes are would not be evident due to the encrytion.

Mechanisms for encryption and decryption is provided by a utility GitHub repository installed on Mac laptops using Homebrew from:

The repository from sobolevn in Moscow, Russia, who specializes in Elixir. His library provides for initalization of a .gitsecrets folder to hold public keys created using the GPG utility. Its “tell” program emails the private keys it creates so it’s off the machine.

Encrypted files do not need to be automatically decrypted into clear text file until secrets need to be edited to change the behavior desired in shell scripts when they are run.

The secreate file can be encrypted automatically on git commit when a git hook program recognizes the need for encryption so the file can be safely pushed into GitHub again.

If the script has code to decrypt the secret files itself based on the public key generated, the clear text file can be removed locally after editing. There is then no need for the clear text file to be referenced.

When someone is out - just delete their public key, re-encrypt the files, and they won’t be able to decrypt secrets anymore.

PROTIP: This is not a totally secure approach for extremely sensitive production data because, any encryption can be hacked given enough time using on supercomputers now commonly available to hackers.


This blog provides an alternative using the openssl utility.

Hashicorp Consul

LFS

NOTE: The Git smudge filter is what converts the LFS pointer stored in Git with the actual large file from the LFS server. If your local repository does not have the LFS object, the smudge filter will have to download it. Network issues can affect downloading and thus smudge filter operation.

Resources

github.com/keybase

More

This is one of a series on Git and GitHub:

  1. Git and GitHub videos

  2. Why Git? (file-based backups vs Git clone)
  3. Git Markdown text
  4. Git basics (script)
  5. Git command shortcuts

  6. Git-client based workflows
  7. Git whoops (correct mistakes)
  8. Git rebase
  9. Git interactive merge (imerge)
  10. Git HEAD (Commitish references)
  11. Git commits with a Tag and Signature

  12. Git custom commands
  13. Git utilities
  14. Git hooks

  15. GitHub data security

  16. TFS vs GitHub
  17. GitHub REST API
  18. GitHub GraphQL API
  19. GitHub PowerShell API Programming
  20. GitHub GraphQL PowerShell Module