Git Guardian

Full-featured free secret scanning for open source and small teams on GitHub and GitLab

                       

Overview

• Social
• Sign up and get API Key from your GG Dashboard
• Get API key
• Other Integrations
• Scan sample code on GUI Dashboard
  • Review bucket_s3.py scan result
  • Review bucket_s3.py scan result
  • Incidents
• Scan my own (vulnerable) repos
  • Open email “uncovered secrets”
• Try Business Collaboration Pricing
• Invoke from CLI scan of github.com repos
  • Edit $HOME/.env file
• Save API to invoke from CLI scan of github.com repos
  • Run
  • Install and update using pip
  • The API
• Use local Git Hooks to run on Pre-commit
• Run GitHub Actions
  • Manual run steps
• Invoke scanning of public repo
• Install and run on Github Actions
  • Configure your .gitguardian.yml
• Run within on-prem GitHub Enterprise instance
  • Create a service account email and GitHub service account
• QUESTION: Adding custom scanning rules
• QUESTION: Enterprise-wide Trend analysis

This is a hands-on introduction to Git Guardian (“GG”), a utility to detect (monitor) API keys and other credentials and secrets exposed in source code on public SaaS or private (internal/on-prem) GitHub.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

GG’s documentation is published at
https://docs.gitguardian.com/internal-repositories-monitoring/home

PROTIP: Know the Glossary of technical terms before you dive into other docs, at:
https://docs.gitguardian.com/internal-repositories-monitoring/glossary

“Why” and FAQ questions are answered in a blog article that’s also called “Learning Center” at
https://www.gitguardian.com/secrets-detection.

All GG’s source is kept under Github account
https://github.com/GitGuardian.

PROTIP: Specific advice is provided on GG’s
https://github.com/GitGuardian/APISecurityBestPractices repository, which is in a zip file you get via email after you sign up to get their whitepaper “Implementing Automated Secrets Detection for Application Security” at
https://www.gitguardian.com/secrets-detection-whitepaper

• GG’s Leak Mitigation Checklist and Good development practices page is useful if you just leaked sensitive information in public source code. It lists what to do for each of the most common services: Alibaba, Algolia, AWS, DigitalOcean, Google, GitHub, GitLab, Heroku, HubSpot, Mailgun, SendGrid, Slack, Twilio, Twitter

• GG’s Good development practices page (leak prevention practices)

Social

sales@gitguardian.com

At time of writing:

• https://twitter.com/gitguardian?lang=en lists 814 followers

• https://www.linkedin.com/company/gitguardian/ lists 36 employees:

People:

• CEO Jérémy Thomas (oo-de-lally) is based in Paris and San Francisco

• Eric Fourier is co-founder.

• https://twitter.com/PonicodeDev at #GitHubUniverse 2020.

• Mackenzie Jackson (@advocatemack), GG’s Dev Advocate, wrote blog articles which are repeated in the website:

  1. https://blog.gitguardian.com/secret-sprawl
     24 Jul 2020

In my blog article here, I wrote down the steps I took on my attempt at installing and making use of the GG web GUI and tools.

Sign up and get API Key from your GG Dashboard

1. Use a browser to be at https://www.gitguardian.com

2. Click the blue “SIGN UP FOR FREE / Internal Monitoring”.

3. Click “GitHub”. ???

4. If you have a repository on GitHub account, specify it.

5. Signing up adds a browser cookie to your account’s dashboard at URL https://dashboard.gitguardian.com

   Logic in the JavaScript would redirect your browser to your own account’s dashboard.

   Get API key

6. Click the “API” icon on the left menu of your GG Dashboard.
7. Scroll down to the bottom of the page for section title “Generate new API key”.
8. In the Name field, type your name, such as “John Doe GUI test 1”.

9. Click “Crate new API key” for an API key such as (for example):

   3BCBC45eb29EA7d6dEb4F7829Bd3e25cdefd43bac74eE0BEC83b7dC4705bE7c559aD0c8

   PROTIP: You don’t need to save that key if you’re only using the GUI page shown in the next section.

   Alternately, if you want to run a scan anywhere else than the GUI you’re on, follow instructions below, then return here.

   Other Integrations

10. Click the “INTEGRATIONS” icon on the left menu of your GG Dashboard.

    By default, you’re installed to use GitHub.com SaaS.

    Scan sample code on GUI Dashboard

    Get a taste of the response output by using the Dashboard to run a scan of GG’s sample Python and JavaScript programming code containing AWS and MongoDb secrets.

11. Click the “+” at the top of your Chrome browser window to open another browser window to the GG’-provided sample Python program files containing AWS and MongoDb secrets at:

    https://github.com/GitGuardian/sample_secrets/blob/main/bucket_s3.py

12. Click “Raw”, click on any line of the file. Press control+A to select all. Press control+C to copy what has been selected.

13. Switch back to the GG Dashboard.
14. Click the “API” icon on the left menu of your Dashboard.
15. Click within the blue box under title “Submit your text content” and press control+V to paste.

16. Click the blue “Scan”.

17. Click the blue icon to the right of the “API response” field to copy GG’s response (in JSON format).

    Review bucket_s3.py scan result

18. In an editor, create empty file bucket_s3.py.json and paste the result file in it…

    Notice the AWS Client Id and Client Secret in this line:

    database = aws_lib.connect("AKIAF6BAFJKR45SAWSZ5", "hjshnk5ex5u34565AWS654/JKGjhz545d89sjkja")
    

    Also notice the database username (“testuser”) and password (“hub24aoeu”) in this line:

    MONGO_URI = "mongodb+srv://testuser:hub24aoeu@gg-is-awesome-gg273.mongodb.net/test?retryWrites=true&w=majority"
    

    PROTIP: Pasting would remove line breaks:

    {"policy_break_count":2,"policies":["Secrets detection","File extensions","Filenames"],"policy_breaks":[{"type":"MongoDB URI","policy":"Secrets detection","matches":[{"type":"connection_uri","match":"mongodb+srv://testuser:hub24aoeu@gg-is-awesome-gg273.mongodb.net/test?retryWrites=true&w=majority","index_start":336,"index_end":432,"line_start":17,"line_end":17},{"type":"scheme","match":"mongodb+srv","index_start":336,"index_end":346,"line_start":17,"line_end":17},{"type":"username","match":"testuser","index_start":350,"index_end":357,"line_start":17,"line_end":17},{"type":"password","match":"hub24aoeu","index_start":359,"index_end":367,"line_start":17,"line_end":17},{"type":"host","match":"gg-is-awesome-gg273.mongodb.net","index_start":369,"index_end":399,"line_start":17,"line_end":17},{"type":"database","match":"test","index_start":401,"index_end":404,"line_start":17,"line_end":17},{"type":"query","match":"retryWrites=true&w=majority","index_start":406,"index_end":432,"line_start":17,"line_end":17}]},{"type":"AWS Keys","policy":"Secrets detection","matches":[{"type":"client_id","match":"AKIAF6BAFJKR45SAWSZ5","index_start":147,"index_end":166,"line_start":10,"line_end":10},{"type":"client_secret","match":"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","index_start":171,"index_end":210,"line_start":10,"line_end":10}]}]}
    

    CAUTION: Responses contains secrets you don’t want others to see (which is why we scanned GG’s sample code). So please don’t format to human-readable (indented) JSON using public web pages such as jsonformatter.curiousconcept.com.

    Review bucket_s3.py scan result

19. Repeat the above procedure, except use the contents at

    https://github.com/GitGuardian/sample_secrets/blob/main/postgres_model.js

    Notice the secret in this line:

    var pg_pass="sup3rstr0ngpass1ForGG";

    There is also an in-line secret in this line:

    var mongo_uri = "mongodb+srv://testuser:hub24aoeu@gg-is-awesome-gg273.mongodb.net/test?retryWrites=true&w=majority";
    

    The result:

    {"policy_break_count":2,"policies":["Secrets detection","File extensions","Filenames"],"policy_breaks":[{"type":"MongoDB URI","policy":"Secrets detection","matches":[{"type":"connection_uri","match":"mongodb+srv://testuser:hub24aoeu@gg-is-awesome-gg273.mongodb.net/test?retryWrites=true&w=majority","index_start":136,"index_end":232,"line_start":6,"line_end":6},{"type":"scheme","match":"mongodb+srv","index_start":136,"index_end":146,"line_start":6,"line_end":6},{"type":"username","match":"testuser","index_start":150,"index_end":157,"line_start":6,"line_end":6},{"type":"password","match":"hub24aoeu","index_start":159,"index_end":167,"line_start":6,"line_end":6},{"type":"host","match":"gg-is-awesome-gg273.mongodb.net","index_start":169,"index_end":199,"line_start":6,"line_end":6},{"type":"database","match":"test","index_start":201,"index_end":204,"line_start":6,"line_end":6},{"type":"query","match":"retryWrites=true&w=majority","index_start":206,"index_end":232,"line_start":6,"line_end":6}]},{"type":"Generic Database Assignment","policy":"Secrets detection","matches":[{"type":"host","match":"gitguardians.com","index_start":31,"index_end":46,"line_start":2,"line_end":2},{"type":"port","match":"9082","index_start":48,"index_end":51,"line_start":2,"line_end":2},{"type":"username","match":"root","index_start":74,"index_end":77,"line_start":3,"line_end":3},{"type":"password","match":"sup3rstr0ngpass1ForGG","index_start":94,"index_end":114,"line_start":4,"line_end":4}]}]}

    Incidents

20. Click the “INCIDENTS” icon at the left menu on your GG Dashboard.

    Note that text scanned in the GUI doesn’t count as incidents of secrets found.

    Scan my own (vulnerable) repos

21. Click the “PERIMETER” icon at the left menu on your GG Dashboard.

22. Click “ADD SOURCES” if you want to specify more repositories to scan.

    I specify to repos which were created to contain vulnerabilities, including secrets:

    WebGoat

    DVWA

    See my website at https://wilsonmar.github.io/webgoat

23. PROTIP: Start a stopwatch to time how long it takes.
24. Click the blue “Launch scan” button.

25. Watch and click your stopwatch when “100%” appears next to “REAL TIME MONITORING”.

    

    NOTE: Sources are removed from the list after scanning.

26. Below the “HEALTH” drop-down, click the arrow next to the list.

27. Check or uncheck the type of health determined for each source to be displayed.

    

    NOTE: Icons under “PROTECTION” identify whether each source is being monitored.

28. Mouse over or click the time icon to see a pop-up of the date/time of last scan.

    Open email “uncovered secrets”

29. In your email client, open the email from “support@gitguardian.com” with subject

    The scan of __ repositories uncovered secrets

30. Click a repo listed under “repository is affected:” to open the repo on your default browser.

    QUESTION: “17 secrets”???

    Try Business Collaboration Pricing

31. View the comparison of plans at https://www.gitguardian.com/pricing.

    PROTIP: GG scanning is free on SaaS for open source and up to 25 developer teams.

    “STANDARD” “Business” accounts are $434 per month per developer, or $1667/month for 100 developers after a 30-day trial.

32. View the comparison of plans at SETTINGS > General.

    PROTIP: The free plan has a limit of 1,000 (1K) calls per month.
    The “BUSINESS” plan has a limit of 10,000 (10K) calls per month.

    Notice the difference between FREE and “BUSINESS” GG accounts is the
    “Private collaboration repositories”.

    “Collaborative” repos have “Teams” on the organization’s menu:

    

    Within a Team is this menu with Members and Repositories associated with that Team:

    

33. Start your business trial at SETTINGS > General by clicking the blue “Start 30-day trial” button.

    PROTIP: Git-guardian does not ask for your credit card up-front.

    A Business plan enables you to more run options:

    • Invoke locally using CLI

    • Use local Git Hooks to run on Pre-commit

    • Scan repos in on-prem GitHub Enterprise instance

    • Install and run on Github Actions

    • https://github.com/GitGuardian/gg-shield#circle-ci

    Invoke from CLI scan of github.com repos

    Edit $HOME/.env file

34. Highlight and copy these two lines into your Clipboard.

    GITGUARDIAN_API_KEY=__________FILL ME__________
    GITGUARDIAN_API_URL=https://api.gitguardian.com/
    

    The API_URL stays the same.

35. In the editor, also open file .env in your laptop’s Home folder.

    If the file doesn’t already exist, create a new file with that name.

36. In the .env file, position your cursor at the end of the file and paste from Clipboard.

37. Save the file.

    NOTE: https://github.com/GitGuardian/gg-shield#installation mentions addition variables:

    GITGUARDIAN_DONT_LOAD_ENV: If set to any value environment variables won't be loaded from a file.
    GUARDIAN_DOTENV_PATH: If set to a path, `ggshield` will attempt to load the environment from the sp
    

    Save API to invoke from CLI scan of github.com repos

38. Follow the steps described in the section above.

39. With your editor open on file .env, highlight “____FILL ME____” and replace it with your invisible Clipboard.

    QUESTION: GG_API_KEY=”123456789…” ???

    Alternately, if you are running on GitHub Actions, navigate to your project settings and paste the GITGUARDIAN_API_KEY secret there.

40. Save the file.

    Run

41. Open a Terminal window.

42. You can run my script that does all the following:

    sh ???.sh

    Install and update using pip

43. Install Python 3.6 and newer

44. In Terminal, navigate to the folder obtained from GitHub.

45. Check if pipenv

    pipenv update

    If that is valid, install gg-shield using pipenv

    pipenv install ggshield

    Alternately,

    pip install ggshield

46. Check if using virtualenv:

    pyenv shell

    The API

    The APIs called are defined at https://api.gitguardian.com/docs which provides an on-line API specification and Authentication description.

    QUESTION: Swagger?

    Use local Git Hooks to run on Pre-commit

    Now that ggshield is installed:

47. Verify

    ggshield -v
    

48. Install git pre-commit hooks in local (not global) mode:

    ggshield install -m local
    

49. Run the last build in CI, with defaults:

    ggshield scan ci
    

    Run GitHub Actions

    To invoke on a server from the GitHub Actions Marketplace at https://github.com/marketplace/actions/gitguardian-shield-action

50. In Terminal, create/navigate to a folder holding repositories created/cloned.

    Manual run steps

51. In Terminal, create/navigate to a folder holding repositories created/cloned.

52. Obtain GG’s repo:

    git clone https://github.com/GitGuardian/py-gitguardian.git && cd py-gitguardian
    

    https://github.com/GitGuardian/py-gitguardian/blob/master/examples/content_scan.py

    https://github.com/GitGuardian/py-gitguardian/blob/master/examples/directory_scan.py

Invoke scanning of public repo

[1] For running on a public Github.com repo, GG has a secrets scanning API library (written in Python 3.5+) at
https://github.com/GitGuardian/py-gitguardian

The README says GG scans for “200 types of secrets”.

Install and run on Github Actions

Source code for the GG scanning engine CLI code is at
https://github.com/GitGuardian/gg-shield

It is invoked either locally on your laptop or invoked from GitHub Actions Marketplace at https://github.com/marketplace/actions/gitguardian-shield-action

1. Create an API key on the API Section of your GG dashboard.

   Configure your .gitguardian.yml

   To configure your .gitguardian.yml file:

2. View the contents based on the contents of the file at https://github.com/GitGuardian/gg-shield/blob/main/.gitguardian.example.yml

3. Click “Raw”, click on any part of the file, press control+A,

4. Add line(s) under paths-ignore to specify wild cards (*)

GG is run using GitHub Actions invoking
https://github.com/GitGuardian/gg-shield-action

Run within on-prem GitHub Enterprise instance

GitGuardian Private Repository Monitoring is a Kubernetes application. You can install the software on an existing cluster or use our installer that has an embedded, production-ready Kubernetes distribution packaged with it. See https://docs.gitguardian.com/internal-repositories-monitoring/self_hosting/replicated_installation

1. Setup Single Sign On (or SSO) allows you to manage your workspace authentication and membership via a third-party identity provider.

2. Install GG app on your GitHub Enterprise on-prem. server.

   GitGuardian integrates with GitHub Enterprise instance through a GitHub app that we need you to create. GG enables you to do so programmatically via GitHub manifest. This will ensure that your GitHub App is created with all the appropriate rights.

3. Provide GG with your GitHub Enterprise url.

   Create a service account email and GitHub service account

   WARNING: The GitHub app will be owned by the GitHub user who created it. We therefore recommend that you subsequently transfer ownership of the GitHub app to a bot user or a GitHub organization.

   Once created, you will be prompted to install the GitHub app on the GitHub organization of your choice. The installation flow even allows you to individually select repositories that you would like to give GitGuardian access to!

QUESTION: Adding custom scanning rules

QUESTION: Enterprise-wide Trend analysis

---