Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

One git commit and you’re hooked

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Here is a tutorial on how to make Git on a local machine automatically run a “hook” script in response to git commands.

Dimensions to Hooks

Hooks scripts run on two locations:

Git and GitHub have been written to look for a hook program before and after each of these commands:

Just so we can use nerdy language, hooks before each event begin with “pre-“ and hooks after each even begin with “post-“. This illustration (by Sarah Goff-Dupont of Atlassian) identifies common uses across four situations:
githooksgrid

Ensure adherance to standards

PROTIP: Some organizations impose a set of hook files on repositories in order to impose some standards, such as:

  • Reject passwords found in files. Doing it on the client is better because once some text in a repository, it is necessary to git rebase, which is rather messy.

  • Reject very short commit messages, such as 10 characters or less.

  • Move media files (.mp4, .mp3, .jpg, .png) from repositories that are only “supposed” to contain text to a repository hold media files that humans can’t read. The job of git-lfs (Large File System) is to move and replace binary files with a (texual) link to binary repositories.
  • etc.

Hooks On the client

The trouble with client hooks is that it’s rather intrusive to install them on each developer’s laptop.

The free Community Edition (CE) of IntelliJ IDEA from Jetbrains provides check-boxes to activate built-in Git hook functionality:

git intellij before commit 302x418-87kb

PROTIP: We don’t see how to get to Version Control in IntelliJ on videos because it’s often accessed via a hotkey - Apple command 9 on macs and Alt+9 on Windows.

See this video about VCS improvements in IntelliJ 2016.3.

Let’s use a sample project that already has .idea folder containing files that define configurations for IntelliJ.

IntelliJ has a project wizard to start from scratch on various languages.

Default Git Hook files

PROTIP: Every Git repository is created with a hooks folder containing sample hook files that are named so they don’t execute.

  1. Navigate into any Git repository you want to automate.

    cd .git/hooks

    Git and GitHub looks into this specific folder name for scripts to run when specific events occur.

    PROTIP: Git and GitHub have been written such that it recognizes specific file names for each internal event.

    PROTIP: Each Git repository is created with a set of sample automation files in the folder. But the file names end with “.sample” so that they won’t run.

    Listed alphabetically:

    applypatch-msg.sample
    commit-msg.sample
    post-update.sample
    pre-applypatch.sample
    pre-commit.sample
    pre-push.sample
    pre-rebase.sample
    pre-receive.sample
    prepare-commit-msg.sample
    update.sample
    

    There are others that Git automatically runs.

    pre-checkout
    pre-applypatch
    post-applypatch
    post-merge
    post-receive
    pre-auto-gc
    post-rewrite runs after a commit is modified by a git commit --amend or git rebase.
    
  2. PROTIP: Click each link above for the lastest version online at:

    https://github.com/git/git/tree/master/templates

    CAUTION: Nobody uses the default files as they are by just removing the “.sample” part of the file name because they probably don’t do what you want.

  3. Delete the sample files in your repo because it’s best to get the latest version from GitHub.

    No file extension

    Due to their Linux origins, Git and GitHub were written to recognize command files that have no file extension, since in Linux the type of command is defined in the first line within each file (with a “SheBang magic number”) rather than in the file name’s extension as in Windows.

    Multiple languages

    Git will run any script that can run on the command line (and properly installed):

    All commands in a particular hook file must be of the same language.

    Bash hook script

    During installation of git-lfs (GitHub’s Large File System), commands like these are placed in hook files:

    #!/bin/sh
    command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-commit.\n"; exit 2; }
    git lfs post-commit "$@"
    

    The above commands first checks if git-lfs is installed, then performs the git lfs command, with “$@” forwarding the attributes passed into the hook file.

    The hook files used by git-lfs are:

    • pre-push
    • post-checkout
    • post-commit
    • post-merge

    Python hook script

    NOTE: The minimal Python hook script (commit-msg):

    #!/usr/bin/env python
    # post-checkout hook
    import argparse
    def parse_args():
      pass
    def main(args=None):
      pass
    if __name__ == "__main__":
      args = parse_args()
    main(args)
    

    TODO: Print out values of arguments supplied when calling the hook.

    PowerShell script

    NOTE: To have Git run a PowerShell script, a shell script invokes an additional PowerShell (.ps1) file added in the hooks folder:

    #!/bin/sh
    # pre-commit (no file extension)
    c:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe \
    -ExecutionPolicy RemoteSigned -Command .\.git\hooks\pre-commit.ps1
    

Demo simple hooks

I have prepared a repository to demo use of local Git hooks to run. The repo contains a “hooks” folder containing script files. Files in the folder are copied into the .git/hooks folder where Git looks.

  1. In an internet browser, go to this repository (which we will later call the “upstream”):

    https://github.com/wilsonmar/git-utilities

  2. Sign in using your account. The rest of this document assumes “hotwilson” is your account name. So replace “hotwilson” with your own account whereever you see it.

  3. Fork the repo under your own account.

  4. Open a Terminal window.
  5. Make and cd into a folder where you will the repository will be downloaded.

    Again, in this example, you would change “hotwilson” to your own account name.

    
    cd ~ && mkdir gits && cd gits && mkdir hotwilson && cd hotwilson
    
  6. Confirm you are indeed where you want to create a new repository:

    
    pwd
    
  7. Clone to download:

    
    git clone https://github.com/hotwilson/git-utilities.git
    cd git-utilities
    
  8. Optionally, to enable updates from the original repo:

    
    git remote add upstream https://github.com/wilsonmar/git-utilities.git
    git remote -v
    

    Global templates

    NOTE Git looks for hooks in a global template folder applicable to all repositories when either a GIT_TEMPLATE_DIR variable is set in .bash_profile or .gitconfig contains a setting defined by command:

    git config --global init.templatedir /path/to/your/templates/
    

    Alternatively, symbolic links or symlinks can be used to link custom hooks to the ones in the .git/hooks folder.

  9. Copy these lines and paste in Terminal (at the root folder) to copy demo hook files from the hooks folder into the .gits/hooks folder:

    cp hooks/* .git/hooks
    chmod +x .git/hooks/*
    

    PROTIP: In Mac and Linux machines, scripts must be authorized to be run using the chmod command.

    PROTIP: It is too difficult and dangerous to edit and add/commit files inside the .git folder, which Git itself uses to track changes. So copying files in is the better approach than error messages such as:

    fatal: This operation must be run in a work tree
  10. Copy these lines and paste in the Terminal to make a change, then add and commit it for the Git Hook to take action:

    echo "random text" >>README.md
     git add .
     git commit -m"try git commit hooks"
     

Prettier node module code

To ensure consistency in formating within code files (without spending time caholing people), automatically format the code using prettier.io (github.com/prettier/prettier).

Here’s why automatica reformatting by Prettier is awesome:

  • No effort spent fixing formatting
  • Code copied in (from another project or from StackOverflow) automatically adjusts

  • No looking up rules in a style guide
  • When the style guide changes (or when it is first introduced) Prettier can automatically apply it across the whole code base

  • No time wasted (ill will created) discussing style in pull requests
  • Differences in coding style by different people won’t appear as different
  • White space can be removed automatically to safe space, and re-formatted for human reading

Below are steps in the shell script that installs prettier for use by Git.

  1. Create or navigate to a folder containing JavaScript NodeJs code (file index.js). That folder should have been initialized as a Git repository (has a .git folder at root). In my example, its:

    ~/gits/wilsonmar/node-sample1
  2. Install yarn or npm. Upgrade if already installed:

    brew upgrade yarn
  3. Install using yarn/npm,

    brew upgrade yarn
    yarn add --dev prettier pretty-quick
    npm install prettier --global

    The above adds file yarn.lock and package.json, which contains a list of dependencies expanded into folder node_modules.

    https://npm.im/pretty-quick (https://github.com/azz/pretty-quick)

  4. Edit file _____ to edit setting to control specific file formats by adding the format between square brackets: javascript, TypeScript, HTML, CSS, JSON, etc. Specify “true” or false for automatic “editor.formatOnSave”:

    {
      "folders": [],
      "settings": {},
      "[javascript]": {
     "editor.formatOnSave": true
      }
    }
  5. Invoke from command line :

    yarn pretty-quick --global

    Alternately, update your current code:

    prettier "*/.ts" --write

    Example response (with either “Everything is awesome” or changefix message):

    yarn run v1.21.1
    $ /Users/wilson_mar/gits/wilsonmar/node-sample1/node_modules/.bin/pretty-quick
    🔍  Finding changed files since git revision null.
    🎯  Found 3 changed files.
    ✍️  Fixing up README.md.
    ✍️  Fixing up index.js.
    ✅  Everything is awesome!
    ✨  Done in 0.31s.
    

    https://www.npmjs.com/package/pretty-quick

  6. Install the Prettier in your code editor. In VS Code, open the Command Palette (under the View submenu, or using Cmd+Shift+P on Mac and Ctrl+Shift+P on Windows) to type “Extensions: Install Extensions”. In place of “Search Extensions in Markeplace”, type “Prettier” (by Esben Petersen), then click “Install”. Click “Install” in the detail screen.

    Search for “Prettier”, click “Install”, and then “Reload” once the installation is complete.

    Alternately, via the command line:

    code --list-extensions
    code --install-extension esbenp.prettier-vscode
    

    To upgrade, install with –force option.

    This is like SublimeLinter

  7. Invoke pretty-quick manually in your code editor opened to a file. In VSCode, Command Palette, type “Format Document” and select it so Prettier will tidy up your code.

  8. For automatic invocation on commit via Git hooks, add a dependency to husky:

    yarn add --dev husky
  9. Edit package.json to add the hook to husky:

    {
      "devDependencies": {
     "husky": "^3.1.0",
     "prettier": "^1.19.1",
     "pretty-quick": "^2.0.1"
      }
      "husky": {
     "hooks": {
       "pre-commit": "pretty-quick --staged"
     }
      }
    }

    The utility automatically adds 21 hook files (with no file extension) to .sample files within folder .git/hooks:

    • applypatch-msg for applypatch-msg.sample
    • commit-msg for commit-msg.sample
    • fsmonitor-watchman.sample
    • post-applypatch
    • post-checkout
    • post-commit
    • post-merge
    • post-receive
    • post-rewrite
    • post-update for post-update.sample
    • pre-applypatch for pre-applypatch.sample
    • pre-auto-gc
    • pre-commit for pre-commit.sample
    • pre-merge-commit
    • pre-push for pre-push.sample
    • pre-rebase for pre-rebase.sample
    • pre-receive for pre-receive.sample
    • prepare-commit-msg for prepare-commit-msg.sample
    • push-to-checkout
    • sendemail-validate
    • update for update.sample

    These are described at https://github.com/typicode/husky#readme

    Variables in hooks

    Each of the demo hook files can contain variables. For example:

    #!/bin/sh
     COMMIT_MSG_FILE=$1  # ".git/COMMIT_EDITMSG"
     echo "prepare-commit-msg: $(cat $COMMIT_MSG_FILE)"
     exit 0
    

    CheatSheet Matrix

    Click to expand this cheatsheet (from Daniel Convissor) to see the 15 hooks:

    git-hooks-cheat-650x292-64103.jpg

    The table lists for each hook what command triggers it, when it runs, what parameters are passed into it, and what happens when the hook exits abnormally (with a 1).

  10. You may delete “.sample” files there.

  11. Make a git commit:

    git commit -m"test commit with husky prettier"

    Skip hook invocation

    One can suppress hooks from firing with the --no-verify command attribute:

    git commit --no-verify
    
  12. Create aliases to make git commands with flags defined in

    https://github.com/azz/pretty-quick

  13. Use git diff utility

    Notice the changes Prettier made to JavaScript coding:

    • Use “ (double quotes) instead of ‘ (single quotes)
    • Add an extra space before the function body (after the parameter list)
    • Add a semicolon at the end of the return statement

    JavaScript Style Guides

    As a starting point for your own organization:

More about Prettier:

  • https://www.codereadability.com/automated-code-formatting-with-prettier/

Flake8 Linter for Python

flake8 is a python wrapper that glues together plugins to check the style and quality of Python code. It runs in GitLab.

  1. To Install, run (within virtualenv):

    pip install flake8
    pip install flake-bugbear   # to add additional rules.
    flake8 --version
    pip install pre-commit  # to $PROJ/.git/hooks/pre-commit
    pre-commit install
    pip install black    # to reformat code
    # bandit  # to check for python code vulnerabilities
    # sign-commit  # to add signature verification
    

    This is explained by Kenneth H. East.

  2. Add a .flake8 configuration file at the root of the project repo.

  3. Variables are used to select and ignore specific warnings or errors:

    flake8 --select W292,E712 --ignore E501,F401,F841,W293,E303,W291,E261,E262,E265,E231,E251,E302 \
    path/to/code/

    Ones that should be fixed:

    • W292 no newline at end of file - this can cause the line to not be read.
    • E712 comparison to False should be ‘if cond is False:’ or ‘if not cond:’

    The whole team should decide whether to enforce:

    • W191 indentation contains tabs
    • E305 expected 2 blank lines after class or function definition, found 1
    • E302 expected 2 blank lines, found 1

    Stylistic choices than can be ignored, but maybe not:

    • E501 line too long
    • F401 module imported but unused
    • F841 local variable ___ is assigned to but never used
    • W293 blank line contains whitespace
    • E303 too many blank lines
    • W291 trailing whitespace
    • E261 at least two spaces before inline comment
    • E262 inline comment should start with ‘# ‘
    • E265 block comment should start with ‘# ‘
    • E231 missing whitespace after ‘,’
    • E251 unexpected spaces around keyword / parameter equals

    Flake8 calls pep8, PyFlakes, pycodestyle, Ned Batchelder’s McCabe, and third-party plugins.

    PROTIP: Flake8 does not issue warnings on lines that contain a # noqa in-line comment at the end.

git commit hooks

First of all, there is already some editing to git commit. Git does not allow empty commit messages. If a commit message does not include a message, your favorite editor is opened for you to enter one. If you still haven’t typed anything in your editor, Git aborts the commit.

The flow of processing on the client is illustrated by this (from Johan Abildskov & Jan Krag of Praqma)
git hooks diagram 650x251(click for full screen image)

Upon git commit, client hooks are executed in this order:

  1. The pre-commit hook runs after a git commit is executed, but before the commit message editor is displayed. Since an exit from this with anything other than zero aborts the commit, it is the place to check assets involved in the commit itself (rather than the commit message), to run linters and unit tests on the local laptop. This program has access to the commit date/time, author name and email.

    NOTE: pre-commit hooks do not support hooks with side effects (such as modifying files and adding them to the index with git add).

    A non-zero exit aborts the commit, which can happen if a file being committed contains “#DONTPUSH” (or other keyword recognized by the hook program) developers put in their code as a reminder to hold off pushing to the team repository.

    The following pre-commit code keeps debugging code (such as “console.log” debugging statements) from reaching the shared code base:

    FILES_PATTERN='\.(js|coffee)(\..+)?$'
    FORBIDDEN='console.log'
    git diff --cached --name-only | \
     grep -E $FILES_PATTERN | \
     GREP_COLOR='4;5;37;41' xargs grep --color --with-filename -n $FORBIDDEN && echo 'COMMIT REJECTED Found "$FORBIDDEN" references. Please remove them before commiting' && exit 1
    

    FILES_PATTERN above contains a Regular Expression.

  2. The prepare-commit-msg hook is invoked after receiving a git commit, just prior to firing up the commit message editor. This hook can edit the commit message in a way that cannot be suppressed. For example, ensuring a capital letter.

    QUESTION: This has access to the commit SHA-1 (when operating on an existing commit)?

  3. The commit-msg hook adjusts the commit message after it has been edited in order to ensure conformity to a standard or to reject based on any criteria. It can abort the commit if it exits with a non-zero value.

  4. The post-commit hook is called after the actual commit is made so that it cannot disrupt the commit. It is mainly used for sending notifications (emails, SMS, Slack, Twitter, etc.). This NOTE describes deploying to a Local Web Server with a Post-Commit Hook.

git push hooks

The order of execution is:

  1. pre-receive run just before pushed files are updated. So it can abort the receive process by exiting with a non-zero status. Thus, it is used to enforce commit policies and reject the entire commit if it is deemed unsatisfactory. (code coverage)

  2. update filters each commit ref made to the remote repository independently. It can be used to reject or accept each ref being pushed.

  3. post-receive is triggered after an update has been done on the remote repository. So it cannot abort the update process. But it can trigger notifications on a successful remote repository update. (send email, run load tests, so that a log of notifications is stored on a remote server.

Sample JavaScript Mocha linter

From https://scotch.io/tutorials/using-git-hooks-in-your-development-workflow#client-side-hooks

#!/bin/bash
 
# Exits with non zero status if tests fail or linting errors exist
num_of_failures=`mocha -R json | grep failures -m 1 | awk '{print $2}' | sed 's/[,]/''/'`
 
errors=`jscs -r inline ./test/test.js`
num_of_linting_errors=`jscs -r junit ./test/test.js | grep failures -m 1 | awk '{print $4}' | sed 's/failures=/''/' | sed s/">"/''/ | sed s/\"/''/ | sed s/\"/''/`
 
if [ $num_of_failures != '0' ]; then
  echo "$num_of_failures tests have failed. You cannot commit until all tests pass.
        Commit exiting with a non-zero status."
  exit 1
fi
 
if [ $num_of_linting_errors !=  '0' ]; then
  echo "Linting errors present. $errors"
  exit 1
fi
   

Useful Hook examples

pre-rebase hooks

  1. Use a text editor to create in the hooks folder a “pre-rebase” containing this text described in YouTube video Life’s better with Git hooks - Grumpy Gits [17:58] Nov 16, 2015 by Deepank Vora.

    #!/bin/sh
    echo "Custom rebasing message."
    exit 1
    

    (Unlike Windows, Linux defines file handling in the first line inside the file).

    PROTIP: Non-zero finish aborts the commit.

  2. Enable the new script:

    
    chmod +x pre-rebase
    

    This would avoid messages such as:

    -bash: ./pre-rebase: No such file or directory
    
  3. Run the new script to make sure it works:

    
    ./pre-rebase
    
  4. Open another Terminal shell window to the working directory of the repo to run the git rebase master command.

    You should see the custom message.

    QUESTION: If instead you see:

    Current branch master is up to date.
    

    View pre-commit.sample

  5. Use a text editor Set to always open .sample files with a text editor.

    #!/bin/sh at the top of the file means that it’s a Bash shell script.

    (Author Junio C Hamano is the primary developer of Git, now works at Google.)

applypatch-msg

GitLab

You can connect to the GitLab API using your API key to perform actions inside GitLab when performing, for example, a commit. This you can do from the client side, but of course, you can also use the server-side hooks, such as post-receive to run programs and scripts after they have reached the remote.

GitLab has a GraphQL API (in alpha as of this writing) among its many APIs

closes or fixes

  1. The following script is saved as .git/hooks/commit-msg (with no file extension):

    #!/bin/sh
    PRIVATE_TOKEN="API_KEY"
    GITLAB_URL="https://your_gitlab_domain.com"
    URL=`git config --get remote.origin.url`
    PROJECT=`basename ${URL} .git | cut -d':' -f2`
    for issue_id in `grep -o -e "\(closes\|fixes\) #[0-9]\+" $1 | cut
    -d'#' -f2`; do
        curl -X PUT -d "closed=1" \
            ${GITLAB_URL}api/v3/projects/${PROJECT}/issues/${issue_
    id}/?private_token=${PRIVATE_TOKEN}
    done
    
  2. Edit the program with your gitlab URL.

    If this script sees a commit message text containing “closes #” or “fixes #”, the hook closes the issue identified by the number after the # symbol.

  3. Make the file executable to enable this functionality.

    chmod +x *
    

Resources

The cananoical resources on Git Hooks:

More Resources

Atlassian’s tutorial on Git Hooks - Local hooks

Git Website: how to

023 Introduction to Git Hooks by Dan Gitschooldude

gated-commit with git pre-commit [12:41] by fullstack

Git hooks man page

Andrew Burgess for Tuts [7:28] June 28, 2012 describes run of mocha test processed by a Bash script.

Git hooks made easy

contains contributor code in the git team’s repo

Git Hooks Practical uses on Windows

Git - How to validate commit messages? 27 May 2019 by Marcell Lipp

https://confluence.atlassian.com/bitbucketserverkb/how-to-scan-for-and-remove-passwords-or-secrets-in-bitbucket-server-repositories-973473524.html

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible
  53. Kubernetes Operators
  54. OPA (Open Policy Agent) in Rego language

  55. MySQL Setup

  56. Threat Modeling
  57. SonarQube & SonarSource static code scan

  58. API Management Microsoft
  59. API Management Amazon

  60. Scenarios for load
  61. Chaos Engineering