Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Extract data from APIs in Bash scripts

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

The jq utility is like the sed Linux utility, but for JSON. They both reformat and transform text.

The “j” in the jq program name means it queries a JSON-formatted file to produce filtered output.

On a Mac, command brew install jq reveals that the utility has a dependency of the oniguruma regular expression handler and is from Stephen Dolan’s stedolan.github.io/jq

Install

On Mac:

    brew install jq

On Ubuntu:

    apt-get install jq

On Windows Git Bash, download jq-win64.exe from github.com/stedolan/jq/releases into a bin folder in the PATH, and specify the program in commands:

    echo '{"foo": 0}' | ./jq-win64.exe .
    {
      "foo": 0
    }

How to jq

https://stedolan.github.io/jq/manual presents a large (and potentially confusing) list of options.

Here’s an abridged version with examples explained step-by-step.

  1. The ipinfo.io website returns the IP address and its metadata: location Lattitude and Longitude, postal (ZIP code), Unix time zone, carrier (org), State of the Union (region), etc.

    curl ipinfo.io

    yields:

    {
      "ip": "184.153.97.231",
      "hostname": "cpe-184-153-97-231.nyc.res.rr.com",
      "city": "Woodside",
      "region": "New York",
      "country": "US",
      "loc": "40.7454,-73.9054",
      "org": "AS12271 Charter Communications Inc",
      "postal": "11377",
      "timezone": "America/New_York",
      "readme": "https://ipinfo.io/missingauth"
    }
    

    NOTE: All JSON responses begin and end with curly braces between { and }.

  2. A dot represents the entire input, but formatted with indents:

    curl ipinfo.io | jq ‘.’

  3. .[] returns each element of the array, one at a time,

    curl ipinfo.io | jq ‘.[]’

  4. If you only want one city name (with no JSON braces or quote marks):

    curl ipinfo.io | jq -r .city

    NOTE: jq receives a file piped in, as specified by the pipe character .
  5. To return the response in a variable within a Bash script:

    CITY=$( curl ipinfo.io | jq -r .city )

More at:

  • https://thoughtbot.com/blog/jq-is-sed-for-json
  • https://shapeshed.com/jq-json/

### GitHub API

https://stedolan.github.io/jq/tutorial/

### Docker Hub API

Docker Hub provides an API to access public Docker images.

curl ‘https://registry.hub.docker.com/v2/repositories/library/debian/tags/’

The response starts with:

{"count": 511, "next": "https://registry.hub.docker.com/v2/repositories/library/debian/tags/?page=2", "previous": null, "results": [{"name": "unstable-slim", "full_size": 27750048, "images": [{"size": 31296808, "architecture": "ppc64le", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}, {"size": 26435103, "architecture": "s390x", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}, {"size": 25698949, "architecture": "arm", "variant": "v5", "features": null ....
   

In JSON, a left bracket ([) begin a list structure:

“results”: [{“name”: “unstable-slim”,:

Note that “name” is an attribute, so would require double-quotes. ???

  1. To return a list of just values beginning with “unstable-slim”:

    curl ‘https://registry.hub.docker.com/v2/repositories/library/debian/tags/’|jq ‘.”results”[][“name”]’

    ???

  2. To list the first element from the root (counting from zero):

    jq ‘.[0]’

More on API Microservices

This is one of a series: