jq (JSON Query utility for Bash CLI scripts)

Define JMESPATH syntax in –query; Extract city and other data from IP address lookup website ipinfo.io, Docker Hub, and other sites

Overview

Install jp
How to jq
More commands
GitHub API
JMESPATH
JMESPATH –query
- AWS EC2 AMI images
- AWS ec2 describ-volumes
–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=available” \
–query ‘sort_by(Volumes[], &VolumeId)[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[].InstanceId, State: Attachments[].State}’
“Name=status,Values=available”

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

The “j” in the jq program name means it queries a JSON-formatted file to filter output.

The jq utility is like the sed Linux utility, but for JSON. They both reformat and transform text.

On a Mac, command brew install jq reveals that the utility has a dependency of the oniguruma regular expression handler and is from Stephen Dolan’s stedolan.github.io/jq

Install jp

On Mac:

brew tap jmespath/jmespath
 brew install jp

Output from an ARM-based mac laptop:

==> Fetching jp
==> Downloading https://ghcr.io/v2/homebrew/core/jp/manifests/1.1.12
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/jp/blobs/sha256:37b85d8d9876ffae1cdf4a3897ca558f2586a826a229d9b85d5799b33e338a89
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:37b85d8d9876ffae1cdf4a3897ca558f2586a826a229d9b85d5799b33e338a89?se=2022-12-24T15%3A45%3A00Z&sig=i%2FoDBRCeCst8TrPsaStWPNkXdv%2FEC00feBgLM5AaDT4%3D&sp=r&spr=https&sr=b&sv=201
######################################################################## 100.0%
==> Pouring jp--1.1.12.arm64_monterey.bottle.tar.gz
🍺  /opt/homebrew/Cellar/jp/1.1.12: 3 files, 3MB

On Ubuntu:
On Windows Git Bash, download jq-win64.exe from github.com/stedolan/jq/releases into a bin folder in the PATH, and specify the program in commands:

How to jq

The jq author’s documentation at
https://stedolan.github.io/jq/manual presents a large (and potentially confusing) list of options.

Here’s an abridged version with examples explained step-by-step.

The ipinfo.io website returns the IP address and its metadata: location Lattitude and Longitude, postal (ZIP code), Unix time zone, carrier (org), State of the Union (region), etc.

curl ipinfo.io

yields:

{
  "ip": "184.153.97.231",
  "hostname": "cpe-184-153-97-231.nyc.res.rr.com",
  "city": "Woodside",
  "region": "New York",
  "country": "US",
  "loc": "40.7454,-73.9054",
  "org": "AS12271 Charter Communications Inc",
  "postal": "11377",
  "timezone": "America/New_York",
  "readme": "https://ipinfo.io/missingauth"
}

NOTE: All JSON responses begin and end with curly braces between { and }.

A dot represents the entire input, but formatted with indents and -Colorized:

curl ipinfo.io | jq -C ‘.’
.[] returns each element of the array, one at a time,

curl ipinfo.io | jq ‘.[]’
If you only want one city name (with no JSON braces or quote marks):

curl ipinfo.io | jq -r .city

NOTE: jq receives a file piped in, as specified by the pipe character .

To return the response in a variable within a Bash script:

CITY=$( curl -s ipinfo.io | jq -r .city )

-s is so you don’t see the progress, such as:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                              Dload  Upload   Total   Spent    Left  Speed
100   265  100   265    0     0   1007      0 --:--:-- --:--:-- --:--:--  1023

On my computer, I can type ipcity to invoke the above because I have the command in my aliases.zsh file defining keyboard shortcuts:
```
alias ipcity="curl -s ipinfo.io | jq -r .city"
```

More at:

https://thoughtbot.com/blog/jq-is-sed-for-json
https://shapeshed.com/jq-json/

### Docker Hub API

Docker Hub provides an API to access public Docker images.

curl ‘https://registry.hub.docker.com/v2/repositories/library/debian/tags/’

The response starts with:

{"count": 511, "next": "https://registry.hub.docker.com/v2/repositories/library/debian/tags/?page=2", "previous": null, "results": [{"name": "unstable-slim", "full_size": 27750048, "images": [{"size": 31296808, "architecture": "ppc64le", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}, {"size": 26435103, "architecture": "s390x", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}, {"size": 25698949, "architecture": "arm", "variant": "v5", "features": null ....

In JSON, a left bracket ([) begin a list structure:

“results”: [{“name”: “unstable-slim”,:

Note that “name” is an attribute, so would require double-quotes. ???

To return a list of just values beginning with “unstable-slim”:


curl -s 'https://registry.hub.docker.com/v2/repositories/library/debian/tags/'|jq '."results"[]["name"]'

The output at time of this writing:

"rc-buggy-20221219"
"rc-buggy"
"experimental-20221219"
"experimental"
"bookworm-20221219-slim"
"bookworm"
"11.6-slim"
"11.6"
"11-slim"
"11"

More commands

To list the first element from the root (counting from zero):

jq ‘.[0]’

GitHub API

https://stedolan.github.io/jq/tutorial/

JMESPATH

### JMESPATH extension in VSCode

https://www.youtube.com/watch?v=InExHoXJ9Xc

https://docs.aws.amazon.com/cli/v1/userguide/cli-usage-filter.html

JMESPATH –query

jq, grep, awk, sed, etc. filter output from text output type obtained from the server.

However, adding the optional –query switch as part of aws AND az commands does filtering BEFORE sending it to your workstation.

The command that lists everything (922 at time of this writing):

aws ec2 describe-instances
Know that the full range of specifications for –query is defined by the industry standard JMESPATH query format website at:

https://jmespath.org/

https://jmespath.org/specification.html

https://jmespath.org/tutorial.html

https://jmespath.org/examples.html

https://github.com/jmespath/jmespath.py

https://opensourceconnections.com/blog/2015/07/27/advanced-aws-cli-jmespath-query/

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html#controlling-output-filter

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

jasoneditoronline.org
A gentle yet deep tutorial is at AzureCitadel.com
- Azure CLI output filter using JMESPath
- VIDEO: “JSON: Filter, sort or Transform JSON Data using JMESPath”
The command sure to return a lot is described by this blog

AWS EC2 AMI images
- https://www.doaws.pl/blog/2021-12-05-how-to-master-aws-cli-in-15-minutes/how-to-master-aws-cli-in-15-minutes
- this command
- https://blog.ashiny.cloud/2017/09/25/useful-jmespath-in-awscli/
- https://winder.ai/how-to-list-all-amis-for-each-region-in-aws/ don’t work
REMEMBER: There is a different ID for each region for the same AWS EC2 image name. And you want the lastest among multiple versions of the same image, with different dates.
```
#!/bin/bash
if [ -z "$1" ] ; then
 echo "Please pass the name of the AMI"
 exit 1
fi
IMAGE_FILTER="${1}"
declare -a AWS_REGIONS=$( aws ec2 describe-regions --output text --query "Regions[].[RegionName]" | tr "\\n" " " )
 # | tr -d "\\""))
for r in "${AWS_REGIONS[@]}" ; do
 ami=$(aws ec2 describe-images --query 'Images[*].[ImageId]' --filters "Name=name,Values=${IMAGE_FILTER}" --region ${r} --output json | jq '.[0][0]')
 printf "\\"${r}\\" = ${ami}\\n"
done
```

From https://blog.scottlowe.org/2019/03/01/advanced-ami-filtering-with-jmespath/

aws ec2 describe-images --owners 099720109477 \
--filters Name=name,Values="*ubuntu-bionic-18.04*" \
Name=virtualization-type,Values=hvm \
Name=root-device-type,Values=ebs \
Name=architecture,Values=x86_64 \
--query 'sort_by(Images,&CreationDate)[?Name!=`null`]|[?contains(Name,`ubuntu-eks`)==`false`]|[?contains(Name,`minimal`)==`false`]|[-1].ImageId'
"ami-0135afc6d226a70a4"

How many images will be returned if no filtering? 81249

aws ec2 describe-images –owner amazon –output text wc -l

View JSON format to identify the names used to specify filtering:

aws ec2 describe-images –owner amazon –output json

  "Images": [
     {
         "Architecture": "x86_64",
         "CreationDate": "2022-08-03T10:55:58.000Z",
         "ImageId": "ami-0292c068b453b70cc",
         "ImageLocation": "amazon/Deep Learning AMI GPU TensorFlow 2.6.5 (Ubuntu 18.04) 20220802",
         "ImageType": "machine",
         "Public": true,
         "OwnerId": "898082745236",
         "PlatformDetails": "Linux/UNIX",
         "UsageOperation": "RunInstances",
         "State": "available",
         "BlockDeviceMappings": [
             {
                 "DeviceName": "/dev/sda1",
                 "Ebs": {
                     "DeleteOnTermination": true,
                     "Iops": 3000,
                     "SnapshotId": "snap-06eee5b870416b8ad",
                     "VolumeSize": 35,
                     "VolumeType": "gp3",
:...skipping...

Count the number of AMI images: q

aws ec2 describe-images –owner amazon –output json –query “Images”

To avoid the “skipping”, get just the first Image’s json structure:

aws ec2 describe-images –owner amazon –output json –query “Images[0]”

{
 "Architecture": "x86_64",
 "CreationDate": "2022-07-27T10:44:11.000Z",
 "ImageId": "ami-0d0d43dd4e2a847da",
 "ImageLocation": "amazon/Deep Learning AMI GPU TensorFlow 2.6.5 (Amazon Linux 2) 20220726",
 "ImageType": "machine",
 "Public": true,
 "OwnerId": "898082745236",
 "PlatformDetails": "Linux/UNIX",
 "UsageOperation": "RunInstances",
 "State": "available",
 "BlockDeviceMappings": [
     {
         "DeviceName": "/dev/xvda",
         "Ebs": {
             "DeleteOnTermination": true,
             "Iops": 3000,
             "SnapshotId": "snap-0ab22834e0079e294",
             "VolumeSize": 35,
             "VolumeType": "gp3",
             "Throughput": 125,
             "Encrypted": false
         }
     }
 ],
 "Description": "Built with AWS optimized TensorFlow, NVIDIA CUDA, cuDNN, NCCL, GPU Driver, Docker, NVIDIA-Docker and EFA support. For a fully managed experience, check: https://aws.amazon.com/sagemaker",
 "EnaSupport": true,
 "Hypervisor": "xen",
 "ImageOwnerAlias": "amazon",
 "Name": "Deep Learning AMI GPU TensorFlow 2.6.5 (Amazon Linux 2) 20220726",
 "RootDeviceName": "/dev/xvda",
 "RootDeviceType": "ebs",
 "SriovNetSupport": "simple",
 "VirtualizationType": "hvm",
 "DeprecationTime": "2024-07-27T10:44:11.000Z"
}

Architecture”: “x86_64”,
“PlatformDetails”: “Linux/UNIX”,
State: Only ones available or not?

Name contains Description contains text “Tensorflow”
For different states:

aws ec2 describe-images –owner amazon –output json –query “Images[0].State”
Sort by date: https://itecnote.com/tecnote/amazon-web-services-sort-by-date-with-jmespath/

aws lambda list-functions –query “reverse(sort_by(Functions, &LastModified))[:2]”
Get just has lots of AMI images, so this command returns a lot of lines:

aws ec2 describe-images –owner amazon –output text wc -l
Return just the first json

aws ec2 describe-images –owner amazon –output json
VIDEO: “Regular Expressions for Performance Engineers #10 - JMESPath”
https://blog.scottlowe.org/2019/03/01/advanced-ami-filtering-with-jmespath/

Videos:

https://www.youtube.com/watch?v=gdDl5DywQGI

AWS ec2 describ-volumes

https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

PROTIP: Specify first root item using [slices]

aws ec2 describe-volumes –query ‘Volumes[0]’

{
 "Attachments": [],
 "AvailabilityZone": "us-west-2c",
 "CreateTime": "2022-12-19T22:00:45.560000+00:00",
 "Encrypted": false,
 "Size": 32,
 "SnapshotId": "",
 "State": "available",
 "VolumeId": "vol-0be808a8215fc5357",
 "Iops": 100,
 "Tags": [
     {
         "Key": "Name",
         "Value": "eks-cluster-with-new-vpc-dynamic-pvc-fdef34a5-b867-4421-9787-b7b14ab810d6"
     },
     {
         "Key": "ebs.csi.aws.com/cluster",
         "Value": "true"
     },
     {
         "Key": "KubernetesCluster",
         "Value": "eks-cluster-with-new-vpc"
     },
     {
         "Key": "CSIVolumeName",
         "Value": "pvc-fdef34a5-b867-4421-9787-b7b14ab810d6"
     },
     {
         "Key": "kubernetes.io/created-for/pvc/namespace",
         "Value": "kubecost"
     },
     {
         "Key": "kubernetes.io/cluster/eks-cluster-with-new-vpc",
         "Value": "owned"
     },
     {
         "Key": "kubernetes.io/created-for/pv/name",
         "Value": "pvc-fdef34a5-b867-4421-9787-b7b14ab810d6"
     },
     {
         "Key": "kubernetes.io/created-for/pvc/name",
         "Value": "kubecost-cost-analyzer"
     }
 ],
 "VolumeType": "gp2",
 "MultiAttachEnabled": false
}

The VolumeId of attached (flatted with []), with no labels:

aws ec2 describe-volumes –query ‘Volumes[3].[VolumeId, State, AvailabilityZone, Size, VolumeType, Attachments[].[InstanceId, State][]][]’
```
[
 "vol-0be808a8215fc5357",
 "available",
 "us-west-2c",
 32,
 "gp2"
]
```
With labels Multi-select:

aws ec2 describe-volumes
–query ‘Volumes[0].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}’
```
{
 "VolumeId": "vol-0be808a8215fc5357",
 "VolumeType": "gp2",
 "InstanceId": null,
 "State": null
}
```
Sorted by Date:

aws ec2 describe-volumes
–output text
–query ‘sort_by(Volumes[], &CreateTime)[].{CreateTime: CreateTime, VolumeId: VolumeId, VolumeType: VolumeType}’ –query ‘sort_by(Volumes[], &CreateTime)[].{VolumeId: VolumeId}’

–query ‘sort_by(Volumes[], &CreateTime)[].{CreateTime: CreateTime, VolumeId: VolumeId, VolumeType: VolumeType}’

–query ‘sort_by(Volumes[], &VolumeId)[].{VolumeId: VolumeId, CreateTime: CreateTime, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}’

–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=available” \

–query ‘sort_by(Volumes[], &VolumeId)[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[].InstanceId, State: Attachments[].State}’

“Name=status,Values=available”

aws ec2 describe-volumes
–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=attached”
–query ‘Volumes[?Size > 0].{Id:VolumeId,Size:Size,Type:VolumeType}’

Filter:

aws ec2 describe-volumes
–filters “Name=availability-zone,Values=us-west-1a”

Volumes[].CreateTime

--filters "Name=status,Values=null"

aws ec2 describe-volumes
–filters “Name=availability-zone,Values=us-west-2a” “Name=status,Values=attached”
–query ‘Volumes[?Size > 50].{Id:VolumeId,Size:Size,Type:VolumeType}’

https://random.ac/cess/2017/10/31/interactive-aws-cli-query-filtering/

https://how.wtf/aws-cli-query-examples.html

jpterm

https://random.ac/cess/2017/10/31/interactive-aws-cli-query-filtering/

https://www.geeksforgeeks.org/how-to-install-jmespath-term-on-macos/

sudo pip install jmespath-terminal

JMESPATH in Azure

https://www.codemag.com/Article/2001021/Azure-CLI

az vm list

az vm start -g rg-vm -n wm-Ubuntu18

JMESPATH in Python

VIDEO: JMESPATH in PYTHON - JSON query language

sudo pip install jmespath-terminal

Add a line within JSON

cat /home/ubuntu/secretid-token.json

jq –arg id approle-secretid-token ‘. + {id: $id}’ > secretid-token.json

From:

    "renewable": true
  }
}

to:

  },
  "id": "approle-secretid-token"
}

Write a secret to Vault’s KV secrets engine. This will give the Chef recipe that authenticates itself to Vault with the AppRole method a secret to read.

References

https://www.youtube.com/watch?v=5KxWfeFPPVY “Episode 1: Using jq to Get the Results You Need From Any Command Line Interface” by Kenneth G. Hartman on the SANS channel references https://headintheclouds.site/episodes/episode1

curl -O https://headintheclouds.site/extras/ep1/policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetEncryptionConfiguration"
            ],
             "Resource": [
                "arn:aws:s3:::khartman-logs-645654/*",
                "arn:aws:s3:::khartman-logs-645654"
            ]
        }
    ]
}

Get Two elements of the array:

cat policy.json | jq '.Version, .Statement[].Effect'

"2012-10-17"
"Allow"

Get the first element of the array:

jq '.Statement[0]' policy.json

{
  "Effect": "Allow",
  "Action": [
 "s3:GetObject",
 "s3:PutObject",
 "s3:PutObjectAcl",
 "s3:GetEncryptionConfiguration"
  ],
  "Resource": [
 "arn:aws:s3:::khartman-logs-645654/*",
 "arn:aws:s3:::khartman-logs-645654"
  ]
}

Get just the Action elements without the brackets:

cat policy.json | jq '.Statement[0].Action[]'

"s3:GetObject",
  "s3:PutObject",
  "s3:PutObjectAcl",
  "s3:GetEncryptionConfiguration"

Sub-claims

https://docs.akeyless.io/docs/sub-claims

Sub-claims are attribute-based access control (ABAC), also known as policy-based access control. They are used with Auth Methods JWT/OIDC, K8s, SAML, and LDAP. Only clients whose token contains these sub-claims (in the case of JWT/OIDC) or attributes (in the case of SAML) will be allowed to access the rules defined in the role.

In this example sub-claim definition using CLI:

akeyless assoc-role-am --role-name r1 \
--am-name Okta 
--sub-claims Groups=Engineering,Security \
--sub-claims Email=james@example.com,linda@example.com

Only JWTs or SAML-XML containing both the Groups= and Email= claims/attributes that match values of Engineering and james@example.com, would be authorized.

More on API Microservices

This is one of a series:

Wilson Mar