search by tags

for the user

adventures into the land of the command line

jq! i just discovered u!

Parsing http JSON response objects in bash is a pain in the ass. Say you have some JSON blob returned from a request somewhere and you store it in foo

$ foo=$(curl 'https://gist.githubusercontent.com/djosephsen/a1a290366b569c5b98e9/raw/c0d01a18e16ba7c75d31a9893dd7fa1b8486a963/docker_issues')

Trying to parse this with just bash is going to be a nightmare. it’s hard in bash to even determine the variable type. We could try:

$ declare -p $foo

But chances are you are gonna end up with something like:

declare: [ { "url":: not a valid identifier
declare: [ { "url":: not found

So what do we do? Does bash see a string? An array? An associative array? Tbh, I don’t wanna reinvent the wheel. Someone already made one and it works.

$ apt-get install jq

If we pipe our response through jq, we get something we can work with

$ echo ${foo} | jq .

[
  {
    "body": "**Description**
    .
    .
    .",
    "closed_at": null,
    "updated_at": "2015-11-17T20:48:17Z",
    "title": "docker daemon crash when starting container after: RequestAddress(LocalDefault/172.17.0.0/16, , map[])",
    "number": 18048,
    "id": 117446711,
    "html_url": "https://github.com/docker/docker/issues/18048",
    "events_url": "https://api.github.com/repos/docker/docker/issues/18048/events",
    "comments_url": "https://api.github.com/repos/docker/docker/issues/18048/comments",
    "labels_url": "https://api.github.com/repos/docker/docker/issues/18048/labels{/name}",
    "url": "https://api.github.com/repos/docker/docker/issues/18048",
    "user": {
      "site_admin": false,
      "following_url": "https://api.github.com/users/bschiffthaler/following{/other_user}",
      "followers_url": "https://api.github.com/users/bschiffthaler/followers",
      "html_url": "https://github.com/bschiffthaler",
      "url": "https://api.github.com/users/bschiffthaler",
      "gravatar_id": "",
      "avatar_url": "https://avatars.githubusercontent.com/u/8308279?v=3",
      "id": 8308279,
      "login": "bschiffthaler",
      "gists_url": "https://api.github.com/users/bschiffthaler/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/bschiffthaler/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/bschiffthaler/subscriptions",
      "organizations_url": "https://api.github.com/users/bschiffthaler/orgs",
      "repos_url": "https://api.github.com/users/bschiffthaler/repos",
      "events_url": "https://api.github.com/users/bschiffthaler/events{/privacy}",
      "received_events_url": "https://api.github.com/users/bschiffthaler/received_events",
      "type": "User"
    },
    "labels": [],
    "state": "open",
    "locked": false,
    "assignee": null,
    "milestone": null,
    "comments": 0,
    "created_at": "2015-11-17T20:48:17Z"
  },
  .
  .
  .
]

jq accepts has some useful arguments which let you determine the type

$ echo '[][]{}' | jq type
"array"
"array"
"object"

As well as the length

$ echo '[][]{}' | jq length
0
0
0

So this curl returns an array of associative arrays. Let’s try and expand those outer square brackets and get to the associative arrays inside.

$ echo ${foo} | jq '.[]'

{
  "body": "**Description**
  .
  .
  .",
  "closed_at": null,
  "updated_at": "2015-11-17T20:48:17Z",
  "title": "docker daemon crash when starting container after: RequestAddress(LocalDefault/172.17.0.0/16, , map[])",
  "number": 18048,
  "id": 117446711,
  "html_url": "https://github.com/docker/docker/issues/18048",
  "events_url": "https://api.github.com/repos/docker/docker/issues/18048/events",
  "comments_url": "https://api.github.com/repos/docker/docker/issues/18048/comments",
  "labels_url": "https://api.github.com/repos/docker/docker/issues/18048/labels{/name}",
  "url": "https://api.github.com/repos/docker/docker/issues/18048",
  "user": {
    "site_admin": false,
    "following_url": "https://api.github.com/users/bschiffthaler/following{/other_user}",
    "followers_url": "https://api.github.com/users/bschiffthaler/followers",
    "html_url": "https://github.com/bschiffthaler",
    "url": "https://api.github.com/users/bschiffthaler",
    "gravatar_id": "",
    "avatar_url": "https://avatars.githubusercontent.com/u/8308279?v=3",
    "id": 8308279,
    "login": "bschiffthaler",
    "gists_url": "https://api.github.com/users/bschiffthaler/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/bschiffthaler/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/bschiffthaler/subscriptions",
    "organizations_url": "https://api.github.com/users/bschiffthaler/orgs",
    "repos_url": "https://api.github.com/users/bschiffthaler/repos",
    "events_url": "https://api.github.com/users/bschiffthaler/events{/privacy}",
    "received_events_url": "https://api.github.com/users/bschiffthaler/received_events",
    "type": "User"
  },
  "labels": [],
  "state": "open",
  "locked": false,
  "assignee": null,
  "milestone": null,
  "comments": 0,
  "created_at": "2015-11-17T20:48:17Z"
}
{
.
.
.
}
{
.
.
.
}

We can get specific elements of the array with numeric index. This example returns the first element.

$ echo ${foo} | jq '.[0]'

{
  "body": "**Description**
  .
  .
  .",
  "closed_at": null,
  "updated_at": "2015-11-17T20:48:17Z",
  "title": "docker daemon crash when starting container after: RequestAddress(LocalDefault/172.17.0.0/16, , map[])",
  "number": 18048,
  "id": 117446711,
  "html_url": "https://github.com/docker/docker/issues/18048",
  "events_url": "https://api.github.com/repos/docker/docker/issues/18048/events",
  "comments_url": "https://api.github.com/repos/docker/docker/issues/18048/comments",
  "labels_url": "https://api.github.com/repos/docker/docker/issues/18048/labels{/name}",
  "url": "https://api.github.com/repos/docker/docker/issues/18048",
  "user": {
    "site_admin": false,
    "following_url": "https://api.github.com/users/bschiffthaler/following{/other_user}",
    "followers_url": "https://api.github.com/users/bschiffthaler/followers",
    "html_url": "https://github.com/bschiffthaler",
    "url": "https://api.github.com/users/bschiffthaler",
    "gravatar_id": "",
    "avatar_url": "https://avatars.githubusercontent.com/u/8308279?v=3",
    "id": 8308279,
    "login": "bschiffthaler",
    "gists_url": "https://api.github.com/users/bschiffthaler/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/bschiffthaler/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/bschiffthaler/subscriptions",
    "organizations_url": "https://api.github.com/users/bschiffthaler/orgs",
    "repos_url": "https://api.github.com/users/bschiffthaler/repos",
    "events_url": "https://api.github.com/users/bschiffthaler/events{/privacy}",
    "received_events_url": "https://api.github.com/users/bschiffthaler/received_events",
    "type": "User"
  },
  "labels": [],
  "state": "open",
  "locked": false,
  "assignee": null,
  "milestone": null,
  "comments": 0,
  "created_at": "2015-11-17T20:48:17Z"
}

We can show all the keys in this associative array:

$ echo ${foo} | jq '.[0] | keys'

[
  "assignee",
  "body",
  "closed_at",
  "comments",
  "comments_url",
  "created_at",
  "events_url",
  "html_url",
  "id",
  "labels",
  "labels_url",
  "locked",
  "milestone",
  "number",
  "state",
  "title",
  "updated_at",
  "url",
  "user"
]

And just the first key:

$ echo ${foo} | jq '.[0] | keys | .[0]'

"assignee"

And if we want a list of all the issue ids:

$ echo ${foo} | jq '.[].id'

117446711
117441002
117435461
117433746
117378598
117375663
117339136
117301367
117300491
117299511
117283601
117274857
117262511
117254147
117215336
117200149
117197379
117182453
117169488
117166943
117163511
117121600
117119586
117112381
117111726
117110707
117110401
117092144
117085605
117084025

And just the first issue id:

$ echo ${foo} | jq '.[0].id'

117446711

And we can select an entire element based on some key in the associative array

$ echo ${foo} | jq '.[] | select(.id==117446711)'

{
  "body": "**Description**
  .
  .
  .",
  "closed_at": null,
  "updated_at": "2015-11-17T20:48:17Z",
  "title": "docker daemon crash when starting container after: RequestAddress(LocalDefault/172.17.0.0/16, , map[])",
  "number": 18048,
  "id": 117446711,
  "html_url": "https://github.com/docker/docker/issues/18048",
  "events_url": "https://api.github.com/repos/docker/docker/issues/18048/events",
  "comments_url": "https://api.github.com/repos/docker/docker/issues/18048/comments",
  "labels_url": "https://api.github.com/repos/docker/docker/issues/18048/labels{/name}",
  "url": "https://api.github.com/repos/docker/docker/issues/18048",
  "user": {
    "site_admin": false,
    "following_url": "https://api.github.com/users/bschiffthaler/following{/other_user}",
    "followers_url": "https://api.github.com/users/bschiffthaler/followers",
    "html_url": "https://github.com/bschiffthaler",
    "url": "https://api.github.com/users/bschiffthaler",
    "gravatar_id": "",
    "avatar_url": "https://avatars.githubusercontent.com/u/8308279?v=3",
    "id": 8308279,
    "login": "bschiffthaler",
    "gists_url": "https://api.github.com/users/bschiffthaler/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/bschiffthaler/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/bschiffthaler/subscriptions",
    "organizations_url": "https://api.github.com/users/bschiffthaler/orgs",
    "repos_url": "https://api.github.com/users/bschiffthaler/repos",
    "events_url": "https://api.github.com/users/bschiffthaler/events{/privacy}",
    "received_events_url": "https://api.github.com/users/bschiffthaler/received_events",
    "type": "User"
  },
  "labels": [],
  "state": "open",
  "locked": false,
  "assignee": null,
  "milestone": null,
  "comments": 0,
  "created_at": "2015-11-17T20:48:17Z"
}

And to get the nested url value

$ echo ${foo} | jq '.[0].user.url )'

"https://api.github.com/users/bschiffthaler"

Oh mann, so good!