How to convert escaped Json into regular Json or variable?

Question

I'm using Ansible to pull the contents of a JSON file from Bitbucket. The problem is the content is escaped JSON so I can't use it in a variable or parse it, etc. Here's an example of what I get back from Bitbucket:

{
    "json": {
      "lines": [
            {
            "text": "    \"AWS\": {"
            },
            {
            "text": "        \"S3\": {"
            },
            {
            "text": "            \"Bucket\": \"my-s3-bucket\","
            }
        ]
    }
}

Here's what I've tried:

json_file.json.lines | map(attribute='text') | join('\n') | from_json

or

json_file.json.lines | map(attribute='text') | join('\n') | to_json

After the join() (before from/to_json), the content looks something like this:

{\\n    \"AWS\": {\\n        \"S3\": {\\n            \"Bucket\": \"my-s3-bucket\"\\n            }\\n    }\\n  }

Neither of above filters return usable Json. As soon as I try to access an item in the Json I get VARIABLE IS UNDEFINED or I get Unexpected failure during module execution: Expecting property name enclosed in double quotes.

So how can I accomplish this?

Answer 1

I see two problems here.

Input is not valid JSON

If we extract the text in your example, we don't get a valid JSON document. We get:

    "AWS": {"
        "S3": {"
            "Bucket": "my-s3-bucket"

That by itself is a fragment of a JSON document. In order to test out a solution, we would need minimally the equivlanet of:

{
    "AWS": {"
        "S3": {"
            "Bucket": "my-s3-bucket"
        }
    }
}

Filter does not result in valid JSON

When you write:

json_file.json.lines | map(attribute='text') | join('\n') | to_json

You are not joining lines with a newline. You are joining lines with the literal two-character sequence \n. That results in a document that is not valid JSON.

The simplest solution is just to join on "" instead.

The solution

Taking both of the above into account, we can put together something like this:

- hosts: localhost
  gather_facts: false
  vars:
    document: {
      "json": {
        "lines": [
              {
              "text": "{"
              },
              {
              "text": "    \"AWS\": {"
              },
              {
              "text": "        \"S3\": {"
              },
              {
              "text": "            \"Bucket\": \"my-s3-bucket\""
              },
              {
                "text": "}}}"
                }
          ]
      }
    }

  tasks:
    - set_fact:
        json: "{{ document.json.lines | map(attribute='text') | join('') }}"

    - debug:
        msg: "Bucket: {{ json.AWS.S3.Bucket }}"

Which produces as output:

PLAY [Testing] *****************************************************************

TASK [set_fact] ****************************************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "Bucket: my-s3-bucket"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0