How To Capture and Format Json Response in Jenkins Pipeline

Question

I'm running a curl command in my Jenkinsfile.

    post {
      success {
        script {
            sh '''
            |SCORE=+1
            |GERRIT_COMMENT="$(cat <<-EOL
            |Sonar result was: SUCCESS
            |Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
            |EOL
            |)"
            |curl -s -u ${apiToken}: ${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children | json_pp -json_opt pretty,canonical > report1.txt
            |echo "Voting unsuccessful"
            '''.stripMargin().stripIndent()
            archiveArtifacts artifacts: 'report1.txt', fingerprint: true
            echo 'I Succeeded'
            }
    }

But I get the error malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/bin/json_pp

I can't use jq as it's not installed and installing it isn't an option.

The curl command works fine on my terminal but is failing in my Jenkins pipeline.

Also, when I do this instead, it works.

    post {
      success {
        script {
            sh '''
            |SCORE=+1
            |GERRIT_COMMENT="$(cat <<-EOL
            |Sonar result was: SUCCESS
            |Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
            |EOL
            |)"
            |echo "Voting unsuccessful"
            '''.stripMargin().stripIndent()
            sh """
            curl -s -u ${apiToken}: '${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children' | json_pp -json_opt pretty,canonical > report1.txt
            """
            archiveArtifacts artifacts: 'report1.txt', fingerprint: true
            echo 'I Succeeded'
            }
    }

But it throws a warning in the console output.

Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]

What am I doing wrong, please?

In a Jenkins pipeline, how would you properly pass a JSON response using curl into a file?

Answer 1

I recommend to not use shell scripts whenever it is possible. Shell scripts are not cross platform and require installing additional tools (e.g. curl).

In your case the curl call could be replaced by the httpRequest step.

First let's replace the curl call and saves the result in a componentTree.json file:

httpRequest(
    url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
    authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable',
    outputFile: 'componentTree.json'
)

You want to format the JSON data in a human-readable format, so let's use the readJSON and writeJSON steps:

def json = readJSON(file: 'componentTree.json')
writeJSON(json: json, file: 'report1.txt', pretty: 4)

Now the report1.txt file contains JSON formatted with indent 4.

The componentTree.json file is written and read only once, so let's decrease the number of the IO operations:

def response = httpRequest(
    url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
    authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable'
)
def json = readJSON(text: response.content)
writeJSON(json: json, file: 'report1.txt', pretty: 4)

---

About the warning:

Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]

Secrets never should be interpolated because they may contain special characters which could be interpreted. Example:

• my secret: My' https://example.org; cat /etc/passwd; echo \
• command: curl -u '${password}' https://private.server/path/file.txt

After the interpolation the following command is called:

curl -u 'My' https://example.org; cat /etc/passwd; echo \' https://private.server/path/file.txt

There are two options to fix it:

• if apiToken is an environment variable:

  sh "curl -s -u \$apiToken: '${Sonar_URL}/api/measures/component..."
  

• if apiToken is a Groovy variable:

  withEnv(["CREDENTIALS=${apiToken}"]) {
      sh "curl -s -u \$CREDENTIALS: '${Sonar_URL}/api/measures/component..."
  }
  

In both cases the dollar sign ($) is escaped before the credentials which means that shell script will resolve it (it will be taken from environment variables).