Search code examples
http-headersazure-data-factory

Headers in Azure Data Factory HTTP Copy data source

We are using Azure Data Factory to source data from an On-Premise JIRA installation. I've managed to get a number of pipelines to work using the JIRA API, but am hitting a wall when trying to source the Organization object.

There's an undocumented API call that can be made, though:

<servername>/jira/rest/servicedeskapi/organization

This will display the following message when attempting to run from a browser:

"This API is experimental. Experimental APIs are not guaranteed to be stable within the preview period. You must set the header 'X-ExperimentalApi: opt-in' to opt into using this API."

Using Postman, I set things up with the additional header, and I manage to get a resultset:

PostMan example

Using the same ADF copy data job I used for all my other API Calls, however, does not seem to work. I'm using the "Additional Headers" field to add a Bearer token we retrieve from our keyvault, like so:

@{concat(
    'Authorization: Bearer '
    , activity('Get Bearer token from Keyvault').output.value
)}

This works fine for all other API calls. I figured adding the extra header would be as simple as simply appending another line like so:

@{concat(
        'Authorization: Bearer '
        , activity('Get Bearer token from Keyvault').output.value,  
        ', X-ExperimentalApi: opt-in')
}

However, that ends up throwing an error:

"ErrorCode=UserErrorInvalidHttpRequestHeaderFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to set addtional http header,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentException,Message=Specified value has invalid HTTP Header characters. Parameter name: name,Source=System,'"

I tried wrapping double quotes (and escaping them) around the key/value pairs, but that did not work. I tried removing the comma, but somehow that leads to the REST API thinking the extra header is part of the Bearer token, as it then throws an "Unauthorized" exception.

I can get the API to return data if I use a WEB component without any issues, but it'd be nice if I somehow would get this to work within the Copy data activity.

Any help is greatly appreciated!

Solution

    • The approaches that are tried to achieve this might be the incorrect way to provide multiple headers while using copy data activity.

    • I have used HTTP source with a sample URL which accepts Authorization: Bearer token. However, giving additional header (even though it is not required) is working same as using just Authorization header.

    • To pass multiple headers, pass each header separated by a new line. I have used the dynamic content with string interpolation (@{...}), instead of using @concat.

    Authorization: Bearer @{pipeline().parameters.token}
X-ExperimentalApi: opt-in

    enter image description here

    • You can see the following debug input how the additional headers are passed.

    enter image description here

    • As an alternative, since there are only 2 headers to be given, you can configure the Authorization in the linked service itself and use the X-ExperimentalApi as a single additional header in Additional Headers section of the copy data activity.

    enter image description here