Search code examples
azuredictionaryazure-stream-analytics

Azure Stream Analytics: "Stream Analytics job has validation errors: The given key was not present in the dictionary."


I burned a couple of hours on a problem today and thought I would share.

I tried to start up a previously-working Azure Stream Analytics job and was greeted by a quick failure:

Failed to start Streaming Job 'shayward10ProcessLogs'.

I looked at the JSON log and found nothing helpful whatsoever. The only description of the problem was:

Stream Analytics job has validation errors: The given key was not present in the dictionary.

Given the error and some changes to our database, I tried the following to no effect:

  • Deleting and Recreating all Inputs
  • Deleting and Recreating all Outputs
  • Running tests against the data (coming from Event Hub) and the output looked good

My query looked as followed:

SELECT
    dateTimeUtc,
    context.tenantId AS tenantId,
    context.userId AS userId,
    context.deviceId AS deviceId,
    changeType,
    dataType,
    changeStatus,
    failureReason,
    ipAddress, 
    UDF.JsonToString(details) AS details
INTO
    [MyOutput]
FROM
    [MyInput]
WHERE
    logType = 'MyLogType';

Nothing made sense so I started deconstructing my query. I took it down to a single field and it succeeded. I went field by field, trying to figure out which field (if any) was the cause.

See my answer below.


Solution

  • The answer was simple (yet frustrating). When I got to the final field, that's where the failure was:

    UDF.JsonToString(details) AS details
    

    This was the only field that used a user-defined function. After futsing around, I noticed that the Function Editor showed the title of the function as:

    udf.JsonToString
    

    It was a casing issue. I had UDF in UPPERCASE and Azure Stream Analytics expected it in lowercase. I changed my final field to:

    udf.JsonToString(details) AS details
    

    It worked.

    The strange thing is, it was previously working. Microsoft may have made a change to Azure Stream Analytics to make it case-sensitive in a place where it seemingly wasn't before.

    It makes sense, though. JavaScript is case-sensitive. Every JavaScript object is basically a dictionary of members. Consider the error:

    Stream Analytics job has validation errors: The given key was not present in the dictionary.

    The "udf" object had a dictionary member with my function in it. The UDF object would be undefined. Undefined doesn't have my function as a member.

    I hope my 2-hour head-banging session helps someone else.