Search code examples
azureazure-data-factorypipeline

Why is my Azure Data Factory data flow failing with 'store is not defined' error?


Getting this error while executing the data flow. I have supplied all the data flow parameters correctly such as PipelineStartTime, PipelineEndTime.

Job failed due to reason: com.microsoft.dataflow.Issues: DF-ARG-007 - store is not defined - EXE-0001,Dataflow cannot be analyzed as a graph,[564 737],
source(
) ~> dummy
DF-EXPR-012 - Parameter value for FileName missing - [418 464 564 737],com.microsoft.dataflow.Parameter@597aa1ed,EXE-0001,Dataflow cannot be analyzed as a graph,
dummy derive(
) ~> derivedColumn1
DF-EXPR-012 - Parameter value for PipelineStartTime missing - [418 473 564 737],EXE-0001,Dataflow cannot be analyzed as a graph,
dummy derive(
) ~> derivedColumn1,com.microsoft.dataflow.Parameter@5954ed58
DF-EXPR-012 - Parameter value for PipelineEndTime missing - EXE-0001,Dataflow cannot be analyzed as a graph,com.microsoft.dataflow.Parameter@1819981a,
dummy derive(
) ~> derivedColumn1,[418 473 564 737]
DF-EXPR-012 - Parameter value for NoRowsRead missing - [418 473 564 737],EXE-0001,Dataflow cannot be analyzed as a graph,com.microsoft.dataflow.Parameter@47c44f0c,
dummy derive(
) ~> derivedColumn1
DF-EXPR-012 - Parameter value for Status missing

What have I done

The data flow that I created takes the values from the copy activity as parameters and I supplied all the parameter values correctly such as PipelineStartTime, PipelineEndTime,Status, Errors etc. But the data flow is failing by giving this error. Before giving the errors variable as parameter it was working fine but after that it started to give such errors.

For the errors variable I supplied this value:

@concat('"',replace(substring(activity('Copy data1').output.errors[0].Message, 0,100 ),'\r\n|\r|\n' ,'' ),'"')

How to resolve this?


Solution

  • I encountered the same error (DF-ARG-007 - store is not defined). In my case, I wanted to hand over a name containing a single quote to a data flow from a pipeline For-Each loop (string parameter). After hours of checking and testing I'd now blame Data Factory to potentially screw up it's internal JSON encoding.

    Summary
    The name field I needed to process contained a quote, e.g. "Miles O'Brian":

    Symptom
    Inside monitor something like below is displayed (note the double-quotation of name):

    {
    "dataflow": {
        "referenceName": "PostProcessContact",
        "type": "DataFlowReference",
        "parameters": {
            "accountid": "'12345678-abcd-1234-abcd-123456789abc'",
            "name": "'Miles O''Brian'",
            ...
        },
        ...
    }
    

    Error

    Job failed due to reason: com.microsoft.dataflow.Issues: DF-ARG-007 - store is not defined - source( ) ~> dummysource,EXE-0001,Dataflow cannot be analyzed as a graph,[564 737] DF-ARG-007 - store is not defined - EXE-0001,Dataflow cannot be analyzed as a graph, operation sink( ) ~> parentaccountid,[564 737]

    Potential, related DF-activity
    In the called flow I am using a dummy source without records as I want to build a record solely on the parameters given.

    "source(useSchema: false,",
    "     allowSchemaDrift: true,",
    "     validateSchema: false,",
    "     inferDriftedColumnTypes: true,",
    "     ignoreNoFilesFound: true,",
    "     format: 'json',",
    "     fileSystem: 'power-platform-dataflows',",
    "     fileName: 'nuxnildummy.json',",
    "     documentForm: 'singleDocument') ~> dummysource",
    

    Workaround
    In my case, I was able to solve the problem by replacing the single quote in my input string with ‘ which is char(145). Not a nice solution but it works and I carry over the name just for reference.

    // Previous Assignment: 
    "'@item()['name']'",
    
    // Fixed Assignment:
    "'@{if(equals(item()['name'], null), null, replace(item()['name'], '''', '‘'))}'",
    

    Conclusion
    Although the workaround is OK in my situation, it is still a bad implementation, therefore @Microsoft:

    • Would be great, if this issue will be addressed in a future update.
    • Would be great, if an in-memory JSON source would be added which doesn't require one to create dummy-sources in Flowlets (e.g. casting an array of records like [{}, {}, ...] into a useable table).