Search code examples
apache-nifidataflowdata-lineage

Apache NiFi instance hangs on the "Computing FlowFile lineage..." window


My Apache NiFi instance just hangs on the "Computing FlowFile lineage..." for a specific flow. Others work, but it won't show the lineage for this specific flow for any data files. The only error message in the log is related to an error in one of the processors, but I can't see how that would affect the lineage, or stop the page from loading.


Solution

  • This was related to two things...

    1) I was using the older (but default) provenance repository, which didn't perform well, resulting in the lag in the UI. So I needed to change it...

    #nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
    nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
    

    2) Fixing #1 exposed the second issue, which was that the EnforceOrder processor was generating hundreds of provenance events per file, because I was ordering on a timestamp, which had large gaps between the values. This is apparently not a proper use case for the EnforceOrder processor. So I'll have to remove it and find another way to do the ordering.