Search code examples
filebigdataapache-nifiapache-minifi

MiNiFi GetFile processor fails to get large files

I'm running apache MiNiFi c++, The flow starts with a GetFile processor.

The input directory includes some large files, and when I run MiNiFi the files above ~1.5 GB fail and do not get queued.

The log file states:

[org::apache::nifi::minifi::processors::GetFile] [Warning] failed to stat large_file_path_here

The other smaller files are queued as expected.

Does anyone have a clue what can be wrong? Why can't the processor manage the larger files?

Thanks in advance.

Solution

  • What you found seems like a bug that is present even in the current MiNiFi implementation even up to today. The issue is that file sizes you mentioned, a narrowing exception happens here when trying to determine the length of the file to be written into the content repository.

    We will try to fix this issue asap.