Search code examples
apacheflume

Flume generated files

I have a couple of questions

I have flume configuration as note here in answer

  1. Flume is generating a file with the name Flume-data.millisecond. Can I assign a custom name? How can I change it?

  2. With this configuration I was able to load a 1.5MB file 5 times but when I ran it for the 6th time I got an OutofMemoryError. How can I resolve this issue?

Thanks

Solution

  • Flume is generating a file and Flume-data.millisecond is the name of the file, can i have a custom name? how can i change it?

    Yes you can have a custom name as a prefix or suffix to the files being generated.

    hdfs.filePrefix : FlumeData : Name prefixed to files created by Flume in hdfs directory

    hdfs.fileSuffix – Suffix to append to file (eg .avro - NOTE: period is not automatically added)

    Check out the HDFS sink in flume user guide for more details.

    With this configuration i was able to load a 1.5MB file 5 times but when i ran it for the 6th time i got an OutofMemoryError. How can i resolve this issue.

    You are getting out of memory error because you are using a memory channel with a insufficient capacity. Better using a File channel.