Search code examples
flume-ng

Difference between below flume sink configuration

I have very confused about below three sink configuration in flume . please clarify me

CONF1

a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/events/

CONF2

a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flume/events/

CONF3

a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.directory = /var/log/flume

what are specific use case for each of this sink?

Solution

  • Conf1 and Conf2 are basically the same. Both of them have hdfs sink and the data is being written to HDFS. The only difference is that in one case you are giving absolute path.

    Conf3 is a local file roll sink where data is being store into local disk and compared to HDFS.