how to use flume for data streaming between two directories?
spool_dir.sources = src-1
spool_dir.channels = channel-1
spool_dir.sinks = sink-1
# source
spool_dir.sources.src-1.type = spooldir
spool_dir.sources.src-1.channels = channel-1
spool_dir.sources.src-1.spoolDir = /usr/lib/flume/source
#sink
spool_dir.sinks.sink-1.type = spooldir
spool_dir.sinks.sink-1.channels = channel-1
spool_dir.sinks.sink-1.spoolDir = /usr/lib/flume/sink
# Bind the source and sink to the channel
spool_dir.sources.src-1.channels = channel-1
spool_dir.sinks.sink-1.channel = channel-1
What i understand your asking is you want to monitor files coming into one folder and copy them to say another folder, if thats the case then your source looks good but for sink use file_Roll, instead of spool dir.
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = file_roll
a1.sinks.k1.channel = c1
a1.sinks.k1.sink.directory = /var/log/flume
Basically in Flume Source and Sink are different interfaces, so you have to choose spool as source saying i want to monitor particular directory, but then i want to write into file or hdfs using one of the sink, so there is no spool sink instead there is file_roll sink, it might or might not work for you. But choose one of the flume sink as target http://flume.apache.org/FlumeUserGuide.html#flume-sinks