Search code examples
flume

Apache Flume ../flume.log permission denied using spooldir as source but ok with other sources

I'm pretty new using Flume, just started testing it using a CDH 4.7.0 distributuion.

I'm configuring Flume through Cloudera Manager.

I've set up an agent using a sequence generator as source and everything went fine, but I've got an error when configured the source as a spooling directory:

log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /var/log/flume-ng/flume-cmf-flume1-AGENT-xxxxx.log (Permission denied)

Didn't change anything else, any ideas?

My .conf:

SOURCE

agent_pbe2.sources.spldir-src1.type = spooldir

agent_pbe2.sources.spldir-src1.spoolDir = /tmp/Flume-PoC

CHANNEL

agent_pbe2.channels.mem-chn1.type = memory

agent_pbe2.channels.mem-chn1.capacity = 1000

agent_pbe2.channels.mem-chn1.transactionCapacity = 100

SINK

agent_pbe2.sinks.hdfs-snk1.type = hdfs

agent_pbe2.sinks.hdfs-snk1.hdfs.path = hdfs://martehadoop/user/hduser/Flume-PoC/pbe2

BIND SOURCES

agent_pbe2.sources.spldir-src1.channels = mem-chn1

BIND SINKS

agent_pbe2.sinks.hdfs-snk1.channel = mem-chn1

Thanks!

Solution

  • I answer my own question.

    The problem was related to user rights. The user used by Flume service must be able to access all resources involved in the agent(s).

    Cheers!