Search code examples
flume-ng

data channel lock error while configuring flume with multiple channels


I have tried to fan out the flow from one source to two channels.Also I specified different dataDirs and checkpointDirs properties for each channel as in the channel lock error while configuring flume's multiple sources using FILE channels question.I have used a multiplexing channel selector. I have get the following error.

18/08/23 16:21:37 **ERROR file.FileChannel: Failed to start the file channel** [channel=fileChannel1_2]
java.io.IOException: Cannot lock /root/.flume/file-channel/data. The directory is already locked. [channel=fileChannel1_2]
    at org.apache.flume.channel.file.Log.lock(Log.java:1169)
    at org.apache.flume.channel.file.Log.<init>(Log.java:336)
    at org.apache.flume.channel.file.Log.<init>(Log.java:76)
    at org.apache.flume.channel.file.Log$Builder.build(Log.java:276)
    at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:281)
    at unAndReset(FutureTask.java:308) .....

My configuration file as follws.

agent1.sinks=hdfs-sink1_1 hdfs-sink1_2 

agent1.sources=source1_1

agent1.channels=fileChannel1_1 fileChannel1_2

agent1.channels.fileChannel1_1.type=file

agent1.channels.fileChannel1_1.checkpointDir=/home/Flume/alpha/001
agent1.channels.fileChannel1_1.dataDir=/mnt/alpha_data/
agent1.channels.fileChannel1_1.checkpointOnClose=true

agent1.channels.fileChannel1_1.dataOnClose=true

agent1.sources.source1_1.type=spooldir

agent1.sources.source1_1.spoolDir=/home/ABC/
agent1.sources.source1_1.recursiveDirectorySearch=true
agent1.sources.source1_1.fileSuffix=.COMPLETED

agent1.sources.source1_1.basenameHeader = true

agent1.sinks.hdfs-sink1_1.type=hdfs

agent1.sinks.hdfs-sink1_1.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_1.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/CA

agent1.sinks.hdfs-sink1_1.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_1.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_1.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_1.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_1.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_1.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_1.hdfs.useLocalTimeStamp=false

agent1.channels.fileChannel1_2.type=file

agent1.channels.fileChannel1_2.capacity=200000
agent1.channels.fileChannel1_2.transactionCapacity=1000
agent1.channels.fileChannel1_2.checkpointDir=/home/Flume/beta/001
agent1.channels.fileChannel1_2.dataDir=/mnt/beta_data/
agent1.channels.fileChannel1_2.checkpointOnClose=true
agent1.channels.fileChannel1_2.dataOnClose=true

agent1.sinks.hdfs-sink1_2.type=hdfs

agent1.sinks.hdfs-sink1_2.hdfs.filePrefix = %{basename}

agent1.sinks.hdfs-sink1_2.hdfs.path=hdfs://10.44.209.44:9000/flume_sink/AZ

agent1.sinks.hdfs-sink1_2.hdfs.batchSize=1000

agent1.sinks.hdfs-sink1_2.hdfs.rollSize=268435456

agent1.sinks.hdfs-sink1_2.hdfs.rollInterval=0

agent1.sinks.hdfs-sink1_2.hdfs.rollCount=50000000

agent1.sinks.hdfs-sink1_2.hdfs.fileType=DataStream

agent1.sinks.hdfs-sink1_2.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_2.hdfs.useLocalTimeStamp=false

agent1.sources.source1_1.channels=fileChannel1_1 fileChannel1_2

agent1.sinks.hdfs-sink1_1.channel=fileChannel1_1

agent1.sinks.hdfs-sink1_2.channel=fileChannel1_2

agent1.sources.source1_1.selector.type=multiplexing
agent1.sources.source1_1.selector.header=basenameHeader
agent1.sources.source1_1.selector.mapping.CA=fileChannel1_1
agent1.sources.source1_1.selector.mapping.AZ=fileChannel1_2

can someone give any solution for that.


Solution

  • Try to set a channel for default property in multiplexing selector

    agent1.sources.source1_1.selector.default=fileChannel1_1