Search code examples
flume-ngflume-twitter

How to pass parameters in flume

I have a twitter flume source defined like this in the flume configuration file

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxx
TwitterAgent.sources.Twitter.accessToken = xxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxx
TwitterAgent.sources.Twitter.keywords = Avengers

The 'keywords' property is hardcoded to 'Avengers'. I wanted to make this property value a variable and pass it in while I start my flume agent. I tried positional parameters but doesn't seem to work.

Any ideas on how to get this done ?

Solution

  • Yes, the proxy feature is missing in twitter-source. I've submitted a patch which is waiting for acceptance. Until then feel free to pull my version from github:

    https://github.com/schmiegelow/flume/blob/trunk/flume-ng-sources/flume-twitter-source/src/main/java/org/apache/flume/source/twitter/TwitterSource.java

    Cheers,

    Erik