I am trying to retrieve facebook data using flume SocialAgent. I have successfully retrieve twitter data using TwitterAgent.
But in case of Facebook I got nothing in hdfs.
My terminal stuck at this stage. When I terminate this process I got this
And my HDFS facebook folder is empty.I am using following flume.conf
SocialAgent.sources = FacebookHttpSource Twitter
SocialAgent.channels = FBmemoryChannel MemChannel
SocialAgent.sinks = fbHDFS HDFS
# For each one of the sources, the type is defined
SocialAgent.sources.FacebookHttpSource.type = org.apache.flume.source.http.HTTPSource
SocialAgent.sources.FacebookHttpSource.port = 51400
SocialAgent.sources.FacebookHttpSource.interceptors = Ts
SocialAgent.sources.FacebookHttpSource.interceptors.Ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
# The channel can be defined as follows.
SocialAgent.sources.FacebookHttpSource.channels = FBmemoryChannel
# Each sink's type must be defined
#Specify the channel the sink should use
SocialAgent.sinks.fbHDFS.channel = FBmemoryChannel
SocialAgent.sinks.fbHDFS.type = hdfs
SocialAgent.sinks.fbHDFS.hdfs.path = hdfs://localhost:9000/user/flume/facebook/%Y/%m/%d/%H/
SocialAgent.sinks.fbHDFS.hdfs.fileType = DataStream
SocialAgent.sinks.fbHDFS.hdfs.writeFormat = Text
SocialAgent.sinks.fbHDFS.hdfs.batchSize = 1000
SocialAgent.sinks.fbHDFS.hdfs.rollSize = 0
SocialAgent.sinks.fbHDFS.hdfs.rollCount = 10000
# Each channel's type is defined.
SocialAgent.channels.FBmemoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
SocialAgent.channels.FBmemoryChannel.capacity = 10000
SocialAgent.channels.FBmemoryChannel.transactionCapacity = 100
Is there any other flume Agent to fetch FaceBook Data.
org.apache.flume.source.http.HTTPSource in your configuration doesn't send any request to Facebook to fetch data. It just listening a port (in your case 51400) and wait for any request (https://flume.apache.org/FlumeUserGuide.html#http-source)
As I know there isn't any Flume FBClient yet.