I am prototyping Apache Flume. My task is to transfer text file contents between two Ubuntu VMs I set up using VirtualBox. I have almost no knowledge of Flume, although I have been going through their documentation.
With VirtualBox I was able to create an Internal Network, and the two VMs are successfully able to ping each other.
I was also able to download and configure flume as shown: https://cwiki.apache.org/confluence/display/FLUME/Getting+Started
My point being that Flume is installed and works on both VMs (or so I think).
This post give me a slight idea: Flume: Data transferring to Server a
So, do I implement exactly that flume.conf file on each VM? And do I need write Java to do this?
Also, I found this video, around the 18 min mark this guy does almost exactly what I want. However, I would like the input to be a text file.
So, could you tell me step-by-step (I am a beginner) how I would go about doing this? Or point me to any useful tutorials.
Thank you!
I solved this problem by using the sink type as file_roll. I also had to identify the second VM's IP so the two could connect.
I found the step-by-step instructions on how to do this at this link: http://thisandthat.io/blog/flume-part3/
Also, you do not need to write any Java to do this. However, as is standard with Flume, ensure your JAVA_HOME is declared properly in the flume-env.sh file. All other parts are handled by the .conf file created (as described in the above link).