Search code examples
hadoopflume

why do i need thrift to build flume?


i have downloaded flume from "https://github.com/apache/flume/downloads"..but i am unable to build it..do i need to install thrift first in order to build flume??if so, what is the reson..i am getting following error when i run mvn compile -

mohammad@ubuntu:~/apache-flume-b01a760$ mvn compile
[INFO] Scanning for projects...
[INFO] Reactor build order: 
[INFO]   Flume
[INFO]   Flume Core
[INFO]   Flume Master Config Web Application
[INFO]   Flume Node Web
[INFO]   Flume Distribution Project
[INFO]   A log4j appender for Flume
[INFO]   Flume Hello World Plugin
[INFO]   Flume HBase Plugin
[INFO] ------------------------------------------------------------------------
[INFO] Building Flume
[INFO]    task-segment: [compile]
[INFO] ------------------------------------------------------------------------
[INFO] No goals needed for project - skipping
[INFO] ------------------------------------------------------------------------
[INFO] Building Flume Core
[INFO]    task-segment: [compile]
[INFO] ------------------------------------------------------------------------
[INFO] [antlr3:antlr {execution: default}]
[INFO] ANTLR: Processing source directory /home/mohammad/apache-flume-b01a760/flume-core/src/main/antlr3
ANTLR Parser Generator  Version 3.2 Sep 23, 2009 14:05:07
com/cloudera/flume/conf/FlumeDeploy.g
com/cloudera/flume/shell/antlr/FlumeShell.g
[INFO] [antrun:run {execution: generate-version-file}]
[INFO] Executing tasks

main:
[copy] Copying 1 file to /home/mohammad/apache-flume-b01a760/flume-core/target/generated-sources/version/com/cloudera/flume
[INFO] Executed tasks
[INFO] [avro:idl-protocol {execution: default}]
[INFO] [thrift:compile {execution: default}]
[ERROR] thrift failed output: 
[ERROR] thrift failed error: /bin/sh: null/bin/thrift: not found

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] thrift did not exit cleanly. Review output for more information.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7 seconds
[INFO] Finished at: Mon Jun 04 16:10:47 IST 2012
[INFO] Final Memory: 28M/243M
[INFO] ------------------------------------------------------------------------

Solution

  • Flume has some code generation that utilizes thrift as a transport communications serialization mechanism.

    In the flume-core/pom.xml, you can see this:

    <plugin>
        <groupId>org.apache.thrift.tools</groupId>
        <artifactId>maven-thrift-plugin</artifactId>
        <version>0.1.10</version>
        <configuration>
          <thriftExecutable>${thrift.executable}</thriftExecutable>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
    </plugin>
    

    You need to install thrift, and then configure the location in the parent pom.xml:

    <properties>
      <!-- NB: The version of the thrift compiler must match that of the dependency 
      on the jar file below. -->
      <thrift.executable>${env.THRIFT_HOME}/bin/thrift</thrift.executable>
    
      <!-- Set default encoding to UTF-8 to remove maven complaints -->
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    
      <!-- defaults for flaky test and focused test exclusions -->
      <test.exclude.pattern>$</test.exclude.pattern> <!-- junk pattern -->
      <test.include.pattern>**/Test*.java</test.include.pattern>
    </properties>