Search code examples
hivedirected-acyclic-graphstez

Can we impose custom DAG plan over generated DAG plan in TEZ? If possible, How?


Instead of using a DAG plan generated by TEZ in Hive, I want to use custom DAG plan created by me. Is it Possible? If Yes, can anyone explain?


Solution

  • If you are using Hive for running queries then you cannot use custom DAG. If you do low level programming Tez application using API, you can. See High level docs here: https://hortonworks.com/blog/expressing-data-processing-in-apache-tez/

    Something like this:

    //Define DAG
    DAG dag = new DAG();
    Vertex map1 = new Vertex(MapProcessor.class) 
    and so on...
    

    See apache project examples: https://git-wip-us.apache.org/repos/asf?p=tez.git;a=blob;f=tez-examples/src/main/java/org/apache/tez/examples/HashJoinExample.java