Search code examples
scalaapache-sparkspark-graphx

Scala - Spark GraphX: debug code that runs Pregel operator

I want to performe Pregel operator on my graphX network with my custom logic to vprog, sendMsg and mergeMsg but results are wrong. So, how can I debugging it? I don't want to post the code but just know how you can do debugging, maybe by learning from your tips.

Solution

  • The simplest way to debug behavior of Pregel is using debug from the standard logger (or even println, if it's not production code):

    initialGraph.pregel(Double.PositiveInfinity)(
  (id, dist, newDist) => math.min(dist, newDist), // Vertex Program
  triplet => {  // Send Message
    debeg(s"Send messege ${triplet.srcAttr} with attribute ${triplet.attr}")
    if (triplet.srcAttr + triplet.attr < triplet.dstAttr) {
      Iterator((triplet.dstId, triplet.srcAttr + triplet.attr))
    } else {
      Iterator.empty
    }
  },
  debeg(s"Compare $a and $b")
  (a, b) => math.min(a, b)
)

    If you don't want to use logger, consider the ability of the embeded debugger in your IDE.