Search code examples
scalaapache-beamspotify-scio

Debugging SCollection contents when running tests

Is there any way to view the contents of an SCollection when running a unit test (PipelineSpec)?

When running something in production on many machines there would be no way to see the entire collection in one machine, but I wonder is there a way to view the contents of an SCollection (for example when running a unit test in debug mode in intellij).

Solution

  • If you want to print debug statements to the console then you can use the debug method which is part of SCollections. A sample code shown below

        val stdOutMock = new MockedPrintStream
    Console.withOut(stdOutMock) {
      runWithContext { sc =>
        val r = sc.parallelize(1 to 3).debug(prefix = "===")
        r should containInAnyOrder(Seq(1, 2, 3))
      }
    }
    stdOutMock.message.filterNot(_ == "\n") should contain theSameElementsAs
      Seq("===1", "===2", "===3")