How to call FileIO.Write.via(Contextful, Contextful) in Scala

Question

I'm using Apache Beam with Scala. To create new ParquetIO.Sink instances with the respective schemas for the different types of streaming data, I'm trying to call .via(Contextful, Contextful) on FileIO.Write. However, IntelliJ cannot find the overloaded method that I'm trying to use and raises this error: Cannot resolve overloaded method 'via'.

FileIO
  .writeDynamic[String, DeserializedEvent]()
  .by(new UDFs.PartitionByEventName())
  .withDestinationCoder(StringUtf8Coder.of())
  .withNumShards(numShards)
  .withNaming(new UDFs.NameFiles())
  .via(
    Contextful.fn[DeserializedEvent, GenericRecord](
      new UDFs.EventToGenericRecord() // SerializableFunction[DeserializedEvent, String]
    ),
    Contextful.fn[String, ParquetIO.Sink](
      new UDFs.SinkParquet() // SerializableFunction[String, ParquetIO.Sink]
    )
  )
  .to(path)

What's the issue here?

Thanks

Answer 1

Scala's type inference system and overload resolution system don't work together quite perfectly. In this case, the overloads of the via() function are preventing type inference from working properly. Why does scala fail to compile when method is overloaded in a seemingly unrelated way? has a good explanation of the problem.

In your case, you should just need to change the type params of the second Contextful.fn to [String, FileIO.Sink] to make it match the signature of the desired overload.