I have multiple function definitions to write a parquet file, but I'd rather have one general function. How can I make one general function work? The problem is that one general function returns error
could not find implicit value for parameter writerFactory: com.github.mjakubowski84.parquet4s.ParquetWriter.ParquetWriterFactory[Product]
Multiple functions:
import com.github.mjakubowski84.parquet4s.ParquetWriter
import org.apache.parquet.hadoop.metadata.CompressionCodecName
import org.apache.parquet.hadoop.ParquetFileWriter
import java.time.format.DateTimeFormatter
import java.time.LocalDate
def writeParquetUnsegmented(rows: List[ACaseClass], date: String) = {
val outPath = s"s3a://redacted/path-$date.parquet"
ParquetWriter.writeAndClose(outPath,
rows,
ParquetWriter.Options(
writeMode = ParquetFileWriter.Mode.OVERWRITE,
compressionCodecName = CompressionCodecName.SNAPPY
))
}
def writeParquetSegmented(rows: List[BCaseClass], date: String) = {
val outPath = s"s3a://redacted/path-$date.parquet"
ParquetWriter.writeAndClose(outPath,
rows,
ParquetWriter.Options(
writeMode = ParquetFileWriter.Mode.OVERWRITE,
compressionCodecName = CompressionCodecName.SNAPPY
))
}
One function:
def writeParquetSegmented(rows: List[Product], date: String) = {
val outPath = s"s3a://redacted/path-$date.parquet"
ParquetWriter.writeAndClose(outPath,
rows,
ParquetWriter.Options(
writeMode = ParquetFileWriter.Mode.OVERWRITE,
compressionCodecName = CompressionCodecName.SNAPPY
))
}
I have also tried different function signatures and got the same error.
def writeParquetSegmented[A](rows: List[A], date: String)
def writeParquetSegmented[A :< RowParent](rows: List[A], date: String)
I would try this signature:
def writeParquetSegmented[A : ParquetWriterFactory](rows: List[A], date: String)
The compiler will turn the context bound [A : ParquetWriterFactory]
into an implicit parameter that the error message complains about.
def writeParquetSegmented[A](rows: List[A], date: String)(implicit pwf: ParquetWriterFactory)