Create Scala case classes for SCIO type safe read/write dynamically

I'm trying to generate Scala classes dynamically, which then should be used as safe types in SCIO read/write to/from GCP BigQuery.

Target example:

import com.spotify.scio.bigquery._
import com.spotify.scio.bigquery.types.BigQueryType


case class TARGET_TABLE(id: String, name: String, desc: String)

def main(cmdlineArgs: Array[String]): Unit = {
  val (sc, args) = ContextAndArgs(cmdlineArgs)
  sc.typedBigQuery[SOURCE_TABLE]()  // Read from BQ
    .map( row => transformation(row) ) // Transform -> SCollection[TARGET_TABLE]
    .saveAsTypedBigQueryTable(Table.Spec(args("TARGET_TABLE")))  // save to BQ

As as input there are dataset, SOURCE_TABLE, TARGET_TABLE, list of target fields, so I can build up a string source of generated classes. All these values are retrieved dynamically from other 3rd party (json, xml, etc. ) and can be mutable by every execution.

So, the source of generated classes can be presented as:

val sourceString =
     |import com.spotify.scio.bigquery.types.BigQueryType
     |class $SOURCE_TABLE

val targetString =
     |import com.spotify.scio.bigquery.types.BigQueryType
     |case class $TARGET_TABLE($fieldDefinitions)

These sources are considered to be translated to classes, which types are required for SCIO BigQuery I/O.

Scala version: 2.12.17

I tried to use Scala runtime Mirror and Toolbox (from this answer, from this, etc.). But all variants throw the same error: enable macro paradise (2.12) or -Ymacro-annotations (2.13) to expand macro annotations It's obvious that the Toolbox's internal compiler doesn't see the build.sbt settings:

addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.full)

Besides that here mentioned that Toolbox is not intended for such complex thing.

So, I decided to apply an approach using the package as described in this answer. But it throws the same error about the lack of macro annotations.

Thus the main question: is there any chance to add required compiler plugin settings to{Global, Settings} or to apply any other approach to generate such annotated classes dynamically?

def compileCode(sources: List[String], classpathDirectories: List[AbstractFile], outputDirectory: AbstractFile): Unit = {
  val settings = new Settings
  classpathDirectories.foreach(dir => settings.classpath.prepend(dir.toString))
  settings.usejavacp.value = true
  // Add macros paradise compiler plugin?
  val global = new Global(settings)
  val files = { case (code, i) => new BatchSourceFile(s"(inline-$i)", code) }
  (new global.Run).compileSources(files)


  • You can switch on the paradise plugin for toolbox by feeding corresponding command-line option

    import scala.reflect.runtime
    val rm = runtime.currentMirror
    val tb = rm.mkToolBox(options = "-Xplugin:/path/to/.cache/coursier/v1/https/")

    in Scala 2.12 or

    val tb = rm.mkToolBox(options = "-Ymacro-annotations")

    in Scala 2.13.

    If you prefer actual compiler over toolbox then you can feed the option in the following way

    settings.plugin.value = List("/path/to/.cache/coursier/v1/https/")

    in Scala 2.12 or

    settings.YmacroAnnotations.value = true

    in Scala 2.13.