Search code examples
scalasbtavroscalapbavrohugger

How to handle clashing plugins generating code in SBT project?


I am stuck on a problem with my Scala SBT (1.7.2) project.

Situation:

I already have working configuration to generate code from *.avsc, *.avdl files via "com.julianpeeters" % "sbt-avrohugger" % "2.0.0-RC19" plugin. Now, I must add code generation from *.proto files. I have followed instruction of setting up ScalaPB (from here). Well, protobuf generation works as intended, but for some reason the avro code generation is affected - some files are created, but it looks like the plugin didn't have time to finish up work and just stopped in the middle, making the tests fail in result.

Code:

plugins.sbt:

addSbtPlugin("com.julianpeeters" % "sbt-avrohugger" % "2.0.0-RC19")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.7")

libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.15"

build.sbt:

lazy val `job` = (project in file("."))
  .settings(avroHuggerSettings)
  .settings(protobufSettings)

lazy val avroHuggerSettings = Seq.concat(
  inConfig(IntegrationTest)(SbtAvrohugger.baseSettings),
  Seq(
    Compile / sourceGenerators += (Compile / avroScalaGenerate).taskValue,
    Test / sourceGenerators += (Test / avroScalaGenerate).taskValue,
    IntegrationTest / sourceGenerators += (IntegrationTest / avroScalaGenerate).taskValue,
    Compile / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/main/resources/avro")),
    Test / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/test/resources/avro")),
    IntegrationTest / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/it/resources/avro")),
    Compile / avroScalaCustomTypes := avroCustomTypes,
    Test / avroScalaCustomTypes := avroCustomTypes,
    IntegrationTest / avroScalaCustomTypes := avroCustomTypes
  )
)

lazy val protobufSettings = Seq(
  Compile / PB.targets := Seq(
    scalapb.gen() -> (Compile / sourceManaged).value
  )

My Avro files are located in src/main/resources/avro, my Proto files are located in src/main/protobuf.

With such setup my proto files are generated correctly, but only some of the avro files are done. Mind you, both plugins generate code into target/scala-2.12/src_managed/main, but into different packages.

Output logs snippet (sbt assembly):

[info] Considering source directories ...resources/avro
[info] Compiling AVSC ... .avsc to .../target/scala-2.12/src_managed/main/compiled_avro
[info] Compiling AVSC ... .avsc to .../target/scala-2.12/src_managed/main/compiled_avro
[info] Compiling Avro IDL ... .avdl
[info] Compiling 4 protobuf files to .../target/scala-2.12/src_managed/main
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] Compiling Avro IDL ... .avdl
[info] compiling 139 Scala sources to .../target/scala-2.12/classes
[info] done compiling
//same as above but for some test avro classes
//failing test output
[error] (Test / compileIncremental) Compilation failed
[error] Total time: 9 s, completed Jun 5, 2024, 4:11:10 PM

My attempts to fix:

I am not good with the SBT so it was a trial and error, but I've tried to make either plugin run before the other one, with no results. I tried to use the examples from this stack thread, like that:

lazy val generateAvro = taskKey[Unit]("Avro task")
lazy val generateProto = taskKey[Unit]("Proto task")

generateAvro := {
  println("Starting avro task ...")
  //tried both with Seq() and without, doesn't matter
    Seq(
      Compile / sourceGenerators += (Compile / avroScalaGenerate).taskValue,
      Test / sourceGenerators += (Test / avroScalaGenerate).taskValue,
      IntegrationTest / sourceGenerators += (IntegrationTest / avroScalaGenerate).taskValue,
      Compile / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/main/resources/avro")),
      Test / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/test/resources/avro")),
      IntegrationTest / avroSourceDirectories := Seq(new java.io.File(s"${baseDirectory.value}/src/it/resources/avro")),
      Compile / avroScalaCustomTypes := avroCustomTypes,
      Test / avroScalaCustomTypes := avroCustomTypes,
      IntegrationTest / avroScalaCustomTypes := avroCustomTypes,
      (Compile / avroScalaGenerate).value // tried with and without it
    )
  println("Avro task ended")
}

generateProto := {
  println("Starting proto task ...")
  Compile / PB.targets := Seq(
    scalapb.gen() -> (Compile / sourceManaged).value

  )
  println("Proto task ended")
}

Compile / compile := Def.taskDyn {
  val result = (Compile / compile).value
  Def.task {
    val _ = generateProto.value
    result
  }
}.dependsOn(generateAvro).value

Unfortunately it does not work, I guess that the content of those tasks is merely a config for the plugins, that work independently from my will during the compilation, right?

Please, show me some guidance how could I make those plugins work together in SBT? Making those plugins work in some orchestrated way, change the configuration for any of it, anything to get all the files generate please.


Solution

  • Scalapb cleans up the output directory. The solution will be to generate to a subdir:

    scalapb.gen() -> (Compile / sourceManaged).value / "scalapb"