Bringing third party .proto files to ScalaPB

Question

How can I tell ScalaPB that it should fetch .proto dependencies from the Internet, e.g.

google/api/annotations.proto from https://github.com/googleapis/googleapis/tree/master/google/api

Background:

The aim is to read etcd v3 API from Scala, via gRPC.

I've picked the etcd specific .proto files from their project, and placed under mine. It works. However, the dependencies start running deep, and there must be a better way.

https://github.com/googleapis/googleapis/tree/master/google/api

Answer 1

Update: sbt-protoc can download and extract jars with protos from maven. It can also be set up to generate Scala code for those third-party protos.

Here is how to do it. In your build.sbt:

import scalapb.compiler.Version.scalapbVersion

val GrpcProtosArtifact = "com.google.api.grpc" % "grpc-google-common-protos" % "1.17.0"

scalaVersion := "2.12.10"

// This sub-project will hold the compiled Scala classes from the external
// jar.
lazy val googleCommonProtos = (project in file("google-common-protos"))
  .settings(
    name := "google-common-protos",

    // Dependencies marked with "protobuf" get extracted to target / protobuf_external
    libraryDependencies ++= Seq(
      GrpcProtosArtifact % "protobuf"
    ),

    // In addition to the JAR we care about, the protobuf_external directory
    // is going to contain protos from Google's standard protos.  
    // In order to avoid compiling things we don't use, we restrict what's
    // compiled to a subdirectory of protobuf_external
    PB.protoSources in Compile += target.value / "protobuf_external" / "google" / "type",

    PB.targets in Compile := Seq(
      scalapb.gen() -> (sourceManaged in Compile).value
    )
  )

// This sub-project is where your code goes. It contains proto file that imports a proto
// from the external proto jar.
lazy val myProject = (project in file("my-project"))
  .settings(
    name := "my-project",

    // The protos in this sub-project depend on the protobufs in
    // GrpcProtosArtifact, so we need to have them extracted here too. This
    // time we do not add them to `PB.protoSources` so they do not compile.
    libraryDependencies ++= Seq(
      GrpcProtosArtifact % "protobuf"
    ),

    PB.targets in Compile := Seq(
      scalapb.gen() -> (sourceManaged in Compile).value
    ),

  )
  .dependsOn(googleCommonProtos)  // brings the compiled Scala classes from googleCommonProtos

Full example here: https://github.com/thesamet/sbt-protoc/tree/master/examples/google-apis-external-jar

---

Old answer, outdated:

ScalaPB does not handle downloading of third party dependencies, but it's fairly easy to make SBT download them for you and tell ScalaPB to build the downloaded protos.

The following sample build.sbt defines an extractProtos task that downloads the master branch of the repo you linked to as a zip file from github and extracts it. Before doing anything, it checks if the target directory does not exist, to prevent downloading the zip over and over each time you compile.

Since there are many protos in there, we filter the zip file. The source root gets extracted to target/scala-2.12/resource_managed/googleapis-master which we add to PB.protocSources in Compile so when protoc is invoked it processes these files.

You can add more sources in src/main/protobuf and have them "import "google/rpc/...".

scalaVersion := "2.12.2"

libraryDependencies ++= Seq(
    "io.grpc" % "grpc-netty" % com.trueaccord.scalapb.compiler.Version.grpcJavaVersion,
    "com.trueaccord.scalapb" %% "scalapb-runtime-grpc" % com.trueaccord.scalapb.compiler.Version.scalapbVersion
)

PB.targets in Compile := Seq(
  scalapb.gen() -> (sourceManaged in Compile).value
)

PB.generate in Compile := (PB.generate in Compile).dependsOn(extractProtos).value

PB.protoSources in Compile += resourceManaged.value / "googleapis-master"

lazy val extractProtos = Def.task {
  if (!(resourceManaged.value / "googleapis-master").exists) {
    val zipUrl = "https://github.com/googleapis/googleapis/archive/master.zip"
    println(s"Unzipping $zipUrl.")
    IO.unzipURL(
        from=url(zipUrl),
        filter=(
          "googleapis-master/google/bigtable/admin/v2/*" |
          "googleapis-master/google/api/*" | 
          "googleapis-master/google/logging/*" |
          "googleapis-master/google/longrunning/*" |
          "googleapis-master/google/rpc/*" |
          "googleapis-master/google/type/*"
        ),
        toDirectory=resourceManaged.value)
  }
}

libraryDependencies += "com.trueaccord.scalapb" %% "scalapb-runtime" %
  com.trueaccord.scalapb.compiler.Version.scalapbVersion % "protobuf"