Search code examples
javagoogle-cloud-platformgoogle-cloud-dataprocgoogle-cloud-dataproc-serverless

Interacting with Dataproc Serverless using Dataproc Client Library


I'm writing a java code in which I run Spark batches using Dataproc Serverless. I have initially used Dataproc Client Library for java and so far, it works just fine. However, by looking at Dataproc Serverless documentation (https://cloud.google.com/dataproc-serverless/docs), there is not a single mention of interacting with it by using Java Client - it mentions REST, RPC and Cloud SDK.

I started wondering if I'm doing something wrong. Is using Dataproc Client Library for Dataproc Serverless related actions actually not recommended or somehow limited?


Solution

  • You're doing it right - the Dataproc Client Library is the correct library to interact with Dataproc Serverless (for context, it uses the gRPC API mentioned in the Dataproc Serverless documentation). Generally, Dataproc Serverless is a part of the overall Dataproc product. The fact that the Dataproc Client Library is not mentioned on the Dataproc Serverless docs page is a mistake.