Google Pub/Sub API gets OAUTH2 authentication failure when trying to connect to a specific endpoint

Question

Recently I have been trying to add ordering-key support to my Google Pub/Sub services. The Google Pub/Sub ordering-key documentation indicates that when using multiple publishers, you should connect to a particular Google endpoint to ensure that the ordering works correctly. Ok fine.

However, when I add the settings.setEndpoint(...) calls to the following code, I get the following exception:

Caused by: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See ...

If I comment out the endpoint calls and connect to the global endpoints then the code works so I don't think there is a problem with the service account key. As far as I can tell there isn't a way to specify an endpoint when creating a service account key.

Here is my code. We are using the com.google.cloud.google-cloud-pubsub-1.114.3.jar API jar although I get the same error when I try 1.120.22 which looks to be the latest.

final String PROJECT_ID = "...";
final String TOPIC_NAME = "test-20221025-153636-7318";
final String ENDPOINT = "us-east1-pubsub.googleapis.com:443";
final String CREDENTIALS = "{\n" //
        + "  \"type\": \"service_account\",\n" //
        + "  \"project_id\": \"" + PROJECT_ID + "\",\n" //
        + "  \"private_key_id\": \"...\",\n" //
        + "  \"private_key\": ...\"" //
        + "  \"client_email\": \"...\",\n" //
        + "  \"client_id\": \"...\",\n" //
        + ...
        + "}\n";

TopicName googleTopicName = ProjectTopicName.of(PROJECT_ID, TOPIC_NAME);
CredentialsProvider credentialsProvider = FixedCredentialsProvider
        .create(GoogleCredentials.fromStream(
            new ByteArrayInputStream(CREDENTIALS.getBytes())));
// create the publisher
Publisher.Builder publisherBuilder = Publisher.newBuilder(googleTopicName);
publisherBuilder.setCredentialsProvider(credentialsProvider);
// if I comment this out then the code works fine
publisherBuilder.setEndpoint(ENDPOINT);
publisherBuilder.setEnableMessageOrdering(true);
Publisher publisher = publisherBuilder.build();
// publish a message
PubsubMessage message = PubsubMessage.newBuilder().setMessageId("id")
        .setData(ByteString.copyFrom("data", Charset.defaultCharset()))
        .setOrderingKey("foo").build();
ApiFuture<String> future = publisher.publish(message);
System.out.println("Publishing message returned: " + future.get());

Any idea what is going on? Thanks in advance for any help.

Answer 1

However, when I add the settings.setEndpoint(...) calls to the following code, I am getting the following exception: ... UnauthenticatedException ...

After more research, I found this github issue thread on the Pub/Sub Github page that matches my problem precisely. Guess I should have started there but hopefully this question/answer will help others.

The problem seems to be that the service account credentials do not have the right scopes. I mean of course they don't. 😑

Changing the credentials creation code to the following makes my little test program work now:

ServiceAccountCredentials sac = ServiceAccountCredentials.fromStream(
    new ByteArrayInputStream(CREDENTIALS.getBytes()));
GoogleCredentials scopedCredentials =
    sac.createScoped(PublisherStubSettings.getDefaultServiceScopes());
CredentialsProvider credentialsProvider =
    FixedCredentialsProvider.create(scopedCredentials);

I have yet to find any mention of this in the Google documentation unfortunately. If anyone finds a mention please add a comment so I can reference it here. The requirement of setting default scopes on the credentials to get it to work seems odd.

I'm also not sure if this is a generic issue. If you are doing anything with the Google API endpoint, do you need to get the default scopes on your credentials? If I have some time I will do some more tests on this. Add a comment if you know the answer.

Hope this helps others.