Gcloud Pub/Sub: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received

Question

I'm trying to publish messages from the cloud function to the cloud Pub/Sub topic using their NodeJS client package. I'm facing the below error:

Total timeout of API google.pubsub.v1.The publisher exceeded 60000 milliseconds before any response was received.

Solutions I have tried:

1. Increased timeout from 60 seconds to 600 seconds in pub/sub client.
2. Increased CPU and RAM for cloud functions.
3. Used batch messaging for efficiency.
4. Create a single topic instance and use it across all messages.
5. Tried to avoid cold start.
6. Tried with admin permission.
7. I tried avoiding JSON messages and publishing buffers.

Here is my code used to publish messages:

export class FeedService {
  private static instance: FeedService;
  private pubsubClient: PubSub;
  private feedTopic: Topic;
  private maxMessages = 50;
  private maxMilliseconds = 10000;
  private messagePromises: Promise<string>[] = [];

  constructor() {
    this.pubsubClient = new PubSub({
      projectId: process.env.PROJECT_ID,
      credentials: {
        client_email: process.env.GOOGLE_CLOUD_CLIENT_EMAIL,
        private_key: process.env.GOOGLE_CLOUD_CLIENT_PRIVATE_KEY,
      },
    });
  }

  public static getInstance() {
    if (!FeedService.instance) {
      FeedService.instance = new FeedService();
    }

    return FeedService.instance;
  }

  // provide global topic instance
  private getFeedTopic() {
    if (!this.feedTopic) {
      this.feedTopic = this.pubsubClient.topic(process.env.FEED_TOPIC_ID, {
        batching: {
          maxMessages: this.maxMessages,
          maxMilliseconds: this.maxMilliseconds,
        },
      });
    }

    return this.feedTopic;
  }

  // Calling this function to publish messages
  async pushNotificationForAggregation(payload: FeedTypes.FeedQueuePayload) {
    try {
      const messageOptions: MessageOptions = {
        json: payload,
        publishTime: {
          seconds: Date.now() / 1000,
        },
      };

      // store promises
      this.messagePromises.push(
        this.getFeedTopic().publishMessage(messageOptions)
      );

      return { success: true };
    } catch (error: any) {
      logger.error(
        `[Failed to push feed to queue] [${payload.userId}]: ${error?.message}`
      );

      return { success: false };
    }
  }

  // await all promises before sending HTTP response 
  async processRemainingMessages() {
    try {
      const messages = await Promise.all(this.messagePromises);

      return { success: true, messages };
    } catch (error: any) {
      logger.error(
        `[Failed to process remaining messages] : ${error?.message}`
      );

      return { success: false };
    }
  }
}

Updates

I have tried the same code with the cloud run instead of the cloud function, and it works fine there.

So I can confirm there are no issues with the code.

Answer 1

I have solved this issue. The problem was with environmental variables. Google Cloud functions parse each variable added from the GUI console, and ads escape sequences. So \n in my service account's private will be converted to \\n resulting in wrong secrets.

It's strange that the Pub/Sub SDK does not return proper error messages; it always returns an API timeout, no matter how much time I increase the timeout.

Removing escape sequences from the account's private key solves the problem.