Search code examples
google-cloud-platformgcloudgoogle-cloud-pubsubstackdrivergoogle-cloud-stackdriver

How can I create a pubsub log sink to a different google cloud project/org?

I would like to define the steps in making a pubsub log export sink from one project to another. As a secondary goal, I would like the sink to bridge organizations. So far I have followed these steps as outlined in the gcloud help pages and the auth documentation.

First I created a sink:

# from project A
gcloud logging sinks create \
    <sink_name> \
    pubsub.googleapis.com/projects/<project_B>/topics/<topic_name> \
    --log-filter <filter>

The CLI returns successfully and gives a little advice on setting up permissions for a service account it created:

Created [https://logging.googleapis.com/v2/projects/<project_A>/sinks/<sink_name>].
Please remember to grant `serviceAccount:<new_user>@logging-<project_A_account_number>.iam.gserviceaccount.com` Pub/Sub Publisher role to the topic.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

Following this advice, I gave the new service account the appropriate permissions for that topic.

gcloud projects add-iam-policy-binding <project_B> \
    --member serviceAccount:<new_user>@logging-<project_A_account_numbe_id>.iam.gserviceaccount.com \
    --role roles/pubsub.publisher

This command returns without issue.

In spite of everything seeming OK, no logs flow through the sink.

Here are some clues: The Exports tab on the Logs Viewer reports a permissions error in the sink. The Project Activity tab reports a permissions issue.

Image: Logs Viewer, Exports

Image: Project, Activity

Is there a solution to make this work? Is it possible to generalize this to send logs to a sink in this project from other gcloud organizations?

Solution

  • I have been able to reproduce the scenario you wanted. Let me set the basics for the scenario description:

    • Project A (where my logs are stored): project-a
    • Project B (where my Pub/Sub Topic and Subscription are): project-b
    • Topic: projects/project-b/topics/myTopic
    • Subscription: mySub
    • Sink: test-sink

    Then, this is the processed I followed:

    1. In project A: create a filter for logs in the Logging > Logs tab in the Google Cloud Console.

    2. Create an export with the elements in the image below (remember to append pubsub.googleapis.com/ to the name of your topic in the other project): enter image description here

    3. Move to the Exports tab and copy the Writer Identity, which should have the format test-sink@XXXXXXXX.iam.gserviceaccount.com

    4. In project B: go to the IAM & admin > IAM tab in the Console and add a new member being the previous service account obtained in step 3 with the role Pub/Sub Editor enabled.

    5. Create a Pub/Sub subscription with the command gcloud beta pubsub subscriptions create --topic myTopic mySub

    6. Do some operation that results in logs read by the filter you specified in Project A.

    7. Consume the logs written to the topic using the subscription, with the command gcloud beta pubsub subscriptions pull mySub.

    There you will find the logs that are written from Project A to your tropic in Project B. I have reproduced this same scenario writing logs from a simple App Engine application (and therefore with the appropriate log filter searching for App Engine logs), and when I make requests to the App Engine app, some logs are created and then written in myTopic, which I can read using mySub.

    Regarding your second question, I cannot make sure whether this same procedure works in a cross-organization scenario but I see no issues with that.