Centralized monitoring solution in Google Cloud

Question

Similar architecture monitoring we implemented in google cloud

Context: We have a single organization, having different folders and each folder has a single project in Google cloud. We want to have a centralized monitoring solution in GCP. We already set up the architecture shown in the picture. In the given picture let's assume:

Terminologies used

1. 'blue' monitoring is our centralized project or a 'Scoping project'
2. 'yellow' all other boxes are 'Monitoring projects'

Queries I have

1. In the 'Scoping project' can we view and query the logs under a single dashboard or interface (I am aware we can view the logs independently for each project or a resource under this scoping project)?
2. Can we stream logs from the 'Scoping project' to a third-party integration service like datadog from the 'scoping project'?
3. Will that third-party integration have live streaming of logs in an external tool like Datadog?
4. As we have centralized logs in the 'Scoping project', do we need to create a 'Log Router Sink' and 'Log Router Storage' under this 'Scoping project' to be able to stream the logs to third-party integration tools?
5. In case we want to add a new project, do we need to create 'Log Router Sink' and 'Log Router Storage' in the 'NEW Monitoring project (to be newly added)' so that we stream logs to the 'Scoping project'?

Answer 1

As the person at least partially responsible for the image and the recommendation....

In the 'Scoping project' can we view and query the logs under a single dashboard or interface (I am aware we can view the logs independently for each project or a resource under this scoping project)?

No - logs aren't part of Monitoring Scopes. They're treated entirely separately. Have a look here.

Can we stream logs from the 'Scoping project' to a third-party integration service like datadog from the 'scoping project'?

Sure - you can do whatever you want with those logs.

Will that third-party integration have live streaming of logs in an external tool like Datadog?

Yes - use e.g the PubSub sink and consume them from there.

As we have centralized logs in the 'Scoping project', do we need to create a 'Log Router Sink' and 'Log Router Storage' under this 'Scoping project' to be able to stream the logs to third-party integration tools?

See above.

In case we want to add a new project, do we need to create 'Log Router Sink' and 'Log Router Storage' in the 'NEW Monitoring project (to be newly added)' so that we stream logs to the 'Scoping project'?

See above. Logs aren't subject to monitoring project configuration and are treated very differently.

Have a look at this video for a walkthrough of log storage and routing.

Additional info in response to comment below:

1. Does the service account used for third-party integration, merely need to have permission on the 'Scoping proj'? BUT NOT on other 'Monitoring proj'?

To see monitoring data, the SA will need appropriate permissions. Have a look here: https://cloud.google.com/monitoring/access-control

2. What's the key diff between 'Cloud Logging' and 'Cloud Monitoring', as seems both offer similar features and functionalities?

Well, Logging is all about logs :) And Monitoring is all about metric/time series data.

3. What are the main advantages of having a 'Centralised Cloud monitoring' proj?

I think I answer this question in this video: https://youtu.be/pcMEFTxcco8?list=PLIivdWyY5sqLOiLXJDlN-wKd0g7hf_9vC

Generally, it's protection from accidental deletion and a single place to manage/control/set things up.

4. How to achieve resilience in a 'Scoping proj'? if something goes wrong in scoping, whole monitoring will be stalled?

Good question - it's not really a problem I've seen anyone have. Remember that the metrics are just visible from the scoping project - they are still available in the actual resource project. So even if something goes wrong, you can just recreate the scoping project and still have all the data.