How can I export a Stack Driver log to a file for local processing?

Question

All I know is, we can fetch logs using stack driver Logging or monitoring services. But from where these logs are being fetched from? If i know where these logs are fetched from then no need of doing API calls or using another service to see my logs. I can simply download them and use my own code to process them. Is there any way to do this?

Answer 1

There is a capability of Stack driver logging called "Exporting". Here is a link to the documentation. At a high level, exporting is the idea that when a new log message is written to a log, a copy of that message is then exported. The targets of the export (called sinks) can be:

• Cloud Storage
• Big Query
• Pub/Sub

From your description, if you set up Cloud Storage as a sink, then you will have new files written to your Cloud Storage bucket that you can then retrieve and process.

The following image (copied from the docs) gives the best overview:

If you don't wish to use exports of new log entries, you can use either the API or gcloud to read the current logs. Realize that GCP held logs (within Stackdriver) expire after a period of time (30 days). See gcloud logging read.