Search code examples
google-cloud-platformlogginggoogle-cloud-storagegoogle-cloud-logging

How can I configure the lifecycle management for automatic deletion of logs in Google Cloud Platform's logging storage?

I have a few Google Cloud projects that accumulate a substantial amount of logs, around 200GB per month, and this is expected to nearly double in the next period. Initially, I assumed it would be straightforward to manage this by adjusting the retention settings: Edit the log bucket, set the retention period in days, and then update the bucket.

I configured this more than six months ago, setting it to one day for a non-production project. However, it seems that nothing gets deleted until the beginning of the next month. The logs continue to accumulate daily and only reset on the first day of the following month.

After reviewing the documentation and various Stack Overflow questions, I discovered that retention isn't the period for log deletion; it's actually the duration to preserve logs. To address this, we need to establish a lifecycle for the objects within the bucket. Unfortunately, the bucket in question (_Default logging bucket) is a system bucket, and I can't configure it.

Even when I delete logs using gcloud logging logs delete, the storage remains unchanged and doesn't reset until the next month. Is there a way to set the logs' lifecycle to be less than a month?

Logs storage

Retention

Edit: Actually, retention should delete the logs, but I don't think this is happening. My logs for all projects.

retentionDays   integer

Logs will be retained by default for this amount of time, after which 
they will automatically be deleted. The minimum retention period is 1 
day. If this value is set to zero at bucket creation time, the 
default time of 30 days will be used.

https://cloud.google.com/logging/docs/reference/v2/rest/v2/locations.buckets

Solution

  • I believe I solved my own question; it is a misunderstanding of how Cloud Logging pricing works.

    • Cloud Logging doesn't charge for the current size of logs stored in the bucket but for the indexed or processed logs. That's why the storage size only goes up, and even if you delete the logs, it doesn't go down.

    ingestion

    That's why it says ingestion, not storage.

    Logging charges for the pre-indexed volume of logs data that is stored in
 the _Default log bucket and in user-defined log buckets, when the total
 volume exceeds the free monthly allotment. Every write of a log entry to
 the _Default log bucket or to a user-defined log bucket counts toward your
 storage allotment. For example, if you have sinks that route a log entry to
 three log buckets, then that log entry is stored three times.
    • Also, retention is insignificant if it's under 30 days since it's free. Keeping data beyond 30 days incurs an extra cost as log storage size.

    retantion

    And the billable storage is 0 because retention was set to 1 day.

    Logging charges retention costs when the logs are retained longer than the
 default retention period. You can't configure the retention period for the
 _Required log bucket. There are no retention costs when logs are stored
 only for the default retention period of the log bucket.

    https://cloud.google.com/stackdriver/pricing#storage-pricing