Google Cloud Storage bucket uploaded file verification when using malware scanner(ClamAV)

Question

So as the title says how do i verify if the file has been successfully uploaded to the clean bucket. We got 3 buckets(unscanned, clean, quarantined) and malware scanner(ClamAV). First the file is uploaded to the unscanned bucket and then it's transferred either to the clean or quarantined bucket. My questions is how do i know if my uploaded file has been uploaded to either clean or the quarantined bucket. I am using Golang as backend language. And i see that we got cloud storage subscription, but when i try to get the messages it says

This method is not supported for this subscription type

seems like we can't use storage subscription this way.

Other way i can think of for confirming the uploaded file is to query the logs of ClamAV scans and find if the file is infected or not. Whats the proper way to do this?

in this example i can't just add:

    err = sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
                fmt.Println(m.Data)
        m.Ack() // Acknowledge that we've consumed the message.
    })

``` it says this method is not supported for this subscription type...

Answer 1

The Cloud Storage subscription is designed to write messages published to Pub/Sub to files in a designated bucket. It is not designed to provide information regarding the status of files that are uploaded via other means. What you are probably looking for is Pub/Sub notifications for Cloud Storage, which publishes messages to a Pub/Sub topic when a file in a bucket is created, deleted, archived, or has its metadata updated.

In your scenario, you would create a Pub/Sub topic, create a Pub/Sub pull subscription, and create a notification on your GCS bucket:

> gcloud pubsub topics create TOPIC_NAME
> gcloud pubsub subscriptions create SUBSCRIPTION_NAME --topic=TOPIC_NAME
> gcloud storage buckets notifications create gs://BUCKET_NAME --topic=TOPIC_NAME

In your subscriber, you would look for messages with the attribute eventType set to OBJECT_FINALIZE:

    err = sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
        event := m.Attributes["eventType"]
        if (event == "OBJECT_FINALIZE") {
            obj := m.Attributes["objectId"]
            fmt.Printf("File %s written successfully.\n", obj)
        }
        m.Ack()
    })

You could also set the subscription up with a filter so that it only delivers the finalize messages:

> gcloud pubsub subscriptions create SUBSCRIPTION_NAME \
  --topic=TOPIC_NAME \
  --message-filter 'attributes.eventType = "OBJECT_FINALIZE"'

Then you wouldn't need the if in your code.