Search code examples
goconcurrencyparallel-processingproducer-consumer

producer consumer in golang - concurrency vs parallelism?

I am working on backend architecture which is purely in Golang. I have an API which is used to upload a file to golang server and then I am transferring the file to cloud storage(from the golang server itself). Now, I want both the transfers to be independent, so that, the end user should not has to wait for the response after uploading a file. 

End User -> Golang Server ->[Concurrency/Parallelism] -> Cloud Storage

Now, I thought of two ways:

  1. Create a goroutine as soon as the user finishes the upload and transfer the file to cloud.
  2. Insert the file handler into a queue, and a different process would read this queue and transfer the file to cloud storage (Multiple producers - Single Consumer model).

I found examples of doing this using goroutine and channels but I think that would create as many goroutines as much there are uploads. I want to use the second option but not able to understand of how to go about it in golang?

Also, do suggest if I am using wrong approach and there is some other efficient method of doing this.

Update

Details about the requirement and constraint:
1. I am using AWS S3 as cloud storage. If at some point, the upload from Go server to Amazon S3 fails, the file handler should be kept as in to keep record of the failed upload.(I am not prioritising this, I might change this based on clients feedback)
2. The file will be deleted from the Go server as soon as the upload completes successfully to Amazon S3, so as to avoid repetitive uploads. Also, if a file is uploaded with same name, it will be replaced at Amazon S3.
3. As pointed out in comments, I can use channel as the queue. Is it possible to design the above architecture using Go's Channels and goroutines?

Solution

  • A User uploading a file could tolerate an error, and try again. But the danger exists when an uploaded file exists only on the machine it was uploaded to, and something goes wrong before it gets uploaded to cloud storage. In that case, the file would be lost, and it would be a bummer for the User.

    This is solved by good architecture. It's a first-in, first out queue pattern.

    A favorite Go implementation of this pattern is go-workers perhaps backed by a Redis database.

    Assume there are n number of servers running your service at any given time. Assume that your backend code compiles two separate binaries, a server binary and a worker binary.

    Ideally, the machines accepting file uploads would all mount a shared Network File System such that:

    1. User uploads a file to a server

      a. server adds a record into the work queue, which contains a unique ID from the Redis storage.

      b. This unique ID is used to create the filename, and the file is piped directly from the User upload to temporary storage on NFS server. Note that the file never resides on the storage of the machine running the server.

    2. File is uploaded to cloud storage by a worker

      a. worker picks up the next to-do record from the work queue, which has a unique ID

      b. Using the unique ID to find the file on NFS server, the worker uploads the file to cloud storage

      c. When successful, worker updates the record in the work queue to reflect success

      d. worker deletes the file on NFS server

    By monitoring the server traffic and work queue size as two separate metrics, it can be determined how many servers ought to run the server/worker services respectively.