How access a set of folders from a list of services without a conflict

Question

I have a set of services running on different servers all accessing a shared resource, e.g; a list of folders containing videos that require some processing.

I want to implement some locking mechanism to prevent the services from accessing the same folder simultaneously, so far, my idea is creating a text file within a folder as an "in-folder-lock" and check if that file exists before processing the video files within.

I'm pretty sure this test won't do well in a real production setting where services look for folders to work on.

Any ideas?

Any help is appreciated, thanks in advance.

Answer 1

You could use some distributed locking mechanism such as Redis Redlock or Consul lock but if you have multiple servers fighting for this shared folder, why don't you consider sharding the files in multiple folders (using a consistent hashing algorithm) and then having your servers (using the same consistent hashing algorithm) process the files only for their dedicated folder? This way you could parallelize the processing instead of having all the N-1 servers waiting for the Nth server to process everything in the single folder you have.