I have a following situation: my application may accept Job from a client. There are many clients submitting separate Jobs. Job consists of multiple Tasks which are executed in a given order, one after another (each Task will use result of previous one). Let's say that each Task is some kind of calculation, however, how long they can last varies greatly - sometimes it is a few minutes and sometimes a few hours. So the whole Job could take up to a day or two. Job can be cancelled or modified by client at any time - for example some Tasks might be removed or even stopped during execution, which would result in moving to the next Task. Client could also ask about the progress of the Job at every moment. So I think that I would need some kind of JobManager and JobProcessor/TaskProcessor to handle given requirements. I was looking at some design patterns and solutions, but I couldn't find any that would fit my problem good enough.
The first thing that came to my mind was using Master-Worker pattern. Master would be a Manager and each Worker would be responsible for executing a single Job. I'm not sure if its a good solution, as workers in this pattern are usually working on different parts of one job, however in my situtaion, each worker would be responsible for completely different job.
Do you know of any patterns or some improvement to my proposed solution that would help me solve this case? It is going to be implemented in Spring WebFlux if that helps.
If we are talking about solution in java you have different options. One of them is to use JSR-352 implementation, for example Spring Batch. This framework allows you to organise your processing in pretty natural way. The domain of JSR-352 is already consists of such terms as Job, Step, Chunk, Execution and so on. To start you have to define the Job and the Steps it consists of. I have simple example of using Spring Batch in my github.
Another option is prepare that processing skeleton by yourself. I'm not sure that there is some ready-to-use design pattern but you can look how Spring Batch internal processing is organised. Also you can take a look into command design pattern.
however in my situtaion, each worker would be responsible for completely different job.
The main trick here is that you have to separate concerns. E.g. worker should be some abstract entity that know nothing about the some specific job. The job itself could implement some interface - contract that all jobs should follow.