Inverted pipeline/chain of responsibility pattern

Question

I am wondering whether there is an establish pattern to control the flow that my application will have.

Simply put, it's supposed to be something like that:

• User provides a file
• File is being processed
• User receives a processed file

There will be several processing steps, lets say PreprocessingOne, PreprocessingTwo, PreprocessingThree and FinalProcessing.

Naturally, we do not control the files that the user provides - they will require a different amount of preprocessing steps.

Since my message handler services will be in separate APIs, I don't want to invoke them just to return 'Cannot process yet' or 'Does not require processing' for performance reason.

Similarily, I don't want to pass the uploaded file around between services.

Ideally, I would like to design the flow for a file dynamically by evaluating the content and inserting only those of the message handlers that make sense.

I am saying 'Inverted' pipeline, because instead of going from A to Z I would rather like to check which stages I need starting from Z and only insert the last ones.

So, if the uploaded file qualifies for FinalProcessing right away, the flow would be just one element.

If the file requires to go from PreprocessingTwo then the flow would be PreprocessingTwo > PreprocessingThree > FinalProcessing

So, I was thinking I could implement something like that, but I am not sure about the details.

public interface IMessageHandler
   {
    void Process(IFile file);
   }


public interface IContentEvaluator 
{
     IList<IMessageHandler> PrepareWorkflow(IFile file);
}

public interface IPipelineExecutor
{
        void ExecuteWorkflow(IList<IMessageHandler> workflow, IFile file);
   }

And then in the application

public void Start(IFile newFile)
{
    var contentEvaluator = new ContentEvaluator(this.availableHandlers); // would be DI
    var workflow = contentEvaluator.PrepareWorkflow(newFile);
    this.executor.ExecuteWorkflow(workflow, newFile);

}

Could you please advise, recommend some approach or further read?

Answer 1

You can consider to use Strategy pattern: ...selects an algorithm at runtime...
But if you have too many combinations of the flow than the number of strategies which needs to be implemented will increase and solution can be complex.

Another approach can be to use SEDA: ...decomposes a complex, event-driven application into a set of stages connected by queues...
PreprocessingOne, PreprocessingTwo, PreprocessingThree and FinalProcessing are the stages, and flows can be defined by directing outgoing messages to different queues.