Best Design Pattern for Large Data processing methods

Question

I have an application that I am refactoring and trying to Follow some of the "Clean Code" principles. I have an application that reads data from multiple different data sources and manipulates/formats that data and inserts it into another database. I have a data layer with the associated DTO's, repositories, interfaces , and helpers for each data source as well as a business layer with the matching entities, repositories and interfaces.

My question comes down to the Import Method. I basically have one method that systematically calls each Business logic method to read, process and save the data. There are a lot of calls that need to be made and even though the Import method itself is not manipulating the data at all, the method is still extremely large. Is there a better way to process this data?

ICustomer<Customer> sourceCustomerList = new CustomerRepository();
foreach (Customer customer in sourceCustomerList.GetAllCustomers())
{

   // Read Some Data
   DataObject object1 = iSourceDataType1.GetDataByCustomerID(customer.ID)
   // Format and save the Data
   iTargetDataType1.InsertDataType1(object1)

   // Read Some Data

   // Format the Data

   // Save the Data

   //...Rinse and repeat
}

Answer 1

You should look into Task Parallel Library (TPL) and Dataflow

ICustomer<Customer> sourceCustomerList = new CustomerRepository();

var customersBuffer = new BufferBlock<Customer>();
var transformBlock = new TransformBlock<Customer, DataObject>(
    customer => iSourceDataType1.GetDataByCustomerID(customer.ID)
);

// Build your block with TransformBlock, ActionBlock, many more... 
customersBuffer.LinkTo(transformBlock);

// Add all the blocks you need here....

// Then feed the first block or use a custom source
foreach (var c in sourceCustomerList.GetAllCustomers())
    customersBuffer.Post(c)
customersBuffer.Complete();