BizTalk 2013 Start Message Processing Before Source File Finishes?

Question

We've got a large & complicated file that takes a long time to disassemble (say, an hour). It would be great if we could spin off messages as they leave the receive pipeline and start on their itinerary immediately before the file is finished. I can tell that it's not easy but is it possible at all?

Answer 1

Not out of the box. Pipeline disassembly is transactional so, as you observe, the entire interchange is debatched and commited to the MessageBox at once.

Here are some options:

1. If you are receiving a Flat File where each row is a message, use SSIS to load it into a Table, then use the SQL Adapter, drain the messages by polling out ~10 at a time.
2. If you are receiving a complex Flat File or Xml, you can wrap eithr XmlDasm or FFDasm in a custom Disassembler Component but instead of returning the debatched messages to the MessageBox, push them somwhere else. A) The file system is easy if order is not required. B) MSMQ will maintain the order of the message as they appear in the file.

I have used both of these where the incoming files have had 100k to 400k records and it does provide a more managable performance profile.