I have a Database Project inside of SSDT 2012 that contains a SSIS project using the package deployment model. The goal of the project is to load a lot of information at once that would normally take too much time if one package did it. So I divided it between 15 children each doing their on separate part, loading data into various sql tables. So, inside this project is one parent package and 15 child packages. Because of the type of data that is loading, I have to use script task to insert it all. Each child package is the same, only differing between parameters that divide the data up between the children. Each child package is executed using a External Reference through the File System.
The problem I'm having is while the parent package is supposed to start all the child packages at once, not all of the children are starting. It's as if there is a limit to how many packages can start at one time (looks like about 10 or 11). Once it hits this limit, the rest don't start. But when one package finishes, another immediately starts.
Is there a property I'm missing that is limiting how packages can run at the same time? Based on what others are able to run at the same time, there seems to be something I'm missing. I read somewhere memory can be a factor, but when I look at Task Manager, I don't see anything above 15% of my memory used.
The problem is solved by looking at the the property MaxConcurrentExecutables on the parent package. In my parent package, this property had a default value of -1, which means it calculates how many tasks that run in parallel (in this case, child packages) based on the number of cores on your PC plus 2.
In my case I have 8 cores on my laptop, plus the number 2 which put me at 10 packages running at the same time. You can override this value by putting a higher positive number in its place to allow more children to run. After putting in 20, all tasks started at once.
More information about this can found here:
https://andrewbutenko.wordpress.com/2012/02/15/ssis-package-maxconcurrentexecutables/