Search code examples
talendtalend-mdm

what are the differences between Multi thread Execution and Parallelization with respect to Job performance in Talend?

Multi Thread Excecutionenter image description here

Parallelization

enter image description here

Solution

  • Multi threading is optimal when the number of threads (in general a Subjob count one thread) do not exceed the number of processors of the machine you use for parallel executions. Otherwise, some of the Subjobs have to wait until any processor is freed up.

    Also note that you cannot parallelize more than your number of CPU, otherwise it will wait for the processors and will be overhead for processors.

    Parallelization helps you to manage complex Job systems. It executes several subjobs simultaneously and synchronizes the execution of a subjob with other sub-jobs within the main Job.

    The exact difference between above two is Parallelize(or multi thread enabled) linked sub jobs run parallel regardless of which ones finish first, on the other hand Synchronize linked sub jobs starts to run only when all other parallelize sub jobs finishes.

    So, Parallelization is best when you have a request that need some of subjobs to run parallel, and a subjob starts to run only when all other parallelize subjobs finishes.

    It also makes your job design more flexible.

    For detailed information you can visit this link