Search code examples
gnu-parallel

parsort using all cores on machine even when --parallel=16 is set

I'm trying to sort a ~200GB text file and I want to use parsort (GNU Parallel) as it seems like it will help with performance.

When I run the following command, instead of running on just 16 cores it quickly starts running across all 56 cores on the machine I'm running it on.

sort --parallel=16 -t ',' -k1,1 -k11,11 -k8,8 -T /data/tmp "${IN_FN}" > "${SORTED_FN}"

I thought the parallel=16 was telling it to use only 16 cores, so does anyone know why it eats up all 56 cores on the machine?

Solution

  • Could you provide the version number you're using (parsort --version)? It seems you might be running on an older version.

    For reference, I was able to reproduce the behavior using version 20220122, but not with 20230423.

    Based on the details provided, it suggests you might be using an outdated version.