Search code examples
javazlibcrccrc32

CRC combine vs update performance

I have a large blob of data that I'm dividing into smaller parts to send over the network. Before sending, I compile the list of smaller parts and want to ensure that there is no data integrity issue.

My question is

  • should I do CRC32C combine for each smaller part and compare with the CRC32C of the large data blob? or
  • just keep updating the CRC32C with update function (defined here) for each smaller part and finally check whether that equals the CRC32C of the large blob?

Also, any pointers on zlib's equivalent of Java's CRC32C combine function implementation would be helpful.

Solution

  • If you are only sending the CRC-32C of the "large blob", then do your second bullet, which is to update a single CRC value for each "smaller part". The only way there would be a speed advantage to combining the CRCs of the smaller parts would be if you were computing those CRCs in parallel on differents cores or threads.