Search code examples
scalaapache-sparksequences

MonotonicallyIncreasingId output widely variable

In usage of this feature I can get low values 0, 1, 2, 3, ... returned or huge values returned when using with a Dataframe. It is unclear why this is so. I read one has no control over values generated.

Solution

  • From a comment in the source code:

    The current implementation puts the partition ID in the upper 31 bits, and the lower 33 bits represent the record number within each partition. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.

    So you will get low values for partition 0, very high values for every other partition.

    But this is an implementation detail that you should not rely on. Only the monotonically increasing nature is guaranteed to remain the same.