Search code examples
datasetpalantir-foundryfoundry-code-repositoriesfoundry-fusion-sheet

Index order break Foundry dataset from fusion sheet


I have created Palantir Foundry dataset from the Fusion sheet. But rows are not in the correct order as the Fusion sheet. What is the correct way to do it?

Fusion sheet data

Fusion sheet data

Dataset created from Fusion sheet

Dataset created from Fusion sheet


Solution

  • Spark doesn't preserve any ordering and Fusion does not try to keep row orders when exporting to a dataset. In general, row orders in Foundry datasets should not be relied upon. You might be able to work around this by adding an index column.

    Previous edit:

    I had tested this and the sync kept the order on the output dataset correctly without any particular configuration. After digging a bit more it seems I was just lucky.

    enter image description here

    enter image description here