Search code examples
palantir-foundryfoundry-code-repositories

How do I save a historical (past) dataset in Foundry

I want to save a version of my dataset called (order_clean). But not the current version. The version I want to save is from the past. I understand this is a historical transaction that still exists within my retention policy window. How would I be able to do this?

Solution

  • This can be done via an API call and hitting a specific endpoint. This answer assumes a working knowledge around authorization tokens and curl requests:

    1. Obtain the desired dataset’s RID (Example in screenshot below), as well as the transaction ID of the transaction corresponding to the version of the dataset you want. [Dataset order_clean

    2. Create a new branch in your specific dataset by running a curl request populated as follows:

      curl -X POST -H "Content-type: application/json" -H "Authorization: Bearer YOUR_AUTH_TOKEN" "STACK_URL/foundry-catalog/api/catalog/datasets/YOUR_DATASET_RID/branchesUnrestricted2/NEW_BRANCH_NAME" -d '{"parentRef": "YOUR_TRANSACTION_ID", "parentBranchId": "master"}'

    3. Replace YOUR_AUTH_TOKEN, STACK_URL, YOUR_DATASET_RID, NEW_BRANCH_NAME and YOUR_TRANSACTION_ID with appropriate values. Short lived authentication token can be generated following these instructions.

    4. Save the branch into a new dataset with a transform in a Code Repo, specifying the Input with the path to the dataset as well as the new branch’s name from #2.