Search code examples
palantir-foundryfoundry-code-repositories

How do I create an output transform with a dynamic date and time in Code Repository?

Is it possible to create an output depending on the date and time? I would like to a use a dynamic name for the datasets that will be built every day for example. This would allow me to keep track of the dataset and the date will be displayed in the pathing. I attached an example below.

output with timestamp

Ideally - the output will be named ".../datasets/dataset_2023_03_09_HH_MM"

Solution

  • Dynamic naming for transform outputs is not possible.

    The technical reason is that the inputs/outputs/transforms are fixed at CI-time. When you press "commit" in Authoring or you merge a PR, a CI job is kicked off. In this CI job, all the relations between inputs and outputs are determined including the links between unique identifiers and datasets names. Output datasets that don't exist yet are created, and a "jobspec" is added to them. A "jobspec" is a snippet of JSON that describes to foundry how a particular dataset is generated.

    Anytime you press the "build" button on a dataset (or build the dataset through a schedule or similar), the jobspec is consulted (not updated). It contains a reference to the repository, revision, source file and entry point of the function that builds this dataset. From there the build is orchestrated and kicks off, invoking your function to produce the final output. Therefore, when the date is changing and the build is triggered, an error will be raised as the job specs are not valid anymore. The link between unique identifier and dataset name is broken.

    To address your needs of further analysis based on the date of the day, the optimal way to proceed is to:

    1. add a new column to your dataset which includes the date at which the build has run
    2. build you dataset incrementally by specifying snapshot_input=[your_dataset] to ensure your are adding each day all exiting rows from your input dataset
    3. perform your analysis by filtering on the date column

    Please find here the complete documentation for Incremental transforms and Snapshot Input.