Spotfire: How to filter 'old' data from data functions running on a data link that appends 'new' data

My situation is this. I have a data-link that continually appends new snapshot data to a table, and upon the arrival of a new snapshot, it runs an R data function (script) which does some calculations with results that append to an output table. The R calculations are quite expensive and the input data is large, and more importantly, the snapshots are independent of each other, so there is no need to re-process previously received snapshots every time a new snapshot arrives.

I can't make a data function that takes it's own results as input (i.e. to filter by previously processed dates), and my other idea also throws up cyclic dependencies (creating a second data function to generate a second table with previously processed dates).

Has anyone experienced this issue, and could you please give me some ideas on safe ways to address it? I'm new to Spotfire (and dash-boarding generally).

Solution

I had the same situation - I got around this with a few steps:

Added a new filtering scheme just for data processing
Adding a calculated column to my output table which was just "X", that I could filter out ie. filter out the entire output table
Added a relationship between my input table and output on a key column which excluded any filtered out columns
In my data function input parameters, selected to filter the input data by the data processing filtering scheme. This meant every time my function runs it only runs off the unprocessed data.

I am current appending the data so that it adds on update not replaces. This works fine when I open the file everyday and save it however is not working with the scheduled updates in the web player- it doesn't add a data function add rows action each time - still looking for a way around this.... possibly an iron python script will do the trick..