PDI - how to skip file that have already processed?

Look at my jobs and transformation below :

I want to process files from FTP and Shared folder. My team will put CSV files in there every day if there are a new one. Files in FTP and Shared Folder will hold until 7 days old before being removed.

My question is if last day I have already process A.csv, B.csv and then today I want process only C.csv without A.csv, B.csv even the file still in the same folder and I don't want move or delete file that have already processed. How I do that?

Solution

Better create one table and store the processed file name. Add step to check the file name exists in that table or not. if not exists then process the file otherwise skip the file

How can I close over variables in kdb/Q?
When is the EACH operator extension necessary in K besides mod/rotate?
Handling single-character strings - in a function or in its caller? ssr()
Kdb+ data fomat when writing to a file
How to convert a symbol to a string in kdb+?
Sum of each two elements using vector functions
A dictionary with a single value and multiple keys
Table transformation, table as list of dicts
Accumulator gives different result then direct function applying
Reshape [cols;table]
FK field over IPC
Protected execution, 2 cases
Enums for tables
Converge (fixed point) syntax difference in q and k
.Q.trp and bt handling
NULLs in q and in k.h
Strange view declaration behaviour
How to build a parse-tree of projections?
Could not evaluate manually created equial ~ parse tree
Select distinct for all columns from keyed table
Parallel execution: blocking receive, deferred synchronous
Multiple variable assignment in q
Select a table from the inside of external select
Select when one of filter-column may not exists
What is the meaning of `s attribute on a table?
On parallel execution - which side reports about an error?
Validate if a keyed table have unique keys
Applying dictionary to dictionary
About xkey implementation
Parse tree built on values from vars