Pentaho kittle - Skip files in root and only move/process the files in subfolder

Question

I need to copy the files from User1 and user2 folder to "process" folder but I need to skip the files that may be dropped by the user directly into Root folder.

-Root + User1 + User2

Is there any way xpression can be used to skip to move those files or can we exclude the files directly under Root folder to be excluded to be processed.

Thanks, Rahul

Answer 1

You can do this by loading your result file list from a transform and then using a Process result filenames step. The job is very simple:

The transform is where the logic happens. Add two Get File Names steps, one for the user1 and user2 files, and one for the exception (root) files. When configuring these be sure to click the 'Filters' tab and uncheck 'Add filenames to result'. Read the exceptions directory with a Stream lookup. Compare the short_filename field from both input steps and specify 'exists' as your lookup field. It should look something like this:

The Filter Rows step flows off all the files that exist in the exceptions directory, and the Set files in result step puts only the ones that don't exist in the exceptions directory into the Job's file results. Be sure to use the 'filename' field rather than the 'short_filename` field here.

Then in the Job, the Process result filenames can be configured to do what you want with the files (move/copy/delete, etc).