Search code examples
pentaho-data-integration

Logging Status of Files Read by Pentaho Text File Input Step

I'm trying to use a Text File Input step with wildcards(regexpressions) to read a bunch of files(say 100) from a directory. This is working fine. But what I'd like to know is whether there is a way to log which all files got processed successfully and which all failed in the step?

Say for example, In directory /home/usr/ I have 10 .txt files like A.txt,B.txt,C.txt,D.txt,E.txt,F.txt,G.txt,H.txt,I.txt,J.txt.

What I would like to know is whether there is way I can log something like,

file A.txt processed successfully 
file B.txt processed successfully 
file C.txt processed successfully 
file D.txt failed //(due to some reason which doesn't matter) 
file E.txt processed successfully 
file F.txt processed successfully 
file G.txt processed successfully 
file H.txt failed 
file I.txt processed successfully 
file J.txt processed successfully

Simply put, I would just like to log the status of the individual files that the Text File Input step is reading.

Is this possible?

I already tried running the transformation with Detailed, Debug and Row Level log level but to no avail.

Would appreciate some help thanks!!

Solution

  • If you are running everything(reading all files) in the same KTR, and any of them causes error, your KTR will stop there.

    What you need is to run this KTR for each file found in the directory, for that you need a job, with a KTR that lists those files, and passes the filename to your KTR through variables. You can then use an Error flow in the Job so your KTR keeps on running for the next files.

    Your Job should look something like this:enter image description here