Search code examples
pentahopdi

Pentaho Text File Input Stuck

I got a TextFile that has Data parsed in a certain Format per line.

Its one of many and this one is quite small with only 34504 lines. Size is 274KB. The same problem happens with every each of them.

I used the Text File Input Step in and put ${line.seperator} as Seperator (Didn't actually found it in the Variable List but I found it in other answers for line breaks). Get Fields got me Field_00 and the Preview showed me all Data in each row as I wanted.

Now if I want to execute the Transformation it gets stuck in that Step and doesn't finish ever. I put a Write to Log File Step after it that never gets reached and even alone the Step never finishes. Even changing the File from "filename.ges" to ".txt" or ".csv" doesn't change the outcome.

If I Stop the Transformation it gets put into Infinite Halting condition and I need to restart the PDI.

I also upped the Java MaxPermSize in the spoon.bat because I got a Null Pointer Exception the first Time:

if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xms1024m" "-Xmx2048m" "-XX:MaxPermSize=8000m"

PDI Version 8.3.0.0-371

Does someone have ideas how I can resolve this?

Content of the Step and the Preview Rows: enter image description here

Solution

  • You are configuring the field separator with the value of line separator. It's probably trying to construct one string with 34504 lines worth of characters.

    If you want to read lines into a single field each, use the Fixed width filetype (top item in the configuration) and define 1 field with length 9999999999 or whatever fits.