I have a main input table containing 'header' data like id, name, date, .... I also have a subjob CALC dat calculates the values by month depending on the date of MAIN --> the subjob stores a new table with columns: [month_1], ...., [month_12], [TOTAL_YEAR] into a new .dat file (stored by tCacheOutput) with the name of the Parent Job and the currentYear.
So for example: Parentjob = PARENT
PARENT saves some header data containing: [id],[name],[date],[value] into a cache file PARENT.dat and then calls the subjob CHILD.
CHILD iterates through years (I supply them through an tFixedFlowInput) and gets data from the PARENT.dat file. Then it checks the year (f.ex. 2016) and month and and generates a new table (using tMap) with [month_1], [month_2],...,[month_12],[total] and stores that into a new .dat file called PARENT_2016. This process is repeated for every year we wish to get the value of...
After returning to the Parent job, i now have to group the data contained in the PARENT.dat, PARENT_2016, ... files.
My question is how can I find all the files generated and simply 'glue them together' into one big table? I get that I can manually create inputs for every year (tCacheInput) and then combine them using tMap, but I don't know the schema before hand (because I don't know how many years the job will have to check) and I don't want manual inputs for every year...
You need to use a tFileList. And put in the filter : "PARENT*"
That will take all files with the pattern PARENT in a folder