Search code examples
mongodbtalend

How to iterate over files for tMongoDBBulkLoad

What is the best way to iterate over files and feed them into tMongoDBBulkLoad? It sees that you cannot feed into this component from a tFileList componet (Iterate) - which would make the most sense.

I want to import 80 files, rather than create one massive file which is too large to open in notepad if I have issues during the import.

Thanks

---Update----

I know how to do this with other components, my issue is I cannot feed an Iterate component into the tMongoBulkLoad

enter image description here

Solution

  • The simplified job will be like this :

    tFileList ---------iterate--------tMongoDBBulkLoad

    and in the tMongoDBBulkLoad settings you set the Data file to :

    ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

    Here, the tFileList will iterate over files, in each iteration, the tMongoDBBulkLoad will be triggered to load the current file, which is indicated by the global variable.

    --- Reply for the Update ---

    To connect an iterate trigger to the component, you can add a dummy tjava with no code, it will be like this :

    tFileList -----(iterate)-----tJava-------(onComponentOk)-------tMongoDBBulkLoad