I am using a batch job to read and update images via API. The images are read from a folder and it can be hundreds images available in the folder. As it is quite heavy on the server I only want to read and process 100 images, then stop the process and read another 100 a few hours later etc.
I have set the batch block size to 100 but the batch job will then continue to run for hours until it has processed all images in the folder. How can I make it just read and process 100 images and then stop again?
Thanks in advance.
You can not and you should not even try. Batch is designed to process all records it receives.
One solution is to send it less records. For example read less records from the folder, or discard all but the first 100.
Another solution could be let batch process all records, but reduce concurrency to 1 so records are processed sequentially block by block.
The sftp:list operation doesn't seem to have a way to configure that but its output is an array of message. You can use a DataWeave script to split the array. For example the take() operation seems to do just whats needed. It will return the first n elements of an array.
Note that you'll have to ensure that those elements are consumed or they will processed all the time.