how to read a file in batches/parts/1000 lines at a time

I am currently trying to make a method that can handle reading in large XML files. All i need is a method that would loading in say 1000 lines at a time or in small batches.

I have been looking at streamreaders, xmlreaders and filestreams, I have seen some mentions of just keeping the stream open while processing data to get what i need but i cant seem to get my head round it.

I have spent a long time checking the similar questions but can seem to find anything that will help me.

ps. first thought i was thinking of doing a for loop around the readline to a counter of 1000 but cant seem to figure out how to continue from that 1000 lines to reading another 1000 etc until the end of the file.

Solution

My feeling is that his will require a custom XML reader implementation.

For example - if your structure looks something like:

root
    item
        stuff
    /item
    item
        stuff
    /item
    item
        stuff
    /item
    item
        stuff
    /item
/root

You'll have to write code that reads a number 'item' blocks (as many as yuo wish to process in a batch), and then converts them into a valid XML doc for further processing.

If, however, your XML doc is one massive sprawling entity - I don't think there's any elegant way you can process it piece-meal.