I try to iterate through a text file using iterators:
1) std::istream_iterator<MyData> itRecordStreamBegin(fileStream);
2) std::istream_iterator<MyData> itRecordStreamEnd;
After the first line it reads some data from the file (it uses operator >> for MyData type). As expected.
Unfortunately after that memory consumption rises and it seems like istream_iterator tried to load whole file in memory. Several seconds later (when there are out_of_memory exceptions thrown) debugger gets into the second line. But file reading failed and I cannot read anything more.
Questions:
Is this correct behaviour for istream_iterator (to load file into memory)?
I do not observe this for smaller files (like 20MB).
Maybe for such large file I need to use common getline way?
The istream_iterator just uses the >> operator on the type. It keeps at most one instance of the type in memory. When using it, you do make copies of the instance. I would suspect (but without seeing the code) that either your copy constructor or destructor is defective, or you leak memory somewhere in your >> operator. I would be very surprised if the problem is related to istream_iterator.