Search code examples
mongodbgomgo

mgo query returns "EOF" for large datasets

I want to perform a query that returns some data from my MongoDB server, but when the amount of data becomes big I get an "EOF" error from the c.Find().All() query.

Basically I have:

activeData := []DataEntry{}
activeDataQuery := bson.M{"data.active": true}
err := sigdb.Find(activeDataQuery).All(&activeData)

Which works fine for a small test with about 50,000 items, but when I try my full dataset, which is more than one million items, it returns "EOF", eventhough there is data there to be queried.

What could be causing this? I am running both the Go program nad the MongoDB server on my laptop running Ubuntu 14.04 using Go 1.3.

Edit: Upon further trials, I am also getting: "write tcp 127.0.0.1:27017: broken pipe" from the same query.

Solution

  • The All method will load all matching data into memory, which is a very bad way to handle large data sets. With luck you'll get such timeouts before the method is finished, and in the worst case the machine will crash out of memory.

    On any sort of non-trivial data set, use normal iteration instead, with the Iter and Next methods.