Search code examples
jsonmongodbmongoimport

mongoimport failing inconsistently with "The _id field cannot be changed from"

I have problems importing data into mongodb with. I have a test.json file like this:

{"_id":{"s":{"$numberLong":"38851448"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38853194"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38760498"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"39099662"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38855558"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38760487"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38760488"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"39099663"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38851450"},"a":5},"someKey":"someValue"}
{"_id":{"s":{"$numberLong":"38853546"},"a":5},"someKey":"someValue"}

I try to import it with following command:

mongoimport --type json --db test --collection coll --file test.json --upsert

The import fails almost always with the same error message:

2015-08-27T17:02:15.510+0200    error inserting documents: The _id field cannot be changed from {_id: { s: 38851448, a: 5 }} to {_id: { a: 5, s: 38851448 }}.
2015-08-27T17:02:15.511+0200    error inserting documents: The _id field cannot be changed from {_id: { a: 5, s: 38760487 }} to {_id: { s: 38760487, a: 5 }}.

It's frustrating that this error isn't even reproducable. The importer seems to change the order of the properties for _id, but I don't know why, is this a BUG that should be reported or is already known? Or is there a problem I don't see.

I even tried to change the order of s and a in _id, but the problem stays the same.

If I trie to run the exact same import command with exactly the same datafile multiple times the documents with errors change and one time it imported all rows as expected, but just one time:

enter image description here

For completeness: I'm using mongo 3.0.5 on Mac OS X 10.10.5 installed by homebrew.

UPDATE: I have created a Ticket with mongoDB-Team: TOOLS-894

UPDATE2:

I tried not to use _id but id for my unique key:

{"id":{"s":{"$numberLong":"38851448"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38853194"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38760498"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"39099662"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38855558"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38760487"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38760488"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"39099663"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38851450"},"a":5},"someKey":"someValue"}
{"id":{"s":{"$numberLong":"38853546"},"a":5},"someKey":"someValue"}

and import this with:

mongoimport --type json --db test --collection coll --file test.json --upsertFields id

Now I don't get any errors, but after two imports I have 15 rows in the collection instead of 10. Again because of the order of the properties in id.

Solution

  • All things I tried lead me to believe that this is a bug within mongoimport and I reported it to the mongoldb team: TOOLS-894

    The workaround that's working for me is to move the the unique key for documents out of _id and to split it up to their own properties like this:

    {"s":{"$numberLong":"38851448"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38853194"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38760498"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"39099662"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38855558"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38760487"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38760488"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"39099663"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38851450"},"a":5,"someKey":"someValue"}
{"s":{"$numberLong":"38853546"},"a":5,"someKey":"someValue"}

    This way I'm able to import the data like this:

    mongoimport --type json --db test --collection coll --upsertFields a,s --file test.json

    with this new documents get inserted and old documents get updated

    UPDATE:

    In TOOLS-894 and TOOLS-899 the mongodb developers describe another workaround: You can use mongoimport from the 2.6 Version, which didn't have this bug. Or you can wait for mongodb 3.0.7 which should contain the fix.

    UPDATE 2:

    New versions of MongoDB fixed this problem and work as expected.