How to handle ReferenceField migration with flask mongoengine?

Question

I've implemented a simple flask application with mongodb that now needs some upgrades.

Let's say to have a class model for Foo and a class model for Bar in which there is a reference field to Foo

class Foo(Document):
  title = StringField()

class Bar(Document):
  name = StringField()
  foo = ReferenceField('Foo')

Let the flask application runs doing its job for a while, so that now there are some data in the DB.

Due to requirements changes, we need to refactor the Foo class subclassing it from a new super class:

class SuperFoo(Document):
  meta = { 'allow_inheritance': True,}
  #[...]

class Foo(SuperFoo):
  #[...]

class Bar(Document):
  name = StringField()
  foo = ReferenceField('Foo')

The code above works well with an empty database.

But in case of some data in it, mongoengine raises an Exception when a flask admin tries to show a Bar instance (in edit mode)

File "[...]/site-packages/mongoengine/fields.py", line 1124, in __get__
    raise DoesNotExist('Trying to dereference unknown document %s' % value)
mongoengine.errors.DoesNotExist: Trying to dereference unknown document DBRef('super_foo', ObjectId('5617a08939c6c70cbaa2af6e'))

I suppose data model needs to be migrated in some way.

How?

thanks, alessandro.

Answer 1

After a little analyis I came up to solve the problem.

Mongoengine creates a new collection super_foo.

Documents of every inherited class goes into this super_foo collection with an additional attribute _cls. The value is the CamelCased hierarchy path of that class. In this example documents will have

'_cls': 'SuperFoo.Foo' field.

What I've done is to copy every document from the old foo collection into the new super_foo one, adding the field {'_cls': u'SuperPlesso.Plesso'} to each.

The migration function should look like:

def migrationFunc():
    from pymongo.errors import DuplicateKeyError
    from my.app import models

    _cls = {'_cls': u'SuperFoo.Foo'}

    fromOldCollection = models.Foo._collection
    toSuperCollection = models.Superfoo._collection

    for doc in fromOldCollection.find():
        doc.update(_cls)
        try:
            toSuperCollection.insert(doc)
        except DuplicateKeyError:
            logger.error('...')

Then I updated the base code of the models with the actual new hierarchy:

class SuperFoo(Document):
  meta = { 'allow_inheritance': True,}
  #[...]

# was class Foo(Document)
class Foo(SuperFoo):
  #[...]

Al back references to Foo in Bar collections, or elsewhere, are preserved.