Search code examples
djangoinheritancedjango-migrations

Renaming a Django superclass model and updating the subclass pointers correctly

I'm having trouble refactoring a superclass in Django v2.2.12 involving three models, with one superclass model and two subclass models:

class BaseProduct(models.Model):
    name = models.CharField()
    description = models.CharField()


class GeneralProduct(BaseProduct):
    pass


class SoftwareProduct(BaseProduct):
    pass

The BaseProduct model needs to be renamed to just Product, so I changed this code to:

class Product(models.Model):
    name = models.CharField()
    description = models.CharField()

class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

And then ran python manage.py makemigrations, in which Django seems to correctly see what changed:

Did you rename the yourapp.BaseProduct model to Product? [y/N] y
Did you rename generalproduct.baseproduct_ptr to generalproduct.product_ptr (a OneToOneField)? [y/N] y
Did you rename softwareproduct.baseproduct_ptr to softwareproduct.product_ptr (a OneToOneField)? [y/N] y

Migrations for 'yourapp':
  .../yourapp/migrations/002_auto_20200507_1830.py
    - Rename model BaseProduct to Product
    - Rename field baseproduct_ptr on generalproduct to product_ptr
    - Rename field baseproduct_ptr on softwareproduct to product_ptr

So far so good. Django sees that the superclass got renamed, and it knows that its own autogenerated ..._ptr values that it uses to track model inheritance need to be updated in the db, too.

The migration it comes up with also looks about as terse as it should be:

# Generated by Django 2.2.12 on 2020-05-07 18:30

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('yourapp', '0001_initial'),
    ]

    operations = [
        migrations.RenameModel(
            old_name='BaseProduct',
            new_name='Product',
        ),
        migrations.RenameField(
            model_name='generalproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),
        migrations.RenameField(
            model_name='softwareproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),
    ]

So this all looks perfect, but we crash out when we try to apply that migration using python manage.py migrate:

Running migrations:
  Applying yourapp.0002_auto_20200507_1830...Traceback (most recent call last):
  [...]
  File ".../python3.7/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File ".../python3.7/site-packages/django/db/migrations/migration.py", line 114, in apply
    operation.state_forwards(self.app_label, project_state)
  File ".../python3.7/site-packages/django/db/migrations/operations/models.py", line 340, in state_forwards
    state.reload_models(to_reload, delay=True)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 165, in reload_models
    self._reload(related_models)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 191, in _reload
    self.apps.render_multiple(states_to_be_rendered)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 308, in render_multiple
    model.render(self)
  File ".../python3.7/site-packages/django/db/migrations/state.py", line 579, in render
    return type(self.name, bases, body)
  File ".../python3.7/site-packages/django/db/models/base.py", line 253, in __new__
    base.__name__,
django.core.exceptions.FieldError: Auto-generated field 'baseproduct_ptr' in class 'SoftwareProduct' for
parent_link to base class 'BaseProduct' clashes with declared field of the same name.

I searched the web for that error, as well as for renaming a Django model that's a superclass for other models, but there does not appear to be any (discoverable) documentation, or blog posts, or SO answers, that talk about this problem.

Solution

  • Canonical answer

    The reason this goes wrong is that even though Django sees that a model got renamed and subclasses need pointer updates, it can't properly execute on those updates. There is a PR open to add this to Django (https://github.com/django/django/pull/17986, originally 13021, itself originally 11222), but until that lands, the solution is to temporarily "trick" Django into thinking the subclasses are in fact plain models without any inheritance, and effecting the changes manually by running through the following steps:

    1. renaming the autogenerated inheritance pointer from superclass_ptr to newsuperclass_ptr manually (in this case, baseproduct_ptr becomes product_prt), then
    2. trick Django into thinking the subclasses are just generic Model implementations by literally rewriting the .bases property for them and telling Django to reload them, then
    3. renaming the superclass to its new name (in this case, BaseProduct becomes Product) and then finally
    4. updating the newsuperclass_ptr fields so that they point to the new superclass name instead, making sure to specify auto_created=True as well as parent_link=True.

    In the last step, the first property should be there mostly because Django auto-generates pointers, and we don't want Django to be able to tell we ever tricked it and did our own thing, and the second property is there because parent_link is the field that Django relies on to correctly hook up model inheritance when it runs.

    So, a few more steps than just manage makemigrations, but each is straight-forward, and we can do all of this by writing a single, custom migration file.

    Using the names from the question post:

    # Custom Django 2.2.12 migration for handling superclass model renaming.

from django.db import migrations, models
import django.db.models.deletion

# with a file called custom_operations.py in our migrations dir:
from .custom_operations import AlterModelBases


class Migration(migrations.Migration):
    dependencies = [
        ('yourapp', '0001_initial'),
        # Note that if the last real migration starts with 0001,
        # this migration file has to start with 0002, etc. 
        #
        # Django simply looks at the initial sequence number in
        # order to build its migration tree, so as long as we
        # name the file correctly, things just work.
    ]

    operations = [
        # Step 1: First, we rename the parent links in our
        # subclasses to match their future name:

        migrations.RenameField(
            model_name='generalproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),
        
        migrations.RenameField(
            model_name='softwareproduct',
            old_name='baseproduct_ptr',
            new_name='product_ptr',
        ),

        # Step 2: then, temporarily set the base model for
        #         our subclassses to just `Model`, which makes
        #         Django think there are no parent links, which
        #         means it won't try to apply crashing logic in step 3.

        AlterModelBases("GeneralProduct", (models.Model,)),
        AlterModelBases("SoftwareProduct", (models.Model,)),

        # Step 3: Now we can safely rename the superclass without
        #         Django trying to fix subclass pointers:

        migrations.RenameModel(
            old_name="BaseProduct",
            new_name="Product"
        ),

        # Step 4: Which means we can now update the `parent_link`
        #         fields for the subclasses: even though we altered
        #         the model bases earlier, this step will restore
        #         the class hierarchy we actually need:

        migrations.AlterField(
            model_name='generalproduct',
            name='product_ptr',
            field=models.OneToOneField(
                auto_created=True,
                on_delete=django.db.models.deletion.CASCADE,
                parent_link=True, primary_key=True,
                serialize=False,
                to='buyersguide.Product'
            ),
        ),

        migrations.AlterField(
            model_name='softwareproduct',
            name='product_ptr',
            field=models.OneToOneField(
                auto_created=True,
                on_delete=django.db.models.deletion.CASCADE,
                parent_link=True,
                primary_key=True,
                serialize=False,
                to='buyersguide.Product'
            ),
        ),
    ]

    The crucial step is the inheritance "destruction": we tell Django that the subclasses inherit from models.Model, so that renaming the superclass will leave the subclasses entirely unaffected (rather than Django trying to update inheritance pointers itself), but we do not actually change anything in the database. We only make that change to the currently running code, so if we exit Django, it'll be as if that change was never made to begin with.

    So to achieve this, we're using a custom ModelOperation that can change the inheritance of a(ny) class to a(ny collection of) different superclass(es) at runtime:

    # contents of yourapp/migrations/custom_operations.py

from django.db.migrations.operations.models import ModelOperation


class AlterModelBases(ModelOperation):
    reduce_to_sql = False
    reversible = True

    def __init__(self, name, bases):
        self.bases = bases
        super().__init__(name)

    def state_forwards(self, app_label, state):
        """
        Overwrite a models base classes with a custom list of
        bases instead, then force Django to reload the model
        with this (probably completely) different class hierarchy.
        """
        state.models[app_label, self.name_lower].bases = self.bases
        state.reload_model(app_label, self.name_lower)

    def database_forwards(self, app_label, schema_editor, from_state, to_state):
        pass

    def database_backwards(self, app_label, schema_editor, from_state, to_state):
        pass

    def describe(self):
        return "Update %s bases to %s" % (self.name, self.bases)

    With this custom migration file and our custom_operations.py in place, all we need to do is update our code to reflect the new naming scheme:

    class Product(models.Model):
    name = models.CharField()
    description = models.CharField()

class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

    And then apply manage migrate, which will run and update everything as needed.

    NOTE: depending on whether or not you "prefactored" your code in preparation for your renaming, using something like this:

    class BaseProduct(models.Model):
    name = models.CharField()
    description = models.CharField()


# "handy" aliasing so that all code can start using `Product`
# even though we haven't renamed actually renamed this class yet:
Product = BaseProduct


class GeneralProduct(Product):
    pass


class SoftwareProduct(Product):
    pass

    you may have to update ForeignKey and ManyToMany relations to Productin other classes, add explicit add models.AlterField instructions to update BaseProduct to Product:

            ...
        migrations.AlterField(
            model_name='productrating',
            name='product',
            field=models.ForeignKey(
                 on_delete=django.db.models.deletion.CASCADE,
                 to='yourapp.Product'
            ),
        ),
        ...

     

     

    Original answer

    Oh, yea, this is a tricky one. But I've solved in my project here is the way I did it.

    1. Delete newly created migration and rollback your model changes

    2. Change your implicit parent link fields to explicit with parent_link option. We need this to manually rename our field to proper name in later steps

    class BaseProduct(models.Model):
    ...
    
class GeneralProduct(BaseProduct):
    baseproduct_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)
    
class SoftwareProduct(BaseProduct):
    baseproduct_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)
    1. Generate migration via makemigrations and get something like this
    ...
migrations.AlterField(
    model_name='generalproduct',
    name='baseproduct_ptr',
    field=models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='BaseProduct'),
),
migrations.AlterField(
    model_name='softwareproduct',
    name='baseproduct_ptr',
    
field=models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='BaseProduct'),
)
...
    1. Now you have explicit links to your parent model you could rename them to product_ptr which will match your desired link name
    class GeneralProduct(BaseProduct):
    product_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)
    
class SoftwareProduct(BaseProduct):
    product_ptr = models.OneToOneField(BaseProduct, django.db.models.deletion.CASCADE, parent_link=True, primary_key=True)
    1. Generate migration via makemigrations and get something like this
    ...
migrations.RenameField(
    model_name='generalproduct',
    old_name='baseproduct_ptr',
    new_name='product_ptr',
),
migrations.RenameField(
    model_name='softwareproduct',
    old_name='baseproduct_ptr',
    new_name='product_ptr',
),
...
    1. Now the trickiest part we need to add new migration operation(source could be found here https://github.com/django/django/pull/11222) and put in our code, I personally have contrib package in my project where I put all stuff like this file in contrib/django/migrations.py:
    # https://github.com/django/django/pull/11222/files
# https://code.djangoproject.com/ticket/26488
# https://code.djangoproject.com/ticket/23521
# https://code.djangoproject.com/ticket/26488#comment:18
# https://github.com/django/django/pull/11222#pullrequestreview-233821387
from django.db.migrations.operations.models import ModelOperation
    
    
class DisconnectModelBases(ModelOperation):
    reduce_to_sql = False
    reversible = True
    
    def __init__(self, name, bases):
        self.bases = bases
        super().__init__(name)
    
    def state_forwards(self, app_label, state):
        state.models[app_label, self.name_lower].bases = self.bases
        state.reload_model(app_label, self.name_lower)
    
    def database_forwards(self, app_label, schema_editor, from_state, to_state):
        pass
    
    def database_backwards(self, app_label, schema_editor, from_state, to_state):
        pass
    
    def describe(self):
        return "Update %s bases to %s" % (self.name, self.bases)
    1. Now we are ready to rename our parent model
    class Product(models.Model):
    ....
    
class GeneralProduct(Product):
    pass
    
class SoftwareProduct(Product):
    pass
    1. Generate migration via makemigrations. Make sure you add DisconnectModelBases step, it will not be added automatically, even if successfully generate migration. If this doesn't help and you could try creating --empty one manually.
    from django.db import migrations, models
import django.db.models.deletion

from contrib.django.migrations import DisconnectModelBases


class Migration(migrations.Migration):

    dependencies = [
        ("contenttypes", "0002_remove_content_type_name"),
        ("products", "0071_auto_20200122_0614"),
    ]

    operations = [
        DisconnectModelBases("GeneralProduct", (models.Model,)),
        DisconnectModelBases("SoftwareProduct", (models.Model,)),
        migrations.RenameModel(
            old_name="BaseProduct", new_name="Product"
        ),
        migrations.AlterField(
            model_name='generalproduct',
            name='product_ptr',
            field=models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='products.Product'),
        ),
        migrations.AlterField(
            model_name='softwareproduct',
            name='product_ptr',
            field=models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='proudcts.Product'),
        ),
    ]

    NOTE: after all this, you don't need explicit parent_link fields. So you can remove them. Which I actually did on step 7.