Django silently removes constraints when removing columns, then arbitrarily chooses to include them in migrations.
I've encountered an odd bug(?) in Django in our production code. Dropping a column manually in migrations without removing its constraints leads to Django being unaware those constraints have been removed, and auto-generating incorrect migrations.
Here's a short example.
The migrations for this schema:
class M():
a = models.IntegerField(default=1)
b = models.IntegerField(default=2)
class Meta:
constraints = [
UniqueConstraint(
name="uniq_ab_1", fields=["a", "b"]
)
]
creates the constraint uniq_ab_1 is Postgres as expected (verified using the \d+ command on the table).
However, this manual migration will remove the constraint, due to one of its member columns being deleted; this is just standard Postgres behavior:
migrations.RemoveField(
model_name="m",
name="a",
),
migrations.AddField(
model_name="m",
name="a",
field=models.IntegerField(default=6),
),
This migration runs just fine. I can even modify the field of M again and run a further migration. However, using \d+ reveals that the uniq_ab_1 constraint is gone from the database.
The only way I found out about this behavior was by renaming the constraint to uniq_ab_2, then auto-generating migrations and getting the error:
django.db.utils.ProgrammingError: constraint "uniq_ab_1" of relation … does not exist.
In other words, on a rename, Django became aware a rename was happening and tried to remove the constraint from the database, in spite of it being gone for a few migrations.
This behavior is rather unexpected. I'd assume Django would either:
M) and try to add it back again.As it stands, it seems like this ghost constraint is observed by some migration operations and not others.
Is this a known bug in Django? Is there a way to guard against this behavior? Is this perfectly normal and I am doing something wrong?
The question asserts several misconceptions about Django.
Django generates migrations in python manage.py makemigrations by comparing the model definition and the model schema built from the existing migrations (i.e. code-first behavior), not by comparing the model definition and the database schema (as it's generally not reproducible).
Django silently removes constraints when removing columns
No, Django doesn't do that. The constraint is automatically dropped by PostgreSQL.
https://postgrespro.com/docs/postgresql/9.6/sql-altertable
You can check what Django does with python manage.py sqlmigrate.
[Django] arbitrarily chooses to include [constraints] in migrations
No, Django always considers the constraints when generating migrations.
In other words, on a rename, Django become aware a rename was happening and tried to remove the constraint from the database, in spite of it being gone for a few migrations.
Django isn't aware a rename is happening. Django simply removes the previous one (in the model schema built from existing migrations) and adds the new one (in the model definition).
This behavior is rather unexpected. I'd assume Django would either:
- Notice the code model differs from the database schema in the original migration (when the constraint gets inadvertently removed) and fail.
- Notice that the constraint is missing in the next migration (e.g. when adding an arbitrary field to M) and try to add it back again.
Django does check that the model definition has no problem at the start of python manage.py migrate, but does not compare that against the database schema — especially not on each step of the operations in a migration.
When you create a manual migration, the above is your responsibility.
Is this a known bug in Django?
As explained above, this is not a bug and is caused by the wrongly planned manual migration.
While it is possible for Django to help guard against this wrong manual operation (shown below), there are simply too many ways that a user can mess up a manual migration for Django to reasonably cover.
Is there a way to guard against this behavior?
You can patch BaseDatabaseSchemaEditor.remove_field to block removing fields that have unique constraints in the model definition:
def _patch_remove_field():
from itertools import chain
from django.core import checks
from django.db.backends.base.schema import BaseDatabaseSchemaEditor
from django.db.models.constraints import UniqueConstraint
old_remove_field = BaseDatabaseSchemaEditor.remove_field
def remove_field(self, model, field):
field_names = set(chain.from_iterable(
(*constraint.fields, *constraint.include)
for constraint in model._meta.constraints if isinstance(constraint, UniqueConstraint)
))
if field.name in field_names:
raise ValueError(checks.Error(
"Cannot remove field '%s' as '%s' refers to it" % (field.name, 'constraints'),
obj=model,
id='models.E012',
))
old_remove_field(self, model, field)
BaseDatabaseSchemaEditor.remove_field = remove_field
Is this perfectly normal and I am doing something wrong?
Yes. Here are the correct migrations.
python manage.py makemigrations preserves the unique constraint, but doesn't have the same behaviour (updating existing rows to be a = 6).
migrations.AlterField(
model_name='m',
name='a',
field=models.IntegerField(default=6),
),
You can get Django to partially help by replacing field a with c, generating a migration and then reverting back to a in both the model definition and the generated migration.
migrations.RemoveConstraint(
model_name='m',
name='uniq_ab_1',
),
migrations.RemoveField(
model_name='m',
name='a',
),
migrations.AddField(
model_name='m',
name='a',
field=models.IntegerField(default=6),
),
migrations.AddConstraint(
model_name='m',
constraint=models.UniqueConstraint(fields=('a', 'b'), name='uniq_ab_1'),
),