loaddata raises IntegrityError when called within migration, but works if called from the CLI

Question

For my Django 1.8.12 site I have an example_content fixture which fills in some initial site content.

If I load this this right after running migrate it imports fine:

$ ./manage.py migrate --noinput
Operations to perform:
  Synchronize unmigrated apps: ckeditor, staticfiles, zinnia_ckeditor, messages, djangocms_admin_style, webapp_creator, template_debug, sekizai, django_pygments, treebeard
  Apply all migrations: djangocms_inherit, djangocms_snippet, api_docs, cmsplugin_zinnia, sites, menus, contenttypes, store_data, zinnia, django_comments, sessions, rev
ersion, auth, djangocms_picture, tagging, developer_portal, admin, djangocms_link, djangocms_video, django_openid_auth, md_importer, cms, djangocms_text_ckeditor
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  ... (skipped for brevity).
  Applying django_comments.0003_add_submit_date_index... OK
$ ./manage.py loaddata example_content
Installed 139582 object(s) from 1 fixture(s)
$

However, I'd like this data to import automatically on migrate, rather than requiring an extra manual loaddata step, so I'm trying to create a migration, my_app.0002_example_content, to call the loaddata command.

I'm making sure all the other data is loading first by adding dependencies on every relevant module.

To confirmed that this migration is indeed being run last (as it was when it succeeded above) I first commented out the RunPython step leaving the migration empty:

# my_app/migrations/0002_example_content.py

from django.core.management import call_command
from django.db import models, migrations

class Migration(migrations.Migration):
    from django.core.management import call_command
    from django.db import models, migrations

    dependencies = [
        ('djangocms_inherit', '0002_auto_20150622_1244'),
        ... all other apps (skipped for brevity)
        ('my_app', '0001_initial'),
    ]
    operations = [
#        migrations.RunPython(
#            lambda apps, schema_editor: call_command("loaddata", "example_content")
#        )
    ]

And sure enough it runs as the last migration:

$ ./manage.py migrate --noinput
Operations to perform:
  Synchronize unmigrated apps: ckeditor, staticfiles, zinnia_ckeditor, messages, djangocms_admin_style, webapp_creator, template_debug, sekizai, django_pygments, treebeard
  Apply all migrations: djangocms_inherit, djangocms_snippet, api_docs, cmsplugin_zinnia, sites, menus, contenttypes, store_data, zinnia, django_comments, sessions, reversion, auth, djangocms_picture, tagging, developer_portal, admin, djangocms_link, djangocms_video, django_openid_auth, md_importer, cms, djangocms_text_ckeditor
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  ... (skipped for brevity)
  Applying django_comments.0003_add_submit_date_index... OK
  Applying my_app.0001_initial... OK
  Applying my_app.0002_example_content... OK
$

However when I try to run it to actually run the loaddata command (commenting the code back in), I get an error:

$ ./manage.py migrate --noinput
Operations to perform:
  Synchronize unmigrated apps: ckeditor, staticfiles, zinnia_ckeditor, messages, djangocms_admin_style, webapp_creator, template_debug, sekizai, django_pygments, treebeard
  Apply all migrations: djangocms_inherit, djangocms_snippet, api_docs, cmsplugin_zinnia, sites, menus, contenttypes, store_data, zinnia, django_comments, sessions, reversion, auth, djangocms_picture, tagging, developer_portal, admin, djangocms_link, djangocms_video, django_openid_auth, md_importer, cms, djangocms_text_ckeditor
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  ... (skipped for brevity)
  Applying django_comments.0003_add_submit_date_index... OK
  Applying my_app.0001_initial... OK
  Applying my_app.0002_example_content...Traceback (most recent call last):
...
django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'django_comments' with primary key '1' has an invalid foreign key: django_comments.content
_type_id contains a value '37' that does not have a corresponding value in django_content_type.id.

Given that my new migration is definitely being run after everything else, the order should be exactly the same as when I called loaddata manually. So I don't understand why it would succeed when being called through the CLI but fail when being called through a migration.

Is there something that happens at the end of the migrate step which I'm missing which might explain the discrepancy here?

Answer 1

Prompted by knbk's answer, I managed to solve this.

Initially I tried manually calling update_contenttypes, but that on its own didn't work. It turns out I need to actually call all the post_migrate signals, as I guess they do more tidying up that's needed.

Here's the my_app.0002_example_content I ended up with:

# my_app/migrations/0002_example_content.py

from django.core.management import call_command
from django.db import migrations
from django.apps import apps
from django.db.models.signals import post_migrate


def finish_previous_migrations(migrate_apps):
    """
    Explicitly run the post_migrate actions
    for all apps
    """

    for app_config in apps.get_app_configs():
        post_migrate.send(
            sender=app_config,
            app_config=app_config
        )


def load_site_content(migrate_apps, schema_editor):
    """
    Load the website content fixture
    """

    finish_previous_migrations(migrate_apps)

    call_command("loaddata", "example_content")


class Migration(migrations.Migration):
    dependencies = [
        ('djangocms_inherit', '0002_auto_20150622_1244'),
        ... all other apps (skipped for brevity)
        ('my_app', '0001_initial'),
    ]
    operations = [
        migrations.RunPython(load_site_content)
    ]