Django 2.0 - test using fixtures from dumpdata gives utf8mb4 encoding errors

Question

I am using utf8mb4 encoding for my mySQL database. All my Django apps work correctly with it even with Cyrillic characters.

But when I run ./manage.py test <module> and try to load a fixture dumped from the same database, I get errors regarding the encoding. One of the errors is:

Could not load advert.Region(pk=1): (1366, "Incorrect string value: '\\xD0\\x9F\\xD0\\xBB\\xD0\\xBE...' for column 'region_name' at row 1")

Then I set DEFAULT_CHARSET in the Django settings and the error changed to LookupError: Problem installing fixture '<path>/test_db_one.xml': unknown encoding: utf8mb4

As you can see I have specified the format of the fixture to be XML as it has better handling of the encoding.

My setting module has the following properties:

DEFAULT_CHARSET = 'utf8mb4'
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'autoplusplus_dev',
        'USER': 'autoplusplus_django',
        'PASSWORD': '1029',
        'HOST': 'localhost',
        'PORT': '3306',
        'STORAGE_ENGINE': 'INNODB',
        'OPTIONS': {'charset': 'utf8mb4'},
        'TEST_CHARSET': 'utf8mb4',
    }
}

I have mysqlclient 1.3.12 installed

Answer 1

Turned out that changing the encoding of the file which Django generates itself to UTF-8 fixes the issue.