I have been trying to migrate the database of my djangocms project from the default sqlite3 to postgreSQL. But at the end of the process I got this error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 119485: invalid start byte.
I am not sure how to handle this issue as I have no experience in databases and most posts about this type of issues on the internet are not database-related from what I have seen.
What I did to reach this point:
I have created a local djangocms website with sqlite. To make the database migration, I have followed the procedure described here: I dumped the database into a db.json file then created the new postgres database on pgAdmin. I modified the field DATABASES in my django settings.py, installed psycopg2 and ran python manage.py makemigrations and python manage.py migrate and then (in the Python shell) I ran:
from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
before python manage.py loaddata db.json.
I also had to re-create a superuser which helped me to log into djangocms admin. But as expected my website was absent (and even the layout of djangocms was wrong).
Any help would be much appreciated! Many thanks.
Here are the tracebacks:
(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py loaddata db.json
Traceback (most recent call last):
File " C:\Users\...\vEnv\myWebsite\project\website\manage.py", line 22, in <module>
main()
File " C:\Users\...\vEnv\myWebsite\project\website\manage.py", line 18, in main
execute_from_command_line(sys.argv)
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
utility.execute()
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\base.py", line 371, in execute
output = self.handle(*args, **options)
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
self.load_label(fixture_label)
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\commands\loaddata.py", line 172, in load_label
for obj in objects:
File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\serializers\json.py", line 67, in Deserializer
stream_or_string = stream_or_string.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 119485: invalid start byte
What I did before and the prompt results:
(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py makemigrations
No changes detected
(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py migrate
Operations to perform:
Apply all migrations: admin, aldryn_background_image, auth, bootstrap4_alerts, bootstrap4_badge, bootstrap4_card, bootstrap4_carousel, bootstrap4_collapse, bootstrap4_content, bootstrap4_grid, bootstrap4_jumbotron, bootstrap4_link, bootstrap4_listgroup, bootstrap4_media, bootstrap4_picture, bootstrap4_tabs, bootstrap4_utilities, cms, cmsplugin_soundcloud, contenttypes, djangocms_file, djangocms_googlemap, djangocms_icon, djangocms_link, djangocms_picture, djangocms_style, djangocms_text_ckeditor, djangocms_video, djangocms_vimeo, djangocms_wow, easy_thumbnails, filer, menus, sessions, sites
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying filer.0001_initial... OK
Applying filer.0002_auto_20150606_2003... OK
Applying sites.0001_initial... OK
Applying cms.0001_initial... OK
Applying cms.0002_auto_20140816_1918... OK
Applying cms.0003_auto_20140926_2347... OK
Applying cms.0004_auto_20140924_1038... OK
Applying cms.0005_auto_20140924_1039... OK
Applying cms.0006_auto_20140924_1110... OK
Applying cms.0007_auto_20141028_1559... OK
Applying cms.0008_auto_20150208_2149... OK
Applying cms.0008_auto_20150121_0059... OK
Applying cms.0009_merge... OK
Applying cms.0010_migrate_use_structure... OK
Applying cms.0011_auto_20150419_1006... OK
Applying cms.0012_auto_20150607_2207... OK
Applying cms.0013_urlconfrevision... OK
Applying aldryn_background_image.0001_initial... OK
Applying aldryn_background_image.0002_backgroundimage_background_attachment... OK
Applying aldryn_background_image.0003_remove_backgroundimage_background_attachment... OK
Applying aldryn_background_image.0004_backgroundimage_background_attachment... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying auth.0012_alter_user_first_name_max_length... OK
Applying cms.0014_auto_20160404_1908... OK
Applying cms.0015_auto_20160421_0000... OK
Applying cms.0016_auto_20160608_1535... OK
Applying bootstrap4_alerts.0001_initial... OK
Applying bootstrap4_badge.0001_initial... OK
Applying bootstrap4_card.0001_initial... OK
Applying filer.0003_thumbnailoption... OK
Applying filer.0004_auto_20160328_1434... OK
Applying filer.0005_auto_20160623_1425... OK
Applying filer.0006_auto_20160623_1627... OK
Applying filer.0007_auto_20161016_1055... OK
Applying filer.0008_auto_20171117_1313... OK
Applying filer.0009_auto_20171220_1635... OK
Applying filer.0010_auto_20180414_2058... OK
Applying filer.0011_auto_20190418_0137... OK
Applying bootstrap4_carousel.0001_initial... OK
Applying bootstrap4_carousel.0002_bootstrap4carousel_carousel_aspect_ratio... OK
Applying bootstrap4_carousel.0003_auto_20180610_1102... OK
Applying bootstrap4_carousel.0004_auto_20190703_0831... OK
Applying bootstrap4_collapse.0001_initial... OK
Applying bootstrap4_content.0001_initial... OK
Applying bootstrap4_content.0002_added_figure... OK
Applying bootstrap4_grid.0001_initial... OK
Applying bootstrap4_grid.0002_auto_20180709_0808... OK
Applying bootstrap4_grid.0003_migrate_column_size... OK
Applying bootstrap4_grid.0004_remove_bootstrap4gridcolumn_column_size... OK
Applying bootstrap4_jumbotron.0001_initial... OK
Applying bootstrap4_link.0001_initial... OK
Applying bootstrap4_link.0002_add_icons... OK
Applying bootstrap4_link.0003_icon_updates... OK
Applying bootstrap4_link.0004_auto_20190703_0831... OK
Applying bootstrap4_listgroup.0001_initial... OK
Applying bootstrap4_media.0001_initial... OK
Applying djangocms_picture.0001_initial... OK
Applying djangocms_picture.0002_auto_20151018_1927... OK
Applying djangocms_picture.0003_migrate_to_filer... OK
Applying djangocms_picture.0004_adapt_fields... OK
Applying djangocms_picture.0005_reset_null_values... OK
Applying djangocms_picture.0006_remove_null_values... OK
Applying djangocms_picture.0007_fix_alignment... OK
Applying djangocms_picture.0008_picture_use_responsive_image... OK
Applying bootstrap4_picture.0001_initial... OK
Applying bootstrap4_picture.0002_bootstrap4picture_use_responsive_image... OK
Applying bootstrap4_picture.0003_auto_20181212_1055... OK
Applying bootstrap4_picture.0004_auto_20190703_0831... OK
Applying bootstrap4_tabs.0001_initial... OK
Applying bootstrap4_tabs.0002_auto_20180610_1106... OK
Applying bootstrap4_utilities.0001_initial... OK
Applying cms.0017_pagetype... OK
Applying cms.0018_create_pagenode... OK
Applying cms.0019_set_pagenode... OK
Applying cms.0020_old_tree_cleanup... OK
Applying cms.0021_auto_20180507_1432... OK
Applying cms.0022_auto_20180620_1551... OK
Applying cmsplugin_soundcloud.0001_initial... OK
Applying cmsplugin_soundcloud.0002_auto_20210621_1556... OK
Applying cmsplugin_soundcloud.0003_auto_20210621_1619... OK
Applying cmsplugin_soundcloud.0004_auto_20210621_1639... OK
Applying cmsplugin_soundcloud.0005_auto_20210621_1732... OK
Applying djangocms_file.0001_initial... OK
Applying djangocms_file.0002_auto_20151202_1551... OK
Applying djangocms_file.0003_remove_related_name_for_cmsplugin_ptr... OK
Applying djangocms_file.0004_set_related_name_for_cmsplugin_ptr... OK
Applying djangocms_file.0005_auto_20160119_1534... OK
Applying djangocms_file.0006_migrate_to_filer... OK
Applying djangocms_file.0007_adapted_fields... OK
Applying djangocms_file.0008_add_folder... OK
Applying djangocms_file.0009_fixed_null_fields... OK
Applying djangocms_file.0010_removed_null_fields... OK
Applying djangocms_file.0011_auto_20181211_0357... OK
Applying djangocms_googlemap.0001_initial... OK
Applying djangocms_googlemap.0002_auto_20160622_1031... OK
Applying djangocms_googlemap.0003_auto_20160825_1829... OK
Applying djangocms_googlemap.0004_adapted_fields... OK
Applying djangocms_googlemap.0005_create_nested_plugins... OK
Applying djangocms_googlemap.0006_remove_fields... OK
Applying djangocms_googlemap.0007_reset_null_values... OK
Applying djangocms_googlemap.0008_removed_null_fields... OK
Applying djangocms_googlemap.0009_googlemapmarker_icon... OK
Applying djangocms_googlemap.0010_auto_20190718_1021... OK
Applying djangocms_icon.0001_initial... OK
Applying djangocms_icon.0002_auto_20190218_2017... OK
Applying djangocms_link.0001_initial... OK
Applying djangocms_link.0002_auto_20140929_1705... OK
Applying djangocms_link.0003_auto_20150212_1310... OK
Applying djangocms_link.0004_auto_20150708_1133... OK
Applying djangocms_link.0005_auto_20151003_1710... OK
Applying djangocms_link.0006_remove_related_name_for_cmsplugin_ptr... OK
Applying djangocms_link.0007_set_related_name_for_cmsplugin_ptr... OK
Applying djangocms_link.0008_link_attributes... OK
Applying djangocms_link.0009_auto_20160705_1344... OK
Applying djangocms_link.0010_adapted_fields... OK
Applying djangocms_link.0011_fixed_null_values... OK
Applying djangocms_link.0012_removed_null... OK
Applying djangocms_link.0013_fix_hostname... OK
Applying djangocms_link.0014_link_file_link... OK
Applying djangocms_link.0015_auto_20190621_0407... OK
Applying djangocms_picture.0009_auto_20181212_1003... OK
Applying djangocms_picture.0010_auto_20190627_0432... OK
Applying djangocms_picture.0011_auto_20190314_1536... OK
Applying djangocms_style.0001_initial... OK
Applying djangocms_style.0002_set_related_name_for_cmsplugin_ptr... OK
Applying djangocms_style.0003_adapted_fields... OK
Applying djangocms_style.0004_use_positive_small_integer_field... OK
Applying djangocms_style.0005_reset_null_values... OK
Applying djangocms_style.0006_removed_null_fields... OK
Applying djangocms_style.0007_style_template... OK
Applying djangocms_text_ckeditor.0001_initial... OK
Applying djangocms_text_ckeditor.0002_remove_related_name_for_cmsplugin_ptr... OK
Applying djangocms_text_ckeditor.0003_set_related_name_for_cmsplugin_ptr... OK
Applying djangocms_text_ckeditor.0004_auto_20160706_1339... OK
Applying djangocms_video.0001_initial... OK
Applying djangocms_video.0002_set_related_name_for_cmsplugin_ptr... OK
Applying djangocms_video.0003_field_adaptions... OK
Applying djangocms_video.0004_move_to_attributes... OK
Applying djangocms_video.0005_migrate_to_filer... OK
Applying djangocms_video.0006_field_adaptions... OK
Applying djangocms_video.0007_create_nested_plugin... OK
Applying djangocms_video.0008_reset_null_values... OK
Applying djangocms_video.0009_removed_null_values... OK
Applying djangocms_video.0010_videoplayer_parameters... OK
Applying djangocms_vimeo.0001_initial... OK
Applying djangocms_vimeo.0002_update_related_name... OK
Applying djangocms_wow.0001_initial... OK
Applying djangocms_wow.0002_auto_20210611_1217... OK
Applying djangocms_wow.0003_auto_20210621_1750... OK
Applying easy_thumbnails.0001_initial... OK
Applying easy_thumbnails.0002_thumbnaildimensions... OK
Applying filer.0012_file_mime_type... OK
Applying menus.0001_initial... OK
Applying sessions.0001_initial... OK
Applying sites.0002_alter_domain_unique... OK
(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py shell
Python 3.9.4 (tags/v3.9.4:1f2e308, Apr 6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
(376, {'auth.Permission': 300, 'contenttypes.ContentType': 76})
And when I log in the website:
(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
August 25, 2021 - 17:21:21
Django version 3.1.11, using settings 'regSite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[25/Aug/2021 17:21:33] "GET /en/ HTTP/1.1" 200 13923
[25/Aug/2021 17:21:33] "GET /static/cms/css/3.8.0/cms.base.css HTTP/1.1" 302 0
[25/Aug/2021 17:21:33] "GET /static/cms/js/dist/3.8.0/bundle.toolbar.min.js HTTP/1.1" 302 0
[25/Aug/2021 17:21:33] "GET /en/static/cms/css/3.8.0/cms.base.css/ HTTP/1.1" 404 179
[25/Aug/2021 17:21:33] "GET /en/static/cms/js/dist/3.8.0/bundle.toolbar.min.js/ HTTP/1.1" 404 179
[25/Aug/2021 17:21:33] "GET /static/cms/css/3.8.0/cms.welcome.css HTTP/1.1" 302 0
[25/Aug/2021 17:21:34] "GET /en/static/cms/css/3.8.0/cms.welcome.css/ HTTP/1.1" 404 179
[25/Aug/2021 17:21:34] "GET /static/cms/js/dist/3.8.0/bundle.toolbar.min.js HTTP/1.1" 302 0
[25/Aug/2021 17:21:34] "GET /en/static/cms/js/dist/3.8.0/bundle.toolbar.min.js/ HTTP/1.1" 404 179
Thanks to Ian Shelvington and Tim Roberts, I opened the db in Notepad and used save as with UTF-8 encoding. Then I ran python manage.py loaddata db.json again.