I have joined a team working on a Django project, and I am trying to setup the local mysql database schema. This project has existed for a long time, so there are existing migrations.
I have created the empty database in mysql, but when I try to apply migrations to create the schema, it complains that the tables don't exist yet.
$ ./manage.py migrate
Traceback (most recent call last):
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
rowcount = self._do_query(q)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
db.query(q)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
_mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1146, "Table 'api_test.headset_product' doesn't exist")
I've also tried syncdb, migrate --run-syncdb, and makemigrations with the same results. I've also tried loaddata with our fixtures, with the same results.
I've read Django : Table doesn't exist, but I didn't drop tables in this case (no tables have ever existed), and I don't want to recreate all the migrations (I'm not the only person on this project).
I have a related project that uses a local postgresql database, and it doesn't seem to have this problem. I can run migrate and it does just what I expect, so it feels that I've run into a problem specific to this codebase, but we aren't certain how to debug it.
"full output of makemigrations, including the full typed command."
$ ./manage.py makemigrations
[WARNING] statsd not configured
[WARNING] SMTP not configured; emails will not be sent
[WARNING] V1 database access not configured
Traceback (most recent call last):
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
rowcount = self._do_query(q)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
db.query(q)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
_mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1146, "Table 'reign_api_test.headset_product' doesn't exist")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./manage.py", line 14, in <module>
execute_from_command_line(sys.argv)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 332, in execute
self.check()
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 364, in check
include_deployment_checks=include_deployment_checks,
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 351, in _run_checks
return checks.run_checks(**kwargs)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/checks/registry.py", line 73, in run_checks
new_errors = check(app_configs=app_configs)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/urls/resolvers.py", line 397, in check
for pattern in self.url_patterns:
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/utils/functional.py", line 36, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/urls/resolvers.py", line 536, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/utils/functional.py", line 36, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/urls/resolvers.py", line 529, in urlconf_module
return import_module(self.urlconf_name)
File "/Users/rnapier/.pyenv/versions/3.6.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/reign_api/urls.py", line 13, in <module>
import newsfeed.urls
File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/urls.py", line 2, in <module>
from newsfeed import views
File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/views.py", line 15, in <module>
from newsfeed import models, serializers
File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/serializers.py", line 27, in <module>
class NewsfeedSubscriptionSerializer(serializers.Serializer):
File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/serializers.py", line 28, in NewsfeedSubscriptionSerializer
PRODUCTS = [f for f in headset.models.Product.objects.values_list("name", flat=True)]
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/query.py", line 272, in __iter__
self._fetch_all()
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/query.py", line 1179, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/query.py", line 178, in __iter__
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1017, in results_iter
results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1066, in execute_sql
cursor.execute(sql, params)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
rowcount = self._do_query(q)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
db.query(q)
File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
_mysql.connection.query(self, query)
django.db.utils.ProgrammingError: (1146, "Table 'reign_api_test.headset_product' doesn't exist")
"model structure for headset_product"
class Product(models.Model):
"""
products supported by the Headset API
"""
name = models.CharField(max_length=128, unique=True)
def __str__(self) -> str:
return "<Product {s.id} {s.name}>".format(s=self)
__repr__ = __str__
The problem is in reign_api/newsfeed/serializers.py, which is trying to make a query at class or module level. This should always be avoided in Django, as the query will be run at first import - before the migrations on a new system will have a chance to run. But this will be causing other problems too, in that that query will only be executed once per process, and will therefore not update when new data is added or changed.
Your colleagues should fix this problem, either by refactoring to move the query into a method, or probably better by using a more appropriate serializer field that takes a queryset and does the query when called.