Search code examples
djangodjango-modelsweb-scrapingscrapyscrapy-pipeline

Using Django's app model in scrapy pipeline gives 'Apps aren't loaded yet' exception

I want to use Django's(version-2.2) Model in scrapy's pipeline.py. I followed these links,

  1. https://medium.com/@ali_oguzhan/how-to-use-scrapy-with-django-application-c16fabd0e62e
  2. Use Django's models in a Scrapy project (in the pipeline)

and added line 'django.setup()' in settings.py of scrpay project. It gives me following error,

File "D:\Technologies\Python_Code\django_develops\django_development\kumo_soft\findchipsData\..\inventory\models.py", line 7, in <module>
  class Product(models.Model):
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\models\base.py", line 101, in _new_
  new_class.add_to_class('_meta', Options(meta, app_label))
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\models\base.py", line 304, in add_to_class
  value.contribute_to_class(cls, name)
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\models\options.py", line 203, in contribute_to_class
  self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\__init_.py", line 33, in _getattr_
  return getattr(connections[DEFAULT_DB_ALIAS], item)
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\utils.py", line 202, in _getitem_
  backend = load_backend(db['ENGINE'])
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\utils.py", line 110, in load_backend
  return import_module('%s.base' % backend_name)
File "c:\program files (x86)\lib\importlib\__init_.py", line 126, in import_module
  return bootstrap.gcd_import(name[level:], package, level)
File "c:\program files (x86)\lib\site-packages\django-2.2-py3.6.egg\django\db\backends\mysql\base.py", line 20, in <module>
  ) from err
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
Did you install mysqlclient?

Please help to resolve this.

Solution

  • The error says what you have to do.

    Since your Django project is configured to use MySQL as a backing database, you'll need to install the mysqlclient package (with e.g. pip install mysqlclient) in the environment you're running your Scrapy project in.