Search code examples
pythondjangomodelsscrapy

How to import django models in scrapy pipelines.py file


I'm trying to import models of one django application in my pipelines.py to save data using django orm. I created a scrapy project scrapy_project in the first involved django application "app1" (is it a good choice by the way?). I added these lines to my scrapy settings file:

def setup_django_env(path):
  import imp, os
  from django.core.management import setup_environ

  f, filename, desc = imp.find_module('settings', [path])
  project = imp.load_module('settings', f, filename, desc)

  setup_environ(project)

current_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
setup_django_env(os.path.join(current_dir, '../../d_project1'))

When I try to import models of my django application app1 I get this error message:

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 4, in <module>
    execute()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 122, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 76, in     _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 129, in     _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/crawl.py", line 43, in     run
    spider = self.crawler.spiders.create(spname, **opts.spargs)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/command.py", line 33, in crawler
    self._crawler.configure()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 41, in configure
    self.engine = ExecutionEngine(self, self._spider_closed)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/core/engine.py", line 63, in     __init__
    self.scraper = Scraper(crawler)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/core/scraper.py", line 66, in     __init__
    self.itemproc = itemproc_cls.from_crawler(crawler)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 50, in     from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/middleware.py", line 29, in     from_settings
    mwcls = load_object(clspath)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 39, in     load_object
    raise ImportError, "Error loading object '%s': %s" % (path, e)
ImportError: Error loading object 'scrapy_project.pipelines.storage.storage': No module         named dydict.models

Why cannot scrapy access django application models (given that app1 in the installed_app ) ?


Solution

  • Try:

    from .. models import MyModel 
    

    OR

    from ... models import MyModel
    

    Every dot represent the location