Search code examples
pythonscrapyscrapyd

Mock in production code


I'll try to be direct here:

I'm a Ruby programmer, beginning with Python, and I'm trying something with Scrapy. I'm looking into a code my company received from a third-party, and one thing is breaking my Scrapyd deploy (on the cloud):

ImportError: no module named mock

I have installed locally, it's on my requirements.txt, but it's breaking upstream, in the deploy process. To be precise, this is the only place I grep'd a reference to mock on production code:

def parse(self, response):
    ...
    with mock.patch('lxml.html', lxml.html):                            
        article.parse()
    ...

Can anyone see some light in the end of this tunnel? Does anyone have any idea why would you use that mock on a production code (and what is it supposed to do?)

Thanks =]

EDIT 1: Maybe I should clarify something: in their Scrapyd interface, they have this place for the developer to upload 'Python eggs'. I'm not sure what that is (sounds like just a Gemfile, or...a pack of modules to be imported by the server), but I ran python setup.py bdist_egg to have this egg generated, I suppose it includes all the dependencies (setup.py is kinda confusing for newcomers to Python). Long story short: I ran this command and uploaded it on the server's Scrapyd interface. So, I have no access to no apt-get or even ssh to the Scrapyd machine.

EDIT 2: I have no access to apt-get on the server, it's a normal Scrapyd server. The answer being called 'duplicate' to this one does not answer my question.


Solution

  • You should create eggs of the dependencies, not of your entire project. To do so, first clone the mock repository:

    hg clone https://code.google.com/p/mock/
    

    Then build the egg:

    python setup.py bdist_egg
    

    and find the egg to be used under dist folder. Upload it to the Scrapyd server and it should be satisfy the dependency.