Search code examples
pythondeploymentscrapyenvironmentscrapinghub

Set variable on shub deploy project


I'm trying to setup scrapy settings to work with test and production environment on local and also on scrapinghub. And I would like to know if there is any way to set this variable (for example as the following) on shub deploy:

enter image description here

And then at settings.py:

if env == "test":
     var1 = some_ip
     var2 = username
elif env == "prod":
     var1 = some_ip
     var2 = username

Or... maybe there is a cleaner way to this?

Thank you for reading!

PS: I want to automate the settings depending of the environment where is the spider/project located instead of changing the variables manually.

EDIT: Found a better solution. Check the answer.


Solution

  • Finally, I solved my problem following these steps:

    So now my settings look like this:

    settings tree

    • And then my __init__.py look like this:

    __init__.py file

    • Next, modified _environment.py as the following:

    _environment.py. this is a function helper that gets programmatically the environment where is Scrapy at the moment

    I hope I have been helpful!

    PS: Remember that you need to have two projects on scrapinghub, one for production and one for testing to do this, also you need to deploy from the same project as the following:

    scrapinghub.yml

    And then you just have to (reference):

    • shub deploy test Deploys to test.
    • shub deploy prod Deploys to production.
    • shub deploy Deploys to test.