Search code examples
pythonscrapyenvironment-variablespython-dotenv

How do you add environment variables to a python scrapy project? dotenv didn't work

I'm having trouble incorporating an IP address into a format string in my Python Scrapy project. I was trying to use python-dotenv to store sensitive information, such as server IPs, in a .env file and load it into my project, instead of hardcoding it.

I added python-dotenv to the settings.py file of my Scrapy project, but when I run a function that should use the values stored in os, I get an error saying that it can't detect dotenv. Can someone help me understand why this is happening and how to properly incorporate an IP address in a format string using python-dotenv in a Python Scrapy project?

Solution

  • It is difficult to help figure out what you might be doing wrong with the limited information you have provided. However here is a general example of how one might use python-dotenv with a scrapy settings.py file.

    First create a .env file and add your IP

    .env

    IPADDRESS = SUPERSECRETIPADDRESS

    Finally in your settings.py file you need to import dotenv and then run dotenv.load_dotenv() and then you can get the ip address from the environment variable.

    settings.py

    import os
import dotenv

dotenv.load_dotenv()

IP_ADDRESS_SCRAPY_SETTING = os.environ["IPADDRESS"]

print(IP_ADDRESS_SCRAPY_SETTING)

    output:

    SUPERSECRETIPADDRESS

    Note: Make sure that the .env file is in the same directory as the settings.py file or it is in one of it's parent/ancestor directories.

    So if your settings.py file is in

    • /home/username/scrapy_project/scrapy_project/settings.py

    then that means the .env file can be in one of the following:

    • /home/username/scrapy_project/scrapy_project/.env
    • /home/username/scrapy_project/.env
    • /home/username/.env
    • /home/.env
    • /.env

    Otherwise it will not be able to find the file.