Search code examples
pythonscrapyscrapy-shell

Contributing to Scrapy - What is the correct way to run from source?


I would like to contribute to the project bu making some patches. However, I'm a little stuck with how to work on the github repo.

I selected a bug on the issue tracker, did some research, and came up with a possible fix. I made some modifications to my fork of the source. However, I have become stuck when trying to run scrapy from source.

I've read the documentation for the Install process, the Contributor guide and the FAQ. I cannot find any information on how to run the project from source code. (Only how to install from official distribution packages, which is not what I want.)

Do you have to create a virtual environment to run this in? Can you provide simple steps on how you normally run the project from the github repo?

Thanks for your time, D.

[Edit]

Thanks to folks on the IRC server I was able to install the repo version using this command:

pip install -e <path/to/scrapy/root/dir>

That answers my question. It was highly recommended to setup and activate a virtual environment to work from, which is described very well in the accepted answer below.

Thanks everyone!


Solution

  • To run the github repo you will need to clone the repo from remote to local using git clone, to get it started the step would be something like this in terminal or git bash

    cd <base_folder>          #folder where repo will be downloaded
    git clone <repo_url>      #this will clone remote repo to local
    cd <repo_folder>          #make repo folder current directory
    

    After that you can work on the project, but to push/merge your local repo to remote you will need the permission from the owner

    Also it is recomended to create separate virtual env for each project, it will be something like this

    python3 -m venv <name_of_venv>  #Creating venv
    <name_of_venv>/bin/activate     #activating venv in linux
    <name_of_venv>/Scripts/activate #activating venv in windows