Search code examples
sql-serverscrapyscrapyd

Scrapy / Python and SQL Server


Is it possible to get the data scraped from websites using Scrapy, and saving that data in an Microsoft SQL Server Database?

If Yes, are there any examples of this being done? Is it mainly a Python issue? i.e. if I find some code of Python saving to an SQL Server database, then Scrapy can do same?


Solution

  • Yes, but you'd have to write the code to do it yourself since scrapy does not provide an item pipeline that writes to a database.

    Have a read of the Item Pipeline page from the scrapy documentation which describes the process in more detail (here's a JSONWriterPipeline as an example). Basically, find some code that writes to a SQL Server database (using something like PyODBC) and you should be able to adapt that to create a custom item pipeline that outputs items directly to a SQL Server database.