Search code examples
scrapyscrapyd

How to store scrapyd items in json format

I am trying to store scrapyd items in a JSON file. Actually by default it store items in a json file but like this: File_1:

{item1}
{item2}
....

And if i run my spider by scrapy crawl spidername -o fileName -t json

it will store item like this: File_2:

[{item1},
{item2},
....]

If i try to copy paste contents of File_1 into jsonParser i got error saying expected [ but File_2 works fine. Can anyone tell me how to store items in proper JSON format using scrapyd

Solution

  • Try to save the items directly into the database rather then storing them on the file....then storing the items from file to database... or try to read line by line from that JSON file and then do what ever you want instead of reading the whole file at a time...