Search code examples
pythonjsondictionaryscrapyscrapy-item

Scrapy - yield nested dictionary to JSON file - doesn't work


EDIT

As Georgiy suggested, I tried to yield dict instead of Item and the results are the same.

EDIT END

Trying to export Scrapy output to a JSON file. An item should have this format:

{'name':'Peter', 'attrs':{'attr1':<VAL>, 'attr2':<VAL>}}

The problem is that Scrapy renders name only. The reason is probably that attrs is a dictionary.

class CookieBotItem(scrapy.Item):
    name = scrapy.Field()
    attrs = scrapy.Field()

For the simplicity I return always this:

    yield CookieBotItem(name='Peter',
                        attrs={
                            'attr1': 'val1',
                            'attr2': 'val2'}
                        )

And the ouput looks like this:

[
{"name": "Peter"},
{"name": "Peter"},
{"name": "Peter"},
{"name": "Peter"},
{"name": "Peter"},
{"name": "Peter"},
{"name": "Peter"},
{"name": "Peter"}
]

Do you know how to make it work?


Solution

  • Not sure about usage of Item classes here for this.. nested items.
    The fastest way to achieve this is to yield dictionaries (not Item class objects):

        yield {
               'name': 'Peter',
               'attrs':{'attr1': 'val1','attr2': 'val2'}
              }