Search code examples
pythonxmlpandaslxmllxml.objectify

Problems parsing XML with lxml

I've been trying to parse an XML feed into a Pandas dataframe and can't work out where I'm going wrong. 

import pandas as pd
import requests
import lxml.objectify
path = "http://www2.cineworld.co.uk/syndication/listings.xml"

xml = lxml.objectify.parse(path)
root = xml.getroot()

The next bit of code is to parse through the bits I want and create a list of show dictionaries. 

shows_list = []
for r in root.cinema:
    rec = {}
    rec['name'] = r.attrib['name']
    rec['info'] = r.attrib["root"] + r.attrib['url']
    listing = r.find("listing")
    for f in listing.film:
        film = rec
        film['title'] = f.attrib['title']
        film['rating'] = f.attrib['rating']
        shows = f.find("shows")
        for s in shows['show']:
            show = rec
            show['time'] = s.attrib['time']
            show['url'] = s.attrib['url']
            #print show
            shows_list.append(rec)

df = pd.DataFrame(show_list)

When I run the code, the film and time field seems to be replicated multiple times within rows. However, if I put a print statement into the code (it's commented out), the dictionaries appear to as I would expect.

What am I doing wrong? Please feel free to let me know if there's a more pythonic way of doing the parsing process.

EDIT: To clarify:

These are the last five rows of the data if I use a print statement to check what's happening as I loop through.

{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729365&seats=STANDARD', 'time': '2016-02-07T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729366&seats=STANDARD', 'time': '2016-02-08T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729367&seats=STANDARD', 'time': '2016-02-09T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729368&seats=STANDARD', 'time': '2016-02-10T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729369&seats=STANDARD', 'time': '2016-02-11T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'PG', 'name': 'Cineworld Stoke-on-Trent', 'title': 'Autism Friendly Screening - Goosebumps', 'url': '/booking?performance=4782937&seats=STANDARD', 'time': '2016-02-07T11:00:00'}

This is the end of the list: ...

{'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'},
 {'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'},
 {'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'},
 {'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'}]
Solution

  • Your code only has one object that keeps getting updated: rec. Try this: 

    from copy import copy
shows_list = []
for r in root.cinema:
    rec = {}
    rec['name'] = r.attrib['name']
    rec['info'] = r.attrib["root"] + r.attrib['url']
    listing = r.find("listing")
    for f in listing.film:
        film = copy(rec) # New object
        film['title'] = f.attrib['title']
        film['rating'] = f.attrib['rating']
        shows = f.find("shows")
        for s in shows['show']:
            show = copy(film) # New object, changed reference
            show['time'] = s.attrib['time']
            show['url'] = s.attrib['url']
            #print show
            shows_list.append(show) # Changed reference

df = pd.DataFrame(show_list)

    With this structure, the data in rec is copied into each film, and the data in each film is copied into each show. Then, at the end, show is added to the shows_list.

    You might want to read this article to learn more about what's happening in your line film = rec, i.e. you are giving another name to the original dictionary rather than creating a new dictionary.