Search code examples
pythonweb-crawlerimdb

python crawler extract url not working


I tried to write a simple python program with imdb package to extract movie information from their database, but I do not know why the code returns empty list. My guess is that the way I extract url info(by using (.*?)) from the website is wrong. I want to extract an url link from the webpage. Here's the code. Thanks!

import urllib
import re
import imdb
imdb_access = imdb.IMDb()

top_num = 5

movie_list = ["The Matrix","The Matrix","The Matrix","The Matrix","The Matrix"]


for x in xrange(0,top_num):
    contain = imdb_access.search_movie(movie_list[x])

    ID = contain[0].movieID #str type

    htmltext = (urllib.urlopen("http://www.imdb.com/title/tt0133093/?ref_=nv_sr_1")).read()
    # a pattern in the website
    regex = regex = '<img alt="The Matrix Poster" title="The Matrix Poster" src="(.*?)" itemprop="image">'
    pattern = re.compile(regex)
    #print (str((pattern)))
    result = re.findall(pattern,htmltext)
    print result 
    #print type(htmltext)

Solution

  • I think the problem is with new lines can you have (.*\n*.*?)