Search code examples
pythonweb-crawlertwill

Python Twill: follow link and grab data

I finally arrived to open a page and make the request I want.

#!/usr/bin/env python
from twill.commands import *

go("http://www.website.com/")
code(200) # assert page loaded fine
showforms()
# Make a search
searchStr = '%s' % n
formvalue(1, 'q', searchStr)
submit('Factorize!')
links = showlinks()

I now see all the links listed and I would like to click on the number 11 and 12 and when I'm there grab 1 value per page.

How to use follow() and what's the command used for grabbing a value?

Solution

  • The solution was:

    data = urllib2.urlopen(follow(str(3)))

    where '3' is is the number of the link listed by links = showlinks()