Search code examples
pythonterminalpython-requestspythonanywherelxml.html

Requests/lxml script works only in terminal but retrieves no data in IDLE or PythonAnywhere? What could cause this?


So my code works exactly like I want it to in the terminal, but I cannot get it to work in IDLE (that comes with homebrew) or PythonAnywhere.

I am getting an error later down the line when I try to call the data that should have been collected, but the function itself does not return an error.

def GetQuantityNDescription(orderID, itemNum):
    payload = {'login_pass': 'password', 'login_user': 'user','submit':'go'}  ## Log in Paramaters
    r = requests.get("http://website.com/?orderID="+str(orderID), params=payload)  ##Get Order Page

    tree = html.fromstring(r.text)  ## turn raw string into html tagged data

    rawdata = tree.xpath('//*[@class="LargeBody"]')   ## Get raw data found based on class LargeBody
    quantity = []  
    description =[]
    even = True
    for item in rawdata:   ## For each item in rawdata, add it to one of the following lists (always multiple of two)
        #print item.text_content()
        if even == True:  ## First take quanity
            quantity.append(item.text_content())
            even = False
        else:             ## Second take desciption
            description.append(item.text_content())
            even = True
        ## Repeat

    orderInfo = [quantity[itemNum-1], description[itemNum-1]]
    print quantity[itemNum-1]
    print description[itemNum-1]
    return  orderInfo

When I run it in in Terminal, rawdata returns [quantity item 1, description item 1, quantity item 2, description item 2, etc, etc]

When I run it in IDLE or PythonAnywhere, It returns [].

I am getting the error:

orderInfo = [quantity[itemNum-1], description[itemNum-1]]
IndexError: list index out of range 

Any idea what could be the cause of this, or how I go about trouble shooting? I need to be able to run this outside of my terminal. Preferably on PythonAnywhere.

I am using:

Python 2.7.6

OS X 10.8.5

requests==2.4.3

lxml ==3.3.6

and the free version of python anywhere

Edit::

IDLE is using Python 2.7.6 (displayed on top of shell.)

Requests is using Python2.7 console 869203

Terminal is using Python 2.7.6 ($ python --version returns Python 2.7.6 )


Solution

  • My guess is that the different environments get different responses from the web site for whatever reason. Try printing the response before you parse it.