Search code examples
pythonwindowscurlproxypycurl

Why Pycurl returns FAILED


I'm writing an application on Python which sends post request via various proxies. I using PycURL library for it. On my Linux machine the same code works well. But in windows response body sometimes contains only "FAILED" instead of any HTTP code or connection failure. I did not saw that on Linux, so I'm really confused, I already spend a day trying to figure out what is wrong. Could someone help me?

Here is my code:

while (success_request != True) and (self.is_need_to_stop() == False):
    current_proxy = self.get_random_proxy() 
    post_data = {'data' : data }
    request_url = 'http://www.example.com'

    _stringio = StringIO()
    _curl = pycurl.Curl()
    _curl.setopt(pycurl.URL, request_url)
    _curl.setopt(pycurl.WRITEFUNCTION, _stringio.write)
    _curl.setopt(pycurl.POSTFIELDS, urlencode(post_data))
    _curl.setopt(pycurl.PROXY, current_proxy['address'])
    _curl.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_HTTP)
    _curl.setopt(pycurl.USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7')
    _curl.setopt(pycurl.CONNECTTIMEOUT, 15)
    _curl.setopt(pycurl.TIMEOUT, 15)
    _curl.setopt(pycurl.VERBOSE, True)             

    try:
        _curl.perform()
    except Exception:
        print "Proxy: '{0}' is not working!".format(current_proxy['address'])
        success_request = False
        self.del_bad_proxy(current_proxy['index'])
        continue

    _curl.close()

    if self.check_responce(_stringio.getvalue().decode('utf-8')) == False:
        print _stringio.getvalue().decode('utf-8')
        print "Proxy: '{0}' returns 'bad' response!".format(current_proxy['address'])
        self.del_bad_proxy(current_proxy['index'])
        continue
    else:
        success_request = True
        self.proxy_list[current_proxy['index']]['requests_number'] += 1
        self.comments_number += 1
        self.window.comment_posted(self.comments_number)

And here is problem output:

* About to connect() to proxy 85.15.234.1 port 8081 (#0)
*   Trying 85.15.234.1... * connected
* Connected to 85.15.234.1 (85.15.234.1) port 8081 (#0)
> POST http://www.example.com HTTP/1.1    
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7    
Host: http://www.example.com    
Accept: */*    
Proxy-Connection: Keep-Alive    
Content-Length: 42    
Content-Type: application/x-www-form-urlencoded    

< HTTP/1.1 200 OK   
< Server: nginx  
< Date: Mon, 29 Dec 2014 11:00:55 GMT   
< Content-Type: text/html  
< P3P: policyref="/p3p.xml", CP="NOI CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT" 
< Vary: Accept-Encoding    
< Content-Length: 6   
< 

* Connection #0 to host 85.15.234.1 left intact
* Closing connection #0
FAILED
Proxy: '85.15.234.1:8081' returns 'bad' response!
* timeout on name lookup is not supported

Solution

  • Well, it turns out, pycurl in windows, and pycurl in Linux just somehow different inside. So, I rewrote my code using requests library and now it works perfectly.