I'm writing an application on Python which sends post request via various proxies. I using PycURL library for it. On my Linux machine the same code works well. But in windows response body sometimes contains only "FAILED" instead of any HTTP code or connection failure. I did not saw that on Linux, so I'm really confused, I already spend a day trying to figure out what is wrong. Could someone help me?
Here is my code:
while (success_request != True) and (self.is_need_to_stop() == False):
current_proxy = self.get_random_proxy()
post_data = {'data' : data }
request_url = 'http://www.example.com'
_stringio = StringIO()
_curl = pycurl.Curl()
_curl.setopt(pycurl.URL, request_url)
_curl.setopt(pycurl.WRITEFUNCTION, _stringio.write)
_curl.setopt(pycurl.POSTFIELDS, urlencode(post_data))
_curl.setopt(pycurl.PROXY, current_proxy['address'])
_curl.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_HTTP)
_curl.setopt(pycurl.USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7')
_curl.setopt(pycurl.CONNECTTIMEOUT, 15)
_curl.setopt(pycurl.TIMEOUT, 15)
_curl.setopt(pycurl.VERBOSE, True)
try:
_curl.perform()
except Exception:
print "Proxy: '{0}' is not working!".format(current_proxy['address'])
success_request = False
self.del_bad_proxy(current_proxy['index'])
continue
_curl.close()
if self.check_responce(_stringio.getvalue().decode('utf-8')) == False:
print _stringio.getvalue().decode('utf-8')
print "Proxy: '{0}' returns 'bad' response!".format(current_proxy['address'])
self.del_bad_proxy(current_proxy['index'])
continue
else:
success_request = True
self.proxy_list[current_proxy['index']]['requests_number'] += 1
self.comments_number += 1
self.window.comment_posted(self.comments_number)
And here is problem output:
* About to connect() to proxy 85.15.234.1 port 8081 (#0)
* Trying 85.15.234.1... * connected
* Connected to 85.15.234.1 (85.15.234.1) port 8081 (#0)
> POST http://www.example.com HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7
Host: http://www.example.com
Accept: */*
Proxy-Connection: Keep-Alive
Content-Length: 42
Content-Type: application/x-www-form-urlencoded
< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 29 Dec 2014 11:00:55 GMT
< Content-Type: text/html
< P3P: policyref="/p3p.xml", CP="NOI CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT"
< Vary: Accept-Encoding
< Content-Length: 6
<
* Connection #0 to host 85.15.234.1 left intact
* Closing connection #0
FAILED
Proxy: '85.15.234.1:8081' returns 'bad' response!
* timeout on name lookup is not supported
Well, it turns out, pycurl in windows, and pycurl in Linux just somehow different inside. So, I rewrote my code using requests library and now it works perfectly.