Typhoeus does not close https connections

Question

I'm using Typhoeus to handle all of the HTTP calls to external APIs on my site, and it's been working beautifully up until recently. My Rails site started to be unresponsive after a while. I noticed that there're a ton of connections in the CLOSE_WAIT state when I do a netstat, and they're generated by the following piece of code.

  requests = []
  hydra = Typhoeus::Hydra.new
  urls.each_with_index do |url, index|
    request = Typhoeus::Request.new(url, :timeout => 5000)
    request.on_complete do |response|
      begin
        resp = JSON.parse(response.body)

        if resp["error"]
          p "no deal found for #{factual_ids[index]}"
          { :deals => nil }
        else
          { :deals => resp["merchant"]["deals"] }
        end
      rescue Exception => e
        p e.message
        { :deals => nil }
      end
    end

    requests << request
    hydra.queue(request)
  end

  hydra.run

The only thing that I find different than how I am using Typhoeus in my other HTTP calls is that the urls above are all HTTPS urls. I don't know if that is of any significance but that's the only thing I can think of at this moment. Has anyone seen this before? Is there a option I can pass into Typheous to force close the connection once it's finished?

Answer 1

Which operating system and Typhoeus version are you using? Some folks with ubuntu seem to run into similar issues. Typhoeus 0.5 is not yet released, but the release candidate which supports the forbid_reuse option.

Typhoeus::Request.new(url, :timeout => 5000, :forbid_reuse => true)

This is the issue: https://github.com/typhoeus/typhoeus/issues/205 and here is the libcurl documentation: http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTFORBIDREUSE.

Your code would look like that for Typhoeus 0.5:

requests = []
Typhoeus.on_complete do |response|
  begin
    resp = JSON.parse(response.body)

    if resp["error"]
      p "no deal found for #{factual_ids[index]}"
      response.options[:response_body] = { :deals => nil }
    else
      response.options[:response_body] = { :deals => resp["merchant"]["deals"] }
    end
  rescue Exception => e
    p e.message
    response.options[:response_body] = { :deals => nil }
  end
end

hydra = Typhoeus::Hydra.new
urls.each_with_index do |url, index|
  request = Typhoeus::Request.new(url, :timeout => 5000)
  requests << request
  hydra.queue(request)
end

hydra.run