Set timeout for get request with HTTP.jl

Question

I got to scan IP ranges and want to reduce waiting time on the timeouts. How do i specify the request timeout, with Julia's HTTP.jl package?

I have found the readtimeout option in the docs for v.0.6.15:

conf = (readtimeout = 10,
        pipeline_limit = 4,
        retry = false,
        redirect = false)

HTTP.get("http://httpbin.org/ip"; conf..)

But in the current stable version v0.8.6 readtimeout seams to only appear on the server side.

Testcode with readtimeout=2 and v.0.8.6:

@time begin
    try
        HTTP.get("http://2.160.0.0:80/"; readtimeout=2)
    catch e
        @info e
    end
end

Output:

113.642150 seconds (6.69 k allocations: 141.328 KiB)
┌ Info: IOError(Base.IOError("connect: connection timed out (ETIMEDOUT)", -4039) during request(http://2.160.0.0:80/))
└ @ Main In[28]:5

So the request took about 114 seconds, hence i think that this option is currently unsupported.

Edit I checked the source code (HTTP.jl) of the stable release:

Timeout options
 - `readtimeout = 60`, close the connection if no data is received for this many
   seconds. Use `readtimeout = 0` to disable.

with this example given:

HTTP.request("GET", "http://httpbin.org/ip"; retries=4, cookies=true)
HTTP.get("http://s3.us-east-1.amazonaws.com/"; aws_authorization=true)
conf = (readtimeout = 10,
        pipeline_limit = 4,
        retry = false,
        redirect = false)
HTTP.get("http://httpbin.org/ip"; conf..)
HTTP.put("http://httpbin.org/put", [], "Hello"; conf..)

So it should be working...

Answer 1

It definitely does not do what is expected. There are multiple things going on here:

First off the default for idempotent requests in HTTP.jl is to retry 4 times. So a HTTP.get will only fail after 5 * readtimeout. You can change this by passing retry = false in the arguments.

Second thing I noticed is that the check for timed out connections is on a very long interval. (See TimeoutRequest Layer) It only checks every 8 to 12 seconds for a time out, so a timeout below 8 seconds is not doing anything. (I suspect these should be 8-12 milliseconds not seconds as implemented)

And finally the docs for 0.8.6 are missing for HTTP.request. I already made a PR to fix this.