Timeout error when making get request on specific site using ruby 2.5.3

Question

Can anyone get a proper response from the site www.coupang.com? I keep making requests to "https://www.coupang.com/" and I get an error 9 out of 10 times.(Sometimes it works! Surprisingly.)

Traceback (most recent call last):
        14: from lib/add_sup/test.rb:7:in `<main>'
        13: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:485:in `get_response'
        12: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:609:in `start'
        11: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:910:in `start'
        10: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:487:in `block in get_response'
         9: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1365:in `request_get'
         8: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1464:in `request'
         7: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1491:in `transport_request'
         6: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1491:in `catch'
         5: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http.rb:1494:in `block in transport_request'
         4: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http/response.rb:29:in `read_new'
         3: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/http/response.rb:40:in `read_status_line'
         2: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:167:in `readline'
         1: from /Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:157:in `readuntil'
/Users/j/.rbenv/versions/2.5.3/lib/ruby/2.5.0/net/protocol.rb:181:in `rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)

I have also tried making request using python3 and it works fine. I think there is something wrong with my ruby configuration or ruby itself.

require 'nokogiri'
require 'open-uri'
require 'net/http'


uri = URI("https://www.coupang.com/")
res = Net::HTTP.get_response(uri)
puts res.body if res.is_a?(Net::HTTPSuccess)

I would appreciate your kind thoughts on this matter. Thank you!

Answer 1

They're using akamai, so first - they expect HTTP/2 (you'll need an http2 gem for that) and they have some fairly tight User-Agent sniffing.

Here's an example that works using net-http2

client = NetHttp2::Client.new "https://www.coupang.com/"
res = client.call :get, '/', headers: { "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6)" }
puts res.body if res.ok?