Why is Net::HTTP returning a 500 when the server is returning a 503?

Question

(This question replaces this one, hopefully with better information.)

I've got three servers that I'm gonna call Alice, Charlie, and Harry (Harry being the server's actual nickname, so as not to confuse myself). They all talk to each other to perform quite a complicated authentication flow:

• A client performs a sequence of three requests through Alice to Harry.
• On the third one, Harry makes a call to Charlie.
• Charlie is prone to timeouts during periods of heavy traffic. If it does, Harry returns a 503 with a Retry-After header to Alice.
• Harry is returning a 503, I have confirmed that in its own logs.
• Alice is not receiving a 503 but a 500, and without the header.
• Alice's other clients (which I'm not in control of) treat a 500 the same as other errors, which is what I'm ultimately trying to fix.

Some extra information, such as I have been able to ascertain:

• Alice proxies calls to Harry using RestClient, which uses Net::HTTP under the hood.
• Using Net::HTTP directly gives the same result.
• It's not environment specific; I have had this problem both in Production and Development.
• I have been trying to simulate Alice using Postman, but haven't had any luck yet; Charlie's traffic is quieter at the moment, and I can't force or simulate a timeout, so so far I've only been getting successful 200 responses from Harry.
• Fixing Charlie's timeouts would obviously be ideal, but I'm not in control of Charlie's hardware either.

Is there something I can change about Alice so it properly detects Harry's 503?

Or, is it possible that something about Harry is changing its 503 to a 500 after it's returned and logged?

Here's Alice's code for that third call, if it's likely to help, but nothing's jumping out at me; I have been wondering if RestClient or Net::HTTP has some configuration that I don't know about.

http_verb = :post
args = [ # actually constructed differently, but this is a reasonable mock up
  'https://api.harry/path?sso_token=token',
  '',
  {
    'content_type' => 'application/json',
    'accept' => '*/*',
    # some other app-specific headers
  }
]

RestClient.send(http_verb, *args) do |response, request, result, &block|
  # `result` is present and has a 500 code at the start of this block if Harry returns a 503.
  @status_code = result.present? ? result.code : :internal_server_error
  cors.merge!( response.headers.slice(:access_control_allow_origin, :access_control_request_method) )
  @body = response.body
end

Answer 1

Turns out it's something way simpler and more obvious than anyone thought.

A separate error was raised in the middleware responsible for returning the 503. As with any other exception, this got rendered as a 500.

The thing that was causing it was a line that was supposed to tell the client to wait five seconds and try again:

response.headers['Retry-After'] = 5

... some middleware component was complaining of undefined method 'each' on 5:Fixnum, because it was expecting an Array where it wasn't a String; it started working when we changed 5 to '5'.