Getting Mechanize::UnauthorizedError: 401 => Net::HTTPUnauthorized when accessing API with Basic Auth

Question

I'm trying to access an API using Basic Auth. It works with HTTParty, but not with 2.7.6 Mechanize.

This is what I tried:

agent = Mechanize.new
agent.log = Logger.new(STDERR)
agent.add_auth("https://website.net/listingapi", "user", "pass")
page = agent.get("https://website.net/listingapi")

And this is what I get:

 INFO -- : Net::HTTP::Get: /listingapi
DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: user-agent => Mechanize/2.7.6 Ruby/2.5.3p105 (http://github.com/sparklemotion/mechanize/)
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: host => website.net
 INFO -- : status: Net::HTTPUnauthorized 1.1 401 Unauthorized
DEBUG -- : response-header: content-type => application/json; charset=utf-8
DEBUG -- : response-header: www-authenticate => Bearer, Basic realm=ListingApi
DEBUG -- : response-header: date => Wed, 13 Mar 2019 14:14:51 GMT
DEBUG -- : response-header: content-length => 61
DEBUG -- : response-header: x-xss-protection => 1; mode=block
DEBUG -- : response-header: strict-transport-security => max-age=31536000
DEBUG -- : response-header: x-content-type-options => nosniff
DEBUG -- : Read 61 bytes (61 total)
Mechanize::UnauthorizedError: 401 => Net::HTTPUnauthorized for https://website.net/listingapi/ -- no credentials found, provide some with #add_auth -- available realms: 
from /Users/nk/.rvm/gems/ruby-2.5.3@mygems/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:749:in `response_authenticate'

What am I doing wrong, or what is wrong with the API response?

PS. I found this, which I think might be related: https://github.com/sparklemotion/mechanize/pull/442

Answer 1

When basic auth is used, username and password are joined together and then encoded using base64. The encoded resulted string is sent to the server in Authorization header using Basic

Now a workaround you can do in case you have issues using add_auth is to pass the Authorization header yourself:

username = 'Radu'
password = 'mypassword'
agent = Mechanize.new do |agent|
  agent.pre_connect_hooks << lambda { |agent, request| request["Authorization"] = "Basic #{Base64.strict_encode64(username + ':' + password)}" }
end
page = agent.get("https://website.net/listingapi")

Edit 1

Now that I read the logs again I can see that www-authenticate header says Bearer, Basic realm=ListingApi. Instead it should say Basic realm=ListingApi.

The problem is response_authenticate can't find any challange most likely because the API you're requesting does not respect this part of RFC7235 regarding the challenge.

The missing challenge raises 401 after this line

[1] pry(main)> authenticate_parser  = Mechanize::HTTP::WWWAuthenticateParser.new
=> #<Mechanize::HTTP::WWWAuthenticateParser:0x00007fe2a5c74ec8 @scanner=nil>

[2] pry(main)> authenticate_parser.parse "Basic realm=ListingApi"
=> [#<struct Mechanize::HTTP::AuthChallenge scheme=nil, params=nil, raw=nil>]

[3] pry(main)> authenticate_parser.parse "Bearer, Basic realm=ListingApi"
=> []

Edit 2

The reason why HTTParty works is that they add the Authorization header upfront directly on Net::HTTP::Get. Mechanize utilizes whole challenge-response authorization and they will only add it if the challenge scheme is Basic.