Search code examples
mysqlcentosclouderaambarihdp

Ambari Admin login hung with correct credentials, throws correct error with invalid credentials


I have setup Ambari 2.7.0 on Centos 7 server with Mysql 5.7.29 as backend. All services are running fine, but when I try to login using admin:admin credential it seems to hang and shows following logs in ambari-server.log file:

2020-03-18 06:55:28,031  INFO [MessageBroker-1] WebSocketMessageBrokerStats:113 - WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannelpool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]

2020-03-18 06:55:28,678  INFO [MessageBroker-1] WebSocketMessageBrokerStats:113 - WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannelpool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]

Interesting thing is, moment I enter invalid user or wrong password for admin user, it throws correct error Invalid credentials but on entering correct credentials, its hanging and logs above mentioned logs in ambari-server.log file.

Previously I used to see following logs in MySQL Log file:

[Note] Aborted connection 4 to db: '' user: '' host: '' (Got an error reading communication packets)

As suggested on some questions, I increased max_connections to 400 and max_allowed_packet close to 2% of RAM available on server to 656MB.

After increasing above mentioned configs, I dont see any Error reading communication packets in MySQL log file but I dont seem to get Ambari login working.

When I turn logging level for Ambari-Server to debug and enter random user it gives logs :

2020-03-18 11:24:16,116 DEBUG [ambari-client-thread-47] FilterChainProxy:325 - /api/v1/users/esrdf?fields=*,privileges/PrivilegeInfo/cluster_name,privileges/PrivilegeInfo/permission_name&_=1599955834198 at position 1 of 13 in additional filter chain; firing Filter: 'WebAsyncManagerIntegrationFilter'

But when I enter correct credentials admin:admin, I dont see any API even being called within Logs. What could be the reason for this?

I tried debugging this using API calls, eg: /api/v1/clusters. When I make the API call from the server itself, it gives correct response but when the API is made from local, it gives out Recv failure: Connection was reset

Can someone point me, where I could be going wrong?

Help Appreciated!


Solution

  • Found the issue. It was a VPN connectivity issue that was preventing login, Ambari UI could be accessed but not login. Login works when connected to a machine on same network.