This is the error that I get: RemoteError: Remote error: TimeoutError QueuePool limit of size x overflow xy reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
We have an issue with metadata agent. One compute node hosts many vms and the vms can't get ip. I've tried to restart on the dhcp metadata server both service, tried to restart our rabbitmq server as well. Also restarted neutron server, but the issue still persist and the vms are down.
Errors like these:
2019-10-24 10:34:47.794 86154 ERROR neutron.agent.metadata.agent
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [-] Unexpected error.: RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 89, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 162, in _get_instance_and_tenant_id
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 155, in _get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 116, in __call__
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.func(target_self, *args, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 137, in _get_ports_for_remote_address
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent ip_address=remote_address)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 106, in _get_ports_from_server
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 72, in get_ports
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 150, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry=self.retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent retry=retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent call_monitor_timeout, retry=retry)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent raise result
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent RemoteError: Remote error: TimeoutError QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/metadata_rpc.py", line 43, in get_ports\n return self.plugin.get_ports(context, filters=filters)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped\n return method(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1435, in get_ports\n items = [self._make_port_dict(c, fields) for c in query]\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, in _execute_and_instances\n close_with_result=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2955, in _get_bind_args\n **kw\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2937, in _connection_from_session\n conn = self.session.connection(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection\n execution_options=execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind\n conn = engine.contextual_connect(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect\n self._wrap_pool_connect(self.pool.connect, None),\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get\n (self.size(), self.overflow(), self._timeout), code="3o7r")\n', u'TimeoutError: QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)\n'].
2019-10-24 10:34:48.433 86149 ERROR neutron.agent.metadata.agent
2019-10-24 10:35:04.436 86151 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
2019-10-24 10:35:04.637 86156 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
2019-10-24 10:35:05.339 86149 WARNING oslo_messaging._drivers.amqpdriver [-] Number of call queues is 21, greater than warning threshold: 20. There could be a leak. Increasing threshold to: 40
What should I do to fix it :( ?
Or how can I finetune this sqlalchemy? I have no idea to be honest what is this.
The solution was to add more neutron server. Memory and cpu increase didn’t help.