I am getting an authentication to MongoDb for every query I run using PyMongo MongoClient. This seems expensive / unnecessary:
2015-02-13T09:38:08.091-0800 [conn375243] authenticate db: { authenticate: 1, user: "", nonce: "xxx", key: "xxx" }
2015-02-13T09:38:08.876-0800 [conn375243] end connection xxx (15 connections now open)
2015-02-13T09:38:08.962-0800 [initandlisten] connection accepted from xxx:42554 #375244 (16 connections now open)
2015-02-13T09:38:08.966-0800 [conn375244] authenticate db: { authenticate: 1, user: "", nonce: "xxx", key: "xxx" }
As far as I can tell, I'm using the same MongoClient (although it's hidden behind MongoEngine) and not intentionally disconnecting it at any point:
19:20:45 {'default': MongoClient('xxx-a0.mongolab.com', 39931)}
19:20:45 [139726027002480]
19:28:35 {'default': MongoClient('xxx-a0.mongolab.com', 39931)} # print mongo_client_instance
19:28:35 [139726027002480] # print id(mongo_Client_instance)
When I set a pdb breakpoint in the authenticate function, this is the stacktrace. I cannot figure out why asking the cursor to refresh requires a fresh authentication. Am I misunderstanding, and that is part of the MongoDb protocol? My goal is to have as few "authenticate" commands sent as possible, since right now they're 50% of my logged commands on the server.
/home/ubuntu/workspace//metadata/jobs.py(24)get()
-> b = Item.objects.get_or_create(id=i['id'])[0]
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.py(241)get_or_create()
-> doc = self.get(*q_objs, **query)
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.py(182)get()
-> result = queryset.next()
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.py(1137)next()
-> raw_doc = self._cursor.next()
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/cursor.py(1058)next()
-> if len(self.__data) or self._refresh():
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/cursor.py(1002)_refresh()
-> self.__uuid_subtype))
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/cursor.py(915)__send_message()
-> res = client._send_message_with_response(message, **kwargs)
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py(1194)_send_message_with_response()
-> sock_info = self.__socket(member)
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py(922)__socket()
-> self.__check_auth(sock_info)
/home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/mongo_client.py(503)__check_auth()
-> sock_info, self.__simple_command)
> /home/ubuntu/workspace//venv/local/lib/python2.7/site-packages/pymongo/auth.py(239)authenticate()
-> mechanism = credentials[0]
Additional information that might be useful is that these calls are from a Python RQ worker. I am trying to set up the connection before the fork step, but it's possible something is happening there to cause this.
(Pdb) os.getpid()
10507
... next query...
(Pdb) os.getpid()
10510
Got it!
The default Python-RQ worker uses the fork model, and the forking blocked PyMongo from sharing connection sockets.
I switched to the GeventWorker and now the sockets are shared by default.