Search code examples
pythonmongodbpymongopicklepython-sacred

How to import the pickle file if sacred failed to connect to MongoDB

The experiment software sacred was run without MongoDB in the background with a configured mongo-observer. When it tried to write the settings to MongoDB, this failed, creating the file /tmp/sacred_mongo_fail__eErwU.pickle, with the message

Warning: saving to MongoDB failed! Stored experiment entry in /tmp/sacred_mongo_fail__eErwU.pickle
Traceback (most recent calls WITHOUT Sacred internals):
  File "/usr/local/lib/python2.7/dist-packages/sacred/observers/mongo.py", line 127, in started_event
    self.run_entry[experiment][sources] = self.save_sources(ex_info)
  File "/usr/local/lib/python2.7/dist-packages/sacred/observers/mongo.py", line 239, in save_sources
    file = self.fs.find_one({filename: abs_path, md5: md5})
  File "/usr/local/lib/python2.7/dist-packages/gridfs/__init__.py", line 261, in find_one
    for f in self.find(filter, *args, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/gridfs/grid_file.py", line 658, in next
    next_file = super(GridOutCursor, self).next()
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

How can this pickle file be imported into MongoDB manually?

Solution
    1. Load the pickle file,
    2. set the _id,
    3. insert 

    db = pymongo.MongoClient().sacred
entry = pickle.load(open('/tmp/sacred_mongo_fail__eErwU.pickle'))
entry['_id'] = list(db.runs.find({}, {"_id": 1}))[-1]['_id']
db.runs.insert_one(entry)

    This is quick and dirty, depends on the find to list objects in order, and could use Cleanest way to get last item from Python iterator instead of list(...)[-1], but it should work.