How to save data to remote mongoDB via ssh tunnel? (connection refused)

Question

I have two computers: Ubuntu1 and Ubuntu2. Ubuntu1 runs MongoDB with database Sacred3. I want to connect from U2 to U1 via ssh and store there my experiment results.

What I tried and failed: 1. I installed mongo DB, created sacred3, I have ssh key to it. I edited /etc/mongod.conf adding:

# network interfaces net: port: 27017 bindIp: 0.0.0.0

Then I enabled port forwarding with

ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:localhost:27017 [email protected] // (with proper ip)

so, as I undertstand, if I connect to my localhost:6666 it will be forwarded to 106.969.696.969:27017

So after that, I'm runnig an experiment with Sacred framework:

python exp1.py -m localhost:6666:sacred3

and this should write experiment to remote DB, HOWEVER i I get:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

which is driving me mad. please help!

#

below contents of exp1.py:

from sacred import Experiment
from sacred.observers import MongoObserver

ex = Experiment()
ex.observers.append(MongoObserver.create())

def compute():
    summ = layer1 - layer2
    return summ


@ex.config
def my_config():

    hp_list = [{"neurons" : [32,32] , "dropout": 1.0},
            {"neurons" : [32,32] , "dropout": 0.7},
            {"neurons" : [32,16] , "dropout": 0.9},
            {"neurons" : [24,16] , "dropout": 0.9},
            {"neurons" : [24,8] , "dropout":  0.9},
            {"neurons" : [16,8] , "dropout":  0.9},
            {"neurons" : [64,64] , "dropout": 0.9},
            {"neurons" : [64,64] , "dropout": 0.7},
            {"neurons" : [64,32] , "dropout": 0.9},
            {"neurons" : [64,32] , "dropout": 0.7},
            {"neurons" : [48,32] , "dropout": 0.9},
            {"neurons" : [48,32] , "dropout": 0.7},
            {"neurons" : [48,16] , "dropout": 0.9},
            {"neurons" : [48,16] , "dropout": 0.7},]

    n_epochs = 2 


@ex.capture
def training_loop(hp_list, n_epochs):
    for j in hp_list:
        print("Epoch: ", n_epochs)
#       layer1 = random.randint(18,68)
#       layer2 = random.randint(18,68)
#       layer3 = random.randint(18,68)
        layer1 = j["neurons"][0]
        layer2 = j["neurons"][1]
        dropout_ratio = j["dropout"]


        print("WHATS UUUUUP",j, layer1, layer2, dropout_ratio, sep="_")
        # vae_training_loop_NN_DO(i, layer1, layer2, dropout_ratio )


@ex.automain
def my_main():
    training_loop()

Answer 1

According to the documentation supplied, it looks like you're creating two observers, or overriding the connection argument you passed with -m, with the MongoObserver.create()specified in the code which uses the default mongo host and port localhost:27017. You either supply the observer connection via the -m argument or in code, not both.

Try removing the MongoObserver.create() line altogether, or hardcoding the connection arguments: MongoObserver(url='localhost:6666', db_name='sacred3')

Also, it looks like your mongo host is not liking the binding to localhost so you should also replace localhost in your ssh command with 127.0.0.1 or [::1], e.g ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:127.0.0.1:27017 [email protected] or ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:[::1]:27017 [email protected]