cephadm: Not able to add nodes to ceph cluster (Error EINVAL: Failed to connect to host)

I followed the following steps from https://docs.ceph.com/en/latest/cephadm/install/ to setup a ceph cluster on Centos 8.1

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
./cephadm add-repo --release octopus
./cephadm install

After the above command I found out that ceph requires either docker or podman to run. So I installed the community version of docker from https://docs.docker.com/engine/install/centos/ and continued the steps below.

./cephadm install
mkdir -p /etc/ceph
cephadm bootstrap --mon-ip *ip_of_the_current_machine (host1)*
cephadm install ceph-common
ssh-copy-id -f -i /etc/ceph/ceph.pub root@host2*
ceph orch host add host2

The above command fails with the error

[root@host1 home]# ceph orch host add host2
INFO:cephadm:Inferring fsid 12345678-2345-6789-1011-000129110013
INFO:cephadm:Inferring config /var/lib/ceph/12345678-2345-6789-1011-000129110013/mon.host1/config
INFO:cephadm:Using recent ceph image ceph/ceph:v15
Error EINVAL: Failed to connect to host2 (host2).
Check that the host is reachable and accepts connections using the cephadm SSH key
 
you may want to run:
> ceph cephadm get-ssh-config > ssh_config
> ceph config-key get mgr/cephadm/ssh_identity_key > key
> ssh -F ssh_config -i key root@host2

I am able to login to host2 using the above steps. Could someone please tell if I am doing something wrong. How do I solve this problem.

Solution

So after days of debugging I figured out that python3 was missing on the node I wanted to add. All I had to do was check the last few logs using the command.

ceph log last cephadm

This gave the following log messages.

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1036, in _remote_connection
    raise execnet.gateway_bootstrap.HostNotFound(msg)
execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `host2`, possibly because python3 is not installed there: cannot send (already closed?)
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 295, in _finalize
    next_result = self._on_complete(self._value)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 103, in <lambda>
    return CephadmCompletion(on_complete=lambda _: f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1201, in add_host
    return self._add_host(spec)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1187, in _add_host
    error_ok=True, no_fsid=True)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1104, in _run_cephadm
    with self._remote_connection(host, addr) as tpl:
  File "/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1055, in _remote_connection
    raise OrchestratorError(msg) from e
orchestrator._interface.OrchestratorError: Failed to connect to host2 (host2).
Check that the host is reachable and accepts connections using the cephadm SSH key

Next to add the node I ran.

ceph orch host add host2 ip_address