Node '[email protected]' not responding to pings

I have no idea what happend with the system, but suddenly when I issue riak start it doesn't start anymore. I get this output.

And if I issue riak console, I get no response for pings. I don't know what to do, so please advise how to put the system back online.

root@aliceubuntu:/home/alice# riak start
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
riak failed to start within 15 seconds,
see the output of 'riak console' for more information.
If you want to wait longer, set the environment variable
WAIT_FOR_ERLANG to the number of seconds to wait.
root@aliceubuntu:/home/alice# riak console
Node '[email protected]' not responding to pings.
config is OK
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot /usr/lib/riak/releases/1.4.1/riak              -config /etc/riak/app.config             -pa /usr/lib/riak/lib/basho-patches             -args_file /etc/riak/vm.args -- console
Root: /usr/lib/riak
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]

/usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
                                                                   Erlang has closed
    {"Kernel pid terminated",application_controller,"{application_start_failure,riak_core,{bad_return,{{riak_core_app,start,[normal,[]]},{'EXIT',{{function_clause,[{orddict,fetch,['[email protected]',[]],[{file,[111,114,100,100,105,99,116,46,101,114,108]},{line,72}]},{riak_core_capability,renegotiate_capabilities,1,[{file,[115,114,99,47,114,105,97,107,95,99,111,114,101,95,99,97,112,97,98,105,108,105,116,121,46,101,114,108]},{line,416}]},{riak_core_capability,handle_call,3,[{file,[115,114,99,47,114,105,97,107,95,99,111,114,101,95,99,97,112,97,98,105,108,105,116,121,46,101,114,108]},{line,202}]},{gen_server,handle_msg,5,[{file,[103,101,110,95,115,101,114,118,101,114,46,101,114,108]},{line,588}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,227}]}]},{gen_server,call,[riak_core_capability,{register,{riak_core,vnode_routing},{capability,[proxy,legacy],legacy,{riak_core,legacy_vnode_routing,[{true,legacy},{false,proxy}]}}},infinity]}}}}}}"}

Crash dump was written to: /var/log/riak/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,riak_core,{bad_return,{{riak_core_app,start,[normal,[]]},{'EXIT',{{function_clause,[{orddict,fetch,['[email protected]',[]
root@aliceubuntu:/home/alice#

Solution

So the whole process here is that

for some reason IP was changed on the machine riak
the IP was hard-coded in /etc/riak/vm.args and /etc/riak/app.config

the process to revive is the following:

edit and set the new IPs in the above files
sudo rm -rf /var/lib/riak/ring/* # delete the riak ring
restart the machine or do the next steps:
riak-admin cluster force-replace [email protected] riak@"hostname -i" # replace the name
riak start # start the node

This topic is also linked to this problem: Riak node no longer working after changing IP address