I am trying to regain access to my databases stored in CouchDB, but the error This database failed to load.
gets shown instead of the databases (see attachments).
This database failed to load.
is normal, when
lacking privileges, 2. The actions to delete
or replicate
a DB are
missing). However, I am logged as admin.Can you help me find the root cause of the error?
Simple PUT and DELETE requests curl -X PUT http://admin:[email protected]:5984/testdb
:
[error] 2018-02-09T15:06:05.221393Z [email protected] <0.1112.0> e84f27d276 Request to create N=3 DB but only 2 node(s)
[error] 2018-02-09T15:06:05.237308Z [email protected] <0.1173.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[notice] 2018-02-09T15:06:05.252506Z [email protected] <0.1112.0> e84f27d276 www.example.com:5984 130.211.68.93 undefined PUT /testdb 500 ok 47
[notice] 2018-02-09T15:06:16.286511Z [email protected] <0.1320.0> 890c465244 www.example.com:5984 130.211.68.93 undefined DELETE /testdb 500 ok 9
[error] 2018-02-09T15:06:16.288230Z [email protected] <0.1350.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.288952Z [email protected] <0.1353.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.289661Z [email protected] <0.1356.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.290339Z [email protected] <0.1359.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.290973Z [email protected] <0.1362.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.291731Z [email protected] <0.1365.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.292422Z [email protected] <0.1368.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[error] 2018-02-09T15:06:16.292870Z [email protected] <0.1371.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
Starting CouchDB:
[info] 2018-02-09T14:54:12.420661Z [email protected] <0.7.0> -------- Application couch_log started on node '[email protected]'
[info] 2018-02-09T14:54:12.423091Z [email protected] <0.7.0> -------- Application folsom started on node '[email protected]'
[info] 2018-02-09T14:54:12.445498Z [email protected] <0.7.0> -------- Application couch_stats started on node '[email protected]'
[info] 2018-02-09T14:54:12.445617Z [email protected] <0.7.0> -------- Application khash started on node '[email protected]'
[info] 2018-02-09T14:54:12.450173Z [email protected] <0.7.0> -------- Application couch_event started on node '[email protected]'
[info] 2018-02-09T14:54:12.453404Z [email protected] <0.7.0> -------- Application ibrowse started on node '[email protected]'
[info] 2018-02-09T14:54:12.456050Z [email protected] <0.7.0> -------- Application ioq started on node '[email protected]'
[info] 2018-02-09T14:54:12.456173Z [email protected] <0.7.0> -------- Application mochiweb started on node '[email protected]'
[info] 2018-02-09T14:54:12.460566Z [email protected] <0.200.0> -------- Apache CouchDB 2.1.1 is starting.
[info] 2018-02-09T14:54:12.460656Z [email protected] <0.201.0> -------- Starting couch_sup
[info] 2018-02-09T14:54:12.523195Z [email protected] <0.200.0> -------- Apache CouchDB has started. Time to relax.
[info] 2018-02-09T14:54:12.523307Z [email protected] <0.200.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/
[info] 2018-02-09T14:54:12.523449Z [email protected] <0.7.0> -------- Application couch started on node '[email protected]'
[info] 2018-02-09T14:54:12.523601Z [email protected] <0.7.0> -------- Application ets_lru started on node '[email protected]'
[info] 2018-02-09T14:54:12.540555Z [email protected] <0.7.0> -------- Application rexi started on node '[email protected]'
[info] 2018-02-09T14:54:12.546300Z [email protected] <0.273.0> -------- Opening index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0
c18d4e574135a5b08"
[info] 2018-02-09T14:54:12.550110Z [email protected] <0.287.0> -------- Opening index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e57413
5a5b08"
[error] 2018-02-09T14:54:12.570392Z [email protected] emulator -------- Error in process <0.302.0> on node '[email protected]' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"myblogdb">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,l
oad_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_sh
ards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_o
pen,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}
[error] 2018-02-09T14:54:12.570593Z [email protected] <0.251.0> -------- Error opening view group `sofa` from database `shards/e0000000-ffffffff/myblogd
b.1518183029`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"myblogdb">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards
.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_
shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shar
ds.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_i
ndex_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{f
ile,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]
},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[
{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},
{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-02-09T14:54:12.576078Z [email protected] <0.331.0> -------- ** System running to use fully qualified hostnames **
** Hostname localhost is illegal **
[info] 2018-02-09T14:54:12.584026Z [email protected] <0.7.0> -------- Application mem3 started on node '[email protected]'
[info] 2018-02-09T14:54:12.584399Z [email protected] <0.7.0> -------- Application fabric started on node '[email protected]'
[info] 2018-02-09T14:54:12.605860Z [email protected] <0.7.0> -------- Application chttpd started on node '[email protected]'
[info] 2018-02-09T14:54:12.614513Z [email protected] <0.7.0> -------- Application couch_index started on node '[email protected]'
[info] 2018-02-09T14:54:12.614673Z [email protected] <0.7.0> -------- Application couch_mrview started on node '[email protected]'
[info] 2018-02-09T14:54:12.614809Z [email protected] <0.7.0> -------- Application couch_plugins started on node '[email protected]'
[notice] 2018-02-09T14:54:12.620187Z [email protected] <0.413.0> -------- Failed to ensure auth ddoc _users/_design/_auth exists for reason: read_failur
e
[error] 2018-02-09T14:54:12.620427Z [email protected] <0.251.0> -------- Error opening view group `_auth` from database `shards/40000000-5fffffff/_users
.1502869354`: {'EXIT',{{badmatch,{error,read_failure}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,ma
ybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_da
emon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_d
atabases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,
[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-02-09T14:54:12.640465Z [email protected] <0.251.0> -------- Error opening view group `_replicator` from database `shards/80000000-9fffffff/
_replicator.1502869354`: {'EXIT',{{badmatch,{error,read_failure}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compactio
n_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_co
mpaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_ser
ver,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fol
d_files2,8,[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[notice] 2018-02-09T14:54:12.652087Z [email protected] <0.88.0> -------- config: [features] scheduler set to true for reason nil
[info] 2018-02-09T14:54:12.672254Z [email protected] <0.7.0> -------- Application couch_replicator started on node '[email protected]'
[info] 2018-02-09T14:54:12.681741Z [email protected] <0.7.0> -------- Application couch_peruser started on node '[email protected]'
[info] 2018-02-09T14:54:12.694711Z [email protected] <0.7.0> -------- Application ddoc_cache started on node '[email protected]'
[info] 2018-02-09T14:54:12.711264Z [email protected] <0.7.0> -------- Application global_changes started on node '[email protected]'
[info] 2018-02-09T14:54:12.711453Z [email protected] <0.7.0> -------- Application jiffy started on node '[email protected]'
[info] 2018-02-09T14:54:12.717159Z [email protected] <0.7.0> -------- Application mango started on node '[email protected]'
[info] 2018-02-09T14:54:12.721969Z [email protected] <0.7.0> -------- Application setup started on node '[email protected]'
[info] 2018-02-09T14:54:12.722104Z [email protected] <0.7.0> -------- Application snappy started on node '[email protected]'
[notice] 2018-02-09T14:54:17.623987Z [email protected] <0.472.0> -------- couch_replicator_clustering : cluster stable
[notice] 2018-02-09T14:54:17.626514Z [email protected] <0.617.0> -------- Started replicator db changes listener <0.835.0>
[notice] 2018-02-09T14:54:27.540971Z [email protected] <0.268.0> -------- rexi_buffer : cluster stable
[notice] 2018-02-09T14:54:27.541041Z [email protected] <0.265.0> -------- rexi_server : cluster stable
Listing DBs via CouchDB's Fauxton UI:
[notice] 2018-02-09T15:02:08.068878Z [email protected] <0.1176.0> 1558dcd7d9 www.example.com:5984 24.134.87.25 admin GET /_utils/index.html 304 ok 1
6
[notice] 2018-02-09T15:02:08.126185Z [email protected] <0.1176.0> 2e1ccbfb10 www.example.com:5984 24.134.87.25 admin GET /_utils/dashboard.assets/cs
s/styles.25bdeac6d2ce92b30461.css 304 ok 1
[notice] 2018-02-09T15:02:08.673483Z [email protected] <0.1176.0> a93b41dba2 www.example.com:5984 24.134.87.25 admin GET / 200 ok 4
[notice] 2018-02-09T15:02:08.738009Z [email protected] <0.1177.0> fe8dfd9e5c www.example.com:5984 24.134.87.25 admin GET /_session 200 ok 0
[notice] 2018-02-09T15:02:08.892143Z [email protected] <0.1177.0> a66af45255 www.example.com:5984 24.134.87.25 admin GET /_scheduler/jobs 200 ok 5
[notice] 2018-02-09T15:02:08.901744Z [email protected] <0.1176.0> e114044236 www.example.com:5984 24.134.87.25 admin GET /_all_dbs?_=1518188528333 2
00 ok 9
[notice] 2018-02-09T15:02:08.986915Z [email protected] <0.1178.0> b69c8dce91 www.example.com:5984 24.134.87.25 admin GET /_all_dbs?startkey=%22%22&e
ndkey=%22%E9%A6%99%22&limit=30&_=1518188528334 200 ok 2
[error] 2018-02-09T15:02:09.049365Z [email protected] <0.1176.0> be6ce99b2d req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.049725Z [email protected] <0.1176.0> be6ce99b2d www.example.com:5984 24.134.87.25 undefined GET /_replicator?_=15181885
28335 500 ok 1
[error] 2018-02-09T15:02:09.056335Z [email protected] <0.1177.0> d07dad801b req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.056610Z [email protected] <0.1177.0> d07dad801b www.example.com:5984 24.134.87.25 undefined GET /_users?_=1518188528336
500 ok 1
[error] 2018-02-09T15:02:09.080553Z [email protected] <0.1178.0> 81dbed0cf9 req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.080854Z [email protected] <0.1178.0> 81dbed0cf9 www.example.com:5984 24.134.87.25 undefined GET /functions?_=1518188528
340 500 ok 1
[error] 2018-02-09T15:02:09.099261Z [email protected] <0.1192.0> 856bef3f1a req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.099639Z [email protected] <0.1192.0> 856bef3f1a www.example.com:5984 24.134.87.25 undefined GET /attribute-groups?_=151
8188528337 500 ok 2
[notice] 2018-02-09T15:02:09.105335Z [email protected] <0.1176.0> ec86ef466a www.example.com:5984 24.134.87.25 admin GET /myblogdb?_=1518188528341 2
00 ok 8
[error] 2018-02-09T15:02:09.182457Z [email protected] <0.1178.0> 07cd60fd59 req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.182864Z [email protected] <0.1178.0> 07cd60fd59 www.example.com:5984 24.134.87.25 undefined GET /products?_=15181885283
43 500 ok 3
[error] 2018-02-09T15:02:09.183128Z [email protected] <0.1198.0> 9965ad9d2d req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
p_do_apply/3 L240">>]
[notice] 2018-02-09T15:02:09.183458Z [email protected] <0.1198.0> 9965ad9d2d www.example.com:5984 24.134.87.25 undefined GET /definitions?_=15181885
28339 500 ok 3
[error] 2018-02-09T15:02:09.183676Z [email protected] <0.1192.0> c29d349706 req_err(1021143351) internal_server_error : No DB shards could be opened.
[<<"fabric_util:get_shard/4 L185">>,<<"fabric:get_security/2 L146">>,<<"chttpd_auth_request:db_authorization_check/1 L91">>,<<"chttpd_auth_request:a
uthorize_request/1 L19">>,<<"chttpd:process_request/1 L293">>,<<"chttpd:handle_request_int/1 L231">>,<<"mochiweb_http:headers/6 L91">>,<<"proc_lib:init_
The error was caused by update to CouchDB 2.1.1
. It's explained on the CouchDB website that the new version brings breaking changes and a line has to be modified in the config file.
So I had to change this file:
$ sudo vim /opt/couchdb/etc/vm.args
-name couchdb@localhost
changed to -name [email protected]
Don't forget to move the DB shards to the new node and restart CouchDB.
This script is useful for migrating the DB shards to the new node:
#!/bin/bash
curl "http://admin:[email protected]:5986/_dbs/$1" > $1.json
cat $1.json | sed s/$2/$3/g > $1_edited.json
curl -X PUT "http://admin:[email protected]:5986/_dbs/$1" -d @$1_edited.json