I have deployed Milvus cluster 2.2.2 in k8s and after some hardware problem two of my three pulsar bookies have their pv/pvc corrupted and failing to start.
I see there is 50 ledgers underreplicated in the healthy bookie1 but when I try to execute:
bookie1# booker shell recover bookie2(failed)
bookie1# booker shell recover bookie3(failed)
there is errors.
So my question: I have loaded the data completely few months ago and I see it in the S3/Minio storage, but can I delete the ledgers and start the bookies from 0, will this affect data availability if the data inside the bookies is lost?
Error 1:
09:43:22.800 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There
are directories without a cookie, and this is neither a new
environment, nor is storage expansion enabled. Empty directories are
[/pulsar/data/bookkeeper/journal/current]
09:43:22.800 [main] INFO
org.apache.bookkeeper.proto.BookieNettyServer - Shutting down
BookieNettyServer
09:43:22.829 [main] ERROR org.apache.bookkeeper.server.Main - Failed
to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:494) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:273) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:731) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
From another thread: If all the pulsar messages were consumed by milvus, then you can relaunch a new/clean pulsar service and stop the old pulsar service. Change the pulsar address in milvus.yaml to the new pulsar service and restart the milvus cluster. Milvus will work well with the new pulsar service.