TSM file xxx size less than index offset xxx

When I upgraded the container, an OOM problem occurred, and then I restricted the resources of the container.

as follow:

docker stop container_name
docker rm container_name

Pull up a new container based on the original data，as follow:

docker run -d -p 1234:8902 -v /xxx/path/:/var/lib/cnosdb/  --name container_name cnosdb/cnosdb:community-2.3.5-nightly  bash -c "cnosdb run -c 8 -m 32 --config /etc/cnosdb/cnosdb.conf -M singleton"

Panic directly after the container is pulled up:

2024-02-28T08:23:05.674986779Z  INFO meta::store::state_machine: METADATA WRITE(ver: 262227): /cluster_xxx/data_nodes_metrics/1001 :{"id":1001,"disk_free":387355713536,"time":1709108585,"status":"Healthy"}
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: ReadTsm { source: Invalid { reason: "TSM file (23111) size less than index offset(18360612730836091900)" } }
Location: /home/runner/work/cnosdb/cnosdb/tskv/src/kvcore.rs:175

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 9 frames hidden ⋮
  10: tskv::kvcore::TsKv::open::{{closure}}::ha75eec53cec8f8f5
      at <unknown source file>:<unknown line>
  11: cnosdb::server::ServiceBuilder::create_tskv::{{closure}}::h8bba0d1c208792ea
      at <unknown source file>:<unknown line>
  12: cnosdb::server::ServiceBuilder::build_query_storage::{{closure}}::hd49aa0ee08fb1e12
      at <unknown source file>:<unknown line>
  13: cnosdb::main::{{closure}}::h6bfa31d83459855d
      at <unknown source file>:<unknown line>
  14: cnosdb::main::hd750d6269300f10a
      at <unknown source file>:<unknown line>
  15: std::sys_common::backtrace::__rust_begin_short_backtrace::h8e567e0f429809ee
      at <unknown source file>:<unknown line>
  16: std::rt::lang_start::{{closure}}::hafc184a31577b26e
      at <unknown source file>:<unknown line>
  17: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h99b789974d08aea6
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/core/src/ops/function.rs:287
  18: std::panicking::try::do_call::hb3a6e017c92fe51c
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/panicking.rs:487
  19: std::panicking::try::h27c8d1fd4a8ceb7a
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/panicking.rs:451
  20: std::panic::catch_unwind::h64d7627f14037ceb
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/panic.rs:140
  21: std::rt::lang_start_internal::{{closure}}::h39c5774e59f5ac41
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/rt.rs:148
  22: std::panicking::try::do_call::haccdca37196c3803
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/panicking.rs:487
  23: std::panicking::try::h22004d35bf8e800a
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/panicking.rs:451
  24: std::panic::catch_unwind::h86f73b299f752c18
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/panic.rs:140
  25: std::rt::lang_start_internal::he600f6bbce69911f
      at /rustc/39f2657d1101b50f9b71ae460b762d330cc8426b/library/std/src/rt.rs:148
  26: main<unknown>
      at <unknown source file>:<unknown line>
  27: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
  28: _start<unknown>
      at /opt/zig-linux-x86_64-0.11.0/lib/libc/glibc/sysdeps/x86_64/start-2.33.S:120

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.

What's going on? Has anyone encountered this error?

Solution

the cnsodb office github PR fix the problem, I retest the above steps.