We have been attempting to backfill 6 months of data into VictoriaMetrics. VictoriaMetrics is set up in a cluster with 2 vmstorage nodes and 1 vminsert and 1 vmselect instance. Everything went smoothly until data stopped showing up grafana after a specific date. Our import script was still running without an error. Checking the logs for vminsert and 2 vmstorage nodes there's no error. Then data started appearing at the end of the range.
So, a big chunk of the data is missing on Grafana. But sometimes zooming in reveals some of the data that doesn't show up on a higher time range. After double-checking the vmstorage logs seems like partitions are not the same on each node. and one of the nodes is missing the latest partition.
Logs from Storage Node 1
2022-12-02T00:21:22.521Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_06" with smallPartsPath="/storage/data/small/2022_06", bigPartsPath="/storage/data/big/2022_06"
2022-12-02T00:21:24.259Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_06" has been created
2022-12-02T13:56:23.722Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_07" with smallPartsPath="/storage/data/small/2022_07", bigPartsPath="/storage/data/big/2022_07"
2022-12-02T13:56:26.533Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_07" has been created
2022-12-03T01:24:45.721Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_08" with smallPartsPath="/storage/data/small/2022_08", bigPartsPath="/storage/data/big/2022_08"
2022-12-03T01:24:46.900Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_08" has been created
2022-12-03T17:57:02.525Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_09" with smallPartsPath="/storage/data/small/2022_09", bigPartsPath="/storage/data/big/2022_09"
2022-12-03T17:57:03.713Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_09" has been created
2022-12-04T07:08:51.722Z info VictoriaMetrics/lib/storage/partition.go:1305 merged 18251 rows across 18251 blocks in 37.328 seconds at 488 rows/sec to "/storage/data/small/2022_09/18251_18251_20220928180000.000_20220928180000.000_172D5A35BF48B011"; sizeBytes: 250837
2022-12-04T08:41:28.530Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_10" with smallPartsPath="/storage/data/small/2022_10", bigPartsPath="/storage/data/big/2022_10"
2022-12-04T08:41:31.022Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_10" has been created
2022-12-04T21:22:29.569Z info VictoriaMetrics/lib/mergeset/table.go:1027 merged 24870725 items across 28659 blocks in 37.205 seconds at 668480 items/sec to "/storage/indexdb/172CAC7059FB5EB3/24860954_28638_172CAC71E7BD8BD0"; sizeBytes: 331467849
2022-12-04T22:37:55.719Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_11" with smallPartsPath="/storage/data/small/2022_11", bigPartsPath="/storage/data/big/2022_11"
2022-12-04T22:37:56.406Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_11" has been created
2022-12-05T00:39:45.097Z info VictoriaMetrics/lib/storage/partition.go:1305 merged 199154 rows across 199154 blocks in 30.426 seconds at 6545 rows/sec to "/storage/data/small/2022_11/199154_113352_20221104180000.000_20221106180000.000_172DB81E235772F4"; sizeBytes: 1909359
2022-12-05T03:29:28.254Z info VictoriaMetrics/lib/storage/partition.go:1305 merged 211814 rows across 211814 blocks in 30.355 seconds at 6977 rows/sec to "/storage/data/small/2022_11/211814_114004_20221108180000.000_20221110180000.000_172DB81E235773BF"; sizeBytes: 1891824
2022-12-05T16:34:53.329Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_12" with smallPartsPath="/storage/data/small/2022_12", bigPartsPath="/storage/data/big/2022_12"
2022-12-05T16:34:54.324Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_12" has been created
Logs from Storage Node 2
2022-12-02T00:21:22.523Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_06" with smallPartsPath="/storage/data/small/2022_06", bigPartsPath="/storage/data/big/2022_06"
2022-12-02T00:21:23.985Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_06" has been created
2022-12-02T13:56:23.727Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_07" with smallPartsPath="/storage/data/small/2022_07", bigPartsPath="/storage/data/big/2022_07"
2022-12-02T13:56:26.533Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_07" has been created
2022-12-03T01:24:45.724Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_08" with smallPartsPath="/storage/data/small/2022_08", bigPartsPath="/storage/data/big/2022_08"
2022-12-03T01:24:46.900Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_08" has been created
2022-12-03T17:57:02.517Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_09" with smallPartsPath="/storage/data/small/2022_09", bigPartsPath="/storage/data/big/2022_09"
2022-12-03T17:57:03.713Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_09" has been created
2022-12-04T07:08:23.316Z info VictoriaMetrics/lib/storage/partition.go:1305 merged 23345 rows across 23345 blocks in 33.611 seconds at 694 rows/sec to "/storage/data/small/2022_09/23345_23345_20220928180000.000_20220928180000.000_172D5A35BF445C06"; sizeBytes: 338817
2022-12-04T08:41:28.524Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_10" with smallPartsPath="/storage/data/small/2022_10", bigPartsPath="/storage/data/big/2022_10"
2022-12-04T08:41:31.022Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_10" has been created
2022-12-04T22:37:55.725Z info VictoriaMetrics/lib/storage/partition.go:200 creating a partition "2022_11" with smallPartsPath="/storage/data/small/2022_11", bigPartsPath="/storage/data/big/2022_11"
2022-12-04T22:37:56.406Z info VictoriaMetrics/lib/storage/partition.go:216 partition "2022_11" has been created
It is likely you need to reset response cache at every vmselect node after the backfilling process is complete. See these docs for details.