I have shutdown every web services using Cassandra.
I have shutdown any ETL using Cassandra.
The last domain-level table compaction is from yesterday (2021-11-18T15:47:00.822). Since then, only compactions on system tables have occured :
Compaction History:
id keyspace_name columnfamily_name compacted_at bytes_in bytes_out rows_merged
c0f4b1e0-4917-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-19T10:04:51.198 78314 19505 {1:12, 4:601}
5cd3e350-490f-11ec-bf5a-0d5dfeeee6e2 system size_estimates 2021-11-19T09:04:47.237 115889 26314 {4:6}
9ba752d0-48fe-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-19T07:04:51.197 77987 19558 {1:12, 4:601}
3786d260-48f6-11ec-bf5a-0d5dfeeee6e2 system size_estimates 2021-11-19T06:04:47.238 115994 26169 {4:6}
765a41e0-48e5-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-19T04:04:51.198 77853 19531 {1:8, 4:601}
12399a60-48dd-11ec-bf5a-0d5dfeeee6e2 system size_estimates 2021-11-19T03:04:47.238 115978 26290 {4:6}
510cbbc0-48cc-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-19T01:04:51.196 78419 19595 {1:12, 4:601}
ecec1440-48c3-11ec-bf5a-0d5dfeeee6e2 system size_estimates 2021-11-19T00:04:47.236 115838 26175 {4:6}
2bbf83c0-48b3-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-18T22:04:51.196 77380 19566 {1:12, 4:601}
c79edc40-48aa-11ec-bf5a-0d5dfeeee6e2 system size_estimates 2021-11-18T21:04:47.236 116007 26208 {4:6}
06735d30-489a-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-18T19:04:51.203 76300 19101 {1:9, 2:3, 3:2, 4:599}
a2517d30-4891-11ec-bf5a-0d5dfeeee6e2 system size_estimates 2021-11-18T18:04:47.235 115858 26258 {4:6}
e3e5a870-4882-11ec-bf5a-0d5dfeeee6e2 system_distributed repair_history 2021-11-18T16:19:14.807 5220983 5232639 {1:49, 2:1, 3:2}
e10c5ba0-4880-11ec-bf5a-0d5dfeeee6e2 system sstable_activity 2021-11-18T16:04:51.034 75302 19166 {1:46, 2:33, 3:50, 4:549}
But still, the Cassandra cluster seems to have high garbadge collector activity :
WARN [Service Thread] 2021-11-18 19:14:17,736 GCInspector.java:283 - ParNew GC in 1073ms. CMS Old Gen: 13461870544 -> 13461916520; Par Eden Space: 1716774456 -> 0; Par Survivor Space: 13116112 -> 57443048
WARN [Service Thread] 2021-11-18 19:14:19,116 GCInspector.java:283 - ParNew GC in 1070ms. CMS Old Gen: 13461916520 -> 13461979400; Par Eden Space: 1714728464 -> 0; Par Survivor Space: 57443048 -> 37282896
WARN [Service Thread] 2021-11-18 19:14:20,466 GCInspector.java:283 - ParNew GC in 1070ms. CMS Old Gen: 13461979400 -> 13462018112; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 37282896 -> 17129408
WARN [Service Thread] 2021-11-18 19:14:21,816 GCInspector.java:283 - ParNew GC in 1070ms. CMS Old Gen: 13462018112 -> 13462045144; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 17129408 -> 39569800
WARN [Service Thread] 2021-11-18 19:14:23,164 GCInspector.java:283 - ParNew GC in 1071ms. CMS Old Gen: 13462045144 -> 13462076376; Par Eden Space: 1717080600 -> 0; Par Survivor Space: 39569800 -> 26910864
WARN [Service Thread] 2021-11-18 19:14:24,524 GCInspector.java:283 - ParNew GC in 1071ms. CMS Old Gen: 13462076376 -> 13462113800; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 26910864 -> 36179936
WARN [Service Thread] 2021-11-18 19:14:25,869 GCInspector.java:283 - ParNew GC in 1069ms. CMS Old Gen: 13462113800 -> 13462137272; Par Eden Space: 1717733528 -> 0; Par Survivor Space: 36179936 -> 30547296
WARN [Service Thread] 2021-11-18 19:14:27,230 GCInspector.java:283 - ParNew GC in 1069ms. CMS Old Gen: 13462137272 -> 13462163256; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 30547296 -> 33604888
WARN [Service Thread] 2021-11-18 19:14:28,574 GCInspector.java:283 - ParNew GC in 1073ms. CMS Old Gen: 13462163256 -> 13462187040; Par Eden Space: 1715261960 -> 0; Par Survivor Space: 33604888 -> 28871272
WARN [Service Thread] 2021-11-18 19:14:29,946 GCInspector.java:283 - ParNew GC in 1069ms. CMS Old Gen: 13462187040 -> 13462216656; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 28871272 -> 37053656
WARN [Service Thread] 2021-11-18 19:14:31,328 GCInspector.java:283 - ParNew GC in 1070ms. CMS Old Gen: 13462216656 -> 13462237976; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 37053656 -> 23342920
WARN [Service Thread] 2021-11-18 19:14:32,743 GCInspector.java:283 - ParNew GC in 1071ms. CMS Old Gen: 13462237976 -> 13462278432; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 23342920 -> 21896200
WARN [Service Thread] 2021-11-18 19:14:34,206 GCInspector.java:283 - ParNew GC in 1071ms. CMS Old Gen: 13462278432 -> 13462343008; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 21896200 -> 20168000
WARN [Service Thread] 2021-11-18 19:14:35,696 GCInspector.java:283 - ParNew GC in 1070ms. CMS Old Gen: 13462343008 -> 13462438104; Par Eden Space: 1717981344 -> 0; Par Survivor Space: 20168000 -> 29781856
WARN [Service Thread] 2021-11-18 19:14:37,115 GCInspector.java:283 - ParNew GC in 1072ms. CMS Old Gen: 13462438104 -> 13462532752; Par Eden Space: 1717180224 -> 0; Par Survivor Space: 29781856 -> 15873392
...
WARN [Service Thread] 2021-11-19 10:34:10,753 GCInspector.java:283 - ParNew GC in 1081ms. CMS Old Gen: 21366236160 -> 22047866248; Par Eden Space: 1692018856 -> 0;
WARN [Service Thread] 2021-11-19 10:34:11,961 GCInspector.java:283 - ParNew GC in 1080ms. CMS Old Gen: 22047866248 -> 22711292400; Par Eden Space: 1718091776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:13,190 GCInspector.java:283 - ParNew GC in 1082ms. CMS Old Gen: 22711292400 -> 23322328920; Par Eden Space: 1718091776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:14,414 GCInspector.java:283 - ParNew GC in 1076ms. CMS Old Gen: 23322328920 -> 23938244632; Par Eden Space: 1710429576 -> 0;
WARN [Service Thread] 2021-11-19 10:34:15,628 GCInspector.java:283 - ParNew GC in 1083ms. CMS Old Gen: 23938244632 -> 24531937352; Par Eden Space: 1718091776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:17,014 GCInspector.java:283 - ParNew GC in 1079ms. CMS Old Gen: 24531937352 -> 25077213400; Par Eden Space: 1718091776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:18,219 GCInspector.java:283 - ParNew GC in 1082ms. CMS Old Gen: 25077213400 -> 25634088464; Par Eden Space: 1689565160 -> 0;
WARN [Service Thread] 2021-11-19 10:34:19,423 GCInspector.java:283 - ParNew GC in 1085ms. CMS Old Gen: 25634088464 -> 26549529728; Par Eden Space: 1714413672 -> 0;
WARN [Service Thread] 2021-11-19 10:34:20,656 GCInspector.java:283 - ParNew GC in 1088ms. CMS Old Gen: 26549529728 -> 27291610392; Par Eden Space: 1707391776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:21,951 GCInspector.java:283 - ParNew GC in 1080ms. CMS Old Gen: 27290538440 -> 27875777144; Par Eden Space: 1718054488 -> 0;
WARN [Service Thread] 2021-11-19 10:34:23,171 GCInspector.java:283 - ParNew GC in 1082ms. CMS Old Gen: 27788203256 -> 28539500224; Par Eden Space: 1717476200 -> 0;
WARN [Service Thread] 2021-11-19 10:34:24,404 GCInspector.java:283 - ParNew GC in 1084ms. CMS Old Gen: 28313984168 -> 28943208880; Par Eden Space: 1690698568 -> 0;
WARN [Service Thread] 2021-11-19 10:34:25,674 GCInspector.java:283 - ParNew GC in 1079ms. CMS Old Gen: 28649641192 -> 29197701416; Par Eden Space: 1667998792 -> 0;
WARN [Service Thread] 2021-11-19 10:34:26,911 GCInspector.java:283 - ParNew GC in 1075ms. CMS Old Gen: 28973128960 -> 29454364992; Par Eden Space: 1718091776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:28,137 GCInspector.java:283 - ParNew GC in 1079ms. CMS Old Gen: 29252627776 -> 29846619728; Par Eden Space: 1718091776 -> 0;
WARN [Service Thread] 2021-11-19 10:34:29,345 GCInspector.java:283 - ParNew GC in 1083ms. CMS Old Gen: 28703301152 -> 29313662360; Par Eden Space: 1684884992 -> 0;
How it is possible ?
Thank you
A contractor of us set the GC to CMS, even if the memory allocated to the heap was > 32 GB. That's why we saw such messages, and long GC pauses.
Turning GC to G1GC solve the issue