Search code examples
postgresqlvacuumautovacuum

Autovacuum struggles with 2000 tables reaching autovacuum_freeze_max_age simultaneously


My goal is to keep autovacuum very responsive to current events, mainly so that tables are always properly analyzed. At the same time, I have huge batches of ~2000 tables that need wraparound-prevention vacuum. These batches delay autovacuum analyze.

What is the best way to solve this (while keeping the app the same)? My current plan is to write a script that does the wraparound-prevention so that autovacuum is free for current events (updates, deletes) so for vacuum and analyze. Is there a better way? Only by changing autovacuum settings maybe? I could just hardcode all analzye so that I do not rely on autovacuum for correct statistics, but that does not see the best way?

I have this problem as my app does the following:

  • Day 1: Insert data into 2000 empty tables (mainly partitions of tables) using many transaction IDs
  • Day 1, but a few minutes later: Read data from these 2000 tables (so statistics need to be correct, otherwise the execution plans are incorect, so analyze must have run in between)
  • Day 2: Insert data into 2000 new empty tables, again using many transaction IDs (this will make that the 2000 tables filled the day before need wraparound-protection vacuum)
  • Day 2, a few minutes later: Read data from the 2000 new tables - Now the problem happens as autovacuum is busy with wraparound-protection of the old 2000 tables and statistics on the new 2000 tables are incorrect

Solution

  • I could just hardcode all analzye so that I do not rely on autovacuum for correct statistics, but that does not see the best way?

    Normally I would agree that that is not the best way. But since you need the ANALYZE to happen within the span of "a few minutes later", I would say it probably is the best way.

    That doesn't mean you can't also issue preemptive 'vacuum freeze' at some convenient time, like over night, or over a weekend.

    Are you really using 200,000,000 transactions every day?