Postgresql 9.5 alter table gets lock timeout without other long running queries

Question

I was trying to execute alter table mydb.r_group drop constraint test; However it took more than 2 mins to get a exclusive lock. The table is busy table with many processes querying against it. (Processes use jdbc to access the db)

I tried to find a long running query using following query statement but couldn't find any except some with idle state.

SELECT
  pid,
  now() - pg_stat_activity.query_start AS duration,
  query,
  state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';

I also tried to find the query which is blocking the altering table query using following query statement but this query was also blocked so couldn't get the data.

SELECT blocked_locks.pid     AS blocked_pid,
         blocked_activity.usename  AS blocked_user,
         blocking_locks.pid     AS blocking_pid,
         blocking_activity.usename AS blocking_user,
         blocked_activity.query    AS blocked_statement,
         blocking_activity.query   AS current_statement_in_blocking_process
   FROM  pg_catalog.pg_locks         blocked_locks
    JOIN pg_catalog.pg_stat_activity blocked_activity  ON blocked_activity.pid = blocked_locks.pid
    JOIN pg_catalog.pg_locks         blocking_locks 
        ON blocking_locks.locktype = blocked_locks.locktype
        AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
        AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
        AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
        AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
        AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
        AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
        AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
        AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
        AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
        AND blocking_locks.pid != blocked_locks.pid

    JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
   WHERE NOT blocked_locks.GRANTED;

Does anyone know what other steps I can take to find root cause?

Answer 1

First, try to find the object ID of the table in question:

SELECT tableoid
FROM mydb.r_group
LIMIT 1;

Then find all the locks on this table:

SELECT pid, mode, granted
FROM pg_locks
WHERE database = current_database()
  AND locktype = 'relation'
  AND relation = <the number from above>;

Then you have a list of all sessions that would block the ALTER TABLE.

Look for these pids in pg_stat_activity to see what they are doing. Particularly interesting is xact_start, the time when the active transaction started.

If you cannot find the blocking process in pg_stat_activity, it could also be a background process, most likely an anti-wraparound autovacuum worker. Look for the blocking process ID using ps on the shell.