Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

M3hm3t/debug upgrade test #7862

Draft
wants to merge 3 commits into
base: naisila/test-upgrade
Choose a base branch
from
Draft

M3hm3t/debug upgrade test #7862

wants to merge 3 commits into from
+82 −2

Conversation

m3hm3t
Copy link
Contributor

@m3hm3t m3hm3t commented Jan 27, 2025

DESCRIPTION: PR description that will go into the change log, up to 78 characters

@m3hm3t m3hm3t changed the base branch from release-13.0 to naisila/test-upgrade January 27, 2025 11:03
@codecov Codecov
Copy link

codecov bot commented Jan 27, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 85.43%. Comparing base (c473445) to head (0115e7e).

❌ Your patch check has failed because the patch coverage (50.00%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (85.43%) is below the target coverage (87.50%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files 
@@                   Coverage Diff                    @@
##           naisila/test-upgrade    #7862      +/-   ##
========================================================
- Coverage                 89.42%   85.43%   -3.99%     
========================================================
  Files                       276      276              
  Lines                     60071    60069       -2     
  Branches                   7524     7526       +2     
========================================================
- Hits                      53718    51322    -2396     
- Misses                     4202     6318    +2116     
- Partials                   2151     2429     +278

@m3hm3t
Copy link
Contributor Author

m3hm3t commented Jan 27, 2025 

         info         
----------------------
 === BEFORE ALTER ===
(1 row)

  pid  |                               query                               | state  | wait_event_type |     wait_event      
-------+-------------------------------------------------------------------+--------+-----------------+---------------------
 95589 |                                                                   |        | Activity        | CheckpointerMain
 95590 |                                                                   |        | Activity        | BgWriterMain
 95592 |                                                                   |        | Activity        | WalWriterMain
 95593 |                                                                   |        | Activity        | AutoVacuumMain
 95594 |                                                                   |        | Activity        | LogicalLauncherMain
 95617 |                                                                  +| active |                 | 
       |             -- Show a marker in output                           +|        |                 | 
       |             SELECT '=== BEFORE ALTER ===' AS info;               +|        |                 | 
       |                                                                  +|        |                 | 
       |             -- Show current queries                              +|        |                 | 
       |             SELECT pid, query, state, wait_event_type, wait_event+|        |                 | 
       |             FROM pg_stat_activity                                +|        |                 | 
       |             ORDER BY pid;                                        +|        |                 | 
       |                                                                  +|        |                 | 
       |             -- Show current locks joined to pg_stat_activity     +|        |                 | 
       |             SELECT l.locktype,                                   +|        |                 | 
       |                 l.relation::regclass AS relname,                 +|        |                 | 
       |                 l.mode,                                          +|        |                 | 
       |                 l.granted,                                       +|        |                 | 
       |                 a.pid,                                           +|        |                 | 
       |                 a.query AS current_query                         +|        |                 | 
       |             FROM pg_locks l                                      +|        |                 | 
       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +|        |                 | 
       |             ORDER BY l.relation, l.pid;                          +|        |                 | 
       |                                                                  +|        |                 | 
       |             -- The actual extension update                       +|        |                 | 
       |             ALTER EXTENSION citus UPDATE;                        +|        |                 | 
       |                                                                   |        |                 | 
 95618 |                                                                   |        |                 | 
(7 rows)

  locktype  |          relname          |           mode           | granted |  pid  |                           current_query                           
------------+---------------------------+--------------------------+---------+-------+-------------------------------------------------------------------
 relation   | pg_authid                 | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_database               | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_database_datname_index | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_database_oid_index     | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_authid_rolname_index   | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_authid_oid_index       | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_locks                  | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_stat_activity          | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_dist_partition         | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_dist_node              | AccessShareLock          | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 relation   | pg_dist_node              | AccessShareLock          | t       | 95618 | 
 virtualxid |                           | ExclusiveLock            | t       | 95617 |                                                                  +
            |                           |                          |         |       |             -- Show a marker in output                           +
            |                           |                          |         |       |             SELECT '=== BEFORE ALTER ===' AS info;               +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current queries                              +
            |                           |                          |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                          |         |       |             FROM pg_stat_activity                                +
            |                           |                          |         |       |             ORDER BY pid;                                        +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                          |         |       |             SELECT l.locktype,                                   +
            |                           |                          |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                          |         |       |                 l.mode,                                          +
            |                           |                          |         |       |                 l.granted,                                       +
            |                           |                          |         |       |                 a.pid,                                           +
            |                           |                          |         |       |                 a.query AS current_query                         +
            |                           |                          |         |       |             FROM pg_locks l                                      +
            |                           |                          |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                          |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                          |         |       |                                                                  +
            |                           |                          |         |       |             -- The actual extension update                       +
            |                           |                          |         |       |             ALTER EXTENSION citus UPDATE;                        +
            |                           |                          |         |       |         
 advisory   |                           | ShareUpdateExclusiveLock | t       | 95618 | 
 object     |                           | AccessShareLock          | t       | 95618 | 
 virtualxid |                           | ExclusiveLock            | t       | 95618 | 
(15 rows)

ERROR:  deadlock detected
DETAIL:  Process 95617 waits for AccessExclusiveLock on relation 16541 of database 5; blocked by process 95618.
Process 95618 waits for AccessShareLock on relation 16679 of database 5; blocked by process 95617.
HINT:  See server log for query details.
[ERROR] Failed to run 'ALTER EXTENSION citus UPDATE;' on port 57635
        psql returned: 1, None
        info         
---------------------
 === AFTER ALTER ===
(1 row)

  pid  |                                                                                                                  query                                                                                                                   | state  | wait_event_type |     wait_event      
-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-----------------+---------------------
 95589 |                                                                                                                                                                                                                                          |        | Activity        | CheckpointerMain
 95590 |                                                                                                                                                                                                                                          |        | Activity        | BgWriterMain
 95592 |                                                                                                                                                                                                                                          |        | Activity        | WalWriterMain
 95593 |                                                                                                                                                                                                                                          |        | Activity        | AutoVacuumMain
 95594 |                                                                                                                                                                                                                                          |        | Activity        | LogicalLauncherMain
 95618 |                                                                                                                                                                                                                                          |        | Extension       | Extension
 95626 | SELECT waiting_pid, waiting_node_id, waiting_transaction_num, waiting_transaction_stamp, blocking_pid, blocking_node_id, blocking_transaction_num, blocking_transaction_stamp, blocking_transaction_waiting FROM dump_local_wait_edges() | idle   | Client          | ClientRead
 95628 | SELECT waiting_pid, waiting_node_id, waiting_transaction_num, waiting_transaction_stamp, blocking_pid, blocking_node_id, blocking_transaction_num, blocking_transaction_stamp, blocking_transaction_waiting FROM dump_local_wait_edges() | idle   | Client          | ClientRead
 95643 | SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_0\_%' and database = current_database()                                                                                                                                         | idle   | Client          | ClientRead
 95645 |                                                                                                                                                                                                                                         +| active |                 | 
       |             -- Show a marker in output                                                                                                                                                                                                  +|        |                 | 
       |             SELECT '=== AFTER ALTER ===' AS info;                                                                                                                                                                                       +|        |                 | 
       |                                                                                                                                                                                                                                         +|        |                 | 
       |             -- Show current queries                                                                                                                                                                                                     +|        |                 | 
       |             SELECT pid, query, state, wait_event_type, wait_event                                                                                                                                                                       +|        |                 | 
       |             FROM pg_stat_activity                                                                                                                                                                                                       +|        |                 | 
       |             ORDER BY pid;                                                                                                                                                                                                               +|        |                 | 
       |                                                                                                                                                                                                                                         +|        |                 | 
       |             -- Show current locks joined to pg_stat_activity                                                                                                                                                                            +|        |                 | 
       |             SELECT l.locktype,                                                                                                                                                                                                          +|        |                 | 
       |                 l.relation::regclass AS relname,                                                                                                                                                                                        +|        |                 | 
       |                 l.mode,                                                                                                                                                                                                                 +|        |                 | 
       |                 l.granted,                                                                                                                                                                                                              +|        |                 | 
       |                 a.pid,                                                                                                                                                                                                                  +|        |                 | 
       |                 a.query AS current_query                                                                                                                                                                                                +|        |                 | 
       |             FROM pg_locks l                                                                                                                                                                                                             +|        |                 | 
       |             JOIN pg_stat_activity a ON (l.pid = a.pid)                                                                                                                                                                                  +|        |                 | 
       |             ORDER BY l.relation, l.pid;                                                                                                                                                                                                 +|        |                 | 
       |                                                                                                                                                                                                                                          |        |                 | 
(10 rows)

  locktype  |          relname          |      mode       | granted |  pid  |                           current_query                           
------------+---------------------------+-----------------+---------+-------+-------------------------------------------------------------------
 relation   | pg_authid                 | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_database               | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_database_datname_index | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_database_oid_index     | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_authid_rolname_index   | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_authid_oid_index       | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_locks                  | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_stat_activity          | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_dist_partition         | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 relation   | pg_dist_node              | AccessShareLock | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
 virtualxid |                           | ExclusiveLock   | t       | 95645 |                                                                  +
            |                           |                 |         |       |             -- Show a marker in output                           +
            |                           |                 |         |       |             SELECT '=== AFTER ALTER ===' AS info;                +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current queries                              +
            |                           |                 |         |       |             SELECT pid, query, state, wait_event_type, wait_event+
            |                           |                 |         |       |             FROM pg_stat_activity                                +
            |                           |                 |         |       |             ORDER BY pid;                                        +
            |                           |                 |         |       |                                                                  +
            |                           |                 |         |       |             -- Show current locks joined to pg_stat_activity     +
            |                           |                 |         |       |             SELECT l.locktype,                                   +
            |                           |                 |         |       |                 l.relation::regclass AS relname,                 +
            |                           |                 |         |       |                 l.mode,                                          +
            |                           |                 |         |       |                 l.granted,                                       +
            |                           |                 |         |       |                 a.pid,                                           +
            |                           |                 |         |       |                 a.query AS current_query                         +
            |                           |                 |         |       |             FROM pg_locks l                                      +
            |                           |                 |         |       |             JOIN pg_stat_activity a ON (l.pid = a.pid)           +
            |                           |                 |         |       |             ORDER BY l.relation, l.pid;                          +
            |                           |                 |         |       |         
(11 rows)

Traceback (most recent call last):
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 281, in <module>
    main(config)
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 47, in main
    run_citus_upgrade_tests(config, before_upgrade_schedule, after_upgrade_schedule)
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 66, in run_citus_upgrade_tests
    run_alter_citus(config.bindir, config.mixed_mode, config)
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 207, in run_alter_citus
    utils.psql(pg_path, port, alter_extension_sql)
  File "/workspaces/citus/src/test/regress/citus_tests/utils.py", line 8, in psql
    return subprocess.run(
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/citus/.pgenv/pgsql-15.10/bin/psql', '-U', 'postgres', '-p', '57635', '-c', "\n            -- Show a marker in output\n            SELECT '=== BEFORE ALTER ===' AS info;\n\n            -- Show current queries\n            SELECT pid, query, state, wait_event_type, wait_event\n            FROM pg_stat_activity\n            ORDER BY pid;\n\n            -- Show current locks joined to pg_stat_activity\n            SELECT l.locktype,\n                l.relation::regclass AS relname,\n                l.mode,\n                l.granted,\n                a.pid,\n                a.query AS current_query\n            FROM pg_locks l\n            JOIN pg_stat_activity a ON (l.pid = a.pid)\n            ORDER BY l.relation, l.pid;\n\n            -- The actual extension update\n            ALTER EXTENSION citus UPDATE;\n        ", '-P', 'pager=off', '--no-psqlrc']' returned non-zero exit status 1.
make: *** [Makefile:285: check-citus-upgrade-local] Error 1

Process 95617 is waiting for an AccessExclusiveLock on relation 16541, which is held by process 95618.
Process 95618 is waiting for an AccessShareLock on relation 16679, which is held by process 95617.

Process 95618 is running as the Citus Maintenance Daemon (state: Extension).
Process 95617 appears to be executing the ALTER EXTENSION citus UPDATE; statement, which requires an AccessExclusiveLock.

Locks are being held on key catalog tables:
pg_dist_partition
pg_dist_node
pg_dist_transaction

The ALTER EXTENSION command needs exclusive access to some Citus catalog tables to make updates, but the Citus Maintenance Daemon has locks on those same tables while doing its regular tasks.

@m3hm3t
Copy link
Contributor Author

m3hm3t commented Jan 27, 2025 

        info         
----------------------
 === BEFORE ALTER ===
(1 row)

 locktype |      relname      |           mode           | granted |                                              query                                              
----------+-------------------+--------------------------+---------+-------------------------------------------------------------------------------------------------
 relation | pg_locks          | AccessShareLock          | t       |                                                                                                +
          |                   |                          |         |             SELECT '=== BEFORE ALTER ===' AS info;                                             +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                          |         |             FROM pg_locks l                                                                    +
          |                   |                          |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                          |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                          |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             ALTER EXTENSION citus UPDATE;                                                      +
          |                   |                          |         |         
 relation | pg_stat_activity  | AccessShareLock          | t       |                                                                                                +
          |                   |                          |         |             SELECT '=== BEFORE ALTER ===' AS info;                                             +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                          |         |             FROM pg_locks l                                                                    +
          |                   |                          |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                          |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                          |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             ALTER EXTENSION citus UPDATE;                                                      +
          |                   |                          |         |         
 relation | pg_dist_partition | AccessShareLock          | t       |                                                                                                +
          |                   |                          |         |             SELECT '=== BEFORE ALTER ===' AS info;                                             +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                          |         |             FROM pg_locks l                                                                    +
          |                   |                          |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                          |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                          |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             ALTER EXTENSION citus UPDATE;                                                      +
          |                   |                          |         |         
 relation | pg_dist_node      | AccessShareLock          | t       |                                                                                                +
          |                   |                          |         |             SELECT '=== BEFORE ALTER ===' AS info;                                             +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                          |         |             FROM pg_locks l                                                                    +
          |                   |                          |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                          |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                          |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |                                                                                                +
          |                   |                          |         |             ALTER EXTENSION citus UPDATE;                                                      +
          |                   |                          |         |         
 relation | pg_dist_node      | AccessShareLock          | t       | 
 object   |                   | AccessShareLock          | t       | 
 advisory |                   | ShareUpdateExclusiveLock | t       | 
(7 rows)

ERROR:  deadlock detected
DETAIL:  Process 15447 waits for AccessExclusiveLock on relation 16541 of database 5; blocked by process 15448.
Process 15448 waits for AccessShareLock on relation 16679 of database 5; blocked by process 15447.
HINT:  See server log for query details.
[ERROR] Failed to run 'ALTER EXTENSION citus UPDATE;' on port 57635
        psql returned: 1, None
        info         
---------------------
 === AFTER ALTER ===
(1 row)

 locktype |      relname      |      mode       | granted |                                              query                                              
----------+-------------------+-----------------+---------+-------------------------------------------------------------------------------------------------
 relation | pg_locks          | AccessShareLock | t       |                                                                                                +
          |                   |                 |         |             SELECT '=== AFTER ALTER ===' AS info;                                              +
          |                   |                 |         |                                                                                                +
          |                   |                 |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                 |         |             FROM pg_locks l                                                                    +
          |                   |                 |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                 |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                 |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                 |         |         
 relation | pg_stat_activity  | AccessShareLock | t       |                                                                                                +
          |                   |                 |         |             SELECT '=== AFTER ALTER ===' AS info;                                              +
          |                   |                 |         |                                                                                                +
          |                   |                 |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                 |         |             FROM pg_locks l                                                                    +
          |                   |                 |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                 |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                 |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                 |         |         
 relation | pg_dist_partition | AccessShareLock | t       |                                                                                                +
          |                   |                 |         |             SELECT '=== AFTER ALTER ===' AS info;                                              +
          |                   |                 |         |                                                                                                +
          |                   |                 |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                 |         |             FROM pg_locks l                                                                    +
          |                   |                 |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                 |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                 |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                 |         |         
 relation | pg_dist_node      | AccessShareLock | t       |                                                                                                +
          |                   |                 |         |             SELECT '=== AFTER ALTER ===' AS info;                                              +
          |                   |                 |         |                                                                                                +
          |                   |                 |         |             SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query     +
          |                   |                 |         |             FROM pg_locks l                                                                    +
          |                   |                 |         |             LEFT JOIN pg_stat_activity a ON l.pid = a.pid                                      +
          |                   |                 |         |             WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())+
          |                   |                 |         |             ORDER BY l.relation, l.mode;                                                       +
          |                   |                 |         |         
(4 rows)

Traceback (most recent call last):
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 252, in <module>
    main(config)
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 47, in main
    run_citus_upgrade_tests(config, before_upgrade_schedule, after_upgrade_schedule)
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 66, in run_citus_upgrade_tests
    run_alter_citus(config.bindir, config.mixed_mode, config)
  File "/workspaces/citus/src/test/regress/citus_tests/upgrade/citus_upgrade_test.py", line 181, in run_alter_citus
    utils.psql(pg_path, port, alter_extension_sql)
  File "/workspaces/citus/src/test/regress/citus_tests/utils.py", line 8, in psql
    return subprocess.run(
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/citus/.pgenv/pgsql-15.10/bin/psql', '-U', 'postgres', '-p', '57635', '-c', "\n            SELECT '=== BEFORE ALTER ===' AS info;\n\n            SELECT l.locktype, l.relation::regclass AS relname, l.mode, l.granted, a.query\n            FROM pg_locks l\n            LEFT JOIN pg_stat_activity a ON l.pid = a.pid\n            WHERE l.database = (SELECT oid FROM pg_database WHERE datname = current_database())\n            ORDER BY l.relation, l.mode;\n\n\n            ALTER EXTENSION citus UPDATE;\n        ", '-P', 'pager=off', '--no-psqlrc']' returned non-zero exit status 1.
make: *** [Makefile:285: check-citus-upgrade-local] Error 1

It seems like the issue comes up because ALTER EXTENSION citus UPDATE; and other processes (probably the Citus Maintenance Daemon) are trying to access the same catalog tables (pg_dist_*) but are running into lock conflicts:

  • The ALTER EXTENSION command needs an AccessExclusiveLock to update the metadata.
  • At the same time, the daemon or other sessions are holding AccessShareLocks, which might be blocking the ALTER EXTENSION command from getting the lock it needs.

I’m guessing this has become more noticeable with PostgreSQL 15, maybe due to changes in locking behavior or deadlock detection, but I’m not 100% sure.

@m3hm3t
Copy link
Contributor Author

m3hm3t commented Jan 27, 2025 

[INFO] Attempt 1: Running 'ALTER EXTENSION citus UPDATE;' on port 57635
ERROR:  deadlock detected
DETAIL:  Process 28620 waits for AccessExclusiveLock on relation 16541 of database 5; blocked by process 28621.
Process 28621 waits for AccessShareLock on relation 16679 of database 5; blocked by process 28620.
HINT:  See server log for query details.
[ERROR] Attempt 1 failed on port 57635 with error: None
[INFO] Retrying after 2 seconds...
[INFO] Attempt 2: Running 'ALTER EXTENSION citus UPDATE;' on port 57635
ALTER EXTENSION
[INFO] Successfully ran 'ALTER EXTENSION citus UPDATE;' on port 57635
[INFO] Attempt 1: Running 'ALTER EXTENSION citus UPDATE;' on port 57636
ALTER EXTENSION
[INFO] Successfully ran 'ALTER EXTENSION citus UPDATE;' on port 57636
[INFO] Attempt 1: Running 'ALTER EXTENSION citus UPDATE;' on port 57637
ALTER EXTENSION
[INFO] Successfully ran 'ALTER EXTENSION citus UPDATE;' on port 57637
port:57635 citus version 13.0
port:57636 citus version 13.0
port:57637 citus version 13.0
(using postmaster on Unix socket, port 57635)

Added retries with an exponential backoff to handle temporary deadlocks in the run_alter_citus function. After waiting 2 seconds, the second try successfully ran ALTER EXTENSION citus UPDATE; on port 57635.

This points to the deadlock being a short-term issue, probably due to some lock contention between the ALTER EXTENSION command and other ongoing tasks.

@colm-mchugh
Copy link
Contributor

Debugging the ALTER EXTENSION citus UPDATE statement, I see that the Citus process utility hook (multi_ProcessUtility() in metadata_cache.c) calls CitusHasBeenLoaded() which in turn calls StartupCitusBackend(), which may start the maintenance daemon, and will if it has not been started. Is this expected for ALTER EXTENSION citus UPDATE @onurctirtir @naisila ?

This comment in StartupCitusBackend() suggests that ALTER EXTENSION citus UPDATE is treated the same way as CREATE EXTENSION citus but in my debug session the variable creating_extension was false so the if (creating_extension) statement did not apply. I just want to confirm that this is expected behavior for ALTER EXTENSION citus UPDATE ? If so we may have the maintenance daemon and the extension update processing running at the same time ?

Also, I didn't observe the issue in my debug session, the ALTER EXTENSION citus UPDATE statement was able to complete successfully, suggesting a timing issue similar to @m3hm3t 's observation above.

@emelsimsek
Copy link
Contributor

emelsimsek commented Jan 28, 2025 •
edited
Loading

creating_extension was false so the if (creating_extension) statement did not apply. I just want to confirm that this is expected behavior for ALTER EXTENSION citus UPDATE

https://github.com/postgres/postgres/blob/75eb9766ec201b62f264554019757716093e2a2f/src/include/commands/extension.h#L22 says

  • creating_extension is only true while running a CREATE EXTENSION or ALTER
  • EXTENSION UPDATE command.

However, looks like maintenance deamon only sporadically checks for this condition.

@m3hm3t
Copy link
Contributor Author

m3hm3t commented Jan 28, 2025

I tried disabling the Citus daemon for the ALTER EXTENSION command as an experiment. It seemed to prevent the deadlock, but it caused other issues that led to test failures.

image

Could you take a look as well? I’m not sure if this is the right approach to pursue.

/workspaces/citus/src/test/regress$ pipenv run make check-citus-upgrade-local citus-old-version=v11.1.0

diff -dU10 -w /workspaces/citus/src/test/regress/expected/upgrade_citus_finish_citus_upgrade_0.out /workspaces/citus/src/test/regress/results/upgrade_citus_finish_citus_upgrade.out
--- /workspaces/citus/src/test/regress/expected/upgrade_citus_finish_citus_upgrade_0.out.modified	2025-01-28 10:42:41.134739660 +0000
+++ /workspaces/citus/src/test/regress/results/upgrade_citus_finish_citus_upgrade.out.modified	2025-01-28 10:42:41.144740205 +0000
@@ -6,25 +6,25 @@
 SELECT substring(:'upgrade_test_old_citus_version', 'v(\d+)\.\d+\.\d+')::int < 11
 AS upgrade_test_old_citus_version_lt_11_0;
  upgrade_test_old_citus_version_lt_11_0 
 ----------------------------------------
  f
 (1 row)
 
 -- this is a transactional procedure, so rollback should be fine
 BEGIN;
 	CALL citus_finish_citus_upgrade();
-NOTICE:  already at the latest distributed schema version (11.3-1)
+NOTICE:  already at the latest distributed schema version (11.1-1)
 ROLLBACK;
 -- do the actual job
 CALL citus_finish_citus_upgrade();
-NOTICE:  already at the latest distributed schema version (11.3-1)
+NOTICE:  already at the latest distributed schema version (11.1-1)
 -- show that the upgrade is successfull
 SELECT metadata->>'last_upgrade_version' = extversion
 FROM pg_dist_node_metadata, pg_extension WHERE extname = 'citus';
  ?column? 
 ----------
  f
 (1 row)
 
 -- idempotent, should be called multiple times
 -- still, do not NOTICE the version as it changes per release
diff -dU10 -w /workspaces/citus/src/test/regress/expected/upgrade_pg_dist_cleanup_after_0.out /workspaces/citus/src/test/regress/results/upgrade_pg_dist_cleanup_after.out
--- /workspaces/citus/src/test/regress/expected/upgrade_pg_dist_cleanup_after_0.out.modified	2025-01-28 10:42:41.714771214 +0000
+++ /workspaces/citus/src/test/regress/results/upgrade_pg_dist_cleanup_after.out.modified	2025-01-28 10:42:41.724771759 +0000
@@ -15,16 +15,14 @@
 -- verify that the orphaned placement is deleted and cleanup record is created
 SELECT COUNT(*) FROM pg_dist_placement WHERE shardid IN (SELECT shardid FROM pg_dist_shard WHERE logicalrelid='table_with_orphaned_shards'::regclass);
  count 
 -------
     32
 (1 row)
 
 SELECT * FROM pg_dist_cleanup;
  record_id | operation_id | object_type | object_name | node_group_id | policy_type 
 -----------+--------------+-------------+-------------+---------------+-------------
-         1 |            0 |           1 | table_with_orphaned_shards_980001 |             1 |           0
-(1 row)
+(0 rows)
 
 CALL citus_cleanup_orphaned_resources();
-NOTICE:  cleaned up 1 orphaned resources
 DROP TABLE table_with_orphaned_shards;

@m3hm3t
Copy link
Contributor Author

m3hm3t commented Jan 29, 2025

extended logs:


2025-01-29 10:38:26.815 UTC [93669] [unknown]@[unknown] LOG:  00000: connection received: host=::1 port=43806
2025-01-29 10:38:26.815 UTC [93669] [unknown]@[unknown] LOCATION:  BackendInitialize, postmaster.c:4403
2025-01-29 10:38:26.816 UTC [93669] postgres@postgres LOG:  00000: connection authorized: user=postgres database=postgres application_name=citus_internal gpid=10000093668
2025-01-29 10:38:26.816 UTC [93669] postgres@postgres LOCATION:  PerformAuthentication, postinit.c:297
2025-01-29 10:38:26.823 UTC [93669] postgres@postgres LOG:  00000: statement: SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_0\_%' and database = current_database()
2025-01-29 10:38:26.823 UTC [93669] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:26.826 UTC [93647] DEBUG:  00000: registering background worker "Citus Maintenance Daemon: 5/10"
2025-01-29 10:38:26.826 UTC [93647] LOCATION:  BackgroundWorkerStateChange, bgworker.c:406
2025-01-29 10:38:26.826 UTC [93647] DEBUG:  00000: starting background worker process "Citus Maintenance Daemon: 5/10"
2025-01-29 10:38:26.826 UTC [93647] LOCATION:  do_start_bgworker, postmaster.c:5781
2025-01-29 10:38:26.829 UTC [93670] LOG:  00000: starting maintenance daemon on database 5 user 10
2025-01-29 10:38:26.829 UTC [93670] CONTEXT:  Citus maintenance daemon for database 5 user 10
2025-01-29 10:38:26.829 UTC [93670] LOCATION:  CitusMaintenanceDaemonMain, maintenanced.c:373
2025-01-29 10:38:26.832 UTC [93669] postgres@postgres LOG:  00000: duration: 9.133 ms
2025-01-29 10:38:26.832 UTC [93669] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:26.833 UTC [93669] postgres@postgres LOG:  00000: statement: SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_0\_%' and database = current_database()
2025-01-29 10:38:26.833 UTC [93669] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:26.835 UTC [93669] postgres@postgres LOG:  00000: duration: 1.203 ms
2025-01-29 10:38:26.835 UTC [93669] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:26.836 UTC [93671] [unknown]@[unknown] LOG:  00000: connection received: host=::1 port=43808
2025-01-29 10:38:26.836 UTC [93671] [unknown]@[unknown] LOCATION:  BackendInitialize, postmaster.c:4403
2025-01-29 10:38:26.837 UTC [93671] postgres@postgres LOG:  00000: connection authorized: user=postgres database=postgres application_name=citus_internal gpid=20000093670
2025-01-29 10:38:26.837 UTC [93671] postgres@postgres LOCATION:  PerformAuthentication, postinit.c:297
2025-01-29 10:38:26.838 UTC [93671] postgres@postgres LOG:  00000: statement: SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_1\_%' and database = current_database()
2025-01-29 10:38:26.838 UTC [93671] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:26.840 UTC [93671] postgres@postgres LOG:  00000: duration: 1.693 ms
2025-01-29 10:38:26.840 UTC [93671] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:26.840 UTC [93671] postgres@postgres LOG:  00000: statement: SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_1\_%' and database = current_database()
2025-01-29 10:38:26.840 UTC [93671] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:26.841 UTC [93671] postgres@postgres LOG:  00000: duration: 0.355 ms
2025-01-29 10:38:26.841 UTC [93671] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:26.854 UTC [93677] [unknown]@[unknown] LOG:  00000: connection received: host=::1 port=43812
2025-01-29 10:38:26.854 UTC [93677] [unknown]@[unknown] LOCATION:  BackendInitialize, postmaster.c:4403
2025-01-29 10:38:26.855 UTC [93677] postgres@postgres LOG:  00000: connection authorized: user=postgres database=postgres application_name=citus_internal gpid=30000093674
2025-01-29 10:38:26.855 UTC [93677] postgres@postgres LOCATION:  PerformAuthentication, postinit.c:297
2025-01-29 10:38:26.856 UTC [93677] postgres@postgres LOG:  00000: statement: SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_2\_%' and database = current_database()
2025-01-29 10:38:26.856 UTC [93677] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:26.857 UTC [93677] postgres@postgres LOG:  00000: duration: 1.555 ms
2025-01-29 10:38:26.857 UTC [93677] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:26.858 UTC [93677] postgres@postgres LOG:  00000: statement: SELECT gid FROM pg_prepared_xacts WHERE gid LIKE 'citus\_2\_%' and database = current_database()
2025-01-29 10:38:26.858 UTC [93677] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:26.858 UTC [93677] postgres@postgres LOG:  00000: duration: 0.413 ms
2025-01-29 10:38:26.858 UTC [93677] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:27.845 UTC [93677] postgres@postgres LOG:  00000: statement: SELECT waiting_pid, waiting_node_id, waiting_transaction_num, waiting_transaction_stamp, blocking_pid, blocking_node_id, blocking_transaction_num, blocking_transaction_stamp, blocking_transaction_waiting FROM dump_local_wait_edges()
2025-01-29 10:38:27.845 UTC [93677] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1054
2025-01-29 10:38:27.846 UTC [93677] postgres@postgres LOG:  00000: duration: 1.134 ms
2025-01-29 10:38:27.846 UTC [93677] postgres@postgres LOCATION:  exec_simple_query, postgres.c:1344
2025-01-29 10:38:27.847 UTC [93669] postgres@postgres LOG:  00000: disconnection: session time: 0:00:01.031 user=postgres database=postgres host=::1 port=43806
2025-01-29 10:38:27.847 UTC [93669] postgres@postgres LOCATION:  log_disconnections, postgres.c:5117
2025-01-29 10:38:27.949 UTC [93647] LOG:  00000: received fast shutdown request
2025-01-29 10:38:27.949 UTC [93647] LOCATION:  pmdie, postmaster.c:2907
2025-01-29 10:38:27.949 UTC [93647] LOG:  00000: aborting any active transactions
2025-01-29 10:38:27.949 UTC [93647] LOCATION:  pmdie, postmaster.c:2925
2025-01-29 10:38:27.950 UTC [93654] DEBUG:  00000: logical replication launcher shutting down
2025-01-29 10:38:27.950 UTC [93654] LOCATION:  ProcessInterrupts, postgres.c:3213
2025-01-29 10:38:27.950 UTC [93677] postgres@postgres FATAL:  57P01: terminating connection due to administrator command
2025-01-29 10:38:27.950 UTC [93677] postgres@postgres LOCATION:  ProcessInterrupts, postgres.c:3246
2025-01-29 10:38:27.950 UTC [93671] postgres@postgres FATAL:  57P01: terminating connection due to administrator command
2025-01-29 10:38:27.950 UTC [93671] postgres@postgres LOCATION:  ProcessInterrupts, postgres.c:3246
2025-01-29 10:38:27.950 UTC [93653] DEBUG:  00000: autovacuum launcher shutting down
2025-01-29 10:38:27.950 UTC [93653] LOCATION:  AutoVacLauncherShutdown, autovacuum.c:849
2025-01-29 10:38:27.950 UTC [93677] postgres@postgres LOG:  00000: disconnection: session time: 0:00:01.096 user=postgres database=postgres host=::1 port=43812
2025-01-29 10:38:27.950 UTC [93677] postgres@postgres LOCATION:  log_disconnections, postgres.c:5117
2025-01-29 10:38:27.950 UTC [93671] postgres@postgres LOG:  00000: disconnection: session time: 0:00:01.114 user=postgres database=postgres host=::1 port=43808
2025-01-29 10:38:27.950 UTC [93671] postgres@postgres LOCATION:  log_disconnections, postgres.c:5117
2025-01-29 10:38:27.952 UTC [93647] LOG:  00000: background worker "logical replication launcher" (PID 93654) exited with exit code 1
2025-01-29 10:38:27.952 UTC [93647] LOCATION:  LogChildExit, postmaster.c:3740
2025-01-29 10:38:27.953 UTC [93647] DEBUG:  00000: unregistering background worker "Citus Maintenance Daemon: 5/10"
2025-01-29 10:38:27.953 UTC [93647] LOCATION:  ForgetBackgroundWorker, bgworker.c:447
2025-01-29 10:38:27.953 UTC [93647] DEBUG:  00000: background worker "Citus Maintenance Daemon: 5/10" (PID 93670) exited with exit code 0
2025-01-29 10:38:27.953 UTC [93647] LOCATION:  LogChildExit, postmaster.c:3740
2025-01-29 10:38:27.953 UTC [93649] LOG:  00000: shutting down
2025-01-29 10:38:27.953 UTC [93649] LOCATION:  ShutdownXLOG, xlog.c:6071
2025-01-29 10:38:27.953 UTC [93649] LOG:  00000: checkpoint starting: shutdown immediate
2025-01-29 10:38:27.953 UTC [93649] LOCATION:  LogCheckpointStart, xlog.c:6121
2025-01-29 10:38:27.953 UTC [93649] DEBUG:  00000: performing replication slot checkpoint
2025-01-29 10:38:27.953 UTC [93649] LOCATION:  CheckPointReplicationSlots, slot.c:1438
2025-01-29 10:38:27.954 UTC [93649] LOG:  00000: checkpoint complete: wrote 9 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.001 s; sync files=0, longest=0.000 s, average=0.000 s; distance=11 kB, estimate=11 kB
2025-01-29 10:38:27.954 UTC [93649] LOCATION:  LogCheckpointEnd, xlog.c:6202
2025-01-29 10:38:27.967 UTC [93647] LOG:  00000: database system is shut down
2025-01-29 10:38:27.967 UTC [93647] LOCATION:  UnlinkLockFiles, miscinit.c:1032

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@m3hm3t @colm-mchugh @emelsimsek