fix(restore): evict active sessions before dropping target DB

PostgreSQL refuses DROP DATABASE while any session is connected:
  ERROR: database "picpeak_prod" is being accessed by other users
  DETAIL: There are 6 other sessions using the database.

The backend's own knex pool holds 5-25 active connections to the
target DB. So even after closing the request that initiated the
restore, the pool keeps the DB busy and the DROP statement fails.

Three-layered cure, all in the restore service's PG branch:

  1. Call `db.destroy()` first to close the in-process knex pool so
     we don't fight ourselves. Knex will lazily re-open on the next
     query via db.js's retry logic, so this is safe to do mid-restore.

  2. SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE
     datname=<target> AND pid<>pg_backend_pid() — evicts any sessions
     from other processes (other server replicas, leftover idle
     transactions, things our own pool destroy missed).

  3. DROP DATABASE IF EXISTS "<target>" WITH (FORCE) — PG13+ kills
     remaining connections atomically with the DROP. Falls back to
     plain DROP on older Postgres where WITH (FORCE) is a syntax error.

Surfaced as the FIFTH latent bug in the restore path tonight: the
DROP DATABASE statement always assumed a quiescent destination, but
the live backend keeps the destination busy at all times. Every
previous PG install of picpeak that ever tried Restore would have
hit this — meaning the disaster-recovery feature has shipped broken
for a long time without anyone exercising it end-to-end.
This commit is contained in:
Luca
2026-05-30 13:20:20 +02:00
parent 4c31a22626
commit a39def672e
+49 -1
View File
@@ -847,12 +847,60 @@ class RestoreService {
// the `postgres` DB is restricted to superusers.
const maintenanceDb = process.env.DB_CHECK_DB || 'postgres';
// The backend's own knex pool holds N active connections to
// the target database (default 5-25 per knexfile.js). PostgreSQL
// refuses DROP DATABASE while any session is connected:
// ERROR: database "X" is being accessed by other users
// DETAIL: There are N other sessions using the database.
// We have to evict those sessions ourselves before issuing the
// DROP. Two-step approach:
// 1. Close knex's own pool so we don't fight ourselves.
// 2. pg_terminate_backend() the rest (other server replicas,
// pg_stat_activity stragglers, leftover idle txns).
//
// After CREATE DATABASE, knex will lazily re-open the pool on
// the next query — handled by db.js's connection retry logic.
this.log('warn', 'Closing knex pool before dropping target database...');
try { await db.destroy(); } catch (poolErr) {
this.log('warn', `Pool destroy threw (continuing): ${poolErr.message}`);
}
this.log('warn', 'Terminating any remaining sessions on target database...', {
target: database,
});
// pg_terminate_backend takes a pid. Kill every session against
// the target DB except our own connection (which is to the
// maintenance DB anyway). Wrapped in `SELECT ... FROM ... WHERE`
// so we get one psql round-trip instead of N.
await spawnAsync('psql', [
'-h', host, '-p', String(port), '-U', user, '-d', maintenanceDb,
'-c',
`SELECT pg_terminate_backend(pid) FROM pg_stat_activity ` +
`WHERE datname = '${database.replace(/'/g, "''")}' AND pid <> pg_backend_pid()`,
], { env });
// Drop and recreate database (extremely dangerous!)
this.log('warn', 'Dropping and recreating PostgreSQL database...', {
target: database, via: maintenanceDb,
});
await spawnAsync('psql', ['-h', host, '-p', String(port), '-U', user, '-d', maintenanceDb, '-c', `DROP DATABASE IF EXISTS "${database}"`], { env });
// WITH (FORCE) on Postgres 13+ kills any remaining connections
// atomically with the DROP. On older Postgres the FORCE option
// doesn't exist, so we fall back to plain DROP IF EXISTS — by
// which point pg_terminate_backend should have cleared the
// table. Try FORCE first, fall back to plain on syntax error.
try {
await spawnAsync('psql', ['-h', host, '-p', String(port), '-U', user, '-d', maintenanceDb,
'-c', `DROP DATABASE IF EXISTS "${database}" WITH (FORCE)`], { env });
} catch (forceErr) {
// PG < 13: WITH (FORCE) is a syntax error. Plain DROP after
// our pg_terminate_backend pass should now succeed.
this.log('info', 'DROP DATABASE WITH (FORCE) not supported — falling back to plain DROP', {
error: forceErr.message,
});
await spawnAsync('psql', ['-h', host, '-p', String(port), '-U', user, '-d', maintenanceDb,
'-c', `DROP DATABASE IF EXISTS "${database}"`], { env });
}
await spawnAsync('psql', ['-h', host, '-p', String(port), '-U', user, '-d', maintenanceDb, '-c', `CREATE DATABASE "${database}"`], { env });