paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(workflows): crash recovery — resume runs orphaned mid-flow

Browse Source 
Closes the crash-safety gap: a run left in running/pending by a crash had
nothing to resume it (the scheduler only wakes 'waiting'). Adds a heartbeat
(workflow_runs.updated_at, stamped on every node advance + start/resume) and a
recoverStaleRuns() sweep that re-enters runs whose heartbeat has gone stale
(>10 min) from their persisted node. Runs on the scheduler tick AND the boot
tick, so a restart catches anything stranded during downtime.

Re-entry is at-least-once (the current node may re-execute) — loop counters +
the late-fee math are idempotent, so the only residual risk is a duplicate
reminder email. An attempts counter (migration 145, cap 5) marks a run failed
instead of recovering a node that reliably crashes the process (crash-loop
backstop). Flag-gated. Tests: orphan-resume + crash-loop cap.
This commit is contained in:
Luca
2026-06-23 13:39:17 +02:00
parent 83dc95a62b
commit 192d2cbc06
4 changed files with 133 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/migrations/core/145_workflow_run_recovery.js
+33
View File
@@ -0,0 +1,33 @@
/**
* Migration 145: crash-recovery fields for workflow runs.
*
* A run left in 'running'/'pending' by a crash has nothing to resume it (the
* scheduler only wakes 'waiting' runs). Add a heartbeat (`updated_at`, stamped
* on every step) so a recovery sweep can detect stale runs, plus an `attempts`
* counter so a node that reliably crashes the process can't be recovered
* forever (crash-loop backstop → marked failed after a cap).
*/
exports.up = async function (knex) {
if (!(await knex.schema.hasTable('workflow_runs'))) return;
const hasUpdated = await knex.schema.hasColumn('workflow_runs', 'updated_at');
const hasAttempts = await knex.schema.hasColumn('workflow_runs', 'attempts');
await knex.schema.alterTable('workflow_runs', (t) => {
if (!hasUpdated) t.timestamp('updated_at').defaultTo(knex.fn.now());
if (!hasAttempts) t.integer('attempts').notNullable().defaultTo(0);
});
// Recovery sweep queries by (status, updated_at).
if (!hasUpdated) {
try { await knex.schema.alterTable('workflow_runs', (t) => t.index(['status', 'updated_at'], 'workflow_runs_recovery_index')); } catch (_) {}
}
};
exports.down = async function (knex) {
if (!(await knex.schema.hasTable('workflow_runs'))) return;
try { await knex.schema.alterTable('workflow_runs', (t) => t.dropIndex(['status', 'updated_at'], 'workflow_runs_recovery_index')); } catch (_) {}
if (await knex.schema.hasColumn('workflow_runs', 'updated_at')) {
await knex.schema.alterTable('workflow_runs', (t) => t.dropColumn('updated_at'));
}
if (await knex.schema.hasColumn('workflow_runs', 'attempts')) {
await knex.schema.alterTable('workflow_runs', (t) => t.dropColumn('attempts'));
}
};