fix(admin): move the maintenance sweeps' run state into the database (#1181) (#1184)

Both photo sweeps tracked whether they were running in a module-level variable.
Correct on one replica, wrong behind a load balancer: the status poll answers
from whichever process it reaches, so an idle replica reports isRunning false
while another is mid-run, the UI re-enables the button, and the next POST lands
elsewhere and starts a second pass over the whole library. The .whereNull()
guards mean nothing is corrupted; the cost is duplicated S3/NAS I/O and an
operator who cannot tell whether a job is running.

Migration 189 adds one row per job. The claim is a conditional UPDATE whose
affected-row count is the answer — the shape backgroundProcessor already uses
to hand a photo to exactly one worker — so two replicas cannot both match.

The lease is fenced on a per-claim token: taking over a stale claim does not
stop the old runner, so without fencing a superseded runner finishing late
cleared the new owner's flag and overwrote its result. heartbeat() reports
renewal failure and the loops stop on it. Renewal runs on a timer spanning the
claim through release, including the candidate query, because one hung NAS read
can outlast the stale window inside a single iteration.

maintenance_jobs is excluded from .picpeak archives — an archive taken mid-sweep
would otherwise restore a live lease with no runner to release it. The importer
filters the same set, so older archives are skipped too.

Response shape is unchanged, so the frontend needs no change.

Co-authored-by: Paul Nothaft <[email protected]>

backend/migrations/core/189_maintenance_job_state.js

@@ -0,0 +1,73 @@
+/**
+ * Migration 189: shared run state for the maintenance sweeps (#1181).
+ *
+ * Both photo maintenance jobs — the dimension repair and the capture-date
+ * backfill — tracked whether they were running in a module-level variable. On
+ * a single-replica install that is correct. Behind a load balancer it is not:
+ * the flag lives in one process, so a status poll routed to any other replica
+ * answers `isRunning: false`, the UI re-enables the button, and the next POST
+ * lands somewhere else and starts a second pass over the whole library. Both
+ * replicas then read and parse every original off S3 or the NAS mount. The
+ * `.whereNull(...)` guards on the writes mean nothing is corrupted — the cost
+ * is the duplicated I/O, and an operator who cannot tell whether a job is
+ * running.
+ *
+ * One row per job, claimed with a conditional UPDATE so the claim itself is
+ * the mutual exclusion — the same UPDATE-with-guard shape backgroundProcessor
+ * already uses to hand a photo to exactly one worker
+ * (services/backgroundProcessor.js:110-116).
+ *
+ * heartbeat_at exists because a lock with no expiry is worse than no lock: a
+ * replica that is OOM-killed mid-run would leave is_running = true forever and
+ * no way to clear it short of editing the database. The runner touches it as
+ * it goes, and a claim is allowed to take over a run whose heartbeat has gone
+ * quiet. See services/maintenanceJobState.js for the read side, which reports
+ * a stale run as not-running so the button comes back on its own.
+ *
+ * Rows are seeded here rather than created on demand so the claim is a plain
+ * UPDATE with no insert race behind it.
+ */
+
+const JOBS = ['photo_dimension_repair', 'photo_capture_date_backfill'];
+
+exports.up = async function (knex) {
+  const exists = await knex.schema.hasTable('maintenance_jobs');
+  if (!exists) {
+    await knex.schema.createTable('maintenance_jobs', (t) => {
+      // The job's identity, not a surrogate key: there is exactly one row per
+      // job and every access is by name, so the name is the primary key.
+      t.string('job_name', 64).primary();
+      t.boolean('is_running').notNullable().defaultTo(false);
+      t.timestamp('started_at').nullable();
+      t.timestamp('heartbeat_at').nullable();
+      t.timestamp('finished_at').nullable();
+      // JSON as text: the shape differs per job (the backfill reports a third
+      // counter the dimension repair has no equivalent for) and nothing
+      // queries into it, so a json column would buy nothing and cost engine
+      // differences between Postgres and SQLite.
+      t.text('last_result').nullable();
+      // Diagnostics only — which process is holding the claim.
+      t.string('owner', 128).nullable();
+      // The fencing token. Unique per claim, not per process: after a stale
+      // takeover the old runner may still be alive and mid-loop, and it can
+      // even be the same process that re-claimed. Every write it makes is
+      // scoped to the token it was handed, so a superseded runner can neither
+      // renew a claim it has lost nor release one it no longer owns.
+      t.string('claim_token', 64).nullable();
+    });
+    console.log('189: created maintenance_jobs');
+  }
+
+  // Idempotent on re-run and safe against a table that already carries rows.
+  for (const jobName of JOBS) {
+    const row = await knex('maintenance_jobs').where({ job_name: jobName }).first();
+    if (!row) {
+      await knex('maintenance_jobs').insert({ job_name: jobName, is_running: false });
+      console.log(`189: seeded job row ${jobName}`);
+    }
+  }
+};
+
+exports.down = async function (knex) {
+  await knex.schema.dropTableIfExists('maintenance_jobs');
+};