* fix(external-media): store external paths from the media root (#1163) Importing a second folder into an event silently invalidated every photo already in it. photos.external_relpath was stored relative to events.external_path, and every import overwrites that column — so the older rows were rebased onto the new folder and their originals resolved to paths that do not exist. Nothing errored, and the grid still looked intact: thumbnails are written to local storage during the import while the base path is still correct. Only what needs the original broke — preview generation, the lightbox, downloads — which presents as a gallery that looks slow rather than one that is broken. The reporter had 7547 of 8004 rows pointing into the void and spent a while chasing it as a CPU problem. - external_relpath is now relative to EXTERNAL_MEDIA_ROOT, so a row is self-describing and nothing an admin does to the event afterwards can move an already-imported photo. - migration 187 folds each event's base path into its rows. Where the current resolution is missing on disk it walks up the base path for an ancestor under which the file IS there — the already-rebased case — and where it finds nothing it leaves the row resolving exactly where it resolves today. Skipped entirely when the media root is unmounted, since every file looks missing then. - the fold also runs after a .picpeak restore: knex_migrations is excluded from the archive, so a pre-#1163 backup would otherwise land base-relative rows on a migrated instance. - drops the duplicate-leaf-segment guess in photoResolver. It papered over this same double-prefixing and actively corrupts a root-relative path whose first segment legitimately repeats (base 'Trip', row 'Trip/x.jpg'). * fix(external-media): verify provenance and fold atomically (#1163) External review found four real defects in the fold. Repair could adopt the wrong file. Existence alone was accepted as proof that an ancestor candidate was the row's original — so a row whose file an admin simply deleted would adopt any same-named file one directory up (base `Trip/Sub`, relpath `photo.jpg`, an unrelated `Trip/photo.jpg`), and downloads would then serve a different photo. Worse than a dead link. An ancestor must now also match photos.size_bytes, which the import recorded from the very file the row describes; rows carrying no size are never repaired from an ancestor. The CURRENT base is still accepted on existence alone, because nothing is being inferred there — that is where the row already resolves. The fold was not atomic. Every UPDATE committed independently and the marker came last, so a process killed mid-fold left converted and unconverted rows with no marker — and the next run folded the converted ones a second time, putting every original one directory deeper with no undo. Probing is now a read-only first phase (so a slow cold NAS does not hold a write transaction open), and every rewrite plus the marker commit together. Failed rewrites certified a partial conversion. The per-row catch counted any error as a collision, carried on, and wrote the marker anyway — leaving that row in the old format for a resolver that now reads it differently. It also could not tell a genuine duplicate from a SQLite lock or I/O fault. Target collisions are now resolved in the planning phase, where they can be identified honestly, and a write that fails rolls the whole fold back. Restore ordering. The fold ran after the face requeue, with the worker live — so a worker could claim an external row while it was still base-relative, resolve it against the wrong path, and burn it to 'failed', a state only an explicit Re-scan clears. The fold now runs first, for the same reason the requeue already sat after restoreFiles. * fix(external-media): close the fold's remaining stranding paths (#1163) Second review round, three findings. A collision loser was left stranded. When an event imported one file through both `Trip` and `Trip/Sub`, two rows folded to the same path and the loser was skipped — keeping a base-relative value that the root-only resolver then reads as `<root>/<relpath>`, permanently wrong, with the marker claiming conversion was complete. It is a duplicate by construction, so it now goes through migration 186's deleteDuplicatePhotos, which reparents its feedback and marks and reconciles the face clusters instead of orphaning them. This branch is rebased onto #1162 for that helper. The other restore path had the same face-ordering bug. restoreService queued face scans in step 6, before step 7c runs pending migrations — so a pre-187 full or database restore handed the live worker rows whose paths were still event-relative, and it burned them to 'failed', a state the later fold does not clear. The requeue now happens after the migrations, where the files already are. A failed conversion was reported as a clean restore. The fold is transactional, so a failure leaves every external path in the old format under a resolver that reads from the media root — every original unreachable. It was logged as a warning and the restore returned success. It now returns externalPathsConverted/externalPathError, and suppresses the face requeue, which would otherwise mark those photos failed on top. * fix(external-media): make the fold safe against its own intermediate states (#1163) Third review round, four findings. A one-pass rewrite could collide with itself. Every FINAL path is distinct, but a final value can equal another row's CURRENT one — `photo.jpg` repairing to `Trip/photo.jpg` while the row already holding `Trip/photo.jpg` folds deeper — so the update violated migration 186's unique index halfway through. On Postgres that surfaces as 23505, which run-migrations-safe.js mistakes for "schema already exists" and records 187 as applied after the rollback, leaving every path unconverted with nothing to retry. Rows now park on a per-row staging value first, and migration 187 re-throws without the driver's code so the runner cannot misread it. The bulk update targeted rows the plan never saw. Phase 1 probes outside the transaction and can run for minutes; an import finishing in that window inserts an already root-relative row, and `where event_id` prefixed it again with the stale base. It now updates by the ids phase 1 captured. The restore UI never showed a conversion failure. The API carried externalPathsConverted, but PicpeakBackupCard neither declared nor read it and showed a green success either way — so an admin whose external originals were all unreachable was told the restore worked. restoreService requeued faces even when the migrations failed. The step 7c catch is deliberately non-fatal, so a pre-187 backup whose fold never ran still handed the live worker event-relative paths to burn to 'failed'. * fix(external-media): the fold's staging value must be storable on Postgres (#1163) External review of the stable twin caught this, and it was on both branches. The two-pass rewrite parks each row on a temporary value, and that value was written with a leading NUL. SQLite stores NUL in TEXT without complaint; Postgres rejects it outright — "invalid byte sequence for encoding UTF8: 0x00" — so migration 187 rolled back on exactly the installs that need the two-pass repair, and only on the engine most of them run. Restores hit the same wall and reported the conversion as failed. The prefix is ordinary text now. It still cannot collide with a real relative path and is still obviously wrong if a crash leaves one behind. Adds a gated Postgres test alongside the existing picpeakRestorePg one, because a SQLite-only suite structurally cannot catch this class: restoring the NUL makes exactly the two-pass repair case fail with that error, and nothing else. --------- Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
co-authored by
Paul Nothaft
Paul Nothaft
parent
36192708ab
commit
a7b74bcd87
@@ -249,6 +249,11 @@ router.post('/picpeak/import', adminAuth, requirePermission('backup.restore'), p
|
||||
filesRestored: result.filesRestored,
|
||||
usesExternalMedia: result.usesExternalMedia,
|
||||
crossEngine: result.crossEngine,
|
||||
// False when the pre-#1163 external-path conversion failed. The rows and
|
||||
// files are in place, but no external original resolves until it is
|
||||
// retried — the UI must say so rather than showing a plain success.
|
||||
externalPathsConverted: result.externalPathsConverted !== false,
|
||||
externalPathError: result.externalPathError || null,
|
||||
sessionInvalidated: true,
|
||||
});
|
||||
} catch (error) {
|
||||
|
||||
@@ -81,6 +81,15 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
|
||||
if (!event) return res.status(404).json({ error: 'Event not found' });
|
||||
|
||||
const baseAbs = resolveExternalPath({ external_path }, '');
|
||||
|
||||
// What gets STORED on the row (#1163). `f.rel` stays relative to the
|
||||
// imported folder because the type inference below reads its first segment
|
||||
// ('individual' / 'collages'); external_relpath is written relative to
|
||||
// EXTERNAL_MEDIA_ROOT so the row does not depend on a column this very
|
||||
// handler is about to overwrite.
|
||||
const basePrefix = String(external_path).replace(/^\/+|\/+$/g, '');
|
||||
const toRootRelative = (rel) => (basePrefix ? path.join(basePrefix, rel) : rel);
|
||||
|
||||
// Collect files
|
||||
const files = recursive ? await walkDir(baseAbs, baseAbs) : (await fs.readdir(baseAbs, { withFileTypes: true }))
|
||||
.filter(e => e.isFile())
|
||||
@@ -131,15 +140,12 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
|
||||
// :52) and is NOT NULL defaulting to 'managed', so flipping source_mode
|
||||
// does not touch them.
|
||||
//
|
||||
// It does NOT isolate existing EXTERNAL rows — resolveExternalPath prefixes
|
||||
// every one of them with event.external_path
|
||||
// (externalMediaService.js:97-102), so importing folder B into an event
|
||||
// that already references folder A rebases the A rows onto B. That is
|
||||
// pre-existing: the update always did this, just after the loop instead of
|
||||
// before it, so moving it changes when the window opens and not whether it
|
||||
// exists. The underlying limitation is that an event carries a single
|
||||
// external base path, which makes importing a second folder into it
|
||||
// incoherent either way — worth its own fix, not this one.
|
||||
// Safe for existing EXTERNAL rows too, as of #1163. It was not: relpaths
|
||||
// were stored relative to external_path, so overwriting the column here
|
||||
// rebased every row already in the event onto the new folder — quietly,
|
||||
// because their thumbnails were already on local disk and the grid carried
|
||||
// on rendering. Rows now carry a root-relative path and this write cannot
|
||||
// reach them.
|
||||
await db('events').where('id', eventId).update({ source_mode: 'reference', external_path });
|
||||
|
||||
let imported = 0;
|
||||
@@ -175,6 +181,8 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
|
||||
if (segs[0] === map.collages) type = 'collage';
|
||||
if (segs[0] === map.individual) type = 'individual';
|
||||
|
||||
const relFromRoot = toRootRelative(f.rel);
|
||||
|
||||
try {
|
||||
// Fast path only. This SELECT settles the common case — a re-import of
|
||||
// a folder already in the event — without paying for a stat and a
|
||||
@@ -184,7 +192,7 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
|
||||
// index from migration 186 is the guard, and the catch below is how
|
||||
// this loop converges when it fires.
|
||||
const exists = await db('photos')
|
||||
.where({ event_id: eventId, external_relpath: f.rel })
|
||||
.where({ event_id: eventId, external_relpath: relFromRoot })
|
||||
.first();
|
||||
if (exists) { skipped++; continue; }
|
||||
const stats = await fs.stat(f.full);
|
||||
@@ -214,7 +222,7 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
|
||||
width,
|
||||
height,
|
||||
source_origin: 'external',
|
||||
external_relpath: f.rel
|
||||
external_relpath: relFromRoot
|
||||
})
|
||||
.returning('id');
|
||||
} catch (insertErr) {
|
||||
|
||||
@@ -94,6 +94,14 @@ async function list(relativePath = '') {
|
||||
return { path: relFromRoot, entries, canNavigateUp };
|
||||
}
|
||||
|
||||
/**
|
||||
* A path under an EVENT's configured base directory.
|
||||
*
|
||||
* Still the right resolver for anything that means "the folder this event was
|
||||
* imported from" — the import walk, and the mount-health probe in
|
||||
* faceProcessor. It is NOT how a photo's original is found any more: see
|
||||
* resolveExternalPhotoPath (#1163).
|
||||
*/
|
||||
function resolveExternalPath(event, relpath) {
|
||||
const root = getExternalMediaRoot();
|
||||
const base = event?.external_path ? path.join(event.external_path) : '';
|
||||
@@ -101,9 +109,29 @@ function resolveExternalPath(event, relpath) {
|
||||
return safePathJoin(root, combined);
|
||||
}
|
||||
|
||||
/**
|
||||
* A photo's original, from the root (#1163).
|
||||
*
|
||||
* photos.external_relpath used to be stored relative to events.external_path,
|
||||
* which made a row's meaning depend on a column on another table that every
|
||||
* import overwrites. Importing a second folder into an event therefore rebased
|
||||
* every row already in it — silently, because thumbnails are written to local
|
||||
* storage during the import and the grid keeps rendering. One reporter had
|
||||
* 7547 of 8004 rows resolving to files that did not exist.
|
||||
*
|
||||
* Root-relative makes a row self-describing: nothing an admin does to the event
|
||||
* afterwards can move an already-imported photo. Migration 187 rewrote the
|
||||
* existing rows.
|
||||
*/
|
||||
function resolveExternalPhotoPath(photo) {
|
||||
const root = getExternalMediaRoot();
|
||||
return safePathJoin(root, photo?.external_relpath || '');
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
getExternalMediaRoot,
|
||||
isUnderRoot,
|
||||
list,
|
||||
resolveExternalPath,
|
||||
resolveExternalPhotoPath,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
/**
|
||||
* Fold each event's base path into its external photo rows (#1163).
|
||||
*
|
||||
* photos.external_relpath used to be stored relative to events.external_path —
|
||||
* a column every import overwrites — so importing a second folder into an
|
||||
* event rebased every photo already in it. Root-relative paths make a row
|
||||
* self-describing.
|
||||
*
|
||||
* This lives in a service rather than inside migration 187 because it has two
|
||||
* callers. The migration is one. The other is a .picpeak restore: knex_migrations
|
||||
* is excluded from the archive, so restoring a pre-#1163 backup onto an
|
||||
* already-migrated instance drops base-relative rows into a schema that no
|
||||
* longer folds them, and every original in the restored library becomes
|
||||
* unreachable with nothing logged.
|
||||
*
|
||||
* REPAIR, and its limits. For a healthy event the correct new value is just
|
||||
* `external_path + relpath` — that is what the app resolves today, so folding
|
||||
* it in changes nothing and can break nothing. For an event that has ALREADY
|
||||
* been rebased, that same rule would bake the broken path in permanently, so
|
||||
* where the current resolution does not exist on disk this walks up the base
|
||||
* path looking for an ancestor under which the file IS there (the shape the
|
||||
* bug produces: a parent imported first, a child second). A row it cannot
|
||||
* place is left resolving exactly where it resolves today — preserving current
|
||||
* behaviour is the floor, never guess below it.
|
||||
*
|
||||
* Existence alone is NOT enough to accept an ancestor. A row whose file an
|
||||
* admin simply deleted would otherwise adopt any same-named file further up —
|
||||
* base `Trip/Sub`, relpath `photo.jpg`, an unrelated `Trip/photo.jpg` — and
|
||||
* downloads would then serve the WRONG original, which is worse than a broken
|
||||
* link. So an ancestor candidate must also match photos.size_bytes, recorded
|
||||
* by the import from the very file the row describes. Rows carrying no size
|
||||
* are never repaired from an ancestor.
|
||||
*
|
||||
* ATOMICITY. Probing is read-only and runs first; every rewrite and the marker
|
||||
* are then committed in ONE transaction. Split across commits, a process
|
||||
* killed mid-fold would leave converted and unconverted rows behind with no
|
||||
* marker, and the next run would fold the converted ones a second time —
|
||||
* putting every original one directory deeper, permanently.
|
||||
*
|
||||
* The probe is skipped entirely when the media root is unreachable or empty:
|
||||
* an unmounted share makes every file look missing, and "repairing" off that
|
||||
* signal would move every original on a healthy install. When it does run it
|
||||
* is one access() per photo, base path first, so a healthy install pays one
|
||||
* stat per row and then a single UPDATE per event.
|
||||
*
|
||||
* Idempotency is recorded explicitly in app_settings rather than inferred from
|
||||
* the data. The tempting inference — "does the relpath already start with the
|
||||
* base path?" — is wrong for any event with a subfolder named after its parent
|
||||
* (base 'Trip', row 'Trip/x.jpg'), and being wrong there corrupts a path in an
|
||||
* operation that has no undo.
|
||||
*/
|
||||
|
||||
const path = require('path');
|
||||
const fsp = require('fs').promises;
|
||||
const { deleteDuplicatePhotos } = require('./externalPhotoDedupe');
|
||||
|
||||
const MARKER = 'external_relpath_root_relative';
|
||||
// Per-row parking value for the two-pass rewrite below.
|
||||
//
|
||||
// NOT a NUL-prefixed string, which is what this was first written as: Postgres
|
||||
// rejects U+0000 in a `text` column outright ("invalid byte sequence for
|
||||
// encoding UTF8"), so the rewrite would abort on exactly the installs that
|
||||
// need the two-pass repair — and only on Postgres, which SQLite-only tests
|
||||
// cannot see. The prefix below is ordinary text, cannot collide with a real
|
||||
// relative path (no import writes a leading dot-segment like this), and stays
|
||||
// obviously wrong if a crash ever leaves one behind.
|
||||
const STAGING_PREFIX = '.picpeak-fold-staging/';
|
||||
const CHUNK = 400; // SQLite caps a statement at 999 bound parameters.
|
||||
const chunk = (arr) => {
|
||||
const out = [];
|
||||
for (let i = 0; i < arr.length; i += CHUNK) out.push(arr.slice(i, i + CHUNK));
|
||||
return out;
|
||||
};
|
||||
|
||||
function normalizeBase(externalPath) {
|
||||
return String(externalPath || '').replace(/\\/g, '/').replace(/^\/+|\/+$/g, '');
|
||||
}
|
||||
|
||||
/** Every prefix of `base`, longest first, then '' (the root itself). */
|
||||
function ancestorPrefixes(base) {
|
||||
const segs = base.split('/').filter(Boolean);
|
||||
const out = [];
|
||||
for (let i = segs.length; i > 0; i--) out.push(segs.slice(0, i).join('/'));
|
||||
out.push('');
|
||||
return out;
|
||||
}
|
||||
|
||||
async function exists(p) {
|
||||
try { await fsp.access(p); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
/**
|
||||
* Is `p` plausibly the file this row was imported from?
|
||||
*
|
||||
* Size is the provenance signal available without re-reading every original:
|
||||
* photos.size_bytes was written by the import from the file the row describes.
|
||||
* Returns false when it cannot be verified, so an unverifiable candidate is
|
||||
* never adopted from somewhere the row did not previously point.
|
||||
*/
|
||||
async function fileMatchesSize(p, expectedSize) {
|
||||
if (expectedSize == null || Number(expectedSize) <= 0) return false;
|
||||
try {
|
||||
const stats = await fsp.stat(p);
|
||||
return stats.isFile() && stats.size === Number(expectedSize);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Joined without the safePathJoin the app serves through: this is reading, and
|
||||
* a stored path that escapes the root is damage worth detecting rather than
|
||||
* throwing on.
|
||||
*/
|
||||
const under = (root, ...parts) => path.join(root, ...parts.filter(Boolean));
|
||||
|
||||
async function rootUsable(root) {
|
||||
if (!root) return false;
|
||||
try {
|
||||
// An unmounted NFS/SMB share usually leaves the mountpoint behind as an
|
||||
// ordinary empty directory, so readdir succeeds on storage that is gone.
|
||||
return (await fsp.readdir(root)).length > 0;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import('knex')} knex a knex instance, or a transaction from a caller
|
||||
* that is already inside one (the restore).
|
||||
* @param {(msg: string) => void} [log]
|
||||
* @returns {Promise<{skipped?: string, folded?: number, repaired?: number, stranded?: number, collided?: number}>}
|
||||
*/
|
||||
async function foldExternalRelpaths(knex, log = () => {}) {
|
||||
if (!(await knex.schema.hasTable('photos'))) return { skipped: 'no photos table' };
|
||||
if (!(await knex.schema.hasColumn('photos', 'external_relpath'))) return { skipped: 'no external_relpath column' };
|
||||
if (!(await knex.schema.hasTable('events'))) return { skipped: 'no events table' };
|
||||
if (!(await knex.schema.hasTable('app_settings'))) return { skipped: 'no app_settings table' };
|
||||
|
||||
if (await knex('app_settings').where('setting_key', MARKER).first()) {
|
||||
return { skipped: 'already folded' };
|
||||
}
|
||||
|
||||
const events = await knex('events').select('id', 'external_path');
|
||||
const byId = new Map(events.map((e) => [e.id, normalizeBase(e.external_path)]));
|
||||
const eventRows = await knex('photos').whereNotNull('external_relpath').distinct('event_id');
|
||||
|
||||
let root = null;
|
||||
try {
|
||||
root = require('./externalMediaService').getExternalMediaRoot();
|
||||
} catch (e) {
|
||||
log(`media root unavailable (${e.message}) — folding without on-disk repair`);
|
||||
}
|
||||
const canProbe = await rootUsable(root);
|
||||
if (!canProbe) log('media root unreachable or empty — folding base paths in without on-disk repair');
|
||||
|
||||
// ---- Phase 1: decide, writing nothing. -------------------------------
|
||||
// Read-only, so the transaction below stays short. Probing a cold NAS can
|
||||
// take minutes and holding a write transaction open for that would block the
|
||||
// app for the duration.
|
||||
let folded = 0; let repaired = 0; let stranded = 0; let collided = 0;
|
||||
const plan = [];
|
||||
|
||||
for (const { event_id: eventId } of eventRows) {
|
||||
// No base path means the rows are already relative to the root.
|
||||
const base = byId.get(eventId);
|
||||
if (!base) continue;
|
||||
|
||||
const rows = await knex('photos')
|
||||
.where('event_id', eventId)
|
||||
.whereNotNull('external_relpath')
|
||||
.select('id', 'external_relpath', 'size_bytes');
|
||||
|
||||
// Deciding health from a SAMPLE was the tempting shortcut and it is not
|
||||
// safe: a rebased event whose first few rows happen to come from the most
|
||||
// recent import reads as healthy, and every older row is baked in wrong.
|
||||
const prefixes = ancestorPrefixes(base);
|
||||
const placements = [];
|
||||
let allUnderBase = true;
|
||||
|
||||
for (const row of rows) {
|
||||
if (!canProbe) { placements.push([row, base]); continue; }
|
||||
|
||||
let chosen = null;
|
||||
// The current base first, on existence alone — nothing is inferred
|
||||
// there, it is where the row already resolves.
|
||||
if (await exists(under(root, base, row.external_relpath))) {
|
||||
chosen = base;
|
||||
} else {
|
||||
// Anywhere else has to prove itself: the name AND the size the import
|
||||
// recorded. Without that a row whose file an admin deleted would adopt
|
||||
// an unrelated same-named file one directory up, and downloads would
|
||||
// serve the wrong original.
|
||||
for (const prefix of prefixes) {
|
||||
if (prefix === base) continue;
|
||||
if (await fileMatchesSize(under(root, prefix, row.external_relpath), row.size_bytes)) {
|
||||
chosen = prefix;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (chosen === null) { chosen = base; stranded++; }
|
||||
if (chosen !== base) allUnderBase = false;
|
||||
placements.push([row, chosen]);
|
||||
}
|
||||
|
||||
if (allUnderBase) {
|
||||
folded += rows.length;
|
||||
plan.push({ eventId, base, bulk: true, rows: [], ids: rows.map((r) => r.id) });
|
||||
continue;
|
||||
}
|
||||
|
||||
// Two rows can now target the same path — the same file imported under two
|
||||
// different bases really is one file. Resolved HERE rather than by letting
|
||||
// the write fail: a caught write error cannot tell a genuine duplicate from
|
||||
// a lock or I/O fault, and continuing past one would certify a partial
|
||||
// conversion by writing the marker anyway.
|
||||
//
|
||||
// The loser is DELETED, not skipped. Skipping leaves it holding a
|
||||
// base-relative path that the root-only resolver then reads as
|
||||
// `<root>/<relpath>` — permanently pointing at the wrong place, or
|
||||
// nowhere, with the marker saying the conversion is done. And it is a
|
||||
// duplicate by construction: two rows that resolve to one file is exactly
|
||||
// what migration 186 removes, so it goes through the same helper, which
|
||||
// reparents the feedback and marks and reconciles the face clusters.
|
||||
const claimed = new Map();
|
||||
const resolved = [];
|
||||
const losers = new Map();
|
||||
for (const [row, chosen] of placements) {
|
||||
const next = chosen ? `${chosen}/${row.external_relpath}` : row.external_relpath;
|
||||
const winner = claimed.get(next);
|
||||
if (winner != null) { losers.set(row.id, winner); collided++; continue; }
|
||||
claimed.set(next, row.id);
|
||||
if (chosen === base) folded++; else repaired++;
|
||||
resolved.push([row.id, next]);
|
||||
}
|
||||
plan.push({ eventId, base, bulk: false, rows: resolved, losers });
|
||||
}
|
||||
|
||||
// ---- Phase 2: write, all or nothing. ---------------------------------
|
||||
// The marker rides in the same transaction as the rewrites, so there is no
|
||||
// window where some rows are folded, the marker is absent, and a second run
|
||||
// folds them again — which would put every original one directory deeper,
|
||||
// permanently.
|
||||
const apply = async (trx) => {
|
||||
for (const step of plan) {
|
||||
if (step.bulk) {
|
||||
// BY ID, not by event. Phase 1 runs outside the transaction and can
|
||||
// take minutes probing a cold mount; an import completing in that
|
||||
// window inserts an already root-relative row, and `where event_id`
|
||||
// would prefix it a second time with the stale base.
|
||||
for (const ids of chunk(step.ids)) {
|
||||
await trx('photos')
|
||||
.whereIn('id', ids)
|
||||
.whereNotNull('external_relpath')
|
||||
.update({ external_relpath: trx.raw('? || external_relpath', [`${step.base}/`]) });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Losers first: while they still hold their old path, the survivor has
|
||||
// not taken the value they would collide with.
|
||||
if (step.losers && step.losers.size) {
|
||||
await deleteDuplicatePhotos(trx, step.losers);
|
||||
}
|
||||
|
||||
// Two passes, through a per-row temporary value. The FINAL values are
|
||||
// all distinct, but a final value can equal another row's CURRENT one —
|
||||
// `photo.jpg` repairing to `Trip/photo.jpg` while the existing
|
||||
// `Trip/photo.jpg` is still waiting to fold — so a single pass violates
|
||||
// migration 186's unique index halfway through. And on Postgres that
|
||||
// surfaces as 23505, which run-migrations-safe.js mistakes for "schema
|
||||
// already exists" and records the migration as applied after the
|
||||
// rollback, leaving every path unconverted with no retry.
|
||||
for (const [id] of step.rows) {
|
||||
await trx('photos').where('id', id).update({ external_relpath: `${STAGING_PREFIX}${id}` });
|
||||
}
|
||||
for (const [id, next] of step.rows) {
|
||||
// No catch: collisions were resolved above, so anything failing here is
|
||||
// a real fault and must roll the whole fold back rather than leave a
|
||||
// half-converted table certified by the marker.
|
||||
await trx('photos').where('id', id).update({ external_relpath: next });
|
||||
}
|
||||
}
|
||||
|
||||
await trx('app_settings').insert({
|
||||
setting_key: MARKER,
|
||||
setting_value: JSON.stringify(true),
|
||||
setting_type: 'system',
|
||||
updated_at: new Date().toISOString(),
|
||||
});
|
||||
};
|
||||
|
||||
// A caller already inside a transaction (the restore) passes its trx in as
|
||||
// `knex`; opening a nested one would deadlock SQLite.
|
||||
if (knex.isTransaction) await apply(knex);
|
||||
else await knex.transaction(apply);
|
||||
|
||||
log(`${folded} folded, ${repaired} repaired, ${stranded} left unresolved, ${collided} skipped as duplicates`);
|
||||
return { folded, repaired, stranded, collided };
|
||||
}
|
||||
|
||||
|
||||
module.exports = { foldExternalRelpaths, MARKER };
|
||||
@@ -1,5 +1,5 @@
|
||||
const path = require('path');
|
||||
const { resolveExternalPath } = require('./externalMediaService');
|
||||
const { resolveExternalPhotoPath } = require('./externalMediaService');
|
||||
const { safePathJoin } = require('../utils/fileSecurityUtils');
|
||||
|
||||
const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
|
||||
@@ -40,7 +40,7 @@ function resolvePhotoStorageKey(event, photo) {
|
||||
/**
|
||||
* Resolve absolute photo file path based on event + photo origin
|
||||
* Managed: storage/events/active + photo.path (legacy variants supported)
|
||||
* External reference: EXTERNAL_MEDIA_ROOT + event.external_path + photo.external_relpath
|
||||
* External reference: EXTERNAL_MEDIA_ROOT + photo.external_relpath
|
||||
*/
|
||||
function resolvePhotoFilePath(event, photo) {
|
||||
if (!event || !photo) throw new Error('resolvePhotoFilePath requires event and photo');
|
||||
@@ -61,20 +61,18 @@ function resolvePhotoFilePath(event, photo) {
|
||||
}
|
||||
throw new Error('Missing external_relpath for external photo');
|
||||
}
|
||||
// Normalize duplicate leaf segments (e.g., event.external_path ends with 'individual'
|
||||
// and external_relpath starts with 'individual/') to avoid double segment like
|
||||
// '/external-media/.../individual/individual/file.jpg'
|
||||
let rel = photo.external_relpath;
|
||||
try {
|
||||
const lastSeg = path.basename(event.external_path || '');
|
||||
const firstSeg = rel.split(path.sep)[0];
|
||||
if (lastSeg && firstSeg && lastSeg === firstSeg) {
|
||||
rel = rel.split(path.sep).slice(1).join(path.sep) || '';
|
||||
}
|
||||
} catch (_) {
|
||||
// ignore normalization errors
|
||||
}
|
||||
return resolveExternalPath(event, rel);
|
||||
// external_relpath is relative to EXTERNAL_MEDIA_ROOT, so the event is not
|
||||
// consulted at all (#1163). It used to be relative to event.external_path,
|
||||
// which meant importing a second folder into an event silently moved every
|
||||
// photo already in it.
|
||||
//
|
||||
// The duplicate-leaf-segment normalisation that used to live here went with
|
||||
// it. It stripped the first segment of the relpath when it matched the last
|
||||
// segment of event.external_path — a guess that papered over the
|
||||
// double-prefixing this class of bug produced, and one that actively
|
||||
// corrupts a root-relative path whose first segment legitimately repeats
|
||||
// (external_path 'Trip', relpath 'Trip/x.jpg').
|
||||
return resolveExternalPhotoPath(photo);
|
||||
}
|
||||
|
||||
const storagePath = getStoragePath();
|
||||
|
||||
@@ -541,6 +541,38 @@ async function importFromPicpeak({ picpeakPath, currentAdminId }) {
|
||||
|
||||
const filesRestored = await restoreFiles(staging);
|
||||
|
||||
// External media paths (#1163). knex_migrations is excluded from the
|
||||
// archive, so migration 187 does not re-run after a restore — a pre-#1163
|
||||
// backup would otherwise drop base-relative rows onto an instance that
|
||||
// resolves them from the media root, and every original in the restored
|
||||
// library would be unreachable with nothing logged. The fold is a no-op
|
||||
// when the restored app_settings already carries the marker.
|
||||
//
|
||||
// BEFORE the face requeue below, and for the same reason that requeue sits
|
||||
// after restoreFiles: the worker is live throughout. Queued first, it can
|
||||
// claim an external row while the row is still base-relative, resolve it
|
||||
// against the wrong path with the root-only resolver, and mark the photo
|
||||
// failed — a state only an explicit Re-scan clears, and one the fold does
|
||||
// not undo. Probing a cold mount takes long enough for that to be likely
|
||||
// rather than theoretical.
|
||||
let externalPathsConverted = true;
|
||||
let externalPathError = null;
|
||||
try {
|
||||
const { foldExternalRelpaths } = require('./externalRelpathFold');
|
||||
const result = await foldExternalRelpaths(db, (msg) => logger.info(`picpeakImport: external paths — ${msg}`));
|
||||
if (result.folded || result.repaired) {
|
||||
logger.info(`picpeakImport: folded ${result.folded} external path(s), repaired ${result.repaired}`);
|
||||
}
|
||||
} catch (err) {
|
||||
// NOT swallowed as a footnote. The fold is transactional, so a failure
|
||||
// leaves every external path in the pre-#1163 format while the running
|
||||
// resolver reads from the media root — meaning every original in the
|
||||
// restored library is unreachable. Reporting that as a clean restore
|
||||
// sends the admin away believing it worked.
|
||||
externalPathsConverted = false;
|
||||
externalPathError = err.message;
|
||||
logger.error(`picpeakImport: external path conversion FAILED — originals will not resolve until this is retried: ${err.message}`);
|
||||
}
|
||||
// Face data (#1074): queue ONLY once the files are on disk. The archive
|
||||
// carries no face rows and the export blanked photos.face_status, but the
|
||||
// event toggles come across enabled, so the "enable" transition that
|
||||
@@ -551,23 +583,44 @@ async function importFromPicpeak({ picpeakPath, currentAdminId }) {
|
||||
// instance's files or marks photos failed for originals that are not
|
||||
// there yet — and nothing re-queues them afterwards.
|
||||
try {
|
||||
const requeued = await db('photos')
|
||||
.whereIn('event_id', db('events').select('id').where('face_recognition_enabled', true))
|
||||
.update({
|
||||
face_status: 'pending', face_count: null, face_started_at: null, face_error: null,
|
||||
});
|
||||
if (requeued > 0) {
|
||||
logger.info(`picpeakImport: queued ${requeued} photo(s) for face detection after import`);
|
||||
if (!externalPathsConverted) {
|
||||
// Queueing now would hand the live worker rows whose paths the
|
||||
// resolver cannot follow, and it would mark them 'failed' — a state
|
||||
// only an explicit Re-scan clears. Leave them unqueued; the operator
|
||||
// re-runs the conversion and then re-scans.
|
||||
logger.warn('picpeakImport: skipping face requeue — external paths are unconverted');
|
||||
} else {
|
||||
const requeued = await db('photos')
|
||||
.whereIn('event_id', db('events').select('id').where('face_recognition_enabled', true))
|
||||
.update({
|
||||
face_status: 'pending', face_count: null, face_started_at: null, face_error: null,
|
||||
});
|
||||
if (requeued > 0) {
|
||||
logger.info(`picpeakImport: queued ${requeued} photo(s) for face detection after import`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
logger.debug?.(`picpeakImport: face requeue skipped: ${err.message}`);
|
||||
}
|
||||
|
||||
const usesExternalMedia = await detectExternalMedia();
|
||||
|
||||
logger.info(
|
||||
`[picpeak-import] restored ${tables.length} tables, ${filesRestored} files (externalMedia=${usesExternalMedia}, crossEngine=${crossEngine})`
|
||||
);
|
||||
return { restored: true, tables: tables.length, filesRestored, usesExternalMedia, crossEngine, manifest };
|
||||
return {
|
||||
restored: true,
|
||||
tables: tables.length,
|
||||
filesRestored,
|
||||
usesExternalMedia,
|
||||
crossEngine,
|
||||
manifest,
|
||||
// Surfaced so the caller can warn rather than report an unqualified
|
||||
// success: the rows and files are in place, but the external originals
|
||||
// do not resolve until the conversion is retried (#1163).
|
||||
externalPathsConverted,
|
||||
externalPathError,
|
||||
};
|
||||
} finally {
|
||||
await fsp.rm(staging, { recursive: true, force: true }).catch(() => {});
|
||||
}
|
||||
|
||||
@@ -209,18 +209,20 @@ class RestoreService {
|
||||
|
||||
// Step 6: Perform the actual restore based on type
|
||||
let restoreResult;
|
||||
// Set by the full/database branches; acted on after step 7c so the
|
||||
// schema — and any data conversion those migrations perform — is in
|
||||
// place before the live face worker can claim a row.
|
||||
let needsFaceRequeue = false;
|
||||
let migrationsApplied = true;
|
||||
switch (options.restoreType) {
|
||||
case 'full':
|
||||
restoreResult = await this.performFullRestore(localBackupPath, manifest, options);
|
||||
// After the FILES too — see requeueFaceScans on why the ordering
|
||||
// matters with a live worker.
|
||||
await this.requeueFaceScans();
|
||||
// Deferred to after step 7c — see the requeue there.
|
||||
needsFaceRequeue = true;
|
||||
break;
|
||||
case 'database':
|
||||
restoreResult = await this.performDatabaseRestore(localBackupPath, manifest, options);
|
||||
// Database-only restore: the existing files stay in place, so there
|
||||
// is nothing to race.
|
||||
await this.requeueFaceScans();
|
||||
needsFaceRequeue = true;
|
||||
break;
|
||||
case 'files':
|
||||
restoreResult = await this.performFilesRestore(localBackupPath, manifest, options);
|
||||
@@ -326,11 +328,31 @@ class RestoreService {
|
||||
}
|
||||
this.log('info', 'Post-restore migrations applied');
|
||||
} catch (migErr) {
|
||||
// Also gates the face requeue below: a pre-#1163 backup whose
|
||||
// migration 187 did not run still holds event-relative external
|
||||
// paths, and queueing those hands the live worker rows it will
|
||||
// resolve from the media root and mark 'failed' — a state the later
|
||||
// retry does not clear.
|
||||
migrationsApplied = false;
|
||||
this.log('warn',
|
||||
'Post-restore migrate:safe failed — restore data is in place but the schema may lag the running image. ' +
|
||||
`A container restart will retry via wait-for-db.sh. Error: ${migErr.message}`);
|
||||
}
|
||||
|
||||
// Faces last (#1163). This used to run in step 6, before the migrations
|
||||
// above. On a backup predating migration 187 that meant queueing rows
|
||||
// whose external_relpath was still relative to events.external_path
|
||||
// while the running code resolves from the media root — so the live
|
||||
// worker resolved them against the wrong path and marked them 'failed',
|
||||
// a state the later fold does not clear and only an explicit Re-scan
|
||||
// does. The files are already in place by step 6, so deferring costs
|
||||
// nothing and closes that window.
|
||||
if (needsFaceRequeue && migrationsApplied) {
|
||||
await this.requeueFaceScans();
|
||||
} else if (needsFaceRequeue) {
|
||||
this.log('warn', 'Skipping face requeue — post-restore migrations did not complete, so photo paths may be unconverted');
|
||||
}
|
||||
|
||||
// Step 8: Clean up temporary files
|
||||
if (localBackupPath !== options.source) {
|
||||
await fs.unlink(localBackupPath).catch(err =>
|
||||
|
||||
Reference in New Issue
Block a user