paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(external-media): watch reference folders and import new files automatically (#1345)

Browse Source 
* feat(external-media): watch reference folders and import new files automatically

Managed uploads dropped into storage/events/active are picked up by the
chokidar watcher; external media had no equivalent, so a NAS folder that
keeps growing needed an admin to open the event and press Import every
time. Relates to issue 1187.

- The import pass moves out of the route into
  services/externalImportService.js. The watcher and the Import button
  now run the identical function; the route only validates and maps
  errors to status codes.
- Mutual exclusion is the per-event claim from maintenanceJobState
  (`external_import:<id>`, seeded on demand by the new ensure()) instead
  of the in-process Set. The Set stopped a double-click in one process;
  the claim also stops the watcher on a second replica, or an admin
  clicking while the watcher is mid-run elsewhere. The run heartbeats so
  a claim from a dead process is taken over.
- services/externalMediaWatcher.js: per-event opt-in via the new
  events.external_watch column (migration 208), chokidar with
  awaitWriteFinish so a copy in flight is not imported half-written,
  debounced full pass per change, a timer sweep every 15 minutes as the
  fallback for NFS/SMB mounts that deliver no inotify events, optional
  stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched
  events is re-read every minute, so the toggle works from any replica.
  A watcher that just started runs one pass immediately.
- Deletions are ignored on purpose: a file vanishing from a NAS is at
  least as likely to be a reorganisation or a dropped mount as an
  intentional removal, and acting on it would delete a guest-visible
  photo. Rows whose file is gone stay, as they do today.
- Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local.
- Quiet system passes stay out of the activity log; runs that imported
  something are logged with actor external-media-watcher.
- Frontend: "Watch folder for new files" checkbox under the external
  folder picker, status line in view mode, EN/DE strings.

* fix(external-media): close the review gaps in the folder watcher

Codex review of the watcher, round 1. All six findings were real:

- Enabling the watcher, or pointing an enabled one at another folder,
  now requires photos.upload — the permission the manual Import already
  requires. events.edit alone was a way around it. Only the transition
  is checked, so a role without photos.upload can still edit an
  already-watched event. The checkbox is disabled for such roles.
- Automatic passes defer files that are still changing: anything
  modified inside the stability window, or whose size moves across one
  wait of that window, is left for the next pass. chokidar's
  awaitWriteFinish only settles the file that fired the event, and the
  sweep sees no events at all, so a sibling still being copied could be
  inserted half-written and then skipped forever.
- Photos an admin deleted are not brought back by the sweep. The delete
  routes record the file in external_import_exclusions (migration 209);
  automatic passes skip the list, the manual Import ignores it and
  clears it for what it imports.
- The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three
  compose files; they were documented but the backend services use
  explicit environment lists, so the kill switch did nothing.
- A pass re-checks is_active / is_archived at run time, not only in the
  minutely reconcile.
- The lease is renewed on a timer for the whole run, walk included, and
  ownership is checked before the event row is touched.

* fix(external-media): make automatic passes follow the row, not rewrite it

Codex review round 2, four findings, all applied:

- The event update route drops non-canonical spellings of external_watch
  and external_path before the permission guard. SQLite resolves column
  names case-insensitively, so `External_Watch` reached the column while
  the guard only looked at the lowercase key.
- Exclusions are checked per file at insert time, not against a
  snapshot taken before the settle wait. A photo deleted during the wait
  was present in the snapshot and got re-inserted by the loop.
- An automatic pass no longer writes source_mode / external_path. It
  re-reads the row after the walk and the settle wait and stops if the
  folder changed or the event went managed; the manual Import is the
  only writer. The options are now `automatic` + `settleMs`.
- A pass that deferred files re-arms the debounced import, so a file
  copied just before the watcher started is not stranded when the sweep
  is disabled.

* fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying

Codex review round 3, both findings applied:

- recordExclusions keys on external_relpath alone. A replaced external
  photo becomes managed but keeps its relpath on purpose, and deleting
  that replacement must not republish the NAS original.
- An automatic pass checks the full watcher predicate (reference mode,
  same folder, watch on, active, not archived) before it inserts and on
  every heartbeat tick during the loop, and stops as soon as the event
  no longer qualifies.

---------

Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
2026-09-07 22:29:55 +02:00
committed by GitHub
co-authored by Paul Nothaft
parent 1b3f721d10
commit 8cc7d7d14a
21 changed files with 1580 additions and 327 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/adminEvents/crud.js
+41 -1
View File
@@ -7,7 +7,7 @@ const { db, logActivity } = require('../../database/db');
const { formatBoolean } = require('../../utils/dbCompat');
const { slugify } = require('../../utils/slug');
const { adminAuth } = require('../../middleware/auth');
const { requirePermission } = require('../../middleware/permissions');
const { requirePermission, userHasAllPermissions } = require('../../middleware/permissions');
const { IDENTITY_PRESERVING_NORMALIZE_EMAIL } = require('../../utils/emailNormalization');
const bcrypt = require('bcrypt');
const crypto = require('crypto');
@@ -1601,6 +1601,7 @@ module.exports = (router) => {
body('allow_presigned_download').optional().isBoolean(),
body('source_mode').optional().isIn(['managed', 'reference']),
body('external_path').optional({ nullable: true }).isString().trim(),
body('external_watch').optional().isBoolean(),
body('require_password').optional().isBoolean(),
// Download protection settings
body('protection_level').optional().isIn(['basic', 'standard', 'enhanced', 'maximum']),
@@ -1811,8 +1812,47 @@ module.exports = (router) => {
updates.external_path = trimmedPath || null;
}
// The permission guard below inspects `external_watch` / `external_path`
// by exact name, but SQLite resolves column names case-insensitively, so
// `External_Watch` would sail past it and still land on the column.
// Anything that is one of these two keys in any spelling other than the
// canonical one is dropped here, before the guard.
for (const key of Object.keys(updates)) {
const lower = key.toLowerCase();
if ((lower === 'external_watch' || lower === 'external_path') && key !== lower) delete updates[key];
}
// Folder watcher opt-in (issue 1187). Written through formatBoolean
// like the other event flags so SQLite gets 0/1; a managed event has
// no folder to watch, so the switch is cleared with the path.
if (Object.prototype.hasOwnProperty.call(updates, 'external_watch')) {
updates.external_watch = formatBoolean(updates.external_watch === true || updates.external_watch === 'true');
}
// Enabling the watcher, or pointing an enabled one at another folder,
// makes the server import on this admin's behalf — which the manual
// Import endpoint requires photos.upload for. events.edit alone must
// not be a way around that. Only transitions are checked: a save that
// leaves an already-watched event as it is stays an events.edit
// operation, so a role without photos.upload can still edit the rest.
if (Object.prototype.hasOwnProperty.call(updates, 'external_watch') || Object.prototype.hasOwnProperty.call(updates, 'external_path')) {
const current = await db('events').where('id', id).select('external_watch', 'external_path').first();
const wasWatched = Boolean(current?.external_watch);
const willWatch = Object.prototype.hasOwnProperty.call(updates, 'external_watch')
? Boolean(updates.external_watch)
: wasWatched;
const pathChanges = Object.prototype.hasOwnProperty.call(updates, 'external_path')
&& (updates.external_path || null) !== (current?.external_path || null);
if (willWatch && ((!wasWatched) || pathChanges)) {
if (!(await userHasAllPermissions(req.admin.id, ['photos.upload']))) {
return res.status(403).json({ error: 'The photos.upload permission is required to enable automatic imports for this folder' });
}
}
}
if (updates.source_mode === 'managed') {
updates.external_path = null;
updates.external_watch = formatBoolean(false);
}
if (updates.source_mode === 'reference' && (updates.external_path === null || updates.external_path === undefined)) {
backend/src/routes/adminExternalMedia.js
+29 -326
View File
@@ -1,32 +1,17 @@
const express = require('express');
const path = require('path');
const fs = require('fs').promises;
const { adminAuth } = require('../middleware/auth');
const { requirePermission } = require('../middleware/permissions');
const { requireEventOwnership } = require('../middleware/ownership');
const { list, resolveExternalPath } = require('../services/externalMediaService');
const { db, logActivity } = require('../database/db');
const sharp = require('sharp');
const { list } = require('../services/externalMediaService');
const logger = require('../utils/logger');
const { generateThumbnail, extractCaptureDate, orientedDimensions } = require('../services/imageProcessor');
const { isUniqueViolation } = require('../utils/dbErrors');
const {
importExternalFolder,
ImportInProgressError,
EventNotFoundError,
} = require('../services/externalImportService');
const router = express.Router();
// Events with an import running in THIS process (#1162).
//
// The second line of defence, not the first: migration 186 puts a unique index
// on (event_id, external_relpath), and that is what actually makes a duplicate
// impossible — it holds across replicas, across restarts, and against anything
// that inserts external rows without going through this route.
//
// This set exists for the reason the duplicates got filed in the first place:
// a large tree takes long enough that the run LOOKS hung, so admins click
// again. Letting that second run walk the whole tree only to have every insert
// bounce off the index wastes minutes of CPU and reports a nonsense
// `skipped: 6012` back. Failing it immediately with 409 says what happened.
const importsInFlight = new Set();
// GET /api/admin/external-media/list?path=relative/dir
router.get('/list', adminAuth, requirePermission('photos.view'), async (req, res) => {
try {
@@ -42,324 +27,42 @@ router.get('/list', adminAuth, requirePermission('photos.view'), async (req, res
}
});
// Helper to recursively collect files under a directory, filtered by image extensions
async function walkDir(dir, baseDir) {
const results = [];
const entries = await fs.readdir(dir, { withFileTypes: true });
for (const e of entries) {
if (e.name.startsWith('.')) continue;
const full = path.join(dir, e.name);
if (e.isDirectory()) {
results.push(...await walkDir(full, baseDir));
} else if (e.isFile()) {
const ext = path.extname(e.name).toLowerCase();
if (['.jpg', '.jpeg', '.png', '.webp'].includes(ext)) {
const rel = path.relative(baseDir, full);
results.push({ full, rel, name: e.name });
}
}
}
return results;
}
// POST /api/admin/events/:id/import-external
// Body: { external_path: string, recursive?: boolean, map?: { individual?: string, collages?: string } }
//
// The import itself lives in services/externalImportService.js so the folder
// watcher (issue 1187) runs the identical pass without an HTTP request. This
// handler only validates, maps the service's errors to status codes, and
// records who asked.
router.post('/events/:id/import-external', adminAuth, requirePermission('photos.upload'), requireEventOwnership, async (req, res) => {
const eventId = parseInt(req.params.id);
if (importsInFlight.has(eventId)) {
return res.status(409).json({
error: 'An import is already running for this event. Wait for it to finish before starting another.'
});
}
importsInFlight.add(eventId);
const { external_path, recursive = true, map = { individual: 'individual', collages: 'collages' } } = req.body || {};
if (!external_path) return res.status(400).json({ error: 'external_path is required' });
try {
const { external_path, recursive = true, map = { individual: 'individual', collages: 'collages' } } = req.body || {};
if (!external_path) return res.status(400).json({ error: 'external_path is required' });
// Load event
const event = await db('events').where('id', eventId).first();
if (!event) return res.status(404).json({ error: 'Event not found' });
const baseAbs = resolveExternalPath({ external_path }, '');
// What gets STORED on the row (#1163). `f.rel` stays relative to the
// imported folder because the type inference below reads its first segment
// ('individual' / 'collages'); external_relpath is written relative to
// EXTERNAL_MEDIA_ROOT so the row does not depend on a column this very
// handler is about to overwrite.
const basePrefix = String(external_path).replace(/^\/+|\/+$/g, '');
const toRootRelative = (rel) => (basePrefix ? path.join(basePrefix, rel) : rel);
// Collect files
const files = recursive ? await walkDir(baseAbs, baseAbs) : (await fs.readdir(baseAbs, { withFileTypes: true }))
.filter(e => e.isFile())
.map(e => ({ full: path.join(baseAbs, e.name), rel: e.name, name: e.name }))
.filter(f => ['.jpg', '.jpeg', '.png', '.webp'].includes(path.extname(f.name).toLowerCase()));
// Prepare file metadata and deduplicate by filename within type (keep largest)
let skipped = 0;
const preparedFiles = [];
for (const f of files) {
try {
const stats = await fs.stat(f.full);
const segs = f.rel.split(path.sep);
let type = 'individual';
if (segs[0] === map.collages) type = 'collage';
if (segs[0] === map.individual) type = 'individual';
preparedFiles.push({ ...f, type, size: stats.size });
} catch (err) {
skipped++;
}
}
const dedupeMap = new Map();
for (const file of preparedFiles) {
const dedupeKey = `${file.type}:${path.basename(file.rel).toLowerCase()}`;
const existing = dedupeMap.get(dedupeKey);
if (!existing || file.size > existing.size) {
if (existing) skipped++;
dedupeMap.set(dedupeKey, file);
} else {
skipped++;
}
}
// Point the event at the new directory BEFORE inserting anything.
//
// Two reasons, both about what a half-finished import leaves behind. The
// update used to run after the loop, so an import that died at photo 500
// of 1000 left those 500 rows carrying external_relpath into the NEW tree
// while the event still resolved against the OLD one — every one of them
// unreadable. And with face detection on, enqueueEvent accepts
// processing_status NULL (faceProcessor.js:243-246), which these inserts
// leave unset, so an admin hitting the toggle or Re-scan mid-import could
// queue those same rows against the stale path and burn them to 'failed'.
//
// Safe to do first for existing MANAGED photos: photo.source_origin takes
// precedence over event.source_mode in both resolvers (photoResolver.js:23,
// :52) and is NOT NULL defaulting to 'managed', so flipping source_mode
// does not touch them.
//
// Safe for existing EXTERNAL rows too, as of #1163. It was not: relpaths
// were stored relative to external_path, so overwriting the column here
// rebased every row already in the event onto the new folder — quietly,
// because their thumbnails were already on local disk and the grid carried
// on rendering. Rows now carry a root-relative path and this write cannot
// reach them.
await db('events').where('id', eventId).update({ source_mode: 'reference', external_path });
let imported = 0;
let thumbnailsGenerated = 0;
let thumbnailsFailed = 0;
// Face detection (#1090). Managed uploads are enqueued by photoProcessor,
// which sets face_status 'pending' once a photo is processed
// (photoProcessor.js:573) — but external media never goes through it, it
// is inserted directly here. Before #1090 that was invisible, because
// faceProcessor skipped externals anyway; now that they are scannable, an
// import into an already-enabled event would still sit unscanned until
// someone pressed Re-scan.
//
// Ids are collected unconditionally and the setting is read at the END,
// not here: this loop can run for many minutes on a large library, and an
// admin who enables detection during it would otherwise leave every photo
// imported after that moment stuck at NULL forever — the toggle endpoint
// only queues rows that already existed when it fired.
//
// The event path is already committed (above), so the enqueue below is
// free of the ordering hazard it used to carry. It stays at the end anyway
// so the setting can be read after the loop, and it only touches rows that
// are still untouched — see the whereNull there. No video guard needed:
// walkDir collects only jpg/jpeg/png/webp.
const importedPhotoIds = [];
// Insert photos
for (const f of dedupeMap.values()) {
// Infer type by subfolder names
const segs = f.rel.split(path.sep);
let type = 'individual';
if (segs[0] === map.collages) type = 'collage';
if (segs[0] === map.individual) type = 'individual';
const relFromRoot = toRootRelative(f.rel);
try {
// Fast path only. This SELECT settles the common case — a re-import of
// a folder already in the event — without paying for a stat and a
// Sharp metadata read per file. It is NOT the guard: those two calls
// sit between here and the INSERT below, which is exactly the window
// two overlapping imports both walked through (#1162). The unique
// index from migration 186 is the guard, and the catch below is how
// this loop converges when it fires.
const exists = await db('photos')
.where({ event_id: eventId, external_relpath: relFromRoot })
.first();
if (exists) { skipped++; continue; }
const stats = await fs.stat(f.full);
// Extract dimensions via Sharp
let width = null;
let height = null;
try {
const metadata = await sharp(f.full).metadata();
// Oriented, not raw: a portrait shot from a body that tags rather
// than rotates reports landscape dimensions, and the grid would size
// its tile from those (#1185).
({ width, height } = orientedDimensions(metadata));
} catch (dimErr) {
logger.warn(`Could not extract dimensions for ${f.rel}: ${dimErr.message}`);
}
// Capture date from EXIF (#1172). Managed uploads get this from
// photoProcessor, which external media never goes through — so
// captured_at stayed NULL for every externally imported photo, and the
// gallery's "Date Taken" sort silently degraded into import order via
// its COALESCE fallback. On a library imported in two batches that put
// the first days of a trip after the last ones.
//
// Read here because the file is already open a few lines above for the
// dimensions, so this costs one more read of the same source rather
// than a second pass over the mount.
//
// Best-effort, exactly like the dimensions: a source without EXIF, or
// one Sharp/exifr cannot parse, imports with captured_at NULL and
// falls back to uploaded_at as before.
let capturedAt = null;
try {
capturedAt = await extractCaptureDate(f.full);
} catch (dateErr) {
logger.warn(`Could not extract capture date for ${f.rel}: ${dateErr.message}`);
}
let inserted;
try {
inserted = await db('photos')
.insert({
event_id: eventId,
filename: f.name,
// The camera-original name (#745). External ingest never sets
// original_filename, and NAS-mounted galleries are among the
// most likely to be driven from Lightroom — without this the
// round-trip has nothing to match a RAW against.
source_filename: f.name,
// Keep path as a hint for legacy code but not used for resolution in external mode
path: path.join(event.slug, f.name),
thumbnail_path: null,
type,
size_bytes: stats.size,
width,
height,
source_origin: 'external',
external_relpath: relFromRoot,
// .toISOString() rather than the Date: inside jest, Dates handed
// to the sqlite3 binding land as the literal string
// "[object Object]" (see CLAUDE.md). Strings round-trip on both
// engines.
captured_at: capturedAt ? capturedAt.toISOString() : null
})
.returning('id');
} catch (insertErr) {
// Another writer inserted this exact path while we were reading
// metadata. That is the outcome the index exists to produce, and it
// is a skip rather than a failure — the row is there, it just isn't
// ours. Counting it as `skipped` keeps the reported totals honest;
// before the index this landed in the outer catch as a nameless
// failure, or (more often) never fired at all and duplicated the row.
if (isUniqueViolation(insertErr)) { skipped++; continue; }
throw insertErr;
}
const photoId = Array.isArray(inserted) && inserted.length
? (typeof inserted[0] === 'object' ? inserted[0].id : inserted[0])
: null;
// Generate the thumbnail right away so the gallery grid can use the
// managed thumbnail endpoint instead of falling back to the full
// NAS-streamed original (#423). Best-effort: a single failure logs
// a warning and leaves thumbnail_path=null — the gallery's
// ensureThumbnail will retry lazily on first view. The cost of
// doing this synchronously is ~100-300ms per image; for the
// worst-case 1000-photo import that's still under the 5-minute
// request timeout typical of the import flow.
if (photoId != null) {
try {
const outputBasename = `ext${photoId}_${path.basename(f.rel)}`;
const thumbnailPath = await generateThumbnail(f.full, { outputBasename });
if (thumbnailPath) {
await db('photos').where({ id: photoId }).update({ thumbnail_path: thumbnailPath });
thumbnailsGenerated++;
} else {
thumbnailsFailed++;
}
} catch (thumbErr) {
thumbnailsFailed++;
logger.warn(`Thumbnail generation failed for external photo ${photoId} (${f.rel}): ${thumbErr.message}`);
}
}
if (photoId != null) importedPhotoIds.push(photoId);
imported += (inserted?.length ? 1 : 0);
} catch (e) {
skipped++;
}
}
// The event already resolves to the new directory (set before the loop),
// so the queue is safe to open. Still done here rather than on insert so
// the setting below is read after the loop. Guarded the same way
// photoProcessor guards it
// (both the global flag and the per-event toggle), so installs without the
// feature still never write a face_status. Re-read here rather than before
// the loop so a toggle flipped mid-import is honoured.
let queueFaces = false;
try {
const { isEnabledForEvent } = require('../services/faceSettings');
const freshEvent = await db('events').where('id', eventId).first();
queueFaces = await isEnabledForEvent(freshEvent);
} catch (err) {
// Never let the face feature break an import — the photos are the point.
logger.warn(`Could not resolve face settings for event ${eventId}: ${err.message}`);
}
// Chunked because SQLite caps a statement at 999 bound parameters and an
// import can be far larger than that.
if (queueFaces && importedPhotoIds.length) {
let queued = 0;
for (let i = 0; i < importedPhotoIds.length; i += 500) {
// whereNull, not a blanket set. Committing the event path before the
// loop means a toggle or Re-scan firing mid-import can now genuinely
// queue and even finish some of these rows — so an unconditional
// update would drag 'done' rows back to 'pending' for a duplicate
// scan, and knock 'processing' rows out from under the worker
// mid-flight. Only rows nothing has touched are ours to queue.
queued += await db('photos')
.whereIn('id', importedPhotoIds.slice(i, i + 500))
.whereNull('face_status')
.update({ face_status: 'pending' });
}
logger.info(`Queued ${queued} of ${importedPhotoIds.length} imported external photo(s) for face scanning (event ${eventId})`);
}
await logActivity(
'external_import_completed',
{ event_id: eventId, imported, skipped, thumbnailsGenerated, thumbnailsFailed, external_path },
const result = await importExternalFolder({
eventId,
{ type: 'admin' }
);
res.json({ imported, skipped, thumbnailsGenerated, thumbnailsFailed });
externalPath: external_path,
recursive,
map,
actor: { type: 'admin', id: req.admin?.id, name: req.admin?.username },
});
res.json(result);
} catch (error) {
if (error instanceof EventNotFoundError) {
return res.status(404).json({ error: 'Event not found' });
}
if (error instanceof ImportInProgressError) {
// Another run holds this event — a double-click, or the watcher on any
// replica mid-pass. Say so rather than walking the tree a second time.
return res.status(409).json({ error: error.message });
}
logger.error('External media import failed', {
eventId: req.params.id,
externalPath: req.body?.external_path,
error: error.message
});
res.status(500).json({ error: 'Failed to import external media' });
} finally {
// In `finally` and not at the end of `try`: an import that throws must
// still release the event, or a single failure locks out every retry
// until the process restarts.
importsInFlight.delete(eventId);
}
});
backend/src/routes/adminPhotos.js
+5
View File
@@ -796,6 +796,9 @@ router.delete('/:eventId/photos/:photoId', adminAuth, requirePermission('photos.
logger.warn(`deletePhoto: face purge failed for photo ${photoId}`, { error: err.message });
}
// An external photo's file stays on the NAS; make sure the folder watcher
// (issue 1187) does not re-import it on its next pass.
await require('../services/externalImportService').recordExclusions(Number(eventId), [photo]);
await db('photos').where({ id: photoId }).delete();
// Log activity (event was fetched above for storage key resolution)
@@ -1037,6 +1040,8 @@ router.post('/:eventId/photos/bulk-delete', adminAuth, requirePermission('photos
}
// Delete from database
// Same as the single delete: keep the watcher from bringing these back.
await require('../services/externalImportService').recordExclusions(Number(eventId), photos);
await db('photos')
.whereIn('id', photoIds)
.where('event_id', eventId)
backend/src/services/externalImportService.js
+585
View File
@@ -0,0 +1,585 @@
/**
* Import the files under an external-media folder into an event.
*
* This used to live inline in POST /events/:id/import-external. It moved here
* because the folder watcher (issue 1187) needs to run the exact same pass
* the Import button runs — same walk, same dedupe, same insert, same
* thumbnail and face handling — without going through HTTP.
*
* Mutual exclusion is the database claim from maintenanceJobState, keyed per
* event. It replaces the in-process Set the route used to keep (#1162): that
* Set stopped a double-click in ONE process, but a watcher on a second
* replica, or an admin clicking Import while the watcher is mid-run on
* another, would still walk the same tree twice. The unique index from
* migration 186 keeps that from duplicating rows; the claim keeps it from
* wasting the walk. The run heartbeats as it goes so a claim from a process
* that died is taken over rather than held forever.
*/
const path = require('path');
const fs = require('fs').promises;
const sharp = require('sharp');
const { db, logActivity } = require('../database/db');
const logger = require('../utils/logger');
const { resolveExternalPath } = require('./externalMediaService');
const { generateThumbnail, extractCaptureDate, orientedDimensions } = require('./imageProcessor');
const { isUniqueViolation } = require('../utils/dbErrors');
const jobState = require('./maintenanceJobState');
const IMAGE_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.webp'];
class ImportInProgressError extends Error {
constructor(eventId) {
super('An import is already running for this event. Wait for it to finish before starting another.');
this.name = 'ImportInProgressError';
this.eventId = eventId;
}
}
class EventNotFoundError extends Error {
constructor(eventId) {
super('Event not found');
this.name = 'EventNotFoundError';
this.eventId = eventId;
}
}
const jobNameFor = (eventId) => `external_import:${eventId}`;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
/**
* Remember that an admin deleted these photos, so an automatic pass does not
* bring them back (migration 209). Called from the photo delete routes.
*
* Keyed on external_relpath alone, not on source_origin: a replaced external
* photo (photoReplacementService) becomes `managed` but deliberately keeps
* its relpath, and deleting that replacement must not republish the NAS
* original either. Rows without a relpath were never imported from the
* folder and are skipped. Insert failures are swallowed on purpose: a
* duplicate means the row is already there, and anything else must not turn
* a successful delete into a 500.
*/
async function recordExclusions(eventId, photos) {
for (const photo of photos) {
if (!photo.external_relpath) continue;
try {
await db('external_import_exclusions').insert({ event_id: eventId, external_relpath: photo.external_relpath });
} catch (err) {
if (!isUniqueViolation(err)) {
logger.warn(`Could not record import exclusion for photo ${photo.id}: ${err.message}`);
}
}
}
}
/**
* Root-relative paths of the rows an event already has, for a set of
* candidates. Chunked for SQLite's bound-parameter cap.
*/
async function existingRelpaths(eventId, relpaths) {
const found = new Set();
for (let i = 0; i < relpaths.length; i += 500) {
const rows = await db('photos')
.where({ event_id: eventId })
.whereIn('external_relpath', relpaths.slice(i, i + 500))
.select('external_relpath');
for (const r of rows) found.add(r.external_relpath);
}
return found;
}
// Helper to recursively collect files under a directory, filtered by image extensions
async function walkDir(dir, baseDir) {
const results = [];
const entries = await fs.readdir(dir, { withFileTypes: true });
for (const e of entries) {
if (e.name.startsWith('.')) continue;
const full = path.join(dir, e.name);
if (e.isDirectory()) {
results.push(...await walkDir(full, baseDir));
} else if (e.isFile()) {
const ext = path.extname(e.name).toLowerCase();
if (IMAGE_EXTENSIONS.includes(ext)) {
const rel = path.relative(baseDir, full);
results.push({ full, rel, name: e.name });
}
}
}
return results;
}
/**
* Run one import pass.
*
* @param {object} opts
* @param {number} opts.eventId
* @param {string} opts.externalPath folder relative to EXTERNAL_MEDIA_ROOT
* @param {boolean} [opts.recursive=true]
* @param {{individual?: string, collages?: string}} [opts.map]
* @param {object|null} [opts.actor] passed through to logActivity
* @param {boolean} [opts.automatic=false] the watcher's pass, as opposed to
* the Import button. An automatic pass never rewrites the event's source
* configuration (it follows the row, and stops if the row moved under it),
* skips files an admin deleted from this event (migration 209), and defers
* files that are still changing. The manual Import writes the folder it was
* given onto the event, ignores the exclusion list and clears it for what it
* imports.
* @param {number} [opts.settleMs=0] automatic passes: a new file modified
* within the last settleMs, or whose size moves during a wait of settleMs,
* is left for the next pass instead of being imported half-copied
* @returns {Promise<{imported:number, skipped:number, deferred:number, excluded:number, thumbnailsGenerated:number, thumbnailsFailed:number}>}
* @throws {EventNotFoundError} when the event does not exist
* @throws {ImportInProgressError} when another run holds the claim for this event
*/
async function importExternalFolder({
eventId,
externalPath,
recursive = true,
map = { individual: 'individual', collages: 'collages' },
actor = null,
automatic = false,
settleMs = 0,
}) {
const external_path = externalPath;
// Load event
const event = await db('events').where('id', eventId).first();
if (!event) throw new EventNotFoundError(eventId);
// The claim comes AFTER the event lookup and BEFORE anything touches the
// filesystem: a large tree takes long enough that a run looks hung and
// admins click again, and letting that second run walk the whole tree only
// to bounce every insert off the unique index wastes minutes of CPU and
// reports a nonsense `skipped: 6012` back. Failing fast says what happened.
const jobName = jobNameFor(eventId);
await jobState.ensure(jobName);
const token = await jobState.claim(jobName);
if (!token) throw new ImportInProgressError(eventId);
// Renew the lease on a clock for the WHOLE run, walk included. A large tree
// on a slow mount can take longer than the stale window just to list, and
// a runner declared stale during that phase would hand the folder to a
// second runner while this one is about to start inserting. `lost` is
// checked before the event is touched and on every loop iteration.
let lost = false;
// Automatic passes follow the row. The watcher decided to run this pass
// from a snapshot of the event that may be a minute old by the time the
// walk and the settle wait are done — and a long pass over a slow mount
// can outlive several admin saves. The pass stops as soon as the event no
// longer qualifies: watcher off, archived, deactivated, switched to
// managed, or pointed at another folder.
const stillEligible = async () => {
const now = await db('events')
.where('id', eventId)
.select('source_mode', 'external_path', 'external_watch', 'is_active', 'is_archived')
.first();
return Boolean(now)
&& now.source_mode === 'reference'
&& (now.external_path || '') === String(external_path)
&& Boolean(now.external_watch)
&& Boolean(now.is_active)
&& !now.is_archived;
};
const heartbeatTimer = setInterval(() => {
jobState.heartbeat(jobName, token).then((ok) => { if (!ok) lost = true; });
if (automatic) {
stillEligible().then((ok) => { if (!ok) lost = true; }).catch(() => {});
}
}, jobState.HEARTBEAT_INTERVAL_MS);
heartbeatTimer.unref?.();
try {
const baseAbs = resolveExternalPath({ external_path }, '');
// What gets STORED on the row (#1163). `f.rel` stays relative to the
// imported folder because the type inference below reads its first segment
// ('individual' / 'collages'); external_relpath is written relative to
// EXTERNAL_MEDIA_ROOT so the row does not depend on a column this very
// function is about to overwrite.
const basePrefix = String(external_path).replace(/^\/+|\/+$/g, '');
const toRootRelative = (rel) => (basePrefix ? path.join(basePrefix, rel) : rel);
// Collect files
const files = recursive ? await walkDir(baseAbs, baseAbs) : (await fs.readdir(baseAbs, { withFileTypes: true }))
.filter(e => e.isFile())
.map(e => ({ full: path.join(baseAbs, e.name), rel: e.name, name: e.name }))
.filter(f => IMAGE_EXTENSIONS.includes(path.extname(f.name).toLowerCase()));
// Prepare file metadata and deduplicate by filename within type (keep largest)
let skipped = 0;
const preparedFiles = [];
for (const f of files) {
try {
const stats = await fs.stat(f.full);
const segs = f.rel.split(path.sep);
let type = 'individual';
if (segs[0] === map.collages) type = 'collage';
if (segs[0] === map.individual) type = 'individual';
preparedFiles.push({ ...f, type, size: stats.size });
} catch (err) {
skipped++;
}
}
const dedupeMap = new Map();
for (const file of preparedFiles) {
const dedupeKey = `${file.type}:${path.basename(file.rel).toLowerCase()}`;
const existing = dedupeMap.get(dedupeKey);
if (!existing || file.size > existing.size) {
if (existing) skipped++;
dedupeMap.set(dedupeKey, file);
} else {
skipped++;
}
}
let deferred = 0;
let excluded = 0;
// Automatic passes: a file still being copied onto the mount is left for
// the next pass. chokidar's awaitWriteFinish settles only the file that
// fired the event, not its siblings, and the sweep sees no events at all
// — so the check is done here, on the candidates that would actually be
// inserted: anything modified within settleMs, or whose size moves across
// a wait of settleMs, is deferred. One wait per pass, not per file. The
// manual Import does not need it: an admin presses it when the copy is
// done.
if (automatic && settleMs > 0) {
const candidates = [...dedupeMap.values()];
const known = await existingRelpaths(eventId, candidates.map((f) => toRootRelative(f.rel)));
const fresh = candidates.filter((f) => !known.has(toRootRelative(f.rel)));
{
const unsettled = [];
const before = new Map();
for (const f of fresh) {
if (!dedupeMap.has(`${f.type}:${path.basename(f.rel).toLowerCase()}`)) continue;
try {
const st = await fs.stat(f.full);
if (Date.now() - st.mtimeMs < settleMs) unsettled.push(f);
else before.set(f.full, st.size);
} catch (_) {
unsettled.push(f);
}
}
if (before.size) {
await sleep(settleMs);
for (const f of fresh) {
if (!before.has(f.full)) continue;
try {
const st = await fs.stat(f.full);
if (st.size !== before.get(f.full) || Date.now() - st.mtimeMs < settleMs) unsettled.push(f);
} catch (_) {
unsettled.push(f);
}
}
}
for (const f of unsettled) {
deferred++;
dedupeMap.delete(`${f.type}:${path.basename(f.rel).toLowerCase()}`);
}
if (deferred) logger.info(`External import for event ${eventId}: ${deferred} file(s) still changing, left for the next pass`);
}
}
if (lost) throw new ImportInProgressError(eventId);
if (automatic) {
// Follow the row, never write it. Writing source_mode/external_path
// here would silently undo a save made while the tree was being walked
// or the settle wait was running; and importing into an event that no
// longer qualifies is wrong too. Re-read and stop if the row moved.
// The heartbeat tick keeps re-checking during the loop.
if (!(await stillEligible())) {
logger.info(`External import for event ${eventId}: event no longer eligible, nothing imported`);
const empty = { imported: 0, skipped, deferred, excluded, thumbnailsGenerated: 0, thumbnailsFailed: 0 };
await jobState.release(jobName, token, null);
return empty;
}
} else {
// Point the event at the new directory BEFORE inserting anything.
//
// Two reasons, both about what a half-finished import leaves behind. The
// update used to run after the loop, so an import that died at photo 500
// of 1000 left those 500 rows carrying external_relpath into the NEW tree
// while the event still resolved against the OLD one — every one of them
// unreadable. And with face detection on, enqueueEvent accepts
// processing_status NULL (faceProcessor.js:243-246), which these inserts
// leave unset, so an admin hitting the toggle or Re-scan mid-import could
// queue those same rows against the stale path and burn them to 'failed'.
//
// Safe to do first for existing MANAGED photos: photo.source_origin takes
// precedence over event.source_mode in both resolvers (photoResolver.js:23,
// :52) and is NOT NULL defaulting to 'managed', so flipping source_mode
// does not touch them.
//
// Safe for existing EXTERNAL rows too, as of #1163. It was not: relpaths
// were stored relative to external_path, so overwriting the column here
// rebased every row already in the event onto the new folder — quietly,
// because their thumbnails were already on local disk and the grid carried
// on rendering. Rows now carry a root-relative path and this write cannot
// reach them.
await db('events').where('id', eventId).update({ source_mode: 'reference', external_path });
}
let imported = 0;
let thumbnailsGenerated = 0;
let thumbnailsFailed = 0;
// Face detection (#1090). Managed uploads are enqueued by photoProcessor,
// which sets face_status 'pending' once a photo is processed
// (photoProcessor.js:573) — but external media never goes through it, it
// is inserted directly here. Before #1090 that was invisible, because
// faceProcessor skipped externals anyway; now that they are scannable, an
// import into an already-enabled event would still sit unscanned until
// someone pressed Re-scan.
//
// Ids are collected unconditionally and the setting is read at the END,
// not here: this loop can run for many minutes on a large library, and an
// admin who enables detection during it would otherwise leave every photo
// imported after that moment stuck at NULL forever — the toggle endpoint
// only queues rows that already existed when it fired.
//
// The event path is already committed (above), so the enqueue below is
// free of the ordering hazard it used to carry. It stays at the end anyway
// so the setting can be read after the loop, and it only touches rows that
// are still untouched — see the whereNull there. No video guard needed:
// walkDir collects only jpg/jpeg/png/webp.
const importedPhotoIds = [];
let superseded = false;
// Insert photos
for (const f of dedupeMap.values()) {
if (lost) {
// Either another process took the claim over — it is walking this
// same folder now, and the unique index makes anything we insert from
// here a wasted stat + decode — or (automatic passes) the event
// stopped qualifying. Stop and let whoever owns it now finish.
superseded = true;
logger.warn(`External import for event ${eventId} stopped mid-run (claim lost or event no longer eligible) after ${imported} imported`);
break;
}
// Infer type by subfolder names
const segs = f.rel.split(path.sep);
let type = 'individual';
if (segs[0] === map.collages) type = 'collage';
if (segs[0] === map.individual) type = 'individual';
const relFromRoot = toRootRelative(f.rel);
try {
// Fast path only. This SELECT settles the common case — a re-import of
// a folder already in the event — without paying for a stat and a
// Sharp metadata read per file. It is NOT the guard: those two calls
// sit between here and the INSERT below, which is exactly the window
// two overlapping imports both walked through (#1162). The unique
// index from migration 186 is the guard, and the catch below is how
// this loop converges when it fires.
const exists = await db('photos')
.where({ event_id: eventId, external_relpath: relFromRoot })
.first();
if (exists) { skipped++; continue; }
// Automatic passes: checked HERE, per file, not against a snapshot
// taken before the loop — an admin can delete a photo (which records
// the exclusion) while this pass is walking or settling, and a
// snapshot would let the loop re-insert it moments later.
if (automatic) {
const excludedRow = await db('external_import_exclusions')
.where({ event_id: eventId, external_relpath: relFromRoot })
.first();
if (excludedRow) { excluded++; continue; }
}
const stats = await fs.stat(f.full);
// Extract dimensions via Sharp
let width = null;
let height = null;
try {
const metadata = await sharp(f.full).metadata();
// Oriented, not raw: a portrait shot from a body that tags rather
// than rotates reports landscape dimensions, and the grid would size
// its tile from those (#1185).
({ width, height } = orientedDimensions(metadata));
} catch (dimErr) {
logger.warn(`Could not extract dimensions for ${f.rel}: ${dimErr.message}`);
}
// Capture date from EXIF (#1172). Managed uploads get this from
// photoProcessor, which external media never goes through — so
// captured_at stayed NULL for every externally imported photo, and the
// gallery's "Date Taken" sort silently degraded into import order via
// its COALESCE fallback. On a library imported in two batches that put
// the first days of a trip after the last ones.
//
// Read here because the file is already open a few lines above for the
// dimensions, so this costs one more read of the same source rather
// than a second pass over the mount.
//
// Best-effort, exactly like the dimensions: a source without EXIF, or
// one Sharp/exifr cannot parse, imports with captured_at NULL and
// falls back to uploaded_at as before.
let capturedAt = null;
try {
capturedAt = await extractCaptureDate(f.full);
} catch (dateErr) {
logger.warn(`Could not extract capture date for ${f.rel}: ${dateErr.message}`);
}
let inserted;
try {
inserted = await db('photos')
.insert({
event_id: eventId,
filename: f.name,
// The camera-original name (#745). External ingest never sets
// original_filename, and NAS-mounted galleries are among the
// most likely to be driven from Lightroom — without this the
// round-trip has nothing to match a RAW against.
source_filename: f.name,
// Keep path as a hint for legacy code but not used for resolution in external mode
path: path.join(event.slug, f.name),
thumbnail_path: null,
type,
size_bytes: stats.size,
width,
height,
source_origin: 'external',
external_relpath: relFromRoot,
// .toISOString() rather than the Date: inside jest, Dates handed
// to the sqlite3 binding land as the literal string
// "[object Object]" (see CLAUDE.md). Strings round-trip on both
// engines.
captured_at: capturedAt ? capturedAt.toISOString() : null
})
.returning('id');
} catch (insertErr) {
// Another writer inserted this exact path while we were reading
// metadata. That is the outcome the index exists to produce, and it
// is a skip rather than a failure — the row is there, it just isn't
// ours. Counting it as `skipped` keeps the reported totals honest;
// before the index this landed in the outer catch as a nameless
// failure, or (more often) never fired at all and duplicated the row.
if (isUniqueViolation(insertErr)) { skipped++; continue; }
throw insertErr;
}
const photoId = Array.isArray(inserted) && inserted.length
? (typeof inserted[0] === 'object' ? inserted[0].id : inserted[0])
: null;
// Generate the thumbnail right away so the gallery grid can use the
// managed thumbnail endpoint instead of falling back to the full
// NAS-streamed original (#423). Best-effort: a single failure logs
// a warning and leaves thumbnail_path=null — the gallery's
// ensureThumbnail will retry lazily on first view. The cost of
// doing this synchronously is ~100-300ms per image; for the
// worst-case 1000-photo import that's still under the 5-minute
// request timeout typical of the import flow.
if (photoId != null) {
try {
const outputBasename = `ext${photoId}_${path.basename(f.rel)}`;
const thumbnailPath = await generateThumbnail(f.full, { outputBasename });
if (thumbnailPath) {
await db('photos').where({ id: photoId }).update({ thumbnail_path: thumbnailPath });
thumbnailsGenerated++;
} else {
thumbnailsFailed++;
}
} catch (thumbErr) {
thumbnailsFailed++;
logger.warn(`Thumbnail generation failed for external photo ${photoId} (${f.rel}): ${thumbErr.message}`);
}
}
if (photoId != null) importedPhotoIds.push(photoId);
imported += (inserted?.length ? 1 : 0);
// The manual Import is the explicit intent the exclusion list exists
// to protect: what it brings back is no longer excluded.
if (!automatic && photoId != null) {
await db('external_import_exclusions').where({ event_id: eventId, external_relpath: relFromRoot }).delete();
}
} catch (e) {
skipped++;
}
}
// The event already resolves to the new directory (set before the loop),
// so the queue is safe to open. Still done here rather than on insert so
// the setting below is read after the loop. Guarded the same way
// photoProcessor guards it
// (both the global flag and the per-event toggle), so installs without the
// feature still never write a face_status. Re-read here rather than before
// the loop so a toggle flipped mid-import is honoured.
let queueFaces = false;
try {
const { isEnabledForEvent } = require('./faceSettings');
const freshEvent = await db('events').where('id', eventId).first();
queueFaces = await isEnabledForEvent(freshEvent);
} catch (err) {
// Never let the face feature break an import — the photos are the point.
logger.warn(`Could not resolve face settings for event ${eventId}: ${err.message}`);
}
// Chunked because SQLite caps a statement at 999 bound parameters and an
// import can be far larger than that.
if (queueFaces && importedPhotoIds.length) {
let queued = 0;
for (let i = 0; i < importedPhotoIds.length; i += 500) {
// whereNull, not a blanket set. Committing the event path before the
// loop means a toggle or Re-scan firing mid-import can now genuinely
// queue and even finish some of these rows — so an unconditional
// update would drag 'done' rows back to 'pending' for a duplicate
// scan, and knock 'processing' rows out from under the worker
// mid-flight. Only rows nothing has touched are ours to queue.
queued += await db('photos')
.whereIn('id', importedPhotoIds.slice(i, i + 500))
.whereNull('face_status')
.update({ face_status: 'pending' });
}
logger.info(`Queued ${queued} of ${importedPhotoIds.length} imported external photo(s) for face scanning (event ${eventId})`);
}
const result = { imported, skipped, deferred, excluded, thumbnailsGenerated, thumbnailsFailed };
// Only a run that changed something goes into the activity log. The
// watcher re-runs this pass on a timer for every watched event, and a
// "0 imported, 6012 skipped" row every fifteen minutes per event would
// bury the entries an admin is actually looking for.
if (imported > 0 || actor?.type !== 'system') {
await logActivity(
'external_import_completed',
{ event_id: eventId, ...result, external_path },
eventId,
actor || { type: 'admin' }
);
}
if (!superseded) await jobState.release(jobName, token, result);
return result;
} catch (error) {
// Release without a result so the last real outcome is kept, and in the
// catch rather than a finally so a run that lost its claim does not clear
// the new owner's flag — release() is token-scoped and refuses that anyway,
// but there is no reason to make the call.
await jobState.release(jobName, token, null);
throw error;
} finally {
clearInterval(heartbeatTimer);
}
}
module.exports = {
importExternalFolder,
recordExclusions,
ImportInProgressError,
EventNotFoundError,
jobNameFor,
IMAGE_EXTENSIONS,
};
backend/src/services/externalMediaWatcher.js
+371
View File
@@ -0,0 +1,371 @@
/**
* Folder watcher for reference-mode events (issue 1187).
*
* Managed uploads dropped into storage/events/active are imported by
* fileWatcher.js without anyone touching the admin UI. External media had no
* equivalent: a reference event's NAS folder keeps growing, and every new
* batch waits for someone to open the event and press Import. This service
* runs that same import pass — literally the same function the button calls —
* whenever a watched folder changes, and again on a timer.
*
* Shape, and why:
*
* - Per-event opt-in (`events.external_watch`, migration 208), not a global
* switch. Each watched tree is a set of inotify handles or, on a mount
* that does not deliver events, a polling stat of every file in it. An
* install with hundreds of reference events should only pay that for the
* ones still receiving files.
*
* - Change events trigger a debounced full pass over the folder, not a
* per-file insert. The import already skips rows it has, so a full pass
* costs one directory walk plus work for the new files — and it keeps
* exactly one code path for external ingest, with the dedupe, the type
* inference, the thumbnail and the face enqueue all in one place.
*
* - A periodic sweep is the fallback, not an optimisation. NFS and SMB
* mounts routinely deliver no inotify events at all for writes made from
* another host, which for a NAS is the normal case. Without the sweep a
* watcher on such a mount would look configured and silently do nothing.
* EXTERNAL_MEDIA_WATCH_POLLING=true switches chokidar to stat-polling for
* installs that want change-driven imports on those mounts anyway.
*
* - Deletions are ignored, deliberately. The manual import only ever adds.
* A file vanishing from the NAS is at least as likely to be a folder
* being reorganised, a mount dropping out, or a copy in progress as it is
* an intentional removal — and acting on it would delete a guest-visible
* photo. Rows whose file is gone stay, exactly as they do today.
*
* - Every replica watches. Which one actually imports is settled by the
* per-event claim inside importExternalFolder; the others get
* ImportInProgressError and stand down. That is what makes the same code
* correct for the AIO container, a two-container install, and a
* multi-replica deploy behind a load balancer.
*
* - Not gated on STORAGE_BACKEND. Unlike the managed watcher,
* EXTERNAL_MEDIA_ROOT is always a local filesystem path — reference
* galleries are not migrated to S3 — so an S3 install can watch too.
*
* Reconciliation runs on a timer against the events table rather than being
* signalled by the toggle endpoint: the endpoint may execute on a different
* replica than the one holding the watcher, and a minute of latency on a
* checkbox is a better trade than cross-process signalling.
*/
const path = require('path');
const fs = require('fs').promises;
const chokidar = require('chokidar');
const { db } = require('../database/db');
const { formatBoolean } = require('../utils/dbCompat');
const logger = require('../utils/logger');
const { resolveExternalPath } = require('./externalMediaService');
const {
importExternalFolder,
ImportInProgressError,
IMAGE_EXTENSIONS,
} = require('./externalImportService');
const envInt = (name, fallback) => {
const parsed = Number.parseInt(process.env[name] || '', 10);
return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
};
// Kill switch. Default on: the feature is already opt-in per event, so an
// install with no watched events runs nothing but the reconcile query.
const ENABLED = (process.env.EXTERNAL_MEDIA_WATCH || 'true').toLowerCase() !== 'false';
// Stat-polling instead of inotify, for mounts that do not deliver events.
const USE_POLLING = (process.env.EXTERNAL_MEDIA_WATCH_POLLING || 'false').toLowerCase() === 'true';
const POLL_INTERVAL_MS = envInt('EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS', 5000);
// How long a file must stop growing before it counts as written. Generous:
// a NAS copy of a 40 MB raw export can stall for seconds mid-file.
const STABILITY_MS = envInt('EXTERNAL_MEDIA_WATCH_STABILITY_MS', 5000);
// Quiet period after the last change before the pass runs, so a batch copy
// of 300 files becomes one import rather than 300.
const DEBOUNCE_MS = envInt('EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS', 10000);
// Timer-driven pass over every watched event. 0 disables the sweep.
const SWEEP_INTERVAL_MS = envInt('EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS', 15 * 60 * 1000);
// How often the set of watched events is re-read from the database.
const RECONCILE_INTERVAL_MS = envInt('EXTERNAL_MEDIA_WATCH_RECONCILE_INTERVAL_MS', 60 * 1000);
const ACTOR = { type: 'system', name: 'external-media-watcher' };
// eventId -> { externalPath, watcher, timer }
const watched = new Map();
// Folders reported missing, so the reconcile loop logs each once rather than
// once a minute until the mount comes back.
const missingLogged = new Set();
let reconcileTimer = null;
let sweepTimer = null;
let started = false;
/**
* Events that asked to be watched and can be: reference mode, a folder set,
* live, not archived. Exported for the test.
*/
async function listWatchedEvents() {
return db('events')
.where({ source_mode: 'reference', external_watch: formatBoolean(true) })
.whereNotNull('external_path')
.where('is_active', formatBoolean(true))
.where(function () {
this.where('is_archived', formatBoolean(false)).orWhereNull('is_archived');
})
.select('id', 'slug', 'external_path');
}
const isImage = (filePath) => IMAGE_EXTENSIONS.includes(path.extname(filePath).toLowerCase());
/**
* One import pass for an event. The event is re-read first: the folder may
* have moved or the toggle may have been cleared since the change that
* scheduled this, and the pass must follow the row, not the scheduler's memory.
*
* Returns the import result, or null when nothing ran.
*/
async function runImport(eventId, reason) {
const event = await db('events').where('id', eventId).first();
// The same eligibility listWatchedEvents() applies, re-checked at run time:
// reconcile only looks once a minute, and an event archived or deactivated
// inside that window must not gain photos from a pass scheduled before.
if (
!event || !event.external_watch || event.source_mode !== 'reference' || !event.external_path
|| !event.is_active || event.is_archived
) {
return null;
}
try {
const result = await importExternalFolder({
eventId,
externalPath: event.external_path,
recursive: true,
actor: ACTOR,
// Automatic pass: follow the row rather than writing it, keep out what
// an admin deleted, leave files still being copied for the next pass
// (see externalImportService).
automatic: true,
settleMs: STABILITY_MS,
});
if (result.imported > 0 || result.deferred > 0) {
logger.info(`[externalMediaWatcher] event ${eventId} (${event.slug}): imported ${result.imported}, skipped ${result.skipped}, deferred ${result.deferred}, excluded ${result.excluded} (${reason})`);
} else {
logger.debug(`[externalMediaWatcher] event ${eventId}: nothing new (${reason})`);
}
// A deferred file gets no second 'add' event (ignoreInitial, and the copy
// that made it unsettled already fired its one), and the sweep may be
// disabled — so the pass re-arms itself until the folder is quiet.
if (result.deferred > 0) scheduleImport(eventId);
return result;
} catch (err) {
if (err instanceof ImportInProgressError) {
// Someone else — the Import button, or this watcher on another replica
// — is already walking this folder. A change-triggered pass re-arms so
// files that landed during that run are not left for the sweep; the
// sweep itself just tries again next tick.
logger.debug(`[externalMediaWatcher] event ${eventId}: import already running, ${reason === 'change' ? 're-arming' : 'skipping'}`);
if (reason === 'change') scheduleImport(eventId);
return null;
}
logger.error(`[externalMediaWatcher] import failed for event ${eventId}: ${err.message}`);
return null;
}
}
/**
* Debounced trigger. Each new change pushes the run back, so a batch copy
* settles into a single pass once the folder has been quiet for DEBOUNCE_MS.
*/
function scheduleImport(eventId) {
const entry = watched.get(eventId);
if (!entry) return;
if (entry.timer) clearTimeout(entry.timer);
entry.timer = setTimeout(() => {
entry.timer = null;
runImport(eventId, 'change').catch((err) => {
logger.error(`[externalMediaWatcher] scheduled import failed for event ${eventId}: ${err.message}`);
});
}, DEBOUNCE_MS);
entry.timer.unref?.();
}
/**
* Start a watcher for one event. Returns true when it is now watched, false
* when it could not be (bad path, folder not there yet).
*/
async function startWatching(event) {
let absPath;
try {
absPath = resolveExternalPath({ external_path: event.external_path }, '');
} catch (err) {
// A path outside EXTERNAL_MEDIA_ROOT cannot be watched, and should not
// have been saved. Log and leave it; nothing to clean up.
logger.warn(`[externalMediaWatcher] event ${event.id}: refusing to watch '${event.external_path}': ${err.message}`);
return false;
}
try {
const stat = await fs.stat(absPath);
if (!stat.isDirectory()) throw new Error('not a directory');
} catch (err) {
// The mount is not there (yet). Reconcile retries every minute, and the
// sweep will not run for an event that is not in `watched` either — a
// folder that cannot be listed has nothing to import.
if (!missingLogged.has(event.id)) {
missingLogged.add(event.id);
logger.warn(`[externalMediaWatcher] event ${event.id} (${event.slug}): folder '${event.external_path}' is not readable (${err.message}); will retry`);
}
return false;
}
missingLogged.delete(event.id);
const watcher = chokidar.watch(absPath, {
// The sweep and the first reconcile cover what is already there; firing
// 'add' for every existing file on boot would schedule an import of a
// folder that was just imported.
ignoreInitial: true,
persistent: true,
ignored: (p) => path.basename(p).startsWith('.'),
usePolling: USE_POLLING,
interval: POLL_INTERVAL_MS,
binaryInterval: POLL_INTERVAL_MS,
awaitWriteFinish: {
stabilityThreshold: STABILITY_MS,
pollInterval: Math.min(1000, STABILITY_MS),
},
});
const entry = { externalPath: event.external_path, watcher, timer: null };
watched.set(event.id, entry);
watcher
.on('add', (filePath) => {
if (isImage(filePath)) scheduleImport(event.id);
})
.on('unlink', (filePath) => {
// Deliberately not acted on — see the header comment.
if (isImage(filePath)) logger.debug(`[externalMediaWatcher] event ${event.id}: file removed, row kept: ${path.relative(absPath, filePath)}`);
})
.on('error', (err) => {
logger.warn(`[externalMediaWatcher] event ${event.id}: watcher error: ${err.message}`);
});
logger.info(`[externalMediaWatcher] watching event ${event.id} (${event.slug}) at '${event.external_path}'${USE_POLLING ? ' (polling)' : ''}`);
return true;
}
async function stopWatching(eventId) {
const entry = watched.get(eventId);
if (!entry) return;
watched.delete(eventId);
if (entry.timer) clearTimeout(entry.timer);
try {
await entry.watcher.close();
} catch (err) {
logger.debug(`[externalMediaWatcher] close failed for event ${eventId}: ${err.message}`);
}
logger.info(`[externalMediaWatcher] stopped watching event ${eventId}`);
}
/**
* Bring the set of live watchers in line with the events table: start for
* events that opted in since the last pass, restart the ones whose folder
* moved, stop the ones that opted out, archived, or went inactive.
*
* A watcher that just started gets one pass straight away. chokidar is told
* to ignore what is already in the folder, so without this an admin ticking
* the box — or a mount coming back after an outage, or a fresh replica
* booting — would see nothing happen until the next sweep. The pass skips
* rows the event already has, so on a folder that was imported by hand it
* costs one directory walk.
*/
async function reconcile() {
let events;
try {
events = await listWatchedEvents();
} catch (err) {
logger.warn(`[externalMediaWatcher] could not list watched events: ${err.message}`);
return;
}
const wanted = new Map(events.map((e) => [e.id, e]));
for (const eventId of [...watched.keys()]) {
const next = wanted.get(eventId);
if (!next || next.external_path !== watched.get(eventId).externalPath) {
await stopWatching(eventId);
}
}
const fresh = [];
for (const event of wanted.values()) {
if (!watched.has(event.id) && await startWatching(event)) {
fresh.push(event.id);
}
}
for (const eventId of fresh) {
await runImport(eventId, 'start');
}
}
/**
* Timer-driven pass over every watched event, one at a time. Sequential on
* purpose: each pass reads and decodes new files off the mount, and running
* them in parallel would only make a slow NAS slower.
*/
async function sweep() {
for (const eventId of [...watched.keys()]) {
await runImport(eventId, 'sweep');
}
}
function startExternalMediaWatcher() {
if (!ENABLED) {
logger.info('[externalMediaWatcher] disabled via EXTERNAL_MEDIA_WATCH=false');
return null;
}
if (started) return null;
started = true;
reconcile().catch((err) => logger.warn(`[externalMediaWatcher] initial reconcile failed: ${err.message}`));
reconcileTimer = setInterval(() => {
reconcile().catch((err) => logger.warn(`[externalMediaWatcher] reconcile failed: ${err.message}`));
}, RECONCILE_INTERVAL_MS);
reconcileTimer.unref?.();
if (SWEEP_INTERVAL_MS > 0) {
sweepTimer = setInterval(() => {
sweep().catch((err) => logger.warn(`[externalMediaWatcher] sweep failed: ${err.message}`));
}, SWEEP_INTERVAL_MS);
sweepTimer.unref?.();
}
logger.info(`[externalMediaWatcher] started (sweep every ${SWEEP_INTERVAL_MS > 0 ? `${Math.round(SWEEP_INTERVAL_MS / 60000)} min` : 'never'}, debounce ${DEBOUNCE_MS} ms${USE_POLLING ? ', polling' : ''})`);
return { reconcile, sweep };
}
/**
* Tear everything down. For tests and for a clean shutdown; the process exits
* fine without it because every timer is unref'd.
*/
async function stopExternalMediaWatcher() {
if (reconcileTimer) clearInterval(reconcileTimer);
if (sweepTimer) clearInterval(sweepTimer);
reconcileTimer = null;
sweepTimer = null;
for (const eventId of [...watched.keys()]) {
await stopWatching(eventId);
}
missingLogged.clear();
started = false;
}
module.exports = {
startExternalMediaWatcher,
stopExternalMediaWatcher,
// Exposed for the test suite.
reconcile,
sweep,
runImport,
listWatchedEvents,
watchedEventIds: () => [...watched.keys()],
};
backend/src/services/maintenanceJobState.js
+22
View File
@@ -40,6 +40,7 @@ const os = require('os');
const crypto = require('crypto');
const { db } = require('../database/db');
const logger = require('../utils/logger');
const { isUniqueViolation } = require('../utils/dbErrors');
const JOB_DIMENSION_REPAIR = 'photo_dimension_repair';
const JOB_CAPTURE_DATE_BACKFILL = 'photo_capture_date_backfill';
@@ -63,6 +64,26 @@ const OWNER = `${os.hostname()}:${process.pid}`;
const nowIso = () => new Date().toISOString();
const cutoffIso = (staleAfterMs) => new Date(Date.now() - staleAfterMs).toISOString();
/**
* Make sure a row exists for `jobName`, so claim() has something to UPDATE.
*
* The maintenance sweeps are seeded by migration 189 and never need this. It
* exists for job names that are only known at runtime — an external-media
* import is claimed per event (`external_import:<id>`), and events are created
* long after any migration ran. Two replicas racing to seed the same name is
* settled by the primary key: the loser's insert bounces and the row is there
* either way.
*/
async function ensure(jobName) {
const row = await db('maintenance_jobs').where({ job_name: jobName }).first();
if (row) return;
try {
await db('maintenance_jobs').insert({ job_name: jobName, is_running: false });
} catch (err) {
if (!isUniqueViolation(err)) throw err;
}
}
/**
* Try to become the one runner of `jobName`.
*
@@ -168,6 +189,7 @@ async function read(jobName, { staleAfterMs = DEFAULT_STALE_MS } = {}) {
}
module.exports = {
ensure,
claim,
heartbeat,
release,