feat(external-media): watch reference folders and import new files automatically (#1345)

* feat(external-media): watch reference folders and import new files automatically

Managed uploads dropped into storage/events/active are picked up by the
chokidar watcher; external media had no equivalent, so a NAS folder that
keeps growing needed an admin to open the event and press Import every
time. Relates to issue 1187.

- The import pass moves out of the route into
  services/externalImportService.js. The watcher and the Import button
  now run the identical function; the route only validates and maps
  errors to status codes.
- Mutual exclusion is the per-event claim from maintenanceJobState
  (`external_import:<id>`, seeded on demand by the new ensure()) instead
  of the in-process Set. The Set stopped a double-click in one process;
  the claim also stops the watcher on a second replica, or an admin
  clicking while the watcher is mid-run elsewhere. The run heartbeats so
  a claim from a dead process is taken over.
- services/externalMediaWatcher.js: per-event opt-in via the new
  events.external_watch column (migration 208), chokidar with
  awaitWriteFinish so a copy in flight is not imported half-written,
  debounced full pass per change, a timer sweep every 15 minutes as the
  fallback for NFS/SMB mounts that deliver no inotify events, optional
  stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched
  events is re-read every minute, so the toggle works from any replica.
  A watcher that just started runs one pass immediately.
- Deletions are ignored on purpose: a file vanishing from a NAS is at
  least as likely to be a reorganisation or a dropped mount as an
  intentional removal, and acting on it would delete a guest-visible
  photo. Rows whose file is gone stay, as they do today.
- Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local.
- Quiet system passes stay out of the activity log; runs that imported
  something are logged with actor external-media-watcher.
- Frontend: "Watch folder for new files" checkbox under the external
  folder picker, status line in view mode, EN/DE strings.

* fix(external-media): close the review gaps in the folder watcher

Codex review of the watcher, round 1. All six findings were real:

- Enabling the watcher, or pointing an enabled one at another folder,
  now requires photos.upload — the permission the manual Import already
  requires. events.edit alone was a way around it. Only the transition
  is checked, so a role without photos.upload can still edit an
  already-watched event. The checkbox is disabled for such roles.
- Automatic passes defer files that are still changing: anything
  modified inside the stability window, or whose size moves across one
  wait of that window, is left for the next pass. chokidar's
  awaitWriteFinish only settles the file that fired the event, and the
  sweep sees no events at all, so a sibling still being copied could be
  inserted half-written and then skipped forever.
- Photos an admin deleted are not brought back by the sweep. The delete
  routes record the file in external_import_exclusions (migration 209);
  automatic passes skip the list, the manual Import ignores it and
  clears it for what it imports.
- The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three
  compose files; they were documented but the backend services use
  explicit environment lists, so the kill switch did nothing.
- A pass re-checks is_active / is_archived at run time, not only in the
  minutely reconcile.
- The lease is renewed on a timer for the whole run, walk included, and
  ownership is checked before the event row is touched.

* fix(external-media): make automatic passes follow the row, not rewrite it

Codex review round 2, four findings, all applied:

- The event update route drops non-canonical spellings of external_watch
  and external_path before the permission guard. SQLite resolves column
  names case-insensitively, so `External_Watch` reached the column while
  the guard only looked at the lowercase key.
- Exclusions are checked per file at insert time, not against a
  snapshot taken before the settle wait. A photo deleted during the wait
  was present in the snapshot and got re-inserted by the loop.
- An automatic pass no longer writes source_mode / external_path. It
  re-reads the row after the walk and the settle wait and stops if the
  folder changed or the event went managed; the manual Import is the
  only writer. The options are now `automatic` + `settleMs`.
- A pass that deferred files re-arms the debounced import, so a file
  copied just before the watcher started is not stranded when the sweep
  is disabled.

* fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying

Codex review round 3, both findings applied:

- recordExclusions keys on external_relpath alone. A replaced external
  photo becomes managed but keeps its relpath on purpose, and deleting
  that replacement must not republish the NAS original.
- An automatic pass checks the full watcher predicate (reference mode,
  same folder, watch on, active, not archived) before it inserts and on
  every heartbeat tick during the loop, and stops as soon as the event
  no longer qualifies.

---------

Co-authored-by: Paul Nothaft <[email protected]>

backend/migrations/core/208_events_external_watch.js

@@ -0,0 +1,36 @@
+/**
+ * `events.external_watch` — per-event opt-in for the external-media folder
+ * watcher (issue 1187).
+ *
+ * Managed uploads are picked up by fileWatcher.js as soon as they land in
+ * storage/events/active. A reference-mode event has no equivalent: new files
+ * copied into its NAS folder sit there until an admin opens the event and
+ * presses Import. services/externalMediaWatcher.js closes that gap for events
+ * that ask for it.
+ *
+ * Opt-in per event rather than a global switch: every watched folder is a set
+ * of inotify handles (or, on a mount that does not deliver events, a polling
+ * stat of the whole tree), and a large install with hundreds of reference
+ * events should not pay that for the ones nobody is still adding files to.
+ *
+ * Boolean with a false default so an existing install changes nothing on
+ * upgrade — the column is read through formatBoolean() so SQLite's 0/1 and
+ * Postgres' true/false both work.
+ */
+
+exports.up = async function (knex) {
+  if (!(await knex.schema.hasColumn('events', 'external_watch'))) {
+    await knex.schema.alterTable('events', (table) => {
+      table.boolean('external_watch').notNullable().defaultTo(false);
+    });
+    console.log('208: added events.external_watch');
+  }
+};
+
+exports.down = async function (knex) {
+  if (await knex.schema.hasColumn('events', 'external_watch')) {
+    await knex.schema.alterTable('events', (table) => {
+      table.dropColumn('external_watch');
+    });
+  }
+};

backend/migrations/core/209_external_import_exclusions.js

@@ -0,0 +1,35 @@
+/**
+ * `external_import_exclusions` — files an admin deleted from a reference
+ * event, so the folder watcher (issue 1187) does not bring them back.
+ *
+ * Deleting an external photo removes its row but leaves the NAS original
+ * alone (resolvePhotoStorageKey returns null for external rows, on purpose).
+ * The manual Import only ran when an admin pressed it, so the deleted file
+ * came back only if they asked. The watcher runs on its own, and a full pass
+ * that skips only rows the event still has would re-import every deleted
+ * photo on the next sweep — republishing what an admin removed, without any
+ * new file arriving.
+ *
+ * One row per (event, root-relative path). Automatic passes skip these; the
+ * manual Import button ignores the list and clears the rows for whatever it
+ * imports, since pressing it is the explicit intent the exclusion exists to
+ * protect.
+ */
+
+exports.up = async function (knex) {
+  if (!(await knex.schema.hasTable('external_import_exclusions'))) {
+    await knex.schema.createTable('external_import_exclusions', (t) => {
+      t.increments('id').primary();
+      t.integer('event_id').notNullable().references('id').inTable('events').onDelete('CASCADE');
+      // Same shape as photos.external_relpath: relative to EXTERNAL_MEDIA_ROOT.
+      t.text('external_relpath').notNullable();
+      t.timestamp('created_at').defaultTo(knex.fn.now());
+      t.unique(['event_id', 'external_relpath'], { indexName: 'external_import_exclusions_event_relpath_unique' });
+    });
+    console.log('209: created external_import_exclusions');
+  }
+};
+
+exports.down = async function (knex) {
+  await knex.schema.dropTableIfExists('external_import_exclusions');
+};