fix(file-watcher): bound concurrent photo processing (#846)

* fix(file-watcher): bound concurrent photo processing

chokidar fires 'add' once per file — with no ignoreInitial option the
boot scan fires it for every existing file, and a bulk drop into the
watch folder fires it for every new one at once. Each handler runs DB
lookups plus (for new files) a full sharp pipeline; sharp.concurrency(2)
only caps libvips threads WITHIN one operation, not the number of
parallel pipelines, so unbounded handlers can OOM small hosts.

Gate both 'add' and 'unlink' through a shared p-limit
(FILE_WATCHER_CONCURRENCY, default 2, floor 1) — mass deletes otherwise
burst DB work and ZIP-cache invalidation the same way. p-limit is pinned
to ^3.1.0, the last CommonJS release.

Adapted from the filpgame fork (426ca491) — thanks @filpgame; extended
to cover 'unlink', documented in .env.example, plus a lock-in test for
the existing Sharp cache/concurrency caps this bound relies on.

* chore(compose): pass FILE_WATCHER_CONCURRENCY into the backend container (codex review of #846)

The backend service uses an explicit environment list (no env_file), so
the documented override never reached the container in the default
compose deployments. Added to both compose files + root .env.example.
This commit is contained in:
Paul Nothaft
2026-07-19 22:00:46 +02:00
committed by GitHub
parent 0310c46fdd
commit 8337a716b1
9 changed files with 195 additions and 14 deletions
+2
View File
@@ -66,6 +66,8 @@ services:
- ADMIN_URL=${ADMIN_URL:-http://localhost:3001}
- TZ=${TZ:-UTC}
- STORAGE_PATH=/app/storage
# Watch-folder auto-import: max photos processed in parallel (default 2).
- FILE_WATCHER_CONCURRENCY=${FILE_WATCHER_CONCURRENCY:-2}
# No `user:` directive — as of #484, the container starts as root,
# chowns the bind mounts to nodejs (UID 1001), then drops privileges
# via su-exec. PUID/PGID env vars are no longer read; if you need