Files
picpeak/docker-compose.production.yml
T
Paul Nothaft 8cc7d7d14a feat(external-media): watch reference folders and import new files automatically (#1345)
* feat(external-media): watch reference folders and import new files automatically

Managed uploads dropped into storage/events/active are picked up by the
chokidar watcher; external media had no equivalent, so a NAS folder that
keeps growing needed an admin to open the event and press Import every
time. Relates to issue 1187.

- The import pass moves out of the route into
  services/externalImportService.js. The watcher and the Import button
  now run the identical function; the route only validates and maps
  errors to status codes.
- Mutual exclusion is the per-event claim from maintenanceJobState
  (`external_import:<id>`, seeded on demand by the new ensure()) instead
  of the in-process Set. The Set stopped a double-click in one process;
  the claim also stops the watcher on a second replica, or an admin
  clicking while the watcher is mid-run elsewhere. The run heartbeats so
  a claim from a dead process is taken over.
- services/externalMediaWatcher.js: per-event opt-in via the new
  events.external_watch column (migration 208), chokidar with
  awaitWriteFinish so a copy in flight is not imported half-written,
  debounced full pass per change, a timer sweep every 15 minutes as the
  fallback for NFS/SMB mounts that deliver no inotify events, optional
  stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched
  events is re-read every minute, so the toggle works from any replica.
  A watcher that just started runs one pass immediately.
- Deletions are ignored on purpose: a file vanishing from a NAS is at
  least as likely to be a reorganisation or a dropped mount as an
  intentional removal, and acting on it would delete a guest-visible
  photo. Rows whose file is gone stay, as they do today.
- Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local.
- Quiet system passes stay out of the activity log; runs that imported
  something are logged with actor external-media-watcher.
- Frontend: "Watch folder for new files" checkbox under the external
  folder picker, status line in view mode, EN/DE strings.

* fix(external-media): close the review gaps in the folder watcher

Codex review of the watcher, round 1. All six findings were real:

- Enabling the watcher, or pointing an enabled one at another folder,
  now requires photos.upload — the permission the manual Import already
  requires. events.edit alone was a way around it. Only the transition
  is checked, so a role without photos.upload can still edit an
  already-watched event. The checkbox is disabled for such roles.
- Automatic passes defer files that are still changing: anything
  modified inside the stability window, or whose size moves across one
  wait of that window, is left for the next pass. chokidar's
  awaitWriteFinish only settles the file that fired the event, and the
  sweep sees no events at all, so a sibling still being copied could be
  inserted half-written and then skipped forever.
- Photos an admin deleted are not brought back by the sweep. The delete
  routes record the file in external_import_exclusions (migration 209);
  automatic passes skip the list, the manual Import ignores it and
  clears it for what it imports.
- The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three
  compose files; they were documented but the backend services use
  explicit environment lists, so the kill switch did nothing.
- A pass re-checks is_active / is_archived at run time, not only in the
  minutely reconcile.
- The lease is renewed on a timer for the whole run, walk included, and
  ownership is checked before the event row is touched.

* fix(external-media): make automatic passes follow the row, not rewrite it

Codex review round 2, four findings, all applied:

- The event update route drops non-canonical spellings of external_watch
  and external_path before the permission guard. SQLite resolves column
  names case-insensitively, so `External_Watch` reached the column while
  the guard only looked at the lowercase key.
- Exclusions are checked per file at insert time, not against a
  snapshot taken before the settle wait. A photo deleted during the wait
  was present in the snapshot and got re-inserted by the loop.
- An automatic pass no longer writes source_mode / external_path. It
  re-reads the row after the walk and the settle wait and stops if the
  folder changed or the event went managed; the manual Import is the
  only writer. The options are now `automatic` + `settleMs`.
- A pass that deferred files re-arms the debounced import, so a file
  copied just before the watcher started is not stranded when the sweep
  is disabled.

* fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying

Codex review round 3, both findings applied:

- recordExclusions keys on external_relpath alone. A replaced external
  photo becomes managed but keeps its relpath on purpose, and deleting
  that replacement must not republish the NAS original.
- An automatic pass checks the full watcher predicate (reference mode,
  same folder, watch on, active, not archived) before it inserts and on
  every heartbeat tick during the loop, and stops as soon as the event
  no longer qualifies.

---------

Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
2026-09-07 22:29:55 +02:00

261 lines
11 KiB
YAML

version: '3.8'
services:
# Generates machine secrets (JWT/DB/Redis) on first run when they aren't set
# in .env, so a fresh install needs zero secret management. Each file is seeded
# from the matching env var when provided (backward-compatible), otherwise a
# strong random value. Idempotent — never overwrites an existing file, so the
# DB password can't drift out from under an already-initialised Postgres volume.
secrets-init:
image: alpine:3.20
container_name: picpeak-secrets-init
env_file: .env
entrypoint:
- sh
- -c
- |
set -e
mkdir -p /run/secrets
if [ ! -s /run/secrets/jwt_secret ]; then
if [ -n "$$JWT_SECRET" ]; then printf '%s' "$$JWT_SECRET" > /run/secrets/jwt_secret;
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/jwt_secret; fi
fi
if [ ! -s /run/secrets/db_password ]; then
if [ -n "$$DB_PASSWORD" ]; then printf '%s' "$$DB_PASSWORD" > /run/secrets/db_password;
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/db_password; fi
fi
if [ ! -s /run/secrets/redis_password ]; then
if [ -n "$$REDIS_PASSWORD" ]; then printf '%s' "$$REDIS_PASSWORD" > /run/secrets/redis_password;
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/redis_password; fi
fi
# 644: the readers run as three different users (postgres, redis, nodejs),
# so a non-root reader must be able to read them. The volume is private to
# these containers and never host-exposed.
chmod 644 /run/secrets/jwt_secret /run/secrets/db_password /run/secrets/redis_password
volumes:
- picpeak-secrets:/run/secrets
restart: "no"
postgres:
image: postgres:15-alpine
container_name: picpeak-postgres
userns_mode: "host"
environment:
POSTGRES_USER: ${DB_USER:-picpeak}
# Reads the generated (or .env-seeded) password from the shared secrets volume.
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
POSTGRES_DB: ${DB_NAME:-picpeak}
volumes:
- postgres-data:/var/lib/postgresql/data
- picpeak-secrets:/run/secrets:ro
depends_on:
secrets-init:
condition: service_completed_successfully
networks:
- picpeak-network
restart: unless-stopped
healthcheck:
# `pg_isready -U <user>` without -d defaults to probing a database
# whose name matches the user — postgres then logs constant
# `FATAL: database "picpeak" does not exist` even though the
# actual DB is `picpeak_prod`. Pinning -d to DB_NAME makes the
# probe hit the real database and silences the log noise that
# made #484's reporter think the install was broken.
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-picpeak} -d ${DB_NAME:-picpeak}"]
interval: 10s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
container_name: picpeak-redis
userns_mode: "host"
# Reads the generated (or .env-seeded) password from the shared secrets volume.
command: sh -c 'exec redis-server --requirepass "$$(cat /run/secrets/redis_password)"'
volumes:
- redis-data:/data
- picpeak-secrets:/run/secrets:ro
depends_on:
secrets-init:
condition: service_completed_successfully
networks:
- picpeak-network
restart: unless-stopped
healthcheck:
test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
interval: 10s
timeout: 5s
retries: 5
backend:
# Use pre-built image from GitHub Container Registry
# PICPEAK_CHANNEL: 'stable' (default), 'beta', or specific version like 'v2.3.0'
image: ghcr.io/picpeak/picpeak/backend:${PICPEAK_CHANNEL:-stable}
container_name: picpeak-backend
env_file: .env
environment:
- NODE_ENV=production
- DB_HOST=${DB_HOST:-postgres}
- REDIS_HOST=redis
- STORAGE_PATH=/app/storage
- PHOTOS_DIR=/app/storage/events
- PICPEAK_RELEASE_CHANNEL=${PICPEAK_CHANNEL:-stable}
# Watch-folder auto-import: max photos processed in parallel (default 2).
- FILE_WATCHER_CONCURRENCY=${FILE_WATCHER_CONCURRENCY:-2}
# External-media folder watcher (issue 1187); all optional, see .env.example.
- EXTERNAL_MEDIA_WATCH=${EXTERNAL_MEDIA_WATCH:-true}
- EXTERNAL_MEDIA_WATCH_POLLING=${EXTERNAL_MEDIA_WATCH_POLLING:-false}
- EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS=${EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS:-5000}
- EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS=${EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS:-900000}
- EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS=${EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS:-10000}
- EXTERNAL_MEDIA_WATCH_STABILITY_MS=${EXTERNAL_MEDIA_WATCH_STABILITY_MS:-5000}
# Face recognition (#1074). Defaults to the sidecar's compose service
# name; nothing touches it until the `faces` feature flag is enabled in
# admin settings, so installs without the picpeak-ml container are
# unaffected. Start the sidecar with `--profile faces`.
- FACE_ML_URL=${FACE_ML_URL:-http://picpeak-ml:8000}
- FACE_ML_TOKEN=${FACE_ML_TOKEN:-}
- FACE_PROCESSOR_CONCURRENCY=${FACE_PROCESSOR_CONCURRENCY:-}
volumes:
# Defaulted so a .env that simply does not set these still works. Without
# them compose aborts with "invalid spec: :/app/storage: empty section
# between colons", which reads like a broken compose file rather than a
# missing variable (#705).
#
# Note this does NOT make `config` work with no .env at all: `env_file`
# above still requires the file. Making it optional needs
# `required: false`, which is Compose 2.24+ syntax that OLDER Compose
# rejects as a schema error — taking the whole stack down rather than
# just losing a default. Not worth it for a case the onboarding flow
# never hits, since it copies .env.example first.
- ${APP_STORAGE:-./storage}:/app/storage
- ${LOGS:-./logs}:/app/logs
- ${APP_DATA:-./data}:/app/data
- picpeak-secrets:/run/secrets:ro
ports:
- "${BACKEND_PORT:-3001}:3000"
networks:
- picpeak-network
depends_on:
secrets-init:
condition: service_completed_successfully
postgres:
condition: service_healthy
redis:
condition: service_healthy
restart: unless-stopped
# Memory cap (optional, recommended on shared / multi-tenant hosts):
# uncomment to bound the backend's RSS. Sharp/libvips decodes the full
# uncompressed image before resize, so a multi-photo upload batch can
# spike memory. With a cap set, the kernel OOM-killer takes the
# container instead of the whole host; restart:unless-stopped brings
# it back. Match this to the RAM budget you've allocated for picpeak
# (`docker stats` shows the live usage).
# mem_limit: 3g
# memswap_limit: 3g
healthcheck:
# Backend exposes /health on internal port 3000.
# The backend image only ships wget (Alpine base) — using curl
# here makes `docker ps` show the container as `unhealthy`
# indefinitely even when /health responds. Mirrors the wget-based
# HEALTHCHECK already declared in backend/Dockerfile so docker
# compose, plain `docker run`, and `docker ps` all agree.
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/health"]
interval: 30s
timeout: 10s
retries: 3
# Optional face-detection sidecar (#1074). Gated behind the `faces` profile:
# `docker compose --profile faces up -d`. Nothing depends on it, and the
# backend never calls it while the `faces` feature flag is off.
#
# The service name is `picpeak-ml` because it doubles as the hostname in
# FACE_ML_URL's default. Renaming it breaks that default for every install
# that never set the variable.
picpeak-ml:
image: ghcr.io/picpeak/picpeak/ml:${PICPEAK_CHANNEL:-stable}
container_name: picpeak-ml
profiles:
- faces
environment:
# The container refuses to start without this rather than serving
# anonymously — it must match the backend's FACE_ML_TOKEN.
- FACE_ML_TOKEN=${FACE_ML_TOKEN:-}
- FACE_ORT_THREADS=${FACE_ORT_THREADS:-1}
- TZ=${TZ:-UTC}
# No volumes and no published ports: stateless, and reachable only from
# the backend on picpeak-network.
networks:
- picpeak-network
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
frontend:
# Use pre-built image from GitHub Container Registry
# Uses same channel as backend for consistency
image: ghcr.io/picpeak/picpeak/frontend:${PICPEAK_CHANNEL:-stable}
container_name: picpeak-frontend
# Note: Pre-built frontend uses Nginx to proxy /api to backend:3000.
# Prefer keeping API base as '/api' in builds to avoid CORS.
environment:
# Substituted into index.html at container start (see frontend/
# docker-entrypoint.sh) so social link previews reaching the
# static SPA shell (WhatsApp Business API, Twilio, LinkPreview,
# etc. — see #521) show the configured brand instead of the
# generic "PicPeak" default. Defaults applied when unset; restart
# the frontend container after changing for the new title to
# take effect.
- BRAND_TITLE=${BRAND_TITLE:-PicPeak}
- BRAND_DESCRIPTION=${BRAND_DESCRIPTION:-Photo gallery shared with PicPeak.}
ports:
- "${FRONTEND_PORT:-3000}:80"
networks:
- picpeak-network
depends_on:
- backend
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost/health"]
interval: 30s
timeout: 10s
retries: 3
# Optional: Nginx reverse proxy for production with SSL
# Uncomment and configure if you want built-in HTTPS support
# nginx:
# image: nginx:alpine
# container_name: picpeak-nginx
# ports:
# - "80:80"
# - "443:443"
# volumes:
# - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
# - ./nginx/ssl:/etc/nginx/ssl:ro
# - ./nginx/conf.d:/etc/nginx/conf.d:ro
# networks:
# - picpeak-network
# depends_on:
# - frontend
# - backend
# restart: unless-stopped
volumes:
postgres-data:
driver: local
redis-data:
driver: local
# Holds the auto-generated machine secrets (jwt_secret, db_password,
# redis_password). Keep it — deleting it orphans the DB password from the
# Postgres volume. Back it up alongside postgres-data.
picpeak-secrets:
driver: local
networks:
picpeak-network:
driver: bridge