Files
picpeak/docker-compose.yml
T
Paul Nothaft 8cc7d7d14a feat(external-media): watch reference folders and import new files automatically (#1345)
* feat(external-media): watch reference folders and import new files automatically

Managed uploads dropped into storage/events/active are picked up by the
chokidar watcher; external media had no equivalent, so a NAS folder that
keeps growing needed an admin to open the event and press Import every
time. Relates to issue 1187.

- The import pass moves out of the route into
  services/externalImportService.js. The watcher and the Import button
  now run the identical function; the route only validates and maps
  errors to status codes.
- Mutual exclusion is the per-event claim from maintenanceJobState
  (`external_import:<id>`, seeded on demand by the new ensure()) instead
  of the in-process Set. The Set stopped a double-click in one process;
  the claim also stops the watcher on a second replica, or an admin
  clicking while the watcher is mid-run elsewhere. The run heartbeats so
  a claim from a dead process is taken over.
- services/externalMediaWatcher.js: per-event opt-in via the new
  events.external_watch column (migration 208), chokidar with
  awaitWriteFinish so a copy in flight is not imported half-written,
  debounced full pass per change, a timer sweep every 15 minutes as the
  fallback for NFS/SMB mounts that deliver no inotify events, optional
  stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched
  events is re-read every minute, so the toggle works from any replica.
  A watcher that just started runs one pass immediately.
- Deletions are ignored on purpose: a file vanishing from a NAS is at
  least as likely to be a reorganisation or a dropped mount as an
  intentional removal, and acting on it would delete a guest-visible
  photo. Rows whose file is gone stay, as they do today.
- Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local.
- Quiet system passes stay out of the activity log; runs that imported
  something are logged with actor external-media-watcher.
- Frontend: "Watch folder for new files" checkbox under the external
  folder picker, status line in view mode, EN/DE strings.

* fix(external-media): close the review gaps in the folder watcher

Codex review of the watcher, round 1. All six findings were real:

- Enabling the watcher, or pointing an enabled one at another folder,
  now requires photos.upload — the permission the manual Import already
  requires. events.edit alone was a way around it. Only the transition
  is checked, so a role without photos.upload can still edit an
  already-watched event. The checkbox is disabled for such roles.
- Automatic passes defer files that are still changing: anything
  modified inside the stability window, or whose size moves across one
  wait of that window, is left for the next pass. chokidar's
  awaitWriteFinish only settles the file that fired the event, and the
  sweep sees no events at all, so a sibling still being copied could be
  inserted half-written and then skipped forever.
- Photos an admin deleted are not brought back by the sweep. The delete
  routes record the file in external_import_exclusions (migration 209);
  automatic passes skip the list, the manual Import ignores it and
  clears it for what it imports.
- The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three
  compose files; they were documented but the backend services use
  explicit environment lists, so the kill switch did nothing.
- A pass re-checks is_active / is_archived at run time, not only in the
  minutely reconcile.
- The lease is renewed on a timer for the whole run, walk included, and
  ownership is checked before the event row is touched.

* fix(external-media): make automatic passes follow the row, not rewrite it

Codex review round 2, four findings, all applied:

- The event update route drops non-canonical spellings of external_watch
  and external_path before the permission guard. SQLite resolves column
  names case-insensitively, so `External_Watch` reached the column while
  the guard only looked at the lowercase key.
- Exclusions are checked per file at insert time, not against a
  snapshot taken before the settle wait. A photo deleted during the wait
  was present in the snapshot and got re-inserted by the loop.
- An automatic pass no longer writes source_mode / external_path. It
  re-reads the row after the walk and the settle wait and stops if the
  folder changed or the event went managed; the manual Import is the
  only writer. The options are now `automatic` + `settleMs`.
- A pass that deferred files re-arms the debounced import, so a file
  copied just before the watcher started is not stranded when the sweep
  is disabled.

* fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying

Codex review round 3, both findings applied:

- recordExclusions keys on external_relpath alone. A replaced external
  photo becomes managed but keeps its relpath on purpose, and deleting
  that replacement must not republish the NAS original.
- An automatic pass checks the full watcher predicate (reference mode,
  same folder, watch on, active, not archived) before it inserts and on
  every heartbeat tick during the loop, and stops as soon as the event
  no longer qualifies.

---------

Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
2026-09-07 22:29:55 +02:00

288 lines
12 KiB
YAML

services:
# Generates machine secrets (JWT/DB/Redis) on first run when they aren't set
# in .env (seeds from the env var when provided, else a random value).
# Idempotent — never overwrites an existing file. See docker-compose.production.yml.
secrets-init:
image: alpine:3.20
container_name: picpeak-secrets-init
env_file: .env
entrypoint:
- sh
- -c
- |
set -e
mkdir -p /run/secrets
if [ ! -s /run/secrets/jwt_secret ]; then
if [ -n "$$JWT_SECRET" ]; then printf '%s' "$$JWT_SECRET" > /run/secrets/jwt_secret;
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/jwt_secret; fi
fi
if [ ! -s /run/secrets/db_password ]; then
if [ -n "$$DB_PASSWORD" ]; then printf '%s' "$$DB_PASSWORD" > /run/secrets/db_password;
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/db_password; fi
fi
if [ ! -s /run/secrets/redis_password ]; then
if [ -n "$$REDIS_PASSWORD" ]; then printf '%s' "$$REDIS_PASSWORD" > /run/secrets/redis_password;
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/redis_password; fi
fi
# 644: the readers run as three different users (postgres, redis, nodejs),
# so a non-root reader must be able to read them. The volume is private to
# these containers and never host-exposed.
chmod 644 /run/secrets/jwt_secret /run/secrets/db_password /run/secrets/redis_password
volumes:
- picpeak-secrets:/run/secrets
restart: "no"
backend:
build:
context: ./backend
dockerfile: Dockerfile
container_name: picpeak-backend
restart: unless-stopped
environment:
- NODE_ENV=${NODE_ENV:-production}
- PORT=3000
- JWT_SECRET=${JWT_SECRET:-}
- ADMIN_USERNAME=${ADMIN_USERNAME:-admin}
- ADMIN_EMAIL=${ADMIN_EMAIL:-admin@example.com}
- ADMIN_PASSWORD=${ADMIN_PASSWORD:-}
# How long a gallery guest stays recognised (#1210). Unset = 30d.
# Listed explicitly because this service takes an environment list, so a
# value in .env that is not named here never reaches the container.
- GUEST_TOKEN_TTL=${GUEST_TOKEN_TTL:-}
- DATABASE_CLIENT=pg
- DB_TYPE=postgresql
- DB_HOST=postgres
- DB_PORT=5432
- DB_USER=${DB_USER}
- DB_PASSWORD=${DB_PASSWORD:-}
- DB_NAME=${DB_NAME}
- EXTERNAL_MEDIA_ROOT=${EXTERNAL_MEDIA_ROOT:-/app/storage/external-media}
- SMTP_HOST=${SMTP_HOST}
- SMTP_PORT=${SMTP_PORT}
- SMTP_SECURE=${SMTP_SECURE:-false}
- SMTP_USER=${SMTP_USER}
- SMTP_PASS=${SMTP_PASS}
- EMAIL_FROM=${EMAIL_FROM:-noreply@picpeak.local}
# Webhook email transport (#1225). Listed here because this service
# declares an explicit `environment:` block — a variable only present in
# .env is NOT passed through, so without these three the documented
# "uncomment in .env and restart" flow silently leaves the transport off.
- EMAIL_WEBHOOK_URL=${EMAIL_WEBHOOK_URL:-}
- EMAIL_WEBHOOK_SECRET=${EMAIL_WEBHOOK_SECRET:-}
- EMAIL_WEBHOOK_ALLOW_PRIVATE_URLS=${EMAIL_WEBHOOK_ALLOW_PRIVATE_URLS:-false}
# Unset by default (#705): an injected value would always win over the
# `general_site_url` admin setting, so the setup wizard could never
# take effect. Set this only to pin the origin from config-as-code.
- FRONTEND_URL=${FRONTEND_URL:-}
# Public API origin for split-origin deployments (#798 SSO redirect_uri).
# Empty = same origin as FRONTEND_URL (the standard proxied setup).
- API_URL=${API_URL:-}
# OIDC SSO (#798): key for the client secret at rest (falls back to
# JWT_SECRET) and the break-glass override that re-enables local
# password login when the IdP is down while SSO-only mode is active.
- OIDC_ENCRYPTION_KEY=${OIDC_ENCRYPTION_KEY:-}
# Optional product usage (#1110); remains off until explicit in-app consent.
- USAGE_COLLECTOR_URL=${USAGE_COLLECTOR_URL:-https://usage.picpeak.app}
- USAGE_ENCRYPTION_KEY=${USAGE_ENCRYPTION_KEY:-}
- OIDC_BREAK_GLASS=${OIDC_BREAK_GLASS:-}
- ADMIN_URL=${ADMIN_URL:-}
- TZ=${TZ:-UTC}
- STORAGE_PATH=/app/storage
# Watch-folder auto-import: max photos processed in parallel (default 2).
- FILE_WATCHER_CONCURRENCY=${FILE_WATCHER_CONCURRENCY:-2}
# External-media folder watcher (issue 1187); all optional, see .env.example.
- EXTERNAL_MEDIA_WATCH=${EXTERNAL_MEDIA_WATCH:-true}
- EXTERNAL_MEDIA_WATCH_POLLING=${EXTERNAL_MEDIA_WATCH_POLLING:-false}
- EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS=${EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS:-5000}
- EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS=${EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS:-900000}
- EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS=${EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS:-10000}
- EXTERNAL_MEDIA_WATCH_STABILITY_MS=${EXTERNAL_MEDIA_WATCH_STABILITY_MS:-5000}
# Face recognition (#1074). The URL defaults to the sidecar's compose
# service name, so the common case needs no configuration. None of this
# is touched until the `faces` feature flag is enabled in admin
# settings — an install without the picpeak-ml container never attempts
# a connection.
- FACE_ML_URL=${FACE_ML_URL:-http://picpeak-ml:8000}
- FACE_ML_TOKEN=${FACE_ML_TOKEN:-}
- FACE_PROCESSOR_CONCURRENCY=${FACE_PROCESSOR_CONCURRENCY:-}
# No `user:` directive — as of #484, the container starts as root,
# chowns the bind mounts to nodejs (UID 1001), then drops privileges
# via su-exec. PUID/PGID env vars are no longer read; if you need
# a different runtime UID, pre-chown the host dirs and pin
# `user: "<uid>:<gid>"` here.
volumes:
- ./events:/app/events
- ./data:/app/data
- ./logs:/app/logs
- ./backup:/backup
- ./storage:/app/storage
- picpeak-secrets:/run/secrets:ro
ports:
- "${BACKEND_PORT:-3001}:3000"
depends_on:
secrets-init:
condition: service_completed_successfully
postgres:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://127.0.0.1:3000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
networks:
- picpeak-network
postgres:
image: postgres:15-alpine
container_name: picpeak-postgres
restart: unless-stopped
userns_mode: "host"
environment:
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD_FILE=/run/secrets/db_password
- POSTGRES_DB=${DB_NAME}
- PGDATA=/var/lib/postgresql/data/pgdata
- TZ=${TZ:-UTC}
volumes:
- postgres-data:/var/lib/postgresql/data
- picpeak-secrets:/run/secrets:ro
depends_on:
secrets-init:
condition: service_completed_successfully
ports:
- "127.0.0.1:${DB_PORT:-5432}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ${DB_NAME}"]
interval: 10s
timeout: 5s
retries: 5
start_period: 20s
networks:
- picpeak-network
redis:
image: redis:7-alpine
container_name: picpeak-redis
restart: unless-stopped
userns_mode: "host"
command: sh -c 'exec redis-server --appendonly yes --requirepass "$$(cat /run/secrets/redis_password)"'
volumes:
- redis-data:/data
- picpeak-secrets:/run/secrets:ro
depends_on:
secrets-init:
condition: service_completed_successfully
ports:
- "127.0.0.1:${REDIS_PORT:-6379}:6379"
healthcheck:
test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
interval: 10s
timeout: 5s
retries: 3
networks:
- picpeak-network
# Local mail catcher for development/testing only — never wanted in a real
# deployment. Gated behind the `dev` profile so a plain `docker compose up -d`
# does NOT start it; opt in with `docker compose --profile dev up -d`. Nothing
# depends on it (SMTP_HOST comes from .env), so gating is safe.
mailhog:
image: mailhog/mailhog:latest
container_name: picpeak-mailhog
restart: unless-stopped
profiles:
- dev
ports:
- "${MAILHOG_SMTP_PORT:-1025}:1025"
- "${MAILHOG_UI_PORT:-8025}:8025"
networks:
- picpeak-network
# Optional face-detection sidecar (#1074). Gated behind the `faces` profile
# so a plain `docker compose up -d` does NOT start it — opt in with
# `docker compose --profile faces up -d`. Nothing depends on it: the backend
# only ever calls it when the `faces` feature flag is on, so an install that
# skips this service behaves exactly as it did before the feature existed.
#
# The service name is `picpeak-ml` (not `ml`) because it doubles as the
# hostname in FACE_ML_URL's default, `http://picpeak-ml:8000`. Renaming this
# service silently breaks that default for every install that never set the
# variable.
#
# Requires FACENET_ONNX_URL / FACENET_ONNX_SHA256 at build time — see
# ml/README.md. The image publishes no host port and mounts no volumes; it
# is reachable only from the backend on picpeak-network.
picpeak-ml:
build:
context: ./ml
dockerfile: Dockerfile
args:
# Defaults live in ml/Dockerfile and point at the canonical published
# model. These pass an override through from .env when set; an empty
# value here would BLANK the Dockerfile default and fail the build,
# so the fallbacks repeat it deliberately.
- FACENET_ONNX_URL=${FACENET_ONNX_URL:-https://github.com/PicPeak/picpeak/releases/download/ml-models-v1/facenet512.onnx}
- FACENET_ONNX_SHA256=${FACENET_ONNX_SHA256:-a1c06dcb79dc17a42af01d5bcbce4822caa148b9c24bf7eb8b8e556b4fd0d5db}
container_name: picpeak-ml
restart: unless-stopped
profiles:
- faces
environment:
# Shared secret with the backend. The container refuses to start
# without it rather than serving anonymously.
- FACE_ML_TOKEN=${FACE_ML_TOKEN:-}
- FACE_ORT_THREADS=${FACE_ORT_THREADS:-1}
- TZ=${TZ:-UTC}
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
networks:
- picpeak-network
frontend:
build:
context: ./frontend
dockerfile: Dockerfile
args:
- VITE_API_URL=${VITE_API_URL:-/api}
- VITE_UMAMI_URL=${VITE_UMAMI_URL:-}
- VITE_UMAMI_WEBSITE_ID=${VITE_UMAMI_WEBSITE_ID:-}
- VITE_UMAMI_SHARE_URL=${VITE_UMAMI_SHARE_URL:-}
container_name: picpeak-frontend
restart: unless-stopped
environment:
- NODE_ENV=${NODE_ENV:-production}
# Static social-preview brand (#521) — substituted into
# index.html at container start; see frontend/docker-entrypoint.sh.
- BRAND_TITLE=${BRAND_TITLE:-PicPeak}
- BRAND_DESCRIPTION=${BRAND_DESCRIPTION:-Photo gallery shared with PicPeak.}
ports:
- "${FRONTEND_PORT:-3000}:80"
depends_on:
- backend
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://127.0.0.1/health"]
interval: 30s
timeout: 10s
retries: 3
networks:
- picpeak-network
volumes:
postgres-data:
driver: local
redis-data:
driver: local
# Auto-generated machine secrets (jwt/db/redis). Keep it — deleting it orphans
# the DB password from the Postgres volume.
picpeak-secrets:
driver: local
networks:
picpeak-network:
driver: bridge