8cc7d7d14a
* feat(external-media): watch reference folders and import new files automatically Managed uploads dropped into storage/events/active are picked up by the chokidar watcher; external media had no equivalent, so a NAS folder that keeps growing needed an admin to open the event and press Import every time. Relates to issue 1187. - The import pass moves out of the route into services/externalImportService.js. The watcher and the Import button now run the identical function; the route only validates and maps errors to status codes. - Mutual exclusion is the per-event claim from maintenanceJobState (`external_import:<id>`, seeded on demand by the new ensure()) instead of the in-process Set. The Set stopped a double-click in one process; the claim also stops the watcher on a second replica, or an admin clicking while the watcher is mid-run elsewhere. The run heartbeats so a claim from a dead process is taken over. - services/externalMediaWatcher.js: per-event opt-in via the new events.external_watch column (migration 208), chokidar with awaitWriteFinish so a copy in flight is not imported half-written, debounced full pass per change, a timer sweep every 15 minutes as the fallback for NFS/SMB mounts that deliver no inotify events, optional stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched events is re-read every minute, so the toggle works from any replica. A watcher that just started runs one pass immediately. - Deletions are ignored on purpose: a file vanishing from a NAS is at least as likely to be a reorganisation or a dropped mount as an intentional removal, and acting on it would delete a guest-visible photo. Rows whose file is gone stay, as they do today. - Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local. - Quiet system passes stay out of the activity log; runs that imported something are logged with actor external-media-watcher. - Frontend: "Watch folder for new files" checkbox under the external folder picker, status line in view mode, EN/DE strings. * fix(external-media): close the review gaps in the folder watcher Codex review of the watcher, round 1. All six findings were real: - Enabling the watcher, or pointing an enabled one at another folder, now requires photos.upload — the permission the manual Import already requires. events.edit alone was a way around it. Only the transition is checked, so a role without photos.upload can still edit an already-watched event. The checkbox is disabled for such roles. - Automatic passes defer files that are still changing: anything modified inside the stability window, or whose size moves across one wait of that window, is left for the next pass. chokidar's awaitWriteFinish only settles the file that fired the event, and the sweep sees no events at all, so a sibling still being copied could be inserted half-written and then skipped forever. - Photos an admin deleted are not brought back by the sweep. The delete routes record the file in external_import_exclusions (migration 209); automatic passes skip the list, the manual Import ignores it and clears it for what it imports. - The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three compose files; they were documented but the backend services use explicit environment lists, so the kill switch did nothing. - A pass re-checks is_active / is_archived at run time, not only in the minutely reconcile. - The lease is renewed on a timer for the whole run, walk included, and ownership is checked before the event row is touched. * fix(external-media): make automatic passes follow the row, not rewrite it Codex review round 2, four findings, all applied: - The event update route drops non-canonical spellings of external_watch and external_path before the permission guard. SQLite resolves column names case-insensitively, so `External_Watch` reached the column while the guard only looked at the lowercase key. - Exclusions are checked per file at insert time, not against a snapshot taken before the settle wait. A photo deleted during the wait was present in the snapshot and got re-inserted by the loop. - An automatic pass no longer writes source_mode / external_path. It re-reads the row after the walk and the settle wait and stops if the folder changed or the event went managed; the manual Import is the only writer. The options are now `automatic` + `settleMs`. - A pass that deferred files re-arms the debounced import, so a file copied just before the watcher started is not stranded when the sweep is disabled. * fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying Codex review round 3, both findings applied: - recordExclusions keys on external_relpath alone. A replaced external photo becomes managed but keeps its relpath on purpose, and deleting that replacement must not republish the NAS original. - An automatic pass checks the full watcher predicate (reference mode, same folder, watch on, active, not archived) before it inserts and on every heartbeat tick during the loop, and stops as soon as the event no longer qualifies. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
261 lines
11 KiB
YAML
261 lines
11 KiB
YAML
version: '3.8'
|
|
|
|
services:
|
|
# Generates machine secrets (JWT/DB/Redis) on first run when they aren't set
|
|
# in .env, so a fresh install needs zero secret management. Each file is seeded
|
|
# from the matching env var when provided (backward-compatible), otherwise a
|
|
# strong random value. Idempotent — never overwrites an existing file, so the
|
|
# DB password can't drift out from under an already-initialised Postgres volume.
|
|
secrets-init:
|
|
image: alpine:3.20
|
|
container_name: picpeak-secrets-init
|
|
env_file: .env
|
|
entrypoint:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -e
|
|
mkdir -p /run/secrets
|
|
if [ ! -s /run/secrets/jwt_secret ]; then
|
|
if [ -n "$$JWT_SECRET" ]; then printf '%s' "$$JWT_SECRET" > /run/secrets/jwt_secret;
|
|
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/jwt_secret; fi
|
|
fi
|
|
if [ ! -s /run/secrets/db_password ]; then
|
|
if [ -n "$$DB_PASSWORD" ]; then printf '%s' "$$DB_PASSWORD" > /run/secrets/db_password;
|
|
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/db_password; fi
|
|
fi
|
|
if [ ! -s /run/secrets/redis_password ]; then
|
|
if [ -n "$$REDIS_PASSWORD" ]; then printf '%s' "$$REDIS_PASSWORD" > /run/secrets/redis_password;
|
|
else tr -dc A-Za-z0-9 < /dev/urandom | head -c 48 > /run/secrets/redis_password; fi
|
|
fi
|
|
# 644: the readers run as three different users (postgres, redis, nodejs),
|
|
# so a non-root reader must be able to read them. The volume is private to
|
|
# these containers and never host-exposed.
|
|
chmod 644 /run/secrets/jwt_secret /run/secrets/db_password /run/secrets/redis_password
|
|
volumes:
|
|
- picpeak-secrets:/run/secrets
|
|
restart: "no"
|
|
|
|
postgres:
|
|
image: postgres:15-alpine
|
|
container_name: picpeak-postgres
|
|
userns_mode: "host"
|
|
environment:
|
|
POSTGRES_USER: ${DB_USER:-picpeak}
|
|
# Reads the generated (or .env-seeded) password from the shared secrets volume.
|
|
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
|
|
POSTGRES_DB: ${DB_NAME:-picpeak}
|
|
volumes:
|
|
- postgres-data:/var/lib/postgresql/data
|
|
- picpeak-secrets:/run/secrets:ro
|
|
depends_on:
|
|
secrets-init:
|
|
condition: service_completed_successfully
|
|
networks:
|
|
- picpeak-network
|
|
restart: unless-stopped
|
|
healthcheck:
|
|
# `pg_isready -U <user>` without -d defaults to probing a database
|
|
# whose name matches the user — postgres then logs constant
|
|
# `FATAL: database "picpeak" does not exist` even though the
|
|
# actual DB is `picpeak_prod`. Pinning -d to DB_NAME makes the
|
|
# probe hit the real database and silences the log noise that
|
|
# made #484's reporter think the install was broken.
|
|
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-picpeak} -d ${DB_NAME:-picpeak}"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
|
|
redis:
|
|
image: redis:7-alpine
|
|
container_name: picpeak-redis
|
|
userns_mode: "host"
|
|
# Reads the generated (or .env-seeded) password from the shared secrets volume.
|
|
command: sh -c 'exec redis-server --requirepass "$$(cat /run/secrets/redis_password)"'
|
|
volumes:
|
|
- redis-data:/data
|
|
- picpeak-secrets:/run/secrets:ro
|
|
depends_on:
|
|
secrets-init:
|
|
condition: service_completed_successfully
|
|
networks:
|
|
- picpeak-network
|
|
restart: unless-stopped
|
|
healthcheck:
|
|
test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
|
|
backend:
|
|
# Use pre-built image from GitHub Container Registry
|
|
# PICPEAK_CHANNEL: 'stable' (default), 'beta', or specific version like 'v2.3.0'
|
|
image: ghcr.io/picpeak/picpeak/backend:${PICPEAK_CHANNEL:-stable}
|
|
container_name: picpeak-backend
|
|
env_file: .env
|
|
environment:
|
|
- NODE_ENV=production
|
|
- DB_HOST=${DB_HOST:-postgres}
|
|
- REDIS_HOST=redis
|
|
- STORAGE_PATH=/app/storage
|
|
- PHOTOS_DIR=/app/storage/events
|
|
- PICPEAK_RELEASE_CHANNEL=${PICPEAK_CHANNEL:-stable}
|
|
# Watch-folder auto-import: max photos processed in parallel (default 2).
|
|
- FILE_WATCHER_CONCURRENCY=${FILE_WATCHER_CONCURRENCY:-2}
|
|
# External-media folder watcher (issue 1187); all optional, see .env.example.
|
|
- EXTERNAL_MEDIA_WATCH=${EXTERNAL_MEDIA_WATCH:-true}
|
|
- EXTERNAL_MEDIA_WATCH_POLLING=${EXTERNAL_MEDIA_WATCH_POLLING:-false}
|
|
- EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS=${EXTERNAL_MEDIA_WATCH_POLL_INTERVAL_MS:-5000}
|
|
- EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS=${EXTERNAL_MEDIA_WATCH_SWEEP_INTERVAL_MS:-900000}
|
|
- EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS=${EXTERNAL_MEDIA_WATCH_DEBOUNCE_MS:-10000}
|
|
- EXTERNAL_MEDIA_WATCH_STABILITY_MS=${EXTERNAL_MEDIA_WATCH_STABILITY_MS:-5000}
|
|
# Face recognition (#1074). Defaults to the sidecar's compose service
|
|
# name; nothing touches it until the `faces` feature flag is enabled in
|
|
# admin settings, so installs without the picpeak-ml container are
|
|
# unaffected. Start the sidecar with `--profile faces`.
|
|
- FACE_ML_URL=${FACE_ML_URL:-http://picpeak-ml:8000}
|
|
- FACE_ML_TOKEN=${FACE_ML_TOKEN:-}
|
|
- FACE_PROCESSOR_CONCURRENCY=${FACE_PROCESSOR_CONCURRENCY:-}
|
|
volumes:
|
|
# Defaulted so a .env that simply does not set these still works. Without
|
|
# them compose aborts with "invalid spec: :/app/storage: empty section
|
|
# between colons", which reads like a broken compose file rather than a
|
|
# missing variable (#705).
|
|
#
|
|
# Note this does NOT make `config` work with no .env at all: `env_file`
|
|
# above still requires the file. Making it optional needs
|
|
# `required: false`, which is Compose 2.24+ syntax that OLDER Compose
|
|
# rejects as a schema error — taking the whole stack down rather than
|
|
# just losing a default. Not worth it for a case the onboarding flow
|
|
# never hits, since it copies .env.example first.
|
|
- ${APP_STORAGE:-./storage}:/app/storage
|
|
- ${LOGS:-./logs}:/app/logs
|
|
- ${APP_DATA:-./data}:/app/data
|
|
- picpeak-secrets:/run/secrets:ro
|
|
ports:
|
|
- "${BACKEND_PORT:-3001}:3000"
|
|
networks:
|
|
- picpeak-network
|
|
depends_on:
|
|
secrets-init:
|
|
condition: service_completed_successfully
|
|
postgres:
|
|
condition: service_healthy
|
|
redis:
|
|
condition: service_healthy
|
|
restart: unless-stopped
|
|
# Memory cap (optional, recommended on shared / multi-tenant hosts):
|
|
# uncomment to bound the backend's RSS. Sharp/libvips decodes the full
|
|
# uncompressed image before resize, so a multi-photo upload batch can
|
|
# spike memory. With a cap set, the kernel OOM-killer takes the
|
|
# container instead of the whole host; restart:unless-stopped brings
|
|
# it back. Match this to the RAM budget you've allocated for picpeak
|
|
# (`docker stats` shows the live usage).
|
|
# mem_limit: 3g
|
|
# memswap_limit: 3g
|
|
healthcheck:
|
|
# Backend exposes /health on internal port 3000.
|
|
# The backend image only ships wget (Alpine base) — using curl
|
|
# here makes `docker ps` show the container as `unhealthy`
|
|
# indefinitely even when /health responds. Mirrors the wget-based
|
|
# HEALTHCHECK already declared in backend/Dockerfile so docker
|
|
# compose, plain `docker run`, and `docker ps` all agree.
|
|
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
|
|
# Optional face-detection sidecar (#1074). Gated behind the `faces` profile:
|
|
# `docker compose --profile faces up -d`. Nothing depends on it, and the
|
|
# backend never calls it while the `faces` feature flag is off.
|
|
#
|
|
# The service name is `picpeak-ml` because it doubles as the hostname in
|
|
# FACE_ML_URL's default. Renaming it breaks that default for every install
|
|
# that never set the variable.
|
|
picpeak-ml:
|
|
image: ghcr.io/picpeak/picpeak/ml:${PICPEAK_CHANNEL:-stable}
|
|
container_name: picpeak-ml
|
|
profiles:
|
|
- faces
|
|
environment:
|
|
# The container refuses to start without this rather than serving
|
|
# anonymously — it must match the backend's FACE_ML_TOKEN.
|
|
- FACE_ML_TOKEN=${FACE_ML_TOKEN:-}
|
|
- FACE_ORT_THREADS=${FACE_ORT_THREADS:-1}
|
|
- TZ=${TZ:-UTC}
|
|
# No volumes and no published ports: stateless, and reachable only from
|
|
# the backend on picpeak-network.
|
|
networks:
|
|
- picpeak-network
|
|
restart: unless-stopped
|
|
healthcheck:
|
|
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 40s
|
|
|
|
frontend:
|
|
# Use pre-built image from GitHub Container Registry
|
|
# Uses same channel as backend for consistency
|
|
image: ghcr.io/picpeak/picpeak/frontend:${PICPEAK_CHANNEL:-stable}
|
|
container_name: picpeak-frontend
|
|
# Note: Pre-built frontend uses Nginx to proxy /api to backend:3000.
|
|
# Prefer keeping API base as '/api' in builds to avoid CORS.
|
|
environment:
|
|
# Substituted into index.html at container start (see frontend/
|
|
# docker-entrypoint.sh) so social link previews reaching the
|
|
# static SPA shell (WhatsApp Business API, Twilio, LinkPreview,
|
|
# etc. — see #521) show the configured brand instead of the
|
|
# generic "PicPeak" default. Defaults applied when unset; restart
|
|
# the frontend container after changing for the new title to
|
|
# take effect.
|
|
- BRAND_TITLE=${BRAND_TITLE:-PicPeak}
|
|
- BRAND_DESCRIPTION=${BRAND_DESCRIPTION:-Photo gallery shared with PicPeak.}
|
|
ports:
|
|
- "${FRONTEND_PORT:-3000}:80"
|
|
networks:
|
|
- picpeak-network
|
|
depends_on:
|
|
- backend
|
|
restart: unless-stopped
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
|
|
# Optional: Nginx reverse proxy for production with SSL
|
|
# Uncomment and configure if you want built-in HTTPS support
|
|
# nginx:
|
|
# image: nginx:alpine
|
|
# container_name: picpeak-nginx
|
|
# ports:
|
|
# - "80:80"
|
|
# - "443:443"
|
|
# volumes:
|
|
# - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
|
# - ./nginx/ssl:/etc/nginx/ssl:ro
|
|
# - ./nginx/conf.d:/etc/nginx/conf.d:ro
|
|
# networks:
|
|
# - picpeak-network
|
|
# depends_on:
|
|
# - frontend
|
|
# - backend
|
|
# restart: unless-stopped
|
|
|
|
volumes:
|
|
postgres-data:
|
|
driver: local
|
|
redis-data:
|
|
driver: local
|
|
# Holds the auto-generated machine secrets (jwt_secret, db_password,
|
|
# redis_password). Keep it — deleting it orphans the DB password from the
|
|
# Postgres volume. Back it up alongside postgres-data.
|
|
picpeak-secrets:
|
|
driver: local
|
|
|
|
networks:
|
|
picpeak-network:
|
|
driver: bridge
|