Paul Nothaft
1b717ce5ed
Lets PicPeak write photos, thumbnails, hero images, watermarks, and archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local filesystem. Selected via STORAGE_BACKEND=local|s3. Architecture - backend/src/services/storage/StorageBackend.js — abstract interface (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/ getToFile) — typedef-only, documents the contract. - LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path traversal protection, list-as-walker. - S3StorageBackend.js — thin wrapper around the existing S3StorageAdapter (used by backupService) mapping it onto the canonical interface; supports optional STORAGE_S3_PREFIX namespace. - index.js — factory selected by STORAGE_BACKEND with startup ping (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast before the first request. Consumer refactors (~12 services + routes), each parametrized over the abstraction: - imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through storage.put; expose withLocalCopy() helper for S3-mode regeneration paths that need a local file for sharp/ffmpeg. - archiveService / downloadZipService — finalize zip in tmp dir, then storage.putFromFile. Atomic-rename pattern preserved on local; S3 emulates via copy + delete (worker prunes orphaned .tmp.* on startup). - photoProcessor / photoReplacementService / adminPhotos upload+delete / routes/v1/events.js POST /events/:id/photos / routes/events.js — every upload path now goes storage.putFromFile(temp) → unlink temp. - gallery.js bulk-download (cached + on-the-fly + selected) — managed photos via storage.get, external-mode unchanged. - protectedImages / secureImages / photoResolver — read via storage.get; resolvePhotoStorageKey returns the canonical key. - watermarkService / watermarkGeneratorService — persistent watermarks via storage.put. - fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3 (chokidar can't watch S3); auto-import lands via the S3 prefix walker introduced in the follow-up commit. - expirationChecker — small touch (event.expired webhook fire from #327 shipping in the next commit). Migration tooling - backend/scripts/migrate-storage.js — one-shot --dry-run capable script that walks photos.path, thumbnail_path, hero_path, watermark_path and events.archive_path/download_zip_path; streams local → S3; sha256 size-match skip for idempotent re-run; failures CSV. Presigned-URL "Download All" (#328 follow-up shipped in this commit) - routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download + downloads enabled + watermark NOT enabled, /download-all returns a 302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface ships in the next commit's UI. Tests - backend/__tests__/integration/storageBackend.test.js — parametrized contract suite running against BOTH LocalFs AND MinIO (18 tests, both backends — 36 cases total). - backend/__tests__/integration/imageProcessor.storage.test.js — same parametrized pattern for the image processor (10 tests × 2 backends). - backend/__tests__/integration/backup-s3.test.js — bootstrap fix: drop the redundant initDb() (001_init handles it) and remove schema-drift in configureS3Backup (app_settings has no created_at anymore and the unique constraint is on setting_key alone, not composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift). - backend/src/services/photoResolver.js — mixed-source events (reference mode with managed-uploaded photos) now fall back to managed when external_relpath is missing instead of throwing. - tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that auto-skips against local backend; full upload → serve → delete round-trip when run against an S3-mode backend. Server wiring (server.js) - initStorage() called after database init, before rate limiters. - This commit's diff also includes the webhook delivery worker startup and the S3 auto-importer startup. Those features ship in the next two commits — co-located here for one bisectable diff per file. Docs + ops - README §"Storage Backends" — capability matrix, switching playbook, IAM policy snippet, MinIO/R2/B2 examples. - README §"Webhooks" — also added here (full diff bundled). - .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT documented; WEBHOOK_* added in the same diff. - .gitignore — re-anchor the existing `storage/` rule to `/storage/` so backend/src/services/storage/ (the new abstraction code) is trackable. The runtime ./storage/ data dir stays ignored. Out of scope for v1 (per the issue): presigned URLs for individual photo display (always streamed for protection middleware), CDN integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket per-event.
201 lines
7.6 KiB
Bash
201 lines
7.6 KiB
Bash
# PicPeak Environment Configuration
|
|
# Copy this file to .env and update with your values
|
|
|
|
# Environment
|
|
NODE_ENV=production
|
|
|
|
# JWT Secret (generate with: openssl rand -base64 64)
|
|
JWT_SECRET=your_very_long_random_jwt_secret_here
|
|
|
|
# Auth cookie Secure flag
|
|
# unset - default: follows NODE_ENV (production=true, dev=false)
|
|
# true - always set Secure (HTTPS-only cookies; breaks plain-HTTP access)
|
|
# false - never set Secure (allows HTTP; cookies not protected on HTTPS)
|
|
# auto - decide per request: Secure on HTTPS, not on HTTP
|
|
#
|
|
# Use COOKIE_SECURE=auto if your deployment is reachable over both HTTPS
|
|
# (via reverse proxy like Nginx Proxy Manager, Traefik, Caddy) AND plain
|
|
# HTTP (e.g. LAN access at http://192.168.x.x:3010). The backend reads
|
|
# req.secure from Express, which respects the X-Forwarded-Proto header
|
|
# when the proxy is in the trust list.
|
|
#
|
|
# Requirements for auto mode:
|
|
# 1. Your reverse proxy MUST send X-Forwarded-Proto: https on HTTPS
|
|
# requests. Standard configs for NPM/Traefik/Caddy do this by default.
|
|
# 2. The proxy must be on a trusted IP range. By default PicPeak trusts
|
|
# loopback and private networks (127.0.0.1, 10.x, 172.16-31.x,
|
|
# 192.168.x, link-local). Proxies outside those ranges need custom
|
|
# trust proxy configuration.
|
|
# COOKIE_SECURE=auto
|
|
|
|
# Cookie SameSite attribute (Lax | Strict | None). Default: Lax
|
|
# COOKIE_SAMESITE=Lax
|
|
|
|
# Cookie Domain — set this if serving auth cookies across subdomains.
|
|
# Leave unset for same-origin setups.
|
|
# COOKIE_DOMAIN=.example.com
|
|
|
|
# Database Configuration (PostgreSQL)
|
|
DATABASE_CLIENT=pg
|
|
DB_USER=picpeak
|
|
# IMPORTANT: Avoid $ character in passwords - Docker Compose interprets it as variable substitution
|
|
# If you must use $, escape it as $$ (e.g., Pass$$word instead of Pass$word)
|
|
DB_PASSWORD=your_secure_postgres_password_here
|
|
DB_NAME=picpeak_prod
|
|
|
|
# Redis Configuration
|
|
# IMPORTANT: Same warning applies - avoid $ or escape as $$
|
|
REDIS_PASSWORD=your_secure_redis_password_here
|
|
|
|
# Admin Account (initial setup)
|
|
ADMIN_USERNAME=admin
|
|
ADMIN_EMAIL=admin@yourdomain.com
|
|
ADMIN_PASSWORD=your_secure_admin_password_here
|
|
|
|
# Email Configuration
|
|
# For Gmail: use app-specific password
|
|
# For SendGrid: SMTP_USER=apikey, SMTP_PASS=your-api-key
|
|
SMTP_HOST=smtp.gmail.com
|
|
SMTP_PORT=587
|
|
SMTP_SECURE=false
|
|
SMTP_USER=your-email@gmail.com
|
|
SMTP_PASS=your-app-specific-password
|
|
EMAIL_FROM=noreply@yourdomain.com
|
|
|
|
# Application URLs
|
|
# Use full origin with scheme, no trailing slash.
|
|
# Admin UI is served by the frontend at /admin.
|
|
FRONTEND_URL=https://yourdomain.com
|
|
ADMIN_URL=https://yourdomain.com
|
|
|
|
# API URL for email assets (logos, images in notification emails)
|
|
# This must be the publicly accessible URL where email recipients can load images.
|
|
# If not set, defaults to http://localhost:3001 which will show broken images in emails.
|
|
API_URL=https://yourdomain.com/api
|
|
|
|
# Frontend API base
|
|
# For pre-built images and production behind a reverse proxy, keep '/api'.
|
|
# If you rebuild the frontend yourself, you may set a full URL at build time.
|
|
VITE_API_URL=/api
|
|
|
|
# Port Configuration (optional)
|
|
# BACKEND_PORT=3001
|
|
# FRONTEND_PORT=3000
|
|
# DB_PORT=5432
|
|
# REDIS_PORT=6379
|
|
|
|
# Release Channel
|
|
# Options: 'stable' (default), 'beta', or specific version like 'v2.3.0'
|
|
# 'stable' uses the :stable tag (same as :latest on main)
|
|
# 'beta' uses the :beta tag for pre-release versions
|
|
PICPEAK_CHANNEL=stable
|
|
|
|
# Update Check Configuration
|
|
# Set to 'false' to disable update notifications in admin UI
|
|
UPDATE_CHECK_ENABLED=true
|
|
|
|
# Timezone
|
|
TZ=UTC
|
|
|
|
# Runtime user mapping for Docker (optional)
|
|
# Set these to your host user's UID/GID to avoid permission issues on bind mounts.
|
|
# Run `id -u` and `id -g` on host to get values. Defaults to 1001.
|
|
PUID=1001
|
|
PGID=1001
|
|
|
|
# Analytics (Optional - Umami)
|
|
VITE_UMAMI_URL=
|
|
VITE_UMAMI_WEBSITE_ID=
|
|
VITE_UMAMI_SHARE_URL=
|
|
|
|
# Storage variables (host paths)
|
|
# These control where data is stored on the host. Defaults are local folders.
|
|
APP_STORAGE=./storage
|
|
APP_DATA=./data
|
|
LOGS=./logs
|
|
|
|
# ─── Storage Backend ────────────────────────────────────────────────────────
|
|
# PicPeak can store photos, thumbnails and archive zips on the local filesystem
|
|
# (default) or on any S3-compatible object store (AWS S3, MinIO, Cloudflare R2,
|
|
# Backblaze B2, Wasabi, DigitalOcean Spaces, …).
|
|
#
|
|
# STORAGE_BACKEND=local (default)
|
|
# Uses STORAGE_PATH on the local filesystem. Backwards compatible — every
|
|
# existing deployment keeps working unchanged.
|
|
#
|
|
# STORAGE_BACKEND=s3
|
|
# Reads STORAGE_S3_* below. Auto-import via the filesystem watcher is
|
|
# disabled in this mode (S3 has no inotify) — every photo must enter via the
|
|
# admin upload UI/API. Run `node backend/scripts/migrate-storage.js` to copy
|
|
# existing local content to S3 before flipping the env.
|
|
#
|
|
# STORAGE_BACKEND=local
|
|
#
|
|
# STORAGE_S3_BUCKET=picpeak
|
|
# STORAGE_S3_REGION=us-east-1
|
|
# STORAGE_S3_ACCESS_KEY=AKIAxxxxxxxxxxxxxxxx
|
|
# STORAGE_S3_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxx
|
|
# Custom endpoint — set this for MinIO / R2 / B2 / Spaces. Leave unset for AWS.
|
|
# STORAGE_S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com
|
|
# Optional namespace prefix inside the bucket — useful for multi-deployment buckets.
|
|
# STORAGE_S3_PREFIX=picpeak
|
|
# STORAGE_S3_FORCE_PATH_STYLE=false # MinIO needs true; auto-on when endpoint is set
|
|
# STORAGE_S3_SSL=true
|
|
#
|
|
# Minimum IAM policy (AWS S3) for the bucket above:
|
|
# {
|
|
# "Version": "2012-10-17",
|
|
# "Statement": [{
|
|
# "Effect": "Allow",
|
|
# "Action": [
|
|
# "s3:GetObject", "s3:PutObject", "s3:DeleteObject",
|
|
# "s3:ListBucket", "s3:GetBucketLocation"
|
|
# ],
|
|
# "Resource": [
|
|
# "arn:aws:s3:::picpeak",
|
|
# "arn:aws:s3:::picpeak/*"
|
|
# ]
|
|
# }]
|
|
# }
|
|
#
|
|
# EXTERNAL_MEDIA_ROOT (above) always lives on the local filesystem regardless
|
|
# of STORAGE_BACKEND — reference-mode galleries are not migrated to S3 in v1.
|
|
|
|
# ─── Outbound Webhooks (#327) ────────────────────────────────────────────────
|
|
# PicPeak POSTs event/photo lifecycle notifications to URLs you configure
|
|
# under Settings → Webhooks. Each delivery is signed HMAC-SHA256 with a
|
|
# per-webhook secret in the X-PicPeak-Signature header.
|
|
#
|
|
# WEBHOOK_ALLOW_PRIVATE_URLS (default: false)
|
|
# Block URLs resolving to private IPs / loopback / .local etc. as an
|
|
# SSRF mitigation. Set to "true" ONLY in dev when your receiver is on
|
|
# the same docker network or localhost. Production deployments must
|
|
# leave this OFF.
|
|
# WEBHOOK_ALLOW_PRIVATE_URLS=false
|
|
#
|
|
# WEBHOOK_DELIVERY_INTERVAL_MS (default: 5000)
|
|
# How often the worker polls webhook_deliveries for pending rows.
|
|
# WEBHOOK_DELIVERY_INTERVAL_MS=5000
|
|
#
|
|
# WEBHOOK_DELIVERY_CONCURRENCY (default: 5)
|
|
# Maximum in-flight deliveries per worker tick. One slow consumer can
|
|
# monopolize all 5 slots — bump this if your receivers are slow OR ship
|
|
# a separate webhook-only deployment.
|
|
# WEBHOOK_DELIVERY_CONCURRENCY=5
|
|
#
|
|
# WEBHOOK_HTTP_TIMEOUT_MS (default: 10000)
|
|
# Per-request timeout. Beyond this, the delivery is recorded as a
|
|
# network error and retried.
|
|
# WEBHOOK_HTTP_TIMEOUT_MS=10000
|
|
#
|
|
# WEBHOOK_MAX_ATTEMPTS (default: 5)
|
|
# Total attempts before a delivery is marked failed. Backoff between
|
|
# attempts is exponential: 1m, 5m, 30m, 2h, 12h.
|
|
# WEBHOOK_MAX_ATTEMPTS=5
|
|
|
|
# Note on FRONTEND_API_URL (documentation only):
|
|
# When using pre-built frontend images, runtime env vars cannot override the built JS.
|
|
# Do NOT rely on FRONTEND_API_URL in Compose. Instead, keep VITE_API_URL=/api and
|
|
# let the frontend Nginx proxy /api to the backend. Only if you rebuild the frontend
|
|
# should you change VITE_API_URL at build time.
|