feat: native S3 storage backend (#328) + presigned download follow-up

Lets PicPeak write photos, thumbnails, hero images, watermarks, and
archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2,
Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local
filesystem. Selected via STORAGE_BACKEND=local|s3.

Architecture
- backend/src/services/storage/StorageBackend.js — abstract interface
  (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/
  getToFile) — typedef-only, documents the contract.
- LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path
  traversal protection, list-as-walker.
- S3StorageBackend.js — thin wrapper around the existing
  S3StorageAdapter (used by backupService) mapping it onto the canonical
  interface; supports optional STORAGE_S3_PREFIX namespace.
- index.js — factory selected by STORAGE_BACKEND with startup ping
  (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast
  before the first request.

Consumer refactors (~12 services + routes), each parametrized over the
abstraction:
- imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through
  storage.put; expose withLocalCopy() helper for S3-mode regeneration
  paths that need a local file for sharp/ffmpeg.
- archiveService / downloadZipService — finalize zip in tmp dir, then
  storage.putFromFile. Atomic-rename pattern preserved on local; S3
  emulates via copy + delete (worker prunes orphaned .tmp.* on startup).
- photoProcessor / photoReplacementService / adminPhotos upload+delete /
  routes/v1/events.js POST /events/:id/photos / routes/events.js — every
  upload path now goes storage.putFromFile(temp) → unlink temp.
- gallery.js bulk-download (cached + on-the-fly + selected) — managed
  photos via storage.get, external-mode unchanged.
- protectedImages / secureImages / photoResolver — read via
  storage.get; resolvePhotoStorageKey returns the canonical key.
- watermarkService / watermarkGeneratorService — persistent watermarks
  via storage.put.
- fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3
  (chokidar can't watch S3); auto-import lands via the S3 prefix walker
  introduced in the follow-up commit.
- expirationChecker — small touch (event.expired webhook fire from #327
  shipping in the next commit).

Migration tooling
- backend/scripts/migrate-storage.js — one-shot --dry-run capable script
  that walks photos.path, thumbnail_path, hero_path, watermark_path and
  events.archive_path/download_zip_path; streams local → S3; sha256
  size-match skip for idempotent re-run; failures CSV.

Presigned-URL "Download All" (#328 follow-up shipped in this commit)
- routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download
  + downloads enabled + watermark NOT enabled, /download-all returns a
  302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface
  ships in the next commit's UI.

Tests
- backend/__tests__/integration/storageBackend.test.js — parametrized
  contract suite running against BOTH LocalFs AND MinIO (18 tests, both
  backends — 36 cases total).
- backend/__tests__/integration/imageProcessor.storage.test.js — same
  parametrized pattern for the image processor (10 tests × 2 backends).
- backend/__tests__/integration/backup-s3.test.js — bootstrap fix:
  drop the redundant initDb() (001_init handles it) and remove
  schema-drift in configureS3Backup (app_settings has no created_at
  anymore and the unique constraint is on setting_key alone, not
  composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift).
- backend/src/services/photoResolver.js — mixed-source events (reference
  mode with managed-uploaded photos) now fall back to managed when
  external_relpath is missing instead of throwing.
- tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that
  auto-skips against local backend; full upload → serve → delete
  round-trip when run against an S3-mode backend.

Server wiring (server.js)
- initStorage() called after database init, before rate limiters.
- This commit's diff also includes the webhook delivery worker startup
  and the S3 auto-importer startup. Those features ship in the next two
  commits — co-located here for one bisectable diff per file.

Docs + ops
- README §"Storage Backends" — capability matrix, switching playbook,
  IAM policy snippet, MinIO/R2/B2 examples.
- README §"Webhooks" — also added here (full diff bundled).
- .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT
  documented; WEBHOOK_* added in the same diff.
- .gitignore — re-anchor the existing `storage/` rule to `/storage/`
  so backend/src/services/storage/ (the new abstraction code) is
  trackable. The runtime ./storage/ data dir stays ignored.

Out of scope for v1 (per the issue): presigned URLs for individual
photo display (always streamed for protection middleware), CDN
integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket
per-event.
This commit is contained in:
Paul Nothaft
2026-04-28 10:06:36 +02:00
parent 3d4ae4d7e9
commit 1b717ce5ed
29 changed files with 2365 additions and 714 deletions
@@ -7,12 +7,15 @@ const crypto = require('crypto');
// Load services
const backupService = require('../../src/services/backupService');
const S3StorageAdapter = require('../../src/services/storage/s3Storage');
const { db, initialize: initDb } = require('../../src/database/db');
const { db, initializeDatabase: initDb } = require('../../src/database/db');
const logger = require('../../src/utils/logger');
// Test configuration
// Defaults match the dev MinIO container in docker-compose.dev.yml (port 7104).
// Override via TEST_S3_ENDPOINT / TEST_S3_ACCESS_KEY / TEST_S3_SECRET_KEY when running
// against a different S3 endpoint (CI, hosted MinIO, real AWS, etc.).
const TEST_CONFIG = {
endpoint: process.env.TEST_S3_ENDPOINT || 'http://localhost:9000',
endpoint: process.env.TEST_S3_ENDPOINT || 'http://localhost:7104',
accessKeyId: process.env.TEST_S3_ACCESS_KEY || 'minioadmin',
secretAccessKey: process.env.TEST_S3_SECRET_KEY || 'minioadmin',
bucket: 'test-backup-bucket-' + Date.now(),
@@ -56,9 +59,17 @@ describe('S3 Backup Integration Tests', () => {
}
}
// Initialize database
await initDb();
await db.migrate.latest();
// Schema is expected to already be applied by `npm run migrate` against
// the dev database. db.migrate.latest() can't be used here because
// PicPeak's custom run-migrations.js tracks state in the `migrations`
// table (not knex's `knex_migrations`), so knex would try to re-apply
// every migration and crash on duplicate-table errors.
const ok = await db.schema.hasTable('events')
&& await db.schema.hasTable('app_settings')
&& await db.schema.hasTable('backup_runs');
if (!ok) {
throw new Error('Required tables missing — run `npm run migrate` against the dev DB first.');
}
// Create test storage directory
testStoragePath = path.join(__dirname, '../fixtures/test-storage');
@@ -468,15 +479,16 @@ describe('S3 Backup Integration Tests', () => {
{ setting_key: 'backup_max_file_size_mb', setting_value: '100' }
];
// Schema drift: app_settings has no created_at column anymore and the
// unique constraint is on setting_key alone, not (setting_type, key).
for (const setting of settings) {
await db('app_settings')
.insert({
setting_type: 'backup',
...setting,
created_at: new Date(),
updated_at: new Date()
updated_at: new Date(),
})
.onConflict(['setting_type', 'setting_key'])
.onConflict('setting_key')
.merge();
}
}