Paul Nothaft
1b717ce5ed
Lets PicPeak write photos, thumbnails, hero images, watermarks, and archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local filesystem. Selected via STORAGE_BACKEND=local|s3. Architecture - backend/src/services/storage/StorageBackend.js — abstract interface (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/ getToFile) — typedef-only, documents the contract. - LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path traversal protection, list-as-walker. - S3StorageBackend.js — thin wrapper around the existing S3StorageAdapter (used by backupService) mapping it onto the canonical interface; supports optional STORAGE_S3_PREFIX namespace. - index.js — factory selected by STORAGE_BACKEND with startup ping (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast before the first request. Consumer refactors (~12 services + routes), each parametrized over the abstraction: - imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through storage.put; expose withLocalCopy() helper for S3-mode regeneration paths that need a local file for sharp/ffmpeg. - archiveService / downloadZipService — finalize zip in tmp dir, then storage.putFromFile. Atomic-rename pattern preserved on local; S3 emulates via copy + delete (worker prunes orphaned .tmp.* on startup). - photoProcessor / photoReplacementService / adminPhotos upload+delete / routes/v1/events.js POST /events/:id/photos / routes/events.js — every upload path now goes storage.putFromFile(temp) → unlink temp. - gallery.js bulk-download (cached + on-the-fly + selected) — managed photos via storage.get, external-mode unchanged. - protectedImages / secureImages / photoResolver — read via storage.get; resolvePhotoStorageKey returns the canonical key. - watermarkService / watermarkGeneratorService — persistent watermarks via storage.put. - fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3 (chokidar can't watch S3); auto-import lands via the S3 prefix walker introduced in the follow-up commit. - expirationChecker — small touch (event.expired webhook fire from #327 shipping in the next commit). Migration tooling - backend/scripts/migrate-storage.js — one-shot --dry-run capable script that walks photos.path, thumbnail_path, hero_path, watermark_path and events.archive_path/download_zip_path; streams local → S3; sha256 size-match skip for idempotent re-run; failures CSV. Presigned-URL "Download All" (#328 follow-up shipped in this commit) - routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download + downloads enabled + watermark NOT enabled, /download-all returns a 302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface ships in the next commit's UI. Tests - backend/__tests__/integration/storageBackend.test.js — parametrized contract suite running against BOTH LocalFs AND MinIO (18 tests, both backends — 36 cases total). - backend/__tests__/integration/imageProcessor.storage.test.js — same parametrized pattern for the image processor (10 tests × 2 backends). - backend/__tests__/integration/backup-s3.test.js — bootstrap fix: drop the redundant initDb() (001_init handles it) and remove schema-drift in configureS3Backup (app_settings has no created_at anymore and the unique constraint is on setting_key alone, not composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift). - backend/src/services/photoResolver.js — mixed-source events (reference mode with managed-uploaded photos) now fall back to managed when external_relpath is missing instead of throwing. - tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that auto-skips against local backend; full upload → serve → delete round-trip when run against an S3-mode backend. Server wiring (server.js) - initStorage() called after database init, before rate limiters. - This commit's diff also includes the webhook delivery worker startup and the S3 auto-importer startup. Those features ship in the next two commits — co-located here for one bisectable diff per file. Docs + ops - README §"Storage Backends" — capability matrix, switching playbook, IAM policy snippet, MinIO/R2/B2 examples. - README §"Webhooks" — also added here (full diff bundled). - .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT documented; WEBHOOK_* added in the same diff. - .gitignore — re-anchor the existing `storage/` rule to `/storage/` so backend/src/services/storage/ (the new abstraction code) is trackable. The runtime ./storage/ data dir stays ignored. Out of scope for v1 (per the issue): presigned URLs for individual photo display (always streamed for protection middleware), CDN integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket per-event.
82 lines
2.2 KiB
JSON
82 lines
2.2 KiB
JSON
{
|
|
"name": "picpeak-backend",
|
|
"version": "3.29.1-beta.0",
|
|
"description": "Backend for PicPeak event photo sharing platform",
|
|
"main": "server.js",
|
|
"scripts": {
|
|
"start": "node server.js",
|
|
"dev": "nodemon server.js",
|
|
"migrate": "node migrations/run-migrations.js",
|
|
"migrate:safe": "node migrations/run-migrations-safe.js",
|
|
"generate:watermarks": "node scripts/generate-watermarks.js",
|
|
"test": "jest",
|
|
"test:s3": "SKIP_S3_TESTS=false jest __tests__/integration/backup-s3",
|
|
"lint": "eslint src/"
|
|
},
|
|
"dependencies": {
|
|
"@aws-sdk/client-s3": "^3.850.0",
|
|
"@aws-sdk/lib-storage": "^3.850.0",
|
|
"@aws-sdk/s3-request-presigner": "^3.850.0",
|
|
"@ffmpeg-installer/ffmpeg": "^1.1.0",
|
|
"adm-zip": "^0.5.16",
|
|
"archiver": "^5.3.1",
|
|
"axios": "1.14.0",
|
|
"bcrypt": "6.0.0",
|
|
"chokidar": "4.0.3",
|
|
"cookie-parser": "^1.4.7",
|
|
"cors": "^2.8.5",
|
|
"dotenv": "^16.0.3",
|
|
"exifr": "^7.1.3",
|
|
"express": "^4.18.2",
|
|
"express-rate-limit": "^6.7.0",
|
|
"express-validator": "^7.0.1",
|
|
"fluent-ffmpeg": "^2.1.3",
|
|
"form-data": "^4.0.4",
|
|
"handlebars": "^4.7.9",
|
|
"helmet": "^7.0.0",
|
|
"i18next": "25.3.2",
|
|
"i18next-browser-languagedetector": "^8.2.0",
|
|
"i18next-http-backend": "^3.0.2",
|
|
"ipaddr.js": "^2.3.0",
|
|
"joi": "^17.9.1",
|
|
"js-yaml": "^4.1.1",
|
|
"jsonwebtoken": "^9.0.0",
|
|
"knex": "^2.4.2",
|
|
"mime-types": "^3.0.1",
|
|
"multer": "^2.0.2",
|
|
"node-cron": "^3.0.2",
|
|
"nodemailer": "^7.0.13",
|
|
"pg": "^8.16.3",
|
|
"react-i18next": "^15.6.0",
|
|
"sanitize-html": "^2.17.0",
|
|
"sharp": "0.34.3",
|
|
"sqlite3": "^5.1.6",
|
|
"swagger-jsdoc": "^6.2.8",
|
|
"swagger-ui-express": "^5.0.1",
|
|
"uuid": "^11.1.0",
|
|
"winston": "^3.8.2",
|
|
"zxcvbn": "^4.4.2"
|
|
},
|
|
"devDependencies": {
|
|
"eslint": "^8.40.0",
|
|
"jest": "^29.5.0",
|
|
"mock-fs": "^5.5.0",
|
|
"nodemon": "^3.1.10",
|
|
"supertest": "^6.3.3"
|
|
},
|
|
"overrides": {
|
|
"prebuild-install": {
|
|
"tar-fs": "2.1.4"
|
|
},
|
|
"glob": "^11.1.0",
|
|
"js-yaml": "^4.1.1",
|
|
"fast-xml-parser": ">=5.5.10",
|
|
"qs": ">=6.14.2",
|
|
"tar": ">=7.5.13",
|
|
"brace-expansion": ">=5.0.5",
|
|
"minimatch": ">=9.0.7",
|
|
"path-to-regexp": "0.1.13",
|
|
"lodash": ">=4.18.1"
|
|
}
|
|
}
|