feat: native S3 storage backend (#328) + presigned download follow-up
Lets PicPeak write photos, thumbnails, hero images, watermarks, and archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local filesystem. Selected via STORAGE_BACKEND=local|s3. Architecture - backend/src/services/storage/StorageBackend.js — abstract interface (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/ getToFile) — typedef-only, documents the contract. - LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path traversal protection, list-as-walker. - S3StorageBackend.js — thin wrapper around the existing S3StorageAdapter (used by backupService) mapping it onto the canonical interface; supports optional STORAGE_S3_PREFIX namespace. - index.js — factory selected by STORAGE_BACKEND with startup ping (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast before the first request. Consumer refactors (~12 services + routes), each parametrized over the abstraction: - imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through storage.put; expose withLocalCopy() helper for S3-mode regeneration paths that need a local file for sharp/ffmpeg. - archiveService / downloadZipService — finalize zip in tmp dir, then storage.putFromFile. Atomic-rename pattern preserved on local; S3 emulates via copy + delete (worker prunes orphaned .tmp.* on startup). - photoProcessor / photoReplacementService / adminPhotos upload+delete / routes/v1/events.js POST /events/:id/photos / routes/events.js — every upload path now goes storage.putFromFile(temp) → unlink temp. - gallery.js bulk-download (cached + on-the-fly + selected) — managed photos via storage.get, external-mode unchanged. - protectedImages / secureImages / photoResolver — read via storage.get; resolvePhotoStorageKey returns the canonical key. - watermarkService / watermarkGeneratorService — persistent watermarks via storage.put. - fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3 (chokidar can't watch S3); auto-import lands via the S3 prefix walker introduced in the follow-up commit. - expirationChecker — small touch (event.expired webhook fire from #327 shipping in the next commit). Migration tooling - backend/scripts/migrate-storage.js — one-shot --dry-run capable script that walks photos.path, thumbnail_path, hero_path, watermark_path and events.archive_path/download_zip_path; streams local → S3; sha256 size-match skip for idempotent re-run; failures CSV. Presigned-URL "Download All" (#328 follow-up shipped in this commit) - routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download + downloads enabled + watermark NOT enabled, /download-all returns a 302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface ships in the next commit's UI. Tests - backend/__tests__/integration/storageBackend.test.js — parametrized contract suite running against BOTH LocalFs AND MinIO (18 tests, both backends — 36 cases total). - backend/__tests__/integration/imageProcessor.storage.test.js — same parametrized pattern for the image processor (10 tests × 2 backends). - backend/__tests__/integration/backup-s3.test.js — bootstrap fix: drop the redundant initDb() (001_init handles it) and remove schema-drift in configureS3Backup (app_settings has no created_at anymore and the unique constraint is on setting_key alone, not composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift). - backend/src/services/photoResolver.js — mixed-source events (reference mode with managed-uploaded photos) now fall back to managed when external_relpath is missing instead of throwing. - tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that auto-skips against local backend; full upload → serve → delete round-trip when run against an S3-mode backend. Server wiring (server.js) - initStorage() called after database init, before rate limiters. - This commit's diff also includes the webhook delivery worker startup and the S3 auto-importer startup. Those features ship in the next two commits — co-located here for one bisectable diff per file. Docs + ops - README §"Storage Backends" — capability matrix, switching playbook, IAM policy snippet, MinIO/R2/B2 examples. - README §"Webhooks" — also added here (full diff bundled). - .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT documented; WEBHOOK_* added in the same diff. - .gitignore — re-anchor the existing `storage/` rule to `/storage/` so backend/src/services/storage/ (the new abstraction code) is trackable. The runtime ./storage/ data dir stays ignored. Out of scope for v1 (per the issue): presigned URLs for individual photo display (always streamed for protection middleware), CDN integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket per-event.
This commit is contained in:
Paul Nothaft
@@ -177,10 +177,111 @@ Perfect for:
|
||||
|
||||
- **Backend**: Node.js, Express, SQLite/PostgreSQL
|
||||
- **Frontend**: React, Tailwind CSS, Framer Motion
|
||||
- **Storage**: File-based with automatic archiving
|
||||
- **Storage**: Local filesystem (default) or S3-compatible object store (AWS S3, MinIO, R2, B2, Wasabi, Spaces) — see [Storage Backends](#storage-backends)
|
||||
- **Email**: SMTP with customizable templates
|
||||
- **Analytics**: Privacy-focused with Umami integration
|
||||
|
||||
## 💾 Storage Backends
|
||||
|
||||
PicPeak supports two storage backends for photos, thumbnails, hero images, watermarks, and archive zips. Both are configured via environment variables; no code change is required to switch.
|
||||
|
||||
| Capability | `STORAGE_BACKEND=local` (default) | `STORAGE_BACKEND=s3` |
|
||||
|---|---|---|
|
||||
| Photo / thumbnail / hero storage | Local filesystem under `STORAGE_PATH` | Bucket on any S3-compatible service |
|
||||
| Admin UI upload | ✅ | ✅ |
|
||||
| Filesystem auto-import (chokidar watcher) | ✅ | ❌ — disabled (use the upload API) |
|
||||
| Watermarks, fingerprinting, fragmentation | ✅ | ✅ (materialized to a tmp file just-in-time) |
|
||||
| Bulk download zips (cached + on-the-fly) | ✅ | ✅ |
|
||||
| Backups | ✅ | ✅ |
|
||||
| External media reference mode (`EXTERNAL_MEDIA_ROOT`) | ✅ (always local) | ✅ (still local — not migrated) |
|
||||
|
||||
### Switching to an S3-compatible backend
|
||||
|
||||
1. Provision a bucket and credentials. The minimum IAM policy is documented in `.env.example`.
|
||||
2. Set `STORAGE_BACKEND=s3` plus `STORAGE_S3_BUCKET`, `STORAGE_S3_REGION`, `STORAGE_S3_ACCESS_KEY`, `STORAGE_S3_SECRET_KEY`. For non-AWS providers (MinIO, R2, B2, …) also set `STORAGE_S3_ENDPOINT`.
|
||||
3. If you have existing local content, copy it first: `node backend/scripts/migrate-storage.js --dry-run` then `node backend/scripts/migrate-storage.js`. The script is idempotent and writes a failures CSV.
|
||||
4. Restart the backend. The startup check pings the bucket and refuses to boot on misconfig.
|
||||
|
||||
Note: presigned-URL serving (zero-bandwidth direct downloads from S3) is intentionally **not** in v1 — every request still streams through the backend so watermarks, devtools-detection, and access logging keep working.
|
||||
|
||||
## 🔔 Webhooks
|
||||
|
||||
PicPeak POSTs event/photo lifecycle notifications to URLs you configure under **Settings → Webhooks**. Each delivery is signed `HMAC-SHA256` with a per-webhook secret in the `X-PicPeak-Signature` header so receivers can verify the request really came from your PicPeak instance.
|
||||
|
||||
### Event types
|
||||
|
||||
| Event | Fires when |
|
||||
|---|---|
|
||||
| `event.created` | Gallery created (admin or API) |
|
||||
| `event.published` | Draft becomes live (`is_draft: true → false`) — also fires when an event is created with `is_draft=false` |
|
||||
| `event.archived` | Bulk-archive, manual archive, or auto-archive on expiry |
|
||||
| `event.expired` | Expiration checker marks the gallery inactive (fires before `event.archived` in the cascade) |
|
||||
| `photo.uploaded` | Admin upload, API upload, guest upload, or auto-import |
|
||||
| `photo.deleted` | Single delete, bulk delete (NOT fired per-photo when an event is archived — receivers infer from `event.archived` to avoid flooding) |
|
||||
|
||||
### Payload shape
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "delivery-uuid",
|
||||
"type": "event.published",
|
||||
"created_at": "2026-04-28T05:25:00.000Z",
|
||||
"data": {
|
||||
"event": { "id": 123, "slug": "wedding-smith", "share_url": "https://..." }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Also sent on every request:
|
||||
- `X-PicPeak-Signature` — `HMAC-SHA256(secret, raw_body)` as hex
|
||||
- `X-PicPeak-Event` — the event type (handy for routing without parsing the body)
|
||||
- `X-PicPeak-Delivery` — UUID for idempotency on the receiver side
|
||||
- `User-Agent: PicPeak-Webhooks/1.0`
|
||||
|
||||
### Verifying signatures
|
||||
|
||||
**Node.js**
|
||||
```js
|
||||
const crypto = require('crypto');
|
||||
function verify(secret, rawBody, signature) {
|
||||
const expected = crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
|
||||
const a = Buffer.from(expected, 'hex');
|
||||
const b = Buffer.from(signature, 'hex');
|
||||
if (a.length !== b.length) return false;
|
||||
return crypto.timingSafeEqual(a, b);
|
||||
}
|
||||
```
|
||||
|
||||
**Python**
|
||||
```python
|
||||
import hmac, hashlib
|
||||
def verify(secret: str, raw_body: bytes, signature: str) -> bool:
|
||||
expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
|
||||
return hmac.compare_digest(expected, signature)
|
||||
```
|
||||
|
||||
**curl + openssl** (one-liner for a quick replay)
|
||||
```sh
|
||||
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')
|
||||
[ "$SIG" = "$RECEIVED_SIG" ] && echo OK || echo MISMATCH
|
||||
```
|
||||
|
||||
### Retries + observability
|
||||
|
||||
- `2xx` → success, recorded with latency
|
||||
- Non-`2xx` or network error → exponential backoff: `1m → 5m → 30m → 2h → 12h`, max 5 attempts
|
||||
- After max attempts: status `failed`, surfaces in **Settings → Webhooks → Deliveries** with a "Replay" button
|
||||
- Up to 5 deliveries in flight at once; one slow consumer can't block others (configurable via `WEBHOOK_DELIVERY_CONCURRENCY`)
|
||||
- Response body truncated to 1KB before storage so chatty receivers don't bloat the audit log
|
||||
|
||||
The deliveries page (`/admin/webhooks/:id/deliveries`) shows every attempt with timestamp, status, HTTP code, latency, payload sent, signature, and response. Click "Send test event" to fire a synthetic delivery for any event type.
|
||||
|
||||
### SSRF protection
|
||||
|
||||
Webhook URLs are validated against the same private-IP blocklist used elsewhere in the app — loopback, private RFC1918 ranges, link-local, `.local`/`.internal` hostnames, cloud metadata endpoints. The check runs both at create time and per-delivery (DNS-rebinding mitigation).
|
||||
|
||||
For local development with a receiver on the same machine or docker network, set `WEBHOOK_ALLOW_PRIVATE_URLS=true`. Production deployments must leave this OFF.
|
||||
|
||||
## 💻 System Requirements
|
||||
|
||||
### Minimum Requirements
|
||||
|
||||
Reference in New Issue
Block a user