paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: native S3 storage backend (#328) + presigned download follow-up

Browse Source 
Lets PicPeak write photos, thumbnails, hero images, watermarks, and
archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2,
Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local
filesystem. Selected via STORAGE_BACKEND=local|s3.

Architecture
- backend/src/services/storage/StorageBackend.js — abstract interface
  (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/
  getToFile) — typedef-only, documents the contract.
- LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path
  traversal protection, list-as-walker.
- S3StorageBackend.js — thin wrapper around the existing
  S3StorageAdapter (used by backupService) mapping it onto the canonical
  interface; supports optional STORAGE_S3_PREFIX namespace.
- index.js — factory selected by STORAGE_BACKEND with startup ping
  (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast
  before the first request.

Consumer refactors (~12 services + routes), each parametrized over the
abstraction:
- imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through
  storage.put; expose withLocalCopy() helper for S3-mode regeneration
  paths that need a local file for sharp/ffmpeg.
- archiveService / downloadZipService — finalize zip in tmp dir, then
  storage.putFromFile. Atomic-rename pattern preserved on local; S3
  emulates via copy + delete (worker prunes orphaned .tmp.* on startup).
- photoProcessor / photoReplacementService / adminPhotos upload+delete /
  routes/v1/events.js POST /events/:id/photos / routes/events.js — every
  upload path now goes storage.putFromFile(temp) → unlink temp.
- gallery.js bulk-download (cached + on-the-fly + selected) — managed
  photos via storage.get, external-mode unchanged.
- protectedImages / secureImages / photoResolver — read via
  storage.get; resolvePhotoStorageKey returns the canonical key.
- watermarkService / watermarkGeneratorService — persistent watermarks
  via storage.put.
- fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3
  (chokidar can't watch S3); auto-import lands via the S3 prefix walker
  introduced in the follow-up commit.
- expirationChecker — small touch (event.expired webhook fire from #327
  shipping in the next commit).

Migration tooling
- backend/scripts/migrate-storage.js — one-shot --dry-run capable script
  that walks photos.path, thumbnail_path, hero_path, watermark_path and
  events.archive_path/download_zip_path; streams local → S3; sha256
  size-match skip for idempotent re-run; failures CSV.

Presigned-URL "Download All" (#328 follow-up shipped in this commit)
- routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download
  + downloads enabled + watermark NOT enabled, /download-all returns a
  302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface
  ships in the next commit's UI.

Tests
- backend/__tests__/integration/storageBackend.test.js — parametrized
  contract suite running against BOTH LocalFs AND MinIO (18 tests, both
  backends — 36 cases total).
- backend/__tests__/integration/imageProcessor.storage.test.js — same
  parametrized pattern for the image processor (10 tests × 2 backends).
- backend/__tests__/integration/backup-s3.test.js — bootstrap fix:
  drop the redundant initDb() (001_init handles it) and remove
  schema-drift in configureS3Backup (app_settings has no created_at
  anymore and the unique constraint is on setting_key alone, not
  composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift).
- backend/src/services/photoResolver.js — mixed-source events (reference
  mode with managed-uploaded photos) now fall back to managed when
  external_relpath is missing instead of throwing.
- tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that
  auto-skips against local backend; full upload → serve → delete
  round-trip when run against an S3-mode backend.

Server wiring (server.js)
- initStorage() called after database init, before rate limiters.
- This commit's diff also includes the webhook delivery worker startup
  and the S3 auto-importer startup. Those features ship in the next two
  commits — co-located here for one bisectable diff per file.

Docs + ops
- README §"Storage Backends" — capability matrix, switching playbook,
  IAM policy snippet, MinIO/R2/B2 examples.
- README §"Webhooks" — also added here (full diff bundled).
- .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT
  documented; WEBHOOK_* added in the same diff.
- .gitignore — re-anchor the existing `storage/` rule to `/storage/`
  so backend/src/services/storage/ (the new abstraction code) is
  trackable. The runtime ./storage/ data dir stays ignored.

Out of scope for v1 (per the issue): presigned URLs for individual
photo display (always streamed for protection middleware), CDN
integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket
per-event.
This commit is contained in:
Paul Nothaft
2026-04-28 10:06:36 +02:00
parent 3d4ae4d7e9
commit 1b717ce5ed
29 changed files with 2365 additions and 714 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/adminPhotos.js
+158 -100
View File
@@ -17,6 +17,7 @@ const watermarkGeneratorService = require('../services/watermarkGeneratorService
const downloadZipService = require('../services/downloadZipService');
const { findReplacementCandidate, replacePhoto } = require('../services/photoReplacementService');
const { requireEventOwnership } = require('../middleware/ownership');
const { getStorage } = require('../services/storage');
const router = express.Router();
// Get storage path from environment or default
@@ -243,9 +244,9 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
categoryName = 'collages';
}
// Create final destination directory
const finalDestPath = path.join(getStoragePath(), 'events/active', event.slug);
await fs.mkdir(finalDestPath, { recursive: true });
// Final destination key prefix under the storage backend (no local mkdir
// needed — LocalFsStorage creates the parent dir on put, S3 has no dirs).
const finalDestPathRel = path.posix.join('events/active', event.slug);
const uploadedPhotos = [];
const replacedPhotos = [];
@@ -330,10 +331,11 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
extension
);
// Calculate final path
const finalPath = path.join(finalDestPath, newFilename);
const storagePath = getStoragePath();
const relativePath = path.relative(path.join(storagePath, 'events/active'), finalPath);
// Storage key: events/active/{slug}/{newFilename}
const finalKey = path.posix.join(finalDestPathRel, newFilename);
// photo.path is stored relative to events/active so resolvePhotoStorageKey
// can rebuild the full key on read.
const relativePath = path.posix.join(event.slug, newFilename);
// Extract capture date from EXIF metadata
let capturedAt = null;
@@ -365,10 +367,10 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
batchPhotos.push(photoData);
// Store move operation for later
// Store upload operation for later (after DB commit)
fileRenameOperations.push({
tempPath: tempPath,
finalPath: finalPath,
finalKey: finalKey,
filename: newFilename,
photoData: photoData
});
@@ -390,34 +392,26 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
await trx.commit();
console.log(`Successfully committed batch of ${batchPhotos.length} photos`);
// Now move files from temp to final location after successful commit
// Now upload files from temp into the storage backend after successful commit
const storage = getStorage();
for (let idx = 0; idx < fileRenameOperations.length; idx++) {
const operation = fileRenameOperations[idx];
try {
// Move the file from temp to final location
await fs.rename(operation.tempPath, operation.finalPath);
console.log(`Moved file from ${operation.tempPath} to ${operation.finalPath}`);
// Verify the file was moved successfully
const finalStats = await fs.stat(operation.finalPath);
if (finalStats.size !== operation.photoData.size_bytes) {
throw new Error(`File size mismatch after move: expected ${operation.photoData.size_bytes}, got ${finalStats.size}`);
}
// Generate thumbnail and extract metadata
// Process source-dependent steps (sharp/ffmpeg) FIRST while the
// tmp file is still on local disk, then upload the original and
// unlink the tmp.
const photoId = insertedIds[idx]?.id || insertedIds[idx];
const isVideoFile = isVideoMimeType(operation.photoData.mime_type);
let thumbnailPath = null;
try {
if (isVideoFile) {
// Process video: extract metadata and generate thumbnail
const thumbnailDir = path.join(getStoragePath(), 'thumbnails');
await fs.mkdir(thumbnailDir, { recursive: true });
const videoThumbnailPath = path.join(thumbnailDir, `thumb_${operation.filename.replace(/\.[^.]+$/, '.jpg')}`);
const result = await processUploadedVideo(operation.finalPath, videoThumbnailPath);
thumbnailPath = path.relative(getStoragePath(), videoThumbnailPath);
const videoThumbnailKey = path.posix.join(
'thumbnails',
`thumb_${operation.filename.replace(/\.[^.]+$/, '.jpg')}`
);
const result = await processUploadedVideo(operation.tempPath, videoThumbnailKey);
thumbnailPath = result.thumbnailKey;
if (photoId && result.metadata) {
await db('photos')
@@ -432,7 +426,7 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
});
}
} else {
thumbnailPath = await generateThumbnail(operation.finalPath);
thumbnailPath = await generateThumbnail(operation.tempPath);
// Update the database with thumbnail path and image dimensions
if (photoId) {
@@ -441,7 +435,7 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
try {
const sharp = require('sharp');
const metadata = await sharp(operation.finalPath).metadata();
const metadata = await sharp(operation.tempPath).metadata();
if (metadata.width && metadata.height) {
updateData.width = metadata.width;
updateData.height = metadata.height;
@@ -461,12 +455,40 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
console.error(`Thumbnail/metadata processing failed for ${operation.filename}:`, thumbError.message);
}
// Upload the original through the storage backend, then drop the
// local tmp file. We do this AFTER thumbnail/metadata processing
// so sharp/ffmpeg still have a local source to work from.
await storage.putFromFile(operation.finalKey, operation.tempPath, {
contentType: operation.photoData.mime_type,
});
await fs.unlink(operation.tempPath).catch(() => {});
// Sanity check: round-trip the size we just wrote.
const stat = await storage.stat(operation.finalKey);
if (!stat || stat.size !== operation.photoData.size_bytes) {
throw new Error(`Size mismatch after upload: expected ${operation.photoData.size_bytes}, got ${stat ? stat.size : 'null'}`);
}
// Queue watermark generation in background (non-blocking, images only)
if (photoId && !isVideoFile) {
watermarkGeneratorService.generateForPhoto(photoId)
.catch(err => console.warn(`Watermark generation queued failed for photo ${photoId}:`, err.message));
}
// Webhook (#327): per-photo upload event.
try {
const webhookService = require('../services/webhookService');
await webhookService.fire('photo.uploaded', {
event: { id: parseInt(eventId, 10), slug: event.slug, event_name: event.event_name },
photo: {
id: insertedIds[idx]?.id || insertedIds[idx],
filename: operation.filename,
original_filename: operation.photoData.original_filename,
size_bytes: operation.photoData.size_bytes,
},
});
} catch (e) { /* non-fatal */ }
// Add to successful uploads
uploadedPhotos.push({
id: insertedIds[idx]?.id || insertedIds[idx],
@@ -475,10 +497,10 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
category_id: operation.photoData.category_id
});
} catch (moveError) {
console.error(`Failed to move file ${operation.tempPath} to ${operation.finalPath}:`, moveError);
errors.push({
filename: operation.filename,
error: `File move failed: ${moveError.message}`
console.error(`Failed to upload ${operation.tempPath} → ${operation.finalKey}:`, moveError);
errors.push({
filename: operation.filename,
error: `File upload failed: ${moveError.message}`
});
// Try to clean up the database entry if file move failed
@@ -604,30 +626,30 @@ router.delete('/:eventId/photos/:photoId', adminAuth, requirePermission('photos.
return res.status(404).json({ error: 'Photo not found' });
}
// Delete physical files
const storagePath = getStoragePath();
const photoPath = path.join(storagePath, 'events/active', photo.path);
// Delete original + thumbnail through the storage backend.
const storage = getStorage();
const { resolvePhotoStorageKey } = require('../services/photoResolver');
const event = await db('events').where({ id: eventId }).first();
try {
await fs.unlink(photoPath);
const originalKey = resolvePhotoStorageKey(event, photo);
if (originalKey) await storage.delete(originalKey);
} catch (error) {
console.error('Error deleting photo file:', error);
}
// Delete thumbnail if exists
// photo.thumbnail_path is stored as the canonical storage key
// (e.g. "thumbnails/thumb_foo.jpg"), so pass it through verbatim.
if (photo.thumbnail_path) {
const thumbPath = path.join(storagePath, 'events/active', photo.thumbnail_path);
try {
// Check if file exists before attempting to delete
await fs.access(thumbPath);
await fs.unlink(thumbPath);
await storage.delete(photo.thumbnail_path);
} catch (error) {
// Only log if it's not a "file not found" error
if (error.code !== 'ENOENT') {
console.error('Error deleting thumbnail:', error);
}
console.error('Error deleting thumbnail:', error);
}
}
if (photo.hero_path) {
await storage.delete(photo.hero_path).catch(() => {});
}
// Delete pre-generated watermark if exists
if (photo.watermark_path) {
@@ -636,15 +658,23 @@ router.delete('/:eventId/photos/:photoId', adminAuth, requirePermission('photos.
// Remove from database
await db('photos').where({ id: photoId }).delete();
// Log activity
const event = await db('events').where({ id: eventId }).first();
// Log activity (event was fetched above for storage key resolution)
await logActivity('photo_deleted',
{ filename: photo.filename, eventName: event.event_name },
eventId,
{ type: 'admin', id: req.admin.id, name: req.admin.username }
);
// Webhook (#327): single-photo delete.
try {
const webhookService = require('../services/webhookService');
await webhookService.fire('photo.deleted', {
event: { id: parseInt(eventId, 10), slug: event?.slug, event_name: event?.event_name },
photo: { id: parseInt(photoId, 10), filename: photo.filename },
});
} catch (e) { /* non-fatal */ }
downloadZipService.invalidate(parseInt(eventId));
res.json({ message: 'Photo deleted successfully' });
} catch (error) {
@@ -735,35 +765,25 @@ router.post('/:eventId/photos/bulk-delete', adminAuth, requirePermission('photos
return res.status(404).json({ error: 'No photos found' });
}
// Delete physical files
const storagePath = getStoragePath();
// Delete original + thumbnail + hero through the storage backend.
const storage = getStorage();
const event = await db('events').where({ id: eventId }).first();
const { resolvePhotoStorageKey } = require('../services/photoResolver');
for (const photo of photos) {
// Delete photo file
const photoPath = path.join(storagePath, 'events/active', photo.path);
try {
await fs.unlink(photoPath);
const originalKey = resolvePhotoStorageKey(event, photo);
if (originalKey) await storage.delete(originalKey);
} catch (error) {
console.error('Error deleting photo file:', error);
}
// Delete thumbnail
if (photo.thumbnail_path) {
const thumbPath = path.join(storagePath, photo.thumbnail_path);
try {
// Check if file exists before attempting to delete
await fs.access(thumbPath);
await fs.unlink(thumbPath);
} catch (error) {
// Only log if it's not a "file not found" error
if (error.code !== 'ENOENT') {
console.error('Error deleting thumbnail:', error);
}
}
}
// Delete pre-generated watermark
if (photo.thumbnail_path) {
await storage.delete(photo.thumbnail_path).catch(() => {});
}
if (photo.hero_path) {
await storage.delete(photo.hero_path).catch(() => {});
}
if (photo.watermark_path) {
await watermarkGeneratorService.deleteForPhoto(photo.id);
}
@@ -774,7 +794,18 @@ router.post('/:eventId/photos/bulk-delete', adminAuth, requirePermission('photos
.whereIn('id', photoIds)
.where('event_id', eventId)
.delete();
// Webhook (#327): one photo.deleted per row in the bulk batch.
try {
const webhookService = require('../services/webhookService');
for (const photo of photos) {
await webhookService.fire('photo.deleted', {
event: { id: parseInt(eventId, 10), slug: event?.slug, event_name: event?.event_name },
photo: { id: photo.id, filename: photo.filename },
});
}
} catch (e) { /* non-fatal */ }
// Log activity
await logActivity('photos_bulk_deleted',
{ count: photos.length, eventName: event.event_name },
@@ -866,18 +897,33 @@ router.get('/:eventId/photos/:photoId/download', adminAuth, requirePermission('p
return res.status(404).json({ error: 'Photo not found' });
}
const { resolvePhotoFilePath } = require('../services/photoResolver');
const { resolvePhotoFilePath, resolvePhotoStorageKey } = require('../services/photoResolver');
const event = await db('events').where('id', eventId).first();
const storage = getStorage();
const storageKey = resolvePhotoStorageKey(event, photo);
if (storageKey) {
const stat = await storage.stat(storageKey);
if (!stat) {
return res.status(404).json({ error: 'Photo file not found' });
}
res.set({
'Content-Type': photo.mime_type || 'application/octet-stream',
'Content-Length': stat.size,
'Content-Disposition': `attachment; filename="${photo.filename}"`,
});
const stream = await storage.get(storageKey);
stream.pipe(res);
return;
}
// External-mode photos still live on local disk.
const filePath = resolvePhotoFilePath(event, photo);
// Check if file exists
try {
await fs.access(filePath);
} catch (error) {
return res.status(404).json({ error: 'Photo file not found' });
}
// Send file
res.download(filePath, photo.filename);
} catch (error) {
console.error('Error downloading photo:', error);
@@ -1033,23 +1079,33 @@ router.get('/:eventId/photo/:photoId', adminAuth, requirePermission('photos.view
return res.status(404).json({ error: 'Photo not found' });
}
const { resolvePhotoFilePath } = require('../services/photoResolver');
const { resolvePhotoFilePath, resolvePhotoStorageKey } = require('../services/photoResolver');
const event = await db('events').where('id', eventId).first();
const storageKey = resolvePhotoStorageKey(event, photo);
res.setHeader('Content-Type', `image/${path.extname(photo.filename).slice(1)}`);
res.setHeader('Cache-Control', 'private, max-age=3600');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
if (storageKey) {
const storage = getStorage();
const stat = await storage.stat(storageKey);
if (!stat) {
return res.status(404).json({ error: 'Photo file not found' });
}
res.setHeader('Content-Length', stat.size);
const stream = await storage.get(storageKey);
stream.pipe(res);
return;
}
// External-mode photos still live on local disk.
const filePath = resolvePhotoFilePath(event, photo);
// Check if file exists
try {
await fs.access(filePath);
} catch (error) {
return res.status(404).json({ error: 'Photo file not found' });
}
// Set appropriate headers
res.setHeader('Content-Type', `image/${path.extname(photo.filename).slice(1)}`);
res.setHeader('Cache-Control', 'private, max-age=3600');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
// Send file (sendFile requires absolute path)
res.sendFile(path.resolve(filePath));
} catch (error) {
console.error('Error serving photo:', error);
@@ -1073,22 +1129,24 @@ router.get('/:eventId/thumbnail/:photoId', adminAuth, requirePermission('photos.
// Ensure thumbnail exists and is valid, regenerate if needed
const thumbnailPath = await ensureThumbnail(photo);
if (!thumbnailPath) {
console.error(`Failed to generate thumbnail for photo ${photoId}`);
return res.status(404).json({ error: 'Thumbnail generation failed' });
}
const storagePath = getStoragePath();
const filePath = path.join(storagePath, thumbnailPath);
// Set appropriate headers
res.setHeader('Content-Type', 'image/jpeg'); // Thumbnails are always JPEG
res.setHeader('Cache-Control', 'private, max-age=3600');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
// Send file (sendFile requires absolute path)
res.sendFile(path.resolve(filePath));
const storage = getStorage();
const stat = await storage.stat(thumbnailPath);
if (!stat) {
return res.status(404).json({ error: 'Thumbnail not found' });
}
res.setHeader('Content-Length', stat.size);
const stream = await storage.get(thumbnailPath);
stream.pipe(res);
} catch (error) {
console.error('Error serving thumbnail:', error);
console.error('Photo ID:', req.params.photoId);
backend/src/routes/gallery.js
+81 -47
View File
@@ -15,6 +15,7 @@ const { handleAsync } = require('../utils/routeHelpers');
const { NotFoundError } = require('../utils/errors');
const { ensureThumbnail, ensureHeroImage } = require('../services/imageProcessor');
const downloadZipService = require('../services/downloadZipService');
const { getStorage } = require('../services/storage');
const fs = require('fs');
// Get storage path from environment or default
@@ -629,10 +630,39 @@ router.get('/:slug/download-all', verifyGalleryAccess, async (req, res) => {
// Try to serve pre-generated zip (instant download with Content-Length)
const zipInfo = await downloadZipService.getZipInfo(req.event.id);
if (zipInfo) {
const storage = getStorage();
// Per-event presigned-URL fast path (#328 follow-up). Conditions:
// 1. STORAGE_BACKEND=s3 (presigned URLs are S3-only)
// 2. event.allow_presigned_download is true (admin opted in)
// 3. Watermarking is OFF for this event — presigned URLs bypass the
// backend, which means no watermark on bytes leaving S3.
// Falls through to streaming on any condition mismatch.
const wantsPresigned = req.event.allow_presigned_download === true || req.event.allow_presigned_download === 1;
const watermarkOnEvent = req.event.watermark_downloads === true || req.event.watermark_downloads === 1;
if (wantsPresigned && storage.kind() === 's3' && !watermarkOnEvent) {
try {
const url = await storage.signedUrl(zipInfo.key, 300); // 5 min
db('access_logs').insert({
event_id: req.event.id,
ip_address: req.ip,
user_agent: req.headers['user-agent'],
action: 'download_all_presigned'
}).catch(() => {});
res.redirect(302, url);
return;
} catch (err) {
logger.warn('presigned download-all failed, falling back to stream', {
eventId: req.event.id,
error: err.message,
});
}
}
res.setHeader('Content-Type', 'application/zip');
res.setHeader('Content-Length', zipInfo.size);
res.setHeader('Content-Disposition', `attachment; filename="${req.event.slug}.zip"`);
const stream = fs.createReadStream(zipInfo.path);
const stream = await storage.get(zipInfo.key);
stream.pipe(res);
// Log bulk download
@@ -686,46 +716,49 @@ router.get('/:slug/download-all', verifyGalleryAccess, async (req, res) => {
text: req.event.watermark_text || watermarkSettings?.text || 'Protected'
} : null;
// Add photos to archive
// Add photos to archive — managed photos via storage backend, external via local path.
const { resolvePhotoStorageKey } = require('../services/photoResolver');
const storage = getStorage();
for (const photo of photos) {
let filePath;
try {
filePath = resolvePhotoFilePath(req.event, photo);
} catch (resolveError) {
logger.warn('Skipping photo in bulk download due to unresolved path', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: resolveError.message,
});
continue;
}
// Determine the file name in the archive
const storageKey = resolvePhotoStorageKey(req.event, photo);
let archiveName;
if (hasMultipleTypes) {
// Use photo type as folder
const folderName = photo.type === 'individual' ? 'Individual Photos' : 'Collages';
archiveName = path.join(folderName, photo.filename);
} else {
// No folders, just the filename
archiveName = photo.filename;
}
if (shouldApplyWatermark && effectiveSettings) {
try {
const watermarkedBuffer = await watermarkService.applyWatermark(filePath, effectiveSettings);
try {
if (shouldApplyWatermark && effectiveSettings) {
// Watermark service operates on a local path. For managed photos in
// S3 mode, materialize a tmp local copy first.
const { withLocalCopy } = require('../services/imageProcessor');
const sourceForWatermark = storageKey
? null
: resolvePhotoFilePath(req.event, photo);
const watermarkedBuffer = storageKey
? await withLocalCopy(storageKey, (localPath) =>
watermarkService.applyWatermark(localPath, effectiveSettings)
)
: await watermarkService.applyWatermark(sourceForWatermark, effectiveSettings);
archive.append(watermarkedBuffer, { name: archiveName });
} catch (watermarkError) {
logger.warn('Failed to watermark photo for bulk download, skipping original to avoid leak', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: watermarkError.message,
});
} else if (storageKey) {
const stream = await storage.get(storageKey);
archive.append(stream, { name: archiveName });
} else {
const filePath = resolvePhotoFilePath(req.event, photo);
archive.file(filePath, { name: archiveName });
}
} else {
archive.file(filePath, { name: archiveName });
} catch (err) {
logger.warn('Skipping photo in bulk download due to error', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: err.message,
});
}
}
@@ -811,31 +844,32 @@ router.post('/:slug/download-selected', verifyGalleryAccess, async (req, res) =>
text: req.event.watermark_text || watermarkSettings?.text || 'Protected'
} : null;
const { resolvePhotoStorageKey: resolveSelectedKey } = require('../services/photoResolver');
const { withLocalCopy: withSelectedLocalCopy } = require('../services/imageProcessor');
const selectedStorage = getStorage();
for (const photo of photos) {
const name = photo.filename || `photo-${photo.id}.jpg`;
const storageKey = resolveSelectedKey(req.event, photo);
try {
const filePath = resolvePhotoFilePath(req.event, photo);
const name = photo.filename || `photo-${photo.id}.jpg`;
if (shouldApplyWatermark && effectiveSettings) {
try {
const watermarkedBuffer = await watermarkService.applyWatermark(filePath, effectiveSettings);
archive.append(watermarkedBuffer, { name });
} catch (watermarkError) {
logger.warn('Failed to watermark selected photo, skipping original to avoid leak', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: watermarkError.message,
});
}
const buf = storageKey
? await withSelectedLocalCopy(storageKey, (lp) =>
watermarkService.applyWatermark(lp, effectiveSettings)
)
: await watermarkService.applyWatermark(resolvePhotoFilePath(req.event, photo), effectiveSettings);
archive.append(buf, { name });
} else if (storageKey) {
const stream = await selectedStorage.get(storageKey);
archive.append(stream, { name });
} else {
archive.file(filePath, { name });
archive.file(resolvePhotoFilePath(req.event, photo), { name });
}
} catch (resolveError) {
logger.warn('Skipping selected photo due to unresolved path', {
} catch (err) {
logger.warn('Skipping selected photo due to error', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: resolveError.message,
error: err.message,
});
}
}
backend/src/routes/protectedImages.js
+30 -18
View File
@@ -1,11 +1,12 @@
const express = require('express');
const path = require('path');
const { db } = require('../database/db');
const { formatBoolean } = require('../utils/dbCompat');
const { verifyGalleryAccess } = require('../middleware/gallery');
const watermarkService = require('../services/watermarkService');
const secureImageService = require('../services/secureImageService');
const { getStoragePath } = require('../config/storage');
const { getStorage } = require('../services/storage');
const { resolvePhotoStorageKey, resolvePhotoFilePath } = require('../services/photoResolver');
const { withLocalCopy } = require('../services/imageProcessor');
const crypto = require('crypto');
const router = express.Router();
@@ -98,11 +99,11 @@ router.get('/:slug/photo/:photoId/view', verifyGalleryAccess, async (req, res) =
fragmentImage: eventProtectionLevel === 'maximum'
};
// Build full path to photo
const photoPath = path.join(getStoragePath(), 'events/active', req.event.slug, photo.path);
// Resolve photo location through the storage backend (managed) or local
// disk (external reference mode).
const storageKey = resolvePhotoStorageKey(req.event, photo);
const storage = getStorage();
// For basic/standard protection without special features, serve original file
// This avoids unnecessary recompression
const needsProcessing = eventProtectionLevel === 'enhanced' ||
eventProtectionLevel === 'maximum' ||
protectionSettings.addFingerprint;
@@ -110,15 +111,25 @@ router.get('/:slug/photo/:photoId/view', verifyGalleryAccess, async (req, res) =
let finalImage;
if (!needsProcessing) {
// Serve original file without processing
const fs = require('fs').promises;
finalImage = await fs.readFile(photoPath);
// Serve original bytes via the storage backend (or local disk for external).
if (storageKey) {
const stream = await storage.get(storageKey);
const chunks = [];
for await (const chunk of stream) chunks.push(chunk);
finalImage = Buffer.concat(chunks);
} else {
const fs = require('fs').promises;
finalImage = await fs.readFile(resolvePhotoFilePath(req.event, photo));
}
} else {
// Process image with protection measures
const processedImage = await secureImageService.processProtectedImage(photoPath, protectionSettings);
// secureImageService.processProtectedImage operates on a local path.
// Materialize a tmp local copy in S3 mode, then run processing.
const runProcessing = (lp) => secureImageService.processProtectedImage(lp, protectionSettings);
const processedImage = storageKey
? await withLocalCopy(storageKey, runProcessing)
: await runProcessing(resolvePhotoFilePath(req.event, photo));
if (processedImage.type === 'fragmented') {
// Return fragmented image data for canvas reconstruction
return res.json({
type: 'fragmented',
fragments: processedImage.fragments.map(f => ({
@@ -273,12 +284,13 @@ router.get('/:slug/photo/:photoId/signed/:token', async (req, res) => {
// Get watermark settings
const watermarkSettings = await watermarkService.getWatermarkSettings();
// Build full path to photo
const photoPath = path.join(getStoragePath(), 'events/active', event.slug, photo.path);
// Apply watermark if enabled
const imageBuffer = await watermarkService.applyWatermark(photoPath, watermarkSettings);
// Apply watermark — managed photos are sourced via the storage backend
// (S3 mode materializes a tmp local copy via withLocalCopy).
const storageKey = resolvePhotoStorageKey(event, photo);
const imageBuffer = storageKey
? await withLocalCopy(storageKey, (lp) => watermarkService.applyWatermark(lp, watermarkSettings))
: await watermarkService.applyWatermark(resolvePhotoFilePath(event, photo), watermarkSettings);
// Set appropriate headers
res.set({
backend/src/routes/secureImages.js
+44 -33
View File
@@ -5,7 +5,9 @@ const secureImageService = require('../services/secureImageService');
const secureImageMiddleware = require('../middleware/secureImageMiddleware');
const logger = require('../utils/logger');
const { formatBoolean } = require('../utils/dbCompat');
const { resolvePhotoFilePath } = require('../services/photoResolver');
const { resolvePhotoFilePath, resolvePhotoStorageKey } = require('../services/photoResolver');
const { withLocalCopy } = require('../services/imageProcessor');
const { getStorage } = require('../services/storage');
const router = express.Router();
@@ -139,18 +141,10 @@ router.get('/:slug/secure/:photoId/:token',
return res.status(404).json({ error: 'Photo not found' });
}
let filePath;
try {
filePath = resolvePhotoFilePath(req.event, photo);
} catch (resolveError) {
logger.error('Failed to resolve photo path for secure token generation', {
slug: req.params.slug,
photoId,
eventId: req.event.id,
error: resolveError.message,
});
return res.status(404).json({ error: 'Photo file not found' });
}
// Resolve photo through storage backend (managed) or fall back to local
// path (external reference mode). secureImageService needs a local file,
// so we materialize a tmp copy via withLocalCopy in S3 mode.
const storageKey = resolvePhotoStorageKey(event, photo);
// Get protection settings for this event
const protectionSettings = {
@@ -160,11 +154,21 @@ router.get('/:slug/secure/:photoId/:token',
fragmentImage: event.use_canvas_rendering === true && fragment !== undefined
};
// Process image with protection measures
const processedImage = await secureImageService.processProtectedImage(
filePath,
protectionSettings
);
let processedImage;
try {
const runProcessing = (lp) => secureImageService.processProtectedImage(lp, protectionSettings);
processedImage = storageKey
? await withLocalCopy(storageKey, runProcessing)
: await runProcessing(resolvePhotoFilePath(event, photo));
} catch (resolveError) {
logger.error('Failed to process secure image', {
slug: req.params.slug,
photoId,
eventId: event.id,
error: resolveError.message,
});
return res.status(404).json({ error: 'Photo file not found' });
}
// Handle fragmented images
if (processedImage.type === 'fragmented') {
@@ -292,11 +296,30 @@ router.get('/:slug/secure-download/:photoId/:token',
return res.status(404).json({ error: 'Photo not found' });
}
let filePath;
// Resolve photo through storage backend (managed) or local disk (external).
const storageKey = resolvePhotoStorageKey(req.event, photo);
const watermarkService = require('../services/watermarkService');
const watermarkSettings = await watermarkService.getWatermarkSettings();
const wantsWatermark = watermarkSettings && watermarkSettings.enabled;
let fileBuffer;
try {
filePath = resolvePhotoFilePath(req.event, photo);
if (wantsWatermark) {
fileBuffer = storageKey
? await withLocalCopy(storageKey, (lp) => watermarkService.applyWatermark(lp, watermarkSettings))
: await watermarkService.applyWatermark(resolvePhotoFilePath(req.event, photo), watermarkSettings);
} else if (storageKey) {
const stream = await getStorage().get(storageKey);
const chunks = [];
for await (const chunk of stream) chunks.push(chunk);
fileBuffer = Buffer.concat(chunks);
} else {
const fs = require('fs').promises;
fileBuffer = await fs.readFile(resolvePhotoFilePath(req.event, photo));
}
} catch (resolveError) {
logger.error('Failed to resolve photo path for secure download', {
logger.error('Failed to fetch photo for secure download', {
slug: req.params.slug,
photoId,
eventId: req.event.id,
@@ -305,18 +328,6 @@ router.get('/:slug/secure-download/:photoId/:token',
return res.status(404).json({ error: 'Photo file not found' });
}
// Apply watermark if enabled
const watermarkService = require('../services/watermarkService');
const watermarkSettings = await watermarkService.getWatermarkSettings();
let fileBuffer;
if (watermarkSettings && watermarkSettings.enabled) {
fileBuffer = await watermarkService.applyWatermark(filePath, watermarkSettings);
} else {
const fs = require('fs').promises;
fileBuffer = await fs.readFile(filePath);
}
// Update download count
await db('photos').where('id', photoId).increment('download_count', 1);
backend/src/routes/v1/events.js
+45 -17
View File
@@ -182,6 +182,18 @@ router.post(
type: 'admin', id: req.admin.id, name: req.admin.username
});
// Webhook lifecycle (#327). v1 events are not draft-aware, so they're
// both created AND published in the same call.
try {
const webhookService = require('../../services/webhookService');
await webhookService.fire('event.created', {
event: { id, slug, event_name, event_type, event_date, share_url: shareUrl },
});
await webhookService.fire('event.published', {
event: { id, slug, event_name, share_url: shareUrl },
});
} catch (e) { /* non-fatal */ }
res.status(201).json({ id, slug, share_url: shareUrl, share_token: shareToken });
} catch (error) {
logger.error('v1 POST /events failed', { error: error.message, stack: error.stack });
@@ -346,33 +358,39 @@ router.post(
const event = await db('events').where({ id: req.params.id }).first();
if (!event) return res.status(404).json({ error: 'Event not found' });
const finalDir = path.join(getStoragePath(), 'events/active', event.slug);
await fs.mkdir(finalDir, { recursive: true });
const ext = path.extname(req.file.originalname);
const finalName = `${Date.now()}_${crypto.randomBytes(4).toString('hex')}${ext}`;
const finalPath = path.join(finalDir, finalName);
await fs.rename(tempPath, finalPath);
tempPath = null;
// photo.path is stored relative to events/active so resolvePhotoStorageKey
// can rebuild the full key on read. Same shape as adminPhotos uploads.
const relPath = path.posix.join(event.slug, finalName);
const finalKey = path.posix.join('events/active', relPath);
const stat = fsSync.statSync(finalPath);
const relPath = path.relative(path.join(getStoragePath(), 'events/active'), finalPath);
const stat = fsSync.statSync(tempPath);
// Read sharp metadata + generate thumbnail FROM the local temp file
// before uploading the original through the storage backend. (Same
// ordering as adminPhotos.js so sharp/ffmpeg always have a real fs path.)
let width = null;
let height = null;
try {
const meta = await sharp(tempPath).metadata();
width = meta.width || null;
height = meta.height || null;
} catch { /* non-fatal */ }
let thumbRel = null;
try {
const thumbPath = await generateThumbnail(finalPath);
thumbRel = path.relative(getStoragePath(), thumbPath);
thumbRel = await generateThumbnail(tempPath);
} catch (err) {
logger.warn('v1 thumbnail generation failed', { err: err.message });
}
// Detect image dimensions for masonry layouts.
let width = null;
let height = null;
try {
const meta = await sharp(finalPath).metadata();
width = meta.width || null;
height = meta.height || null;
} catch { /* non-fatal */ }
// Upload the original via the storage backend (local fs OR S3),
// then drop the multer temp file.
const { getStorage } = require('../../services/storage');
await getStorage().putFromFile(finalKey, tempPath, { contentType: req.file.mimetype });
await fs.unlink(tempPath).catch(() => {});
tempPath = null;
const insertResult = await db('photos').insert({
event_id: event.id,
@@ -394,6 +412,16 @@ router.post(
type: 'admin', id: req.admin.id, name: req.admin.username
});
// Webhook (#327): one event per uploaded photo so receivers get a
// 1:1 stream they can react to.
try {
const webhookService = require('../../services/webhookService');
await webhookService.fire('photo.uploaded', {
event: { id: event.id, slug: event.slug, event_name: event.event_name },
photo: { id, filename: finalName, original_filename: req.file.originalname, size_bytes: stat.size, width, height },
});
} catch (e) { /* non-fatal */ }
res.status(201).json({ id, filename: finalName, path: relPath, thumbnail_path: thumbRel, size_bytes: stat.size });
} catch (error) {
logger.error('v1 POST /events/:id/photos failed', { error: error.message });