paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: native S3 storage backend (#328) + presigned download follow-up

Browse Source 
Lets PicPeak write photos, thumbnails, hero images, watermarks, and
archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2,
Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local
filesystem. Selected via STORAGE_BACKEND=local|s3.

Architecture
- backend/src/services/storage/StorageBackend.js — abstract interface
  (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/
  getToFile) — typedef-only, documents the contract.
- LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path
  traversal protection, list-as-walker.
- S3StorageBackend.js — thin wrapper around the existing
  S3StorageAdapter (used by backupService) mapping it onto the canonical
  interface; supports optional STORAGE_S3_PREFIX namespace.
- index.js — factory selected by STORAGE_BACKEND with startup ping
  (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast
  before the first request.

Consumer refactors (~12 services + routes), each parametrized over the
abstraction:
- imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through
  storage.put; expose withLocalCopy() helper for S3-mode regeneration
  paths that need a local file for sharp/ffmpeg.
- archiveService / downloadZipService — finalize zip in tmp dir, then
  storage.putFromFile. Atomic-rename pattern preserved on local; S3
  emulates via copy + delete (worker prunes orphaned .tmp.* on startup).
- photoProcessor / photoReplacementService / adminPhotos upload+delete /
  routes/v1/events.js POST /events/:id/photos / routes/events.js — every
  upload path now goes storage.putFromFile(temp) → unlink temp.
- gallery.js bulk-download (cached + on-the-fly + selected) — managed
  photos via storage.get, external-mode unchanged.
- protectedImages / secureImages / photoResolver — read via
  storage.get; resolvePhotoStorageKey returns the canonical key.
- watermarkService / watermarkGeneratorService — persistent watermarks
  via storage.put.
- fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3
  (chokidar can't watch S3); auto-import lands via the S3 prefix walker
  introduced in the follow-up commit.
- expirationChecker — small touch (event.expired webhook fire from #327
  shipping in the next commit).

Migration tooling
- backend/scripts/migrate-storage.js — one-shot --dry-run capable script
  that walks photos.path, thumbnail_path, hero_path, watermark_path and
  events.archive_path/download_zip_path; streams local → S3; sha256
  size-match skip for idempotent re-run; failures CSV.

Presigned-URL "Download All" (#328 follow-up shipped in this commit)
- routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download
  + downloads enabled + watermark NOT enabled, /download-all returns a
  302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface
  ships in the next commit's UI.

Tests
- backend/__tests__/integration/storageBackend.test.js — parametrized
  contract suite running against BOTH LocalFs AND MinIO (18 tests, both
  backends — 36 cases total).
- backend/__tests__/integration/imageProcessor.storage.test.js — same
  parametrized pattern for the image processor (10 tests × 2 backends).
- backend/__tests__/integration/backup-s3.test.js — bootstrap fix:
  drop the redundant initDb() (001_init handles it) and remove
  schema-drift in configureS3Backup (app_settings has no created_at
  anymore and the unique constraint is on setting_key alone, not
  composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift).
- backend/src/services/photoResolver.js — mixed-source events (reference
  mode with managed-uploaded photos) now fall back to managed when
  external_relpath is missing instead of throwing.
- tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that
  auto-skips against local backend; full upload → serve → delete
  round-trip when run against an S3-mode backend.

Server wiring (server.js)
- initStorage() called after database init, before rate limiters.
- This commit's diff also includes the webhook delivery worker startup
  and the S3 auto-importer startup. Those features ship in the next two
  commits — co-located here for one bisectable diff per file.

Docs + ops
- README §"Storage Backends" — capability matrix, switching playbook,
  IAM policy snippet, MinIO/R2/B2 examples.
- README §"Webhooks" — also added here (full diff bundled).
- .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT
  documented; WEBHOOK_* added in the same diff.
- .gitignore — re-anchor the existing `storage/` rule to `/storage/`
  so backend/src/services/storage/ (the new abstraction code) is
  trackable. The runtime ./storage/ data dir stays ignored.

Out of scope for v1 (per the issue): presigned URLs for individual
photo display (always streamed for protection middleware), CDN
integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket
per-event.
This commit is contained in:
Paul Nothaft
2026-04-28 10:06:36 +02:00
parent 3d4ae4d7e9
commit 1b717ce5ed
29 changed files with 2365 additions and 714 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/adminPhotos.js
+158 -100
View File
@@ -17,6 +17,7 @@ const watermarkGeneratorService = require('../services/watermarkGeneratorService
const downloadZipService = require('../services/downloadZipService');
const { findReplacementCandidate, replacePhoto } = require('../services/photoReplacementService');
const { requireEventOwnership } = require('../middleware/ownership');
const { getStorage } = require('../services/storage');
const router = express.Router();
// Get storage path from environment or default
@@ -243,9 +244,9 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
categoryName = 'collages';
}
// Create final destination directory
const finalDestPath = path.join(getStoragePath(), 'events/active', event.slug);
await fs.mkdir(finalDestPath, { recursive: true });
// Final destination key prefix under the storage backend (no local mkdir
// needed — LocalFsStorage creates the parent dir on put, S3 has no dirs).
const finalDestPathRel = path.posix.join('events/active', event.slug);
const uploadedPhotos = [];
const replacedPhotos = [];
@@ -330,10 +331,11 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
extension
);
// Calculate final path
const finalPath = path.join(finalDestPath, newFilename);
const storagePath = getStoragePath();
const relativePath = path.relative(path.join(storagePath, 'events/active'), finalPath);
// Storage key: events/active/{slug}/{newFilename}
const finalKey = path.posix.join(finalDestPathRel, newFilename);
// photo.path is stored relative to events/active so resolvePhotoStorageKey
// can rebuild the full key on read.
const relativePath = path.posix.join(event.slug, newFilename);
// Extract capture date from EXIF metadata
let capturedAt = null;
@@ -365,10 +367,10 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
batchPhotos.push(photoData);
// Store move operation for later
// Store upload operation for later (after DB commit)
fileRenameOperations.push({
tempPath: tempPath,
finalPath: finalPath,
finalKey: finalKey,
filename: newFilename,
photoData: photoData
});
@@ -390,34 +392,26 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
await trx.commit();
console.log(`Successfully committed batch of ${batchPhotos.length} photos`);
// Now move files from temp to final location after successful commit
// Now upload files from temp into the storage backend after successful commit
const storage = getStorage();
for (let idx = 0; idx < fileRenameOperations.length; idx++) {
const operation = fileRenameOperations[idx];
try {
// Move the file from temp to final location
await fs.rename(operation.tempPath, operation.finalPath);
console.log(`Moved file from ${operation.tempPath} to ${operation.finalPath}`);
// Verify the file was moved successfully
const finalStats = await fs.stat(operation.finalPath);
if (finalStats.size !== operation.photoData.size_bytes) {
throw new Error(`File size mismatch after move: expected ${operation.photoData.size_bytes}, got ${finalStats.size}`);
}
// Generate thumbnail and extract metadata
// Process source-dependent steps (sharp/ffmpeg) FIRST while the
// tmp file is still on local disk, then upload the original and
// unlink the tmp.
const photoId = insertedIds[idx]?.id || insertedIds[idx];
const isVideoFile = isVideoMimeType(operation.photoData.mime_type);
let thumbnailPath = null;
try {
if (isVideoFile) {
// Process video: extract metadata and generate thumbnail
const thumbnailDir = path.join(getStoragePath(), 'thumbnails');
await fs.mkdir(thumbnailDir, { recursive: true });
const videoThumbnailPath = path.join(thumbnailDir, `thumb_${operation.filename.replace(/\.[^.]+$/, '.jpg')}`);
const result = await processUploadedVideo(operation.finalPath, videoThumbnailPath);
thumbnailPath = path.relative(getStoragePath(), videoThumbnailPath);
const videoThumbnailKey = path.posix.join(
'thumbnails',
`thumb_${operation.filename.replace(/\.[^.]+$/, '.jpg')}`
);
const result = await processUploadedVideo(operation.tempPath, videoThumbnailKey);
thumbnailPath = result.thumbnailKey;
if (photoId && result.metadata) {
await db('photos')
@@ -432,7 +426,7 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
});
}
} else {
thumbnailPath = await generateThumbnail(operation.finalPath);
thumbnailPath = await generateThumbnail(operation.tempPath);
// Update the database with thumbnail path and image dimensions
if (photoId) {
@@ -441,7 +435,7 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
try {
const sharp = require('sharp');
const metadata = await sharp(operation.finalPath).metadata();
const metadata = await sharp(operation.tempPath).metadata();
if (metadata.width && metadata.height) {
updateData.width = metadata.width;
updateData.height = metadata.height;
@@ -461,12 +455,40 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
console.error(`Thumbnail/metadata processing failed for ${operation.filename}:`, thumbError.message);
}
// Upload the original through the storage backend, then drop the
// local tmp file. We do this AFTER thumbnail/metadata processing
// so sharp/ffmpeg still have a local source to work from.
await storage.putFromFile(operation.finalKey, operation.tempPath, {
contentType: operation.photoData.mime_type,
});
await fs.unlink(operation.tempPath).catch(() => {});
// Sanity check: round-trip the size we just wrote.
const stat = await storage.stat(operation.finalKey);
if (!stat || stat.size !== operation.photoData.size_bytes) {
throw new Error(`Size mismatch after upload: expected ${operation.photoData.size_bytes}, got ${stat ? stat.size : 'null'}`);
}
// Queue watermark generation in background (non-blocking, images only)
if (photoId && !isVideoFile) {
watermarkGeneratorService.generateForPhoto(photoId)
.catch(err => console.warn(`Watermark generation queued failed for photo ${photoId}:`, err.message));
}
// Webhook (#327): per-photo upload event.
try {
const webhookService = require('../services/webhookService');
await webhookService.fire('photo.uploaded', {
event: { id: parseInt(eventId, 10), slug: event.slug, event_name: event.event_name },
photo: {
id: insertedIds[idx]?.id || insertedIds[idx],
filename: operation.filename,
original_filename: operation.photoData.original_filename,
size_bytes: operation.photoData.size_bytes,
},
});
} catch (e) { /* non-fatal */ }
// Add to successful uploads
uploadedPhotos.push({
id: insertedIds[idx]?.id || insertedIds[idx],
@@ -475,10 +497,10 @@ router.post('/:eventId/upload', adminAuth, requirePermission('photos.upload'), r
category_id: operation.photoData.category_id
});
} catch (moveError) {
console.error(`Failed to move file ${operation.tempPath} to ${operation.finalPath}:`, moveError);
errors.push({
filename: operation.filename,
error: `File move failed: ${moveError.message}`
console.error(`Failed to upload ${operation.tempPath} → ${operation.finalKey}:`, moveError);
errors.push({
filename: operation.filename,
error: `File upload failed: ${moveError.message}`
});
// Try to clean up the database entry if file move failed
@@ -604,30 +626,30 @@ router.delete('/:eventId/photos/:photoId', adminAuth, requirePermission('photos.
return res.status(404).json({ error: 'Photo not found' });
}
// Delete physical files
const storagePath = getStoragePath();
const photoPath = path.join(storagePath, 'events/active', photo.path);
// Delete original + thumbnail through the storage backend.
const storage = getStorage();
const { resolvePhotoStorageKey } = require('../services/photoResolver');
const event = await db('events').where({ id: eventId }).first();
try {
await fs.unlink(photoPath);
const originalKey = resolvePhotoStorageKey(event, photo);
if (originalKey) await storage.delete(originalKey);
} catch (error) {
console.error('Error deleting photo file:', error);
}
// Delete thumbnail if exists
// photo.thumbnail_path is stored as the canonical storage key
// (e.g. "thumbnails/thumb_foo.jpg"), so pass it through verbatim.
if (photo.thumbnail_path) {
const thumbPath = path.join(storagePath, 'events/active', photo.thumbnail_path);
try {
// Check if file exists before attempting to delete
await fs.access(thumbPath);
await fs.unlink(thumbPath);
await storage.delete(photo.thumbnail_path);
} catch (error) {
// Only log if it's not a "file not found" error
if (error.code !== 'ENOENT') {
console.error('Error deleting thumbnail:', error);
}
console.error('Error deleting thumbnail:', error);
}
}
if (photo.hero_path) {
await storage.delete(photo.hero_path).catch(() => {});
}
// Delete pre-generated watermark if exists
if (photo.watermark_path) {
@@ -636,15 +658,23 @@ router.delete('/:eventId/photos/:photoId', adminAuth, requirePermission('photos.
// Remove from database
await db('photos').where({ id: photoId }).delete();
// Log activity
const event = await db('events').where({ id: eventId }).first();
// Log activity (event was fetched above for storage key resolution)
await logActivity('photo_deleted',
{ filename: photo.filename, eventName: event.event_name },
eventId,
{ type: 'admin', id: req.admin.id, name: req.admin.username }
);
// Webhook (#327): single-photo delete.
try {
const webhookService = require('../services/webhookService');
await webhookService.fire('photo.deleted', {
event: { id: parseInt(eventId, 10), slug: event?.slug, event_name: event?.event_name },
photo: { id: parseInt(photoId, 10), filename: photo.filename },
});
} catch (e) { /* non-fatal */ }
downloadZipService.invalidate(parseInt(eventId));
res.json({ message: 'Photo deleted successfully' });
} catch (error) {
@@ -735,35 +765,25 @@ router.post('/:eventId/photos/bulk-delete', adminAuth, requirePermission('photos
return res.status(404).json({ error: 'No photos found' });
}
// Delete physical files
const storagePath = getStoragePath();
// Delete original + thumbnail + hero through the storage backend.
const storage = getStorage();
const event = await db('events').where({ id: eventId }).first();
const { resolvePhotoStorageKey } = require('../services/photoResolver');
for (const photo of photos) {
// Delete photo file
const photoPath = path.join(storagePath, 'events/active', photo.path);
try {
await fs.unlink(photoPath);
const originalKey = resolvePhotoStorageKey(event, photo);
if (originalKey) await storage.delete(originalKey);
} catch (error) {
console.error('Error deleting photo file:', error);
}
// Delete thumbnail
if (photo.thumbnail_path) {
const thumbPath = path.join(storagePath, photo.thumbnail_path);
try {
// Check if file exists before attempting to delete
await fs.access(thumbPath);
await fs.unlink(thumbPath);
} catch (error) {
// Only log if it's not a "file not found" error
if (error.code !== 'ENOENT') {
console.error('Error deleting thumbnail:', error);
}
}
}
// Delete pre-generated watermark
if (photo.thumbnail_path) {
await storage.delete(photo.thumbnail_path).catch(() => {});
}
if (photo.hero_path) {
await storage.delete(photo.hero_path).catch(() => {});
}
if (photo.watermark_path) {
await watermarkGeneratorService.deleteForPhoto(photo.id);
}
@@ -774,7 +794,18 @@ router.post('/:eventId/photos/bulk-delete', adminAuth, requirePermission('photos
.whereIn('id', photoIds)
.where('event_id', eventId)
.delete();
// Webhook (#327): one photo.deleted per row in the bulk batch.
try {
const webhookService = require('../services/webhookService');
for (const photo of photos) {
await webhookService.fire('photo.deleted', {
event: { id: parseInt(eventId, 10), slug: event?.slug, event_name: event?.event_name },
photo: { id: photo.id, filename: photo.filename },
});
}
} catch (e) { /* non-fatal */ }
// Log activity
await logActivity('photos_bulk_deleted',
{ count: photos.length, eventName: event.event_name },
@@ -866,18 +897,33 @@ router.get('/:eventId/photos/:photoId/download', adminAuth, requirePermission('p
return res.status(404).json({ error: 'Photo not found' });
}
const { resolvePhotoFilePath } = require('../services/photoResolver');
const { resolvePhotoFilePath, resolvePhotoStorageKey } = require('../services/photoResolver');
const event = await db('events').where('id', eventId).first();
const storage = getStorage();
const storageKey = resolvePhotoStorageKey(event, photo);
if (storageKey) {
const stat = await storage.stat(storageKey);
if (!stat) {
return res.status(404).json({ error: 'Photo file not found' });
}
res.set({
'Content-Type': photo.mime_type || 'application/octet-stream',
'Content-Length': stat.size,
'Content-Disposition': `attachment; filename="${photo.filename}"`,
});
const stream = await storage.get(storageKey);
stream.pipe(res);
return;
}
// External-mode photos still live on local disk.
const filePath = resolvePhotoFilePath(event, photo);
// Check if file exists
try {
await fs.access(filePath);
} catch (error) {
return res.status(404).json({ error: 'Photo file not found' });
}
// Send file
res.download(filePath, photo.filename);
} catch (error) {
console.error('Error downloading photo:', error);
@@ -1033,23 +1079,33 @@ router.get('/:eventId/photo/:photoId', adminAuth, requirePermission('photos.view
return res.status(404).json({ error: 'Photo not found' });
}
const { resolvePhotoFilePath } = require('../services/photoResolver');
const { resolvePhotoFilePath, resolvePhotoStorageKey } = require('../services/photoResolver');
const event = await db('events').where('id', eventId).first();
const storageKey = resolvePhotoStorageKey(event, photo);
res.setHeader('Content-Type', `image/${path.extname(photo.filename).slice(1)}`);
res.setHeader('Cache-Control', 'private, max-age=3600');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
if (storageKey) {
const storage = getStorage();
const stat = await storage.stat(storageKey);
if (!stat) {
return res.status(404).json({ error: 'Photo file not found' });
}
res.setHeader('Content-Length', stat.size);
const stream = await storage.get(storageKey);
stream.pipe(res);
return;
}
// External-mode photos still live on local disk.
const filePath = resolvePhotoFilePath(event, photo);
// Check if file exists
try {
await fs.access(filePath);
} catch (error) {
return res.status(404).json({ error: 'Photo file not found' });
}
// Set appropriate headers
res.setHeader('Content-Type', `image/${path.extname(photo.filename).slice(1)}`);
res.setHeader('Cache-Control', 'private, max-age=3600');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
// Send file (sendFile requires absolute path)
res.sendFile(path.resolve(filePath));
} catch (error) {
console.error('Error serving photo:', error);
@@ -1073,22 +1129,24 @@ router.get('/:eventId/thumbnail/:photoId', adminAuth, requirePermission('photos.
// Ensure thumbnail exists and is valid, regenerate if needed
const thumbnailPath = await ensureThumbnail(photo);
if (!thumbnailPath) {
console.error(`Failed to generate thumbnail for photo ${photoId}`);
return res.status(404).json({ error: 'Thumbnail generation failed' });
}
const storagePath = getStoragePath();
const filePath = path.join(storagePath, thumbnailPath);
// Set appropriate headers
res.setHeader('Content-Type', 'image/jpeg'); // Thumbnails are always JPEG
res.setHeader('Cache-Control', 'private, max-age=3600');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
// Send file (sendFile requires absolute path)
res.sendFile(path.resolve(filePath));
const storage = getStorage();
const stat = await storage.stat(thumbnailPath);
if (!stat) {
return res.status(404).json({ error: 'Thumbnail not found' });
}
res.setHeader('Content-Length', stat.size);
const stream = await storage.get(thumbnailPath);
stream.pipe(res);
} catch (error) {
console.error('Error serving thumbnail:', error);
console.error('Photo ID:', req.params.photoId);
backend/src/routes/gallery.js
+81 -47
View File
@@ -15,6 +15,7 @@ const { handleAsync } = require('../utils/routeHelpers');
const { NotFoundError } = require('../utils/errors');
const { ensureThumbnail, ensureHeroImage } = require('../services/imageProcessor');
const downloadZipService = require('../services/downloadZipService');
const { getStorage } = require('../services/storage');
const fs = require('fs');
// Get storage path from environment or default
@@ -629,10 +630,39 @@ router.get('/:slug/download-all', verifyGalleryAccess, async (req, res) => {
// Try to serve pre-generated zip (instant download with Content-Length)
const zipInfo = await downloadZipService.getZipInfo(req.event.id);
if (zipInfo) {
const storage = getStorage();
// Per-event presigned-URL fast path (#328 follow-up). Conditions:
// 1. STORAGE_BACKEND=s3 (presigned URLs are S3-only)
// 2. event.allow_presigned_download is true (admin opted in)
// 3. Watermarking is OFF for this event — presigned URLs bypass the
// backend, which means no watermark on bytes leaving S3.
// Falls through to streaming on any condition mismatch.
const wantsPresigned = req.event.allow_presigned_download === true || req.event.allow_presigned_download === 1;
const watermarkOnEvent = req.event.watermark_downloads === true || req.event.watermark_downloads === 1;
if (wantsPresigned && storage.kind() === 's3' && !watermarkOnEvent) {
try {
const url = await storage.signedUrl(zipInfo.key, 300); // 5 min
db('access_logs').insert({
event_id: req.event.id,
ip_address: req.ip,
user_agent: req.headers['user-agent'],
action: 'download_all_presigned'
}).catch(() => {});
res.redirect(302, url);
return;
} catch (err) {
logger.warn('presigned download-all failed, falling back to stream', {
eventId: req.event.id,
error: err.message,
});
}
}
res.setHeader('Content-Type', 'application/zip');
res.setHeader('Content-Length', zipInfo.size);
res.setHeader('Content-Disposition', `attachment; filename="${req.event.slug}.zip"`);
const stream = fs.createReadStream(zipInfo.path);
const stream = await storage.get(zipInfo.key);
stream.pipe(res);
// Log bulk download
@@ -686,46 +716,49 @@ router.get('/:slug/download-all', verifyGalleryAccess, async (req, res) => {
text: req.event.watermark_text || watermarkSettings?.text || 'Protected'
} : null;
// Add photos to archive
// Add photos to archive — managed photos via storage backend, external via local path.
const { resolvePhotoStorageKey } = require('../services/photoResolver');
const storage = getStorage();
for (const photo of photos) {
let filePath;
try {
filePath = resolvePhotoFilePath(req.event, photo);
} catch (resolveError) {
logger.warn('Skipping photo in bulk download due to unresolved path', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: resolveError.message,
});
continue;
}
// Determine the file name in the archive
const storageKey = resolvePhotoStorageKey(req.event, photo);
let archiveName;
if (hasMultipleTypes) {
// Use photo type as folder
const folderName = photo.type === 'individual' ? 'Individual Photos' : 'Collages';
archiveName = path.join(folderName, photo.filename);
} else {
// No folders, just the filename
archiveName = photo.filename;
}
if (shouldApplyWatermark && effectiveSettings) {
try {
const watermarkedBuffer = await watermarkService.applyWatermark(filePath, effectiveSettings);
try {
if (shouldApplyWatermark && effectiveSettings) {
// Watermark service operates on a local path. For managed photos in
// S3 mode, materialize a tmp local copy first.
const { withLocalCopy } = require('../services/imageProcessor');
const sourceForWatermark = storageKey
? null
: resolvePhotoFilePath(req.event, photo);
const watermarkedBuffer = storageKey
? await withLocalCopy(storageKey, (localPath) =>
watermarkService.applyWatermark(localPath, effectiveSettings)
)
: await watermarkService.applyWatermark(sourceForWatermark, effectiveSettings);
archive.append(watermarkedBuffer, { name: archiveName });
} catch (watermarkError) {
logger.warn('Failed to watermark photo for bulk download, skipping original to avoid leak', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: watermarkError.message,
});
} else if (storageKey) {
const stream = await storage.get(storageKey);
archive.append(stream, { name: archiveName });
} else {
const filePath = resolvePhotoFilePath(req.event, photo);
archive.file(filePath, { name: archiveName });
}
} else {
archive.file(filePath, { name: archiveName });
} catch (err) {
logger.warn('Skipping photo in bulk download due to error', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: err.message,
});
}
}
@@ -811,31 +844,32 @@ router.post('/:slug/download-selected', verifyGalleryAccess, async (req, res) =>
text: req.event.watermark_text || watermarkSettings?.text || 'Protected'
} : null;
const { resolvePhotoStorageKey: resolveSelectedKey } = require('../services/photoResolver');
const { withLocalCopy: withSelectedLocalCopy } = require('../services/imageProcessor');
const selectedStorage = getStorage();
for (const photo of photos) {
const name = photo.filename || `photo-${photo.id}.jpg`;
const storageKey = resolveSelectedKey(req.event, photo);
try {
const filePath = resolvePhotoFilePath(req.event, photo);
const name = photo.filename || `photo-${photo.id}.jpg`;
if (shouldApplyWatermark && effectiveSettings) {
try {
const watermarkedBuffer = await watermarkService.applyWatermark(filePath, effectiveSettings);
archive.append(watermarkedBuffer, { name });
} catch (watermarkError) {
logger.warn('Failed to watermark selected photo, skipping original to avoid leak', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: watermarkError.message,
});
}
const buf = storageKey
? await withSelectedLocalCopy(storageKey, (lp) =>
watermarkService.applyWatermark(lp, effectiveSettings)
)
: await watermarkService.applyWatermark(resolvePhotoFilePath(req.event, photo), effectiveSettings);
archive.append(buf, { name });
} else if (storageKey) {
const stream = await selectedStorage.get(storageKey);
archive.append(stream, { name });
} else {
archive.file(filePath, { name });
archive.file(resolvePhotoFilePath(req.event, photo), { name });
}
} catch (resolveError) {
logger.warn('Skipping selected photo due to unresolved path', {
} catch (err) {
logger.warn('Skipping selected photo due to error', {
slug: req.params.slug,
photoId: photo.id,
eventId: req.event.id,
error: resolveError.message,
error: err.message,
});
}
}
backend/src/routes/protectedImages.js
+30 -18
View File
@@ -1,11 +1,12 @@
const express = require('express');
const path = require('path');
const { db } = require('../database/db');
const { formatBoolean } = require('../utils/dbCompat');
const { verifyGalleryAccess } = require('../middleware/gallery');
const watermarkService = require('../services/watermarkService');
const secureImageService = require('../services/secureImageService');
const { getStoragePath } = require('../config/storage');
const { getStorage } = require('../services/storage');
const { resolvePhotoStorageKey, resolvePhotoFilePath } = require('../services/photoResolver');
const { withLocalCopy } = require('../services/imageProcessor');
const crypto = require('crypto');
const router = express.Router();
@@ -98,11 +99,11 @@ router.get('/:slug/photo/:photoId/view', verifyGalleryAccess, async (req, res) =
fragmentImage: eventProtectionLevel === 'maximum'
};
// Build full path to photo
const photoPath = path.join(getStoragePath(), 'events/active', req.event.slug, photo.path);
// Resolve photo location through the storage backend (managed) or local
// disk (external reference mode).
const storageKey = resolvePhotoStorageKey(req.event, photo);
const storage = getStorage();
// For basic/standard protection without special features, serve original file
// This avoids unnecessary recompression
const needsProcessing = eventProtectionLevel === 'enhanced' ||
eventProtectionLevel === 'maximum' ||
protectionSettings.addFingerprint;
@@ -110,15 +111,25 @@ router.get('/:slug/photo/:photoId/view', verifyGalleryAccess, async (req, res) =
let finalImage;
if (!needsProcessing) {
// Serve original file without processing
const fs = require('fs').promises;
finalImage = await fs.readFile(photoPath);
// Serve original bytes via the storage backend (or local disk for external).
if (storageKey) {
const stream = await storage.get(storageKey);
const chunks = [];
for await (const chunk of stream) chunks.push(chunk);
finalImage = Buffer.concat(chunks);
} else {
const fs = require('fs').promises;
finalImage = await fs.readFile(resolvePhotoFilePath(req.event, photo));
}
} else {
// Process image with protection measures
const processedImage = await secureImageService.processProtectedImage(photoPath, protectionSettings);
// secureImageService.processProtectedImage operates on a local path.
// Materialize a tmp local copy in S3 mode, then run processing.
const runProcessing = (lp) => secureImageService.processProtectedImage(lp, protectionSettings);
const processedImage = storageKey
? await withLocalCopy(storageKey, runProcessing)
: await runProcessing(resolvePhotoFilePath(req.event, photo));
if (processedImage.type === 'fragmented') {
// Return fragmented image data for canvas reconstruction
return res.json({
type: 'fragmented',
fragments: processedImage.fragments.map(f => ({
@@ -273,12 +284,13 @@ router.get('/:slug/photo/:photoId/signed/:token', async (req, res) => {
// Get watermark settings
const watermarkSettings = await watermarkService.getWatermarkSettings();
// Build full path to photo
const photoPath = path.join(getStoragePath(), 'events/active', event.slug, photo.path);
// Apply watermark if enabled
const imageBuffer = await watermarkService.applyWatermark(photoPath, watermarkSettings);
// Apply watermark — managed photos are sourced via the storage backend
// (S3 mode materializes a tmp local copy via withLocalCopy).
const storageKey = resolvePhotoStorageKey(event, photo);
const imageBuffer = storageKey
? await withLocalCopy(storageKey, (lp) => watermarkService.applyWatermark(lp, watermarkSettings))
: await watermarkService.applyWatermark(resolvePhotoFilePath(event, photo), watermarkSettings);
// Set appropriate headers
res.set({
backend/src/routes/secureImages.js
+44 -33
View File
@@ -5,7 +5,9 @@ const secureImageService = require('../services/secureImageService');
const secureImageMiddleware = require('../middleware/secureImageMiddleware');
const logger = require('../utils/logger');
const { formatBoolean } = require('../utils/dbCompat');
const { resolvePhotoFilePath } = require('../services/photoResolver');
const { resolvePhotoFilePath, resolvePhotoStorageKey } = require('../services/photoResolver');
const { withLocalCopy } = require('../services/imageProcessor');
const { getStorage } = require('../services/storage');
const router = express.Router();
@@ -139,18 +141,10 @@ router.get('/:slug/secure/:photoId/:token',
return res.status(404).json({ error: 'Photo not found' });
}
let filePath;
try {
filePath = resolvePhotoFilePath(req.event, photo);
} catch (resolveError) {
logger.error('Failed to resolve photo path for secure token generation', {
slug: req.params.slug,
photoId,
eventId: req.event.id,
error: resolveError.message,
});
return res.status(404).json({ error: 'Photo file not found' });
}
// Resolve photo through storage backend (managed) or fall back to local
// path (external reference mode). secureImageService needs a local file,
// so we materialize a tmp copy via withLocalCopy in S3 mode.
const storageKey = resolvePhotoStorageKey(event, photo);
// Get protection settings for this event
const protectionSettings = {
@@ -160,11 +154,21 @@ router.get('/:slug/secure/:photoId/:token',
fragmentImage: event.use_canvas_rendering === true && fragment !== undefined
};
// Process image with protection measures
const processedImage = await secureImageService.processProtectedImage(
filePath,
protectionSettings
);
let processedImage;
try {
const runProcessing = (lp) => secureImageService.processProtectedImage(lp, protectionSettings);
processedImage = storageKey
? await withLocalCopy(storageKey, runProcessing)
: await runProcessing(resolvePhotoFilePath(event, photo));
} catch (resolveError) {
logger.error('Failed to process secure image', {
slug: req.params.slug,
photoId,
eventId: event.id,
error: resolveError.message,
});
return res.status(404).json({ error: 'Photo file not found' });
}
// Handle fragmented images
if (processedImage.type === 'fragmented') {
@@ -292,11 +296,30 @@ router.get('/:slug/secure-download/:photoId/:token',
return res.status(404).json({ error: 'Photo not found' });
}
let filePath;
// Resolve photo through storage backend (managed) or local disk (external).
const storageKey = resolvePhotoStorageKey(req.event, photo);
const watermarkService = require('../services/watermarkService');
const watermarkSettings = await watermarkService.getWatermarkSettings();
const wantsWatermark = watermarkSettings && watermarkSettings.enabled;
let fileBuffer;
try {
filePath = resolvePhotoFilePath(req.event, photo);
if (wantsWatermark) {
fileBuffer = storageKey
? await withLocalCopy(storageKey, (lp) => watermarkService.applyWatermark(lp, watermarkSettings))
: await watermarkService.applyWatermark(resolvePhotoFilePath(req.event, photo), watermarkSettings);
} else if (storageKey) {
const stream = await getStorage().get(storageKey);
const chunks = [];
for await (const chunk of stream) chunks.push(chunk);
fileBuffer = Buffer.concat(chunks);
} else {
const fs = require('fs').promises;
fileBuffer = await fs.readFile(resolvePhotoFilePath(req.event, photo));
}
} catch (resolveError) {
logger.error('Failed to resolve photo path for secure download', {
logger.error('Failed to fetch photo for secure download', {
slug: req.params.slug,
photoId,
eventId: req.event.id,
@@ -305,18 +328,6 @@ router.get('/:slug/secure-download/:photoId/:token',
return res.status(404).json({ error: 'Photo file not found' });
}
// Apply watermark if enabled
const watermarkService = require('../services/watermarkService');
const watermarkSettings = await watermarkService.getWatermarkSettings();
let fileBuffer;
if (watermarkSettings && watermarkSettings.enabled) {
fileBuffer = await watermarkService.applyWatermark(filePath, watermarkSettings);
} else {
const fs = require('fs').promises;
fileBuffer = await fs.readFile(filePath);
}
// Update download count
await db('photos').where('id', photoId).increment('download_count', 1);
backend/src/routes/v1/events.js
+45 -17
View File
@@ -182,6 +182,18 @@ router.post(
type: 'admin', id: req.admin.id, name: req.admin.username
});
// Webhook lifecycle (#327). v1 events are not draft-aware, so they're
// both created AND published in the same call.
try {
const webhookService = require('../../services/webhookService');
await webhookService.fire('event.created', {
event: { id, slug, event_name, event_type, event_date, share_url: shareUrl },
});
await webhookService.fire('event.published', {
event: { id, slug, event_name, share_url: shareUrl },
});
} catch (e) { /* non-fatal */ }
res.status(201).json({ id, slug, share_url: shareUrl, share_token: shareToken });
} catch (error) {
logger.error('v1 POST /events failed', { error: error.message, stack: error.stack });
@@ -346,33 +358,39 @@ router.post(
const event = await db('events').where({ id: req.params.id }).first();
if (!event) return res.status(404).json({ error: 'Event not found' });
const finalDir = path.join(getStoragePath(), 'events/active', event.slug);
await fs.mkdir(finalDir, { recursive: true });
const ext = path.extname(req.file.originalname);
const finalName = `${Date.now()}_${crypto.randomBytes(4).toString('hex')}${ext}`;
const finalPath = path.join(finalDir, finalName);
await fs.rename(tempPath, finalPath);
tempPath = null;
// photo.path is stored relative to events/active so resolvePhotoStorageKey
// can rebuild the full key on read. Same shape as adminPhotos uploads.
const relPath = path.posix.join(event.slug, finalName);
const finalKey = path.posix.join('events/active', relPath);
const stat = fsSync.statSync(finalPath);
const relPath = path.relative(path.join(getStoragePath(), 'events/active'), finalPath);
const stat = fsSync.statSync(tempPath);
// Read sharp metadata + generate thumbnail FROM the local temp file
// before uploading the original through the storage backend. (Same
// ordering as adminPhotos.js so sharp/ffmpeg always have a real fs path.)
let width = null;
let height = null;
try {
const meta = await sharp(tempPath).metadata();
width = meta.width || null;
height = meta.height || null;
} catch { /* non-fatal */ }
let thumbRel = null;
try {
const thumbPath = await generateThumbnail(finalPath);
thumbRel = path.relative(getStoragePath(), thumbPath);
thumbRel = await generateThumbnail(tempPath);
} catch (err) {
logger.warn('v1 thumbnail generation failed', { err: err.message });
}
// Detect image dimensions for masonry layouts.
let width = null;
let height = null;
try {
const meta = await sharp(finalPath).metadata();
width = meta.width || null;
height = meta.height || null;
} catch { /* non-fatal */ }
// Upload the original via the storage backend (local fs OR S3),
// then drop the multer temp file.
const { getStorage } = require('../../services/storage');
await getStorage().putFromFile(finalKey, tempPath, { contentType: req.file.mimetype });
await fs.unlink(tempPath).catch(() => {});
tempPath = null;
const insertResult = await db('photos').insert({
event_id: event.id,
@@ -394,6 +412,16 @@ router.post(
type: 'admin', id: req.admin.id, name: req.admin.username
});
// Webhook (#327): one event per uploaded photo so receivers get a
// 1:1 stream they can react to.
try {
const webhookService = require('../../services/webhookService');
await webhookService.fire('photo.uploaded', {
event: { id: event.id, slug: event.slug, event_name: event.event_name },
photo: { id, filename: finalName, original_filename: req.file.originalname, size_bytes: stat.size, width, height },
});
} catch (e) { /* non-fatal */ }
res.status(201).json({ id, filename: finalName, path: relPath, thumbnail_path: thumbRel, size_bytes: stat.size });
} catch (error) {
logger.error('v1 POST /events/:id/photos failed', { error: error.message });
backend/src/services/archiveService.js
+116 -80
View File
@@ -1,57 +1,47 @@
const archiver = require('archiver');
const fs = require('fs').promises;
const fs = require('fs');
const fsp = require('fs').promises;
const path = require('path');
const os = require('os');
const crypto = require('crypto');
const { db } = require('../database/db');
const { queueEmail } = require('./emailProcessor');
const logger = require('../utils/logger');
const feedbackService = require('./feedbackService');
const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
const ACTIVE_PATH = () => path.join(getStoragePath(), 'events/active');
const ARCHIVE_PATH = () => path.join(getStoragePath(), 'events/archived');
const { getStorage } = require('./storage');
async function archiveEvent(event) {
const storage = getStorage();
const archiveName = `${event.slug}.zip`;
const archiveRelKey = path.posix.join('events/archived', archiveName);
const eventPrefix = path.posix.join('events/active', event.slug);
const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'picpeak-archive-'));
const tmpArchive = path.join(tmpDir, `${crypto.randomBytes(4).toString('hex')}-${archiveName}`);
try {
const eventPath = path.join(ACTIVE_PATH(), event.slug);
const archiveName = `${event.slug}.zip`;
const archivePath = path.join(ARCHIVE_PATH(), archiveName);
// Ensure archive directory exists
await fs.mkdir(ARCHIVE_PATH(), { recursive: true });
// Create archive
const output = require('fs').createWriteStream(archivePath);
const archive = archiver('zip', {
zlib: { level: 9 } // Maximum compression
});
archive.on('error', (err) => {
throw err;
});
// Export feedback data before archiving
// Collect feedback data first so it can be included as in-memory entries.
const feedbackEntries = [];
const feedbackSettings = await feedbackService.getEventFeedbackSettings(event.id);
if (feedbackSettings.feedback_enabled) {
try {
logger.info(`Exporting feedback data for event ${event.slug}`);
const feedbackData = await feedbackService.exportEventFeedback(event.id);
if (feedbackData && feedbackData.length > 0) {
// Create feedback JSON file
const feedbackJson = JSON.stringify(feedbackData, null, 2);
const feedbackJsonPath = path.join(eventPath, 'feedback_data.json');
await fs.writeFile(feedbackJsonPath, feedbackJson, 'utf8');
// Create feedback CSV file
const feedbackCsv = convertToCSV(feedbackData);
const feedbackCsvPath = path.join(eventPath, 'feedback_data.csv');
await fs.writeFile(feedbackCsvPath, feedbackCsv, 'utf8');
// Create feedback summary
feedbackEntries.push({
name: 'feedback_data.json',
buffer: Buffer.from(JSON.stringify(feedbackData, null, 2), 'utf8'),
});
feedbackEntries.push({
name: 'feedback_data.csv',
buffer: Buffer.from(convertToCSV(feedbackData), 'utf8'),
});
const summary = await feedbackService.getEventFeedbackSummary(event.id);
const summaryPath = path.join(eventPath, 'feedback_summary.json');
await fs.writeFile(summaryPath, JSON.stringify(summary, null, 2), 'utf8');
feedbackEntries.push({
name: 'feedback_summary.json',
buffer: Buffer.from(JSON.stringify(summary, null, 2), 'utf8'),
});
logger.info(`Feedback data exported: ${feedbackData.length} entries`);
}
} catch (error) {
@@ -59,64 +49,110 @@ async function archiveEvent(event) {
// Continue with archiving even if feedback export fails
}
}
output.on('close', async () => {
try {
logger.info(`Archive created: ${archiveName} (${archive.pointer()} bytes)`);
// Update database
await db('events').where('id', event.id).update({
is_archived: true,
archive_path: path.relative(getStoragePath(), archivePath),
archived_at: new Date()
});
// Stream every photo (and any other content under events/active/{slug}/) into
// the zip directly from the storage backend.
const photoEntries = await storage.list(eventPrefix);
// Delete original files
await fs.rm(eventPath, { recursive: true });
let totalBytes = 0;
await new Promise((resolve, reject) => {
const output = fs.createWriteStream(tmpArchive);
const archive = archiver('zip', { zlib: { level: 9 } });
// Delete thumbnails
const photos = await db('photos').where('event_id', event.id);
for (const photo of photos) {
if (photo.thumbnail_path) {
const thumbPath = path.join(getStoragePath(), photo.thumbnail_path);
await fs.unlink(thumbPath).catch(() => {}); // Ignore if already deleted
}
output.on('close', () => {
totalBytes = archive.pointer();
resolve();
});
archive.on('error', reject);
archive.pipe(output);
const append = async () => {
for (const entry of photoEntries) {
const nameInZip = entry.key.startsWith(`${eventPrefix}/`)
? entry.key.slice(eventPrefix.length + 1)
: entry.key;
const stream = await storage.get(entry.key);
archive.append(stream, { name: nameInZip });
}
// Queue completion email — admin_email is nullable on events (migration 073);
// skip queueing rather than violating email_queue.recipient_email NOT NULL.
if (event.admin_email) {
await queueEmail(event.id, event.admin_email, 'archive_complete', {
event_name: event.event_name,
archive_size: (archive.pointer() / 1024 / 1024).toFixed(2) + ' MB'
});
} else {
logger.info(`Skipping archive_complete email for event ${event.slug}: no admin_email set`);
for (const f of feedbackEntries) {
archive.append(f.buffer, { name: f.name });
}
} catch (err) {
// Never let the close handler reject — it runs detached from the caller,
// and an unhandled rejection here crashes the backend process.
logger.error(`Post-archive cleanup failed for event ${event.slug}:`, err);
}
archive.finalize();
};
append().catch(reject);
});
archive.pipe(output);
archive.directory(eventPath, false);
await archive.finalize();
// Upload the finalized zip to the storage backend.
await storage.putFromFile(archiveRelKey, tmpArchive, { contentType: 'application/zip' });
logger.info(`Archive created: ${archiveName} (${totalBytes} bytes)`);
// Update DB BEFORE deleting originals so a crash mid-cleanup leaves the
// archive accessible rather than orphaning the photos.
await db('events').where('id', event.id).update({
is_archived: true,
archive_path: archiveRelKey,
archived_at: new Date(),
});
// Fire event.archived webhook (#327). Receivers infer per-photo loss
// from this event — we deliberately do NOT fire photo.deleted for each
// archived photo to avoid flooding subscribers on bulk archives.
try {
const webhookService = require('./webhookService');
await webhookService.fire('event.archived', {
event: { id: event.id, slug: event.slug, event_name: event.event_name, archive_path: archiveRelKey },
});
} catch (e) { /* non-fatal */ }
// Delete the originals from storage.
for (const entry of photoEntries) {
await storage.delete(entry.key).catch((err) =>
logger.warn(`Failed to delete archived original ${entry.key}: ${err.message}`)
);
}
// Delete thumbnails for this event's photos.
const photos = await db('photos').where('event_id', event.id);
for (const photo of photos) {
if (photo.thumbnail_path) {
await storage.delete(photo.thumbnail_path).catch(() => {});
}
if (photo.hero_path) {
await storage.delete(photo.hero_path).catch(() => {});
}
// Best effort: remove watermarked variants too if a refactor added them.
if (photo.watermark_path) {
await storage.delete(photo.watermark_path).catch(() => {});
}
}
// Queue completion email — admin_email is nullable on events (migration 073);
// skip queueing rather than violating email_queue.recipient_email NOT NULL.
if (event.admin_email) {
await queueEmail(event.id, event.admin_email, 'archive_complete', {
event_name: event.event_name,
archive_size: (totalBytes / 1024 / 1024).toFixed(2) + ' MB',
});
} else {
logger.info(`Skipping archive_complete email for event ${event.slug}: no admin_email set`);
}
} catch (error) {
logger.error(`Error archiving event ${event.slug}:`, error);
throw error;
} finally {
await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
}
}
// Helper function to convert JSON to CSV
function convertToCSV(data) {
if (!data || data.length === 0) return '';
const headers = Object.keys(data[0]);
const csvHeaders = headers.join(',');
const csvRows = data.map(row => {
return headers.map(header => {
const value = row[header];
@@ -127,7 +163,7 @@ function convertToCSV(data) {
return value || '';
}).join(',');
});
return [csvHeaders, ...csvRows].join('\n');
}
backend/src/services/downloadZipService.js
+65 -40
View File
@@ -7,16 +7,22 @@
*
* Pattern follows watermarkGeneratorService.js — singleton with
* in-memory locking and debounced background regeneration.
*
* Storage: zips are written to a local tmp file then uploaded to the
* configured storage backend (local fs or S3) via storage.putFromFile.
* The cached zip is served via the storage backend on download.
*/
const fs = require('fs');
const fsp = require('fs/promises');
const path = require('path');
const os = require('os');
const crypto = require('crypto');
const archiver = require('archiver');
const { db } = require('../database/db');
const watermarkService = require('./watermarkService');
const { resolvePhotoFilePath } = require('./photoResolver');
const { getStoragePath } = require('../config/storage');
const { resolvePhotoStorageKey, resolvePhotoFilePath } = require('./photoResolver');
const { getStorage } = require('./storage');
const logger = require('../utils/logger');
const DEBOUNCE_MS = 5000;
@@ -29,15 +35,16 @@ class DownloadZipService {
}
/**
* Absolute path to the cached zip for an event slug.
* Relative storage key for the cached zip.
*/
getCachePath(slug) {
return path.join(getStoragePath(), 'events', 'active', slug, '.download-cache', 'all.zip');
getCacheKey(slug) {
return path.posix.join('events/active', slug, '.download-cache', 'all.zip');
}
/**
* Check if a valid cached zip exists.
* Returns { path, size, generatedAt } or null.
* Returns { key, size, generatedAt } or null. The key is a relative storage
* key — callers stream it via storage.get() rather than reading directly.
*/
async getZipInfo(eventId) {
try {
@@ -48,15 +55,10 @@ class DownloadZipService {
if (!event || !event.download_zip_path) return null;
const absPath = this.getCachePath(event.slug);
try {
const stat = await fsp.stat(absPath);
return {
path: absPath,
size: stat.size,
generatedAt: event.download_zip_generated_at,
};
} catch {
const storage = getStorage();
const key = this.getCacheKey(event.slug);
const stat = await storage.stat(key);
if (!stat) {
// File gone — clear stale DB record
await db('events').where({ id: eventId }).update({
download_zip_path: null,
@@ -64,6 +66,11 @@ class DownloadZipService {
});
return null;
}
return {
key,
size: stat.size,
generatedAt: event.download_zip_generated_at,
};
} catch (err) {
logger.warn('downloadZipService.getZipInfo error', { eventId, error: err.message });
return null;
@@ -71,7 +78,7 @@ class DownloadZipService {
}
/**
* Generate the pre-zip for an event. Returns { success, path, size } or { success: false }.
* Generate the pre-zip for an event. Returns { success, key, size } or { success: false }.
* Concurrent calls for the same eventId share one in-flight build.
*/
async generateZip(eventId) {
@@ -97,6 +104,9 @@ class DownloadZipService {
}
async _build(eventId, version) {
const storage = getStorage();
let tmpDir;
try {
const event = await db('events').where({ id: eventId }).first();
if (!event) return { success: false, error: 'Event not found' };
@@ -119,11 +129,10 @@ class DownloadZipService {
text: event.watermark_text || watermarkSettings?.text || 'Protected',
} : null;
const cacheDir = path.dirname(this.getCachePath(event.slug));
await fsp.mkdir(cacheDir, { recursive: true });
const finalKey = this.getCacheKey(event.slug);
const tmpPath = this.getCachePath(event.slug) + `.tmp.${Date.now()}`;
const finalPath = this.getCachePath(event.slug);
tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'picpeak-zipbuild-'));
const tmpPath = path.join(tmpDir, `${crypto.randomBytes(4).toString('hex')}-all.zip`);
// Build zip — level 0 (store only) since photos are already compressed
await new Promise((resolve, reject) => {
@@ -145,13 +154,6 @@ class DownloadZipService {
return reject(new Error('Build invalidated'));
}
let filePath;
try {
filePath = resolvePhotoFilePath(event, photo);
} catch {
continue;
}
let archiveName;
if (hasMultipleTypes) {
const folderName = photo.type === 'individual' ? 'Individual Photos' : 'Collages';
@@ -160,14 +162,36 @@ class DownloadZipService {
archiveName = photo.filename;
}
// External-mode photos still live on local disk; managed photos go
// through the storage backend. resolvePhotoStorageKey returns null
// for external, in which case fall back to resolvePhotoFilePath.
const storageKey = resolvePhotoStorageKey(event, photo);
if (shouldApplyWatermark && effectiveSettings) {
try {
const buf = await watermarkService.applyWatermark(filePath, effectiveSettings);
let sourcePath;
if (storageKey) {
// Stream the original to a tmp file just long enough for sharp
// (watermarkService) to operate on it. Avoids buffering the
// entire image in memory for huge originals.
sourcePath = path.join(tmpDir, `wm-${crypto.randomBytes(4).toString('hex')}`);
await storage.getToFile(storageKey, sourcePath);
} else {
sourcePath = resolvePhotoFilePath(event, photo);
}
const buf = await watermarkService.applyWatermark(sourcePath, effectiveSettings);
archive.append(buf, { name: archiveName });
if (storageKey) {
await fsp.unlink(sourcePath).catch(() => {});
}
} catch (err) {
logger.warn('Skipping watermark in pre-zip', { photoId: photo.id, error: err.message });
}
} else if (storageKey) {
const stream = await storage.get(storageKey);
archive.append(stream, { name: archiveName });
} else {
const filePath = resolvePhotoFilePath(event, photo);
archive.file(filePath, { name: archiveName });
}
}
@@ -180,29 +204,33 @@ class DownloadZipService {
// Check version again — another invalidation may have arrived
if (this.versions.get(eventId) !== version) {
await fsp.unlink(tmpPath).catch(() => {});
return { success: false, error: 'Build invalidated' };
}
// Atomic rename
await fsp.rename(tmpPath, finalPath);
// Upload to storage (atomic from caller's perspective: storage.put writes
// to a tmp file/object first then commits in LocalFs; in S3 the key only
// exists after the multipart upload completes).
await storage.putFromFile(finalKey, tmpPath, { contentType: 'application/zip' });
const stat = await fsp.stat(finalPath);
const stat = await storage.stat(finalKey);
// Update DB
await db('events').where({ id: eventId }).update({
download_zip_path: `events/active/${event.slug}/.download-cache/all.zip`,
download_zip_path: finalKey,
download_zip_generated_at: new Date(),
});
logger.info('Pre-zip generated', { eventId, slug: event.slug, size: stat.size, photos: photos.length });
return { success: true, path: finalPath, size: stat.size };
return { success: true, key: finalKey, size: stat.size };
} catch (err) {
if (err.message === 'Build invalidated') {
return { success: false, error: 'Build invalidated' };
}
logger.error('downloadZipService._build error', { eventId, error: err.message });
return { success: false, error: err.message };
} finally {
if (tmpDir) {
await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
}
}
}
@@ -264,17 +292,14 @@ class DownloadZipService {
async _cleanup(eventId) {
try {
const storage = getStorage();
const event = await db('events')
.where({ id: eventId })
.select('slug', 'download_zip_path')
.first();
if (event && event.download_zip_path) {
const absPath = this.getCachePath(event.slug);
await fsp.unlink(absPath).catch(() => {});
// Also try to remove the cache directory if empty
const cacheDir = path.dirname(absPath);
await fsp.rmdir(cacheDir).catch(() => {});
await storage.delete(this.getCacheKey(event.slug)).catch(() => {});
}
await db('events').where({ id: eventId }).update({
backend/src/services/fileWatcher.js
+33 -2
View File
@@ -13,6 +13,16 @@ const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '.
const WATCH_PATH = () => path.join(getStoragePath(), 'events/active');
function startFileWatcher() {
// Auto-import via filesystem watching only works with the local storage
// backend. In S3 mode there is no local directory to watch — every photo
// must enter through the admin upload API. Skip cleanly with a clear log
// so operators aren't surprised by the missing feature.
const backend = (process.env.STORAGE_BACKEND || 'local').toLowerCase();
if (backend !== 'local') {
logger.warn(`[fileWatcher] auto-import disabled — STORAGE_BACKEND=${backend}. Use the admin upload API instead.`);
return null;
}
const watcher = chokidar.watch(WATCH_PATH(), {
ignored: /(^|[\/\\])\../, // ignore dotfiles
persistent: true,
@@ -93,7 +103,7 @@ async function processNewPhoto(filePath) {
if (!existingPhoto) {
// Add to database
await db('photos').insert({
const insertResult = await db('photos').insert({
event_id: event.id,
filename: path.basename(filePath),
path: relativePath,
@@ -101,10 +111,21 @@ async function processNewPhoto(filePath) {
type: isVideo ? 'video' : photoType,
size_bytes: stats.size,
mime_type: mimeType
});
}).returning('id');
const photoId = insertResult[0]?.id || insertResult[0];
logger.info(`Added new photo: ${relativePath}`);
downloadZipService.invalidate(event.id);
// Webhook (#327) — auto-import path. Only fires in local mode since
// the watcher is disabled in S3 mode.
try {
const webhookService = require('./webhookService');
await webhookService.fire('photo.uploaded', {
event: { id: event.id, slug: event.slug, event_name: event.event_name },
photo: { id: photoId, filename: path.basename(filePath), size_bytes: stats.size, source: 'auto-import' },
});
} catch (e) { /* non-fatal */ }
} else {
logger.debug(`Photo already exists: ${relativePath}`);
}
@@ -121,6 +142,16 @@ async function removePhoto(filePath) {
if (photo) {
downloadZipService.invalidate(photo.event_id);
// Webhook (#327) — fire only if the row actually existed.
try {
const event = await db('events').where({ id: photo.event_id }).first();
const webhookService = require('./webhookService');
await webhookService.fire('photo.deleted', {
event: { id: photo.event_id, slug: event?.slug, event_name: event?.event_name },
photo: { id: photo.id, filename: photo.filename, source: 'auto-import' },
});
} catch (e) { /* non-fatal */ }
}
logger.info(`Removed photo: ${relativePath}`);
backend/src/services/imageProcessor.js
+133 -157
View File
@@ -1,9 +1,12 @@
const sharp = require('sharp');
const exifr = require('exifr');
const path = require('path');
const fs = require('fs').promises;
const fsp = require('fs').promises;
const os = require('os');
const crypto = require('crypto');
const logger = require('../utils/logger');
const { db } = require('../database/db');
const { getStorage } = require('./storage');
// Configure sharp for better memory management with large batches
sharp.cache(false); // Disable cache to prevent memory buildup
@@ -16,8 +19,10 @@ const DEFAULT_THUMBNAIL_FIT = 'cover'; // 'cover' for square crops
const DEFAULT_THUMBNAIL_QUALITY = 85;
const DEFAULT_THUMBNAIL_FORMAT = 'jpeg';
const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
const getThumbnailPath = () => path.join(getStoragePath(), 'thumbnails');
// Hero image settings - optimized for large displays
const DEFAULT_HERO_WIDTH = 1920;
const DEFAULT_HERO_HEIGHT = 1080;
const DEFAULT_HERO_QUALITY = 85;
// Helper to parse setting value (handles both JSON-encoded and plain values)
function parseSettingValue(value) {
@@ -83,60 +88,62 @@ async function getThumbnailSettings() {
}
}
const contentTypeFor = (format) => {
if (format === 'png') return 'image/png';
if (format === 'webp') return 'image/webp';
return 'image/jpeg';
};
/**
* Generate a thumbnail from a local source image path. The output is written
* to the storage backend (local fs or S3) under `thumbnails/thumb_<filename>`
* and the relative storage key is returned for DB persistence.
*
* Callers must ensure the source is on the local filesystem. For S3 mode
* regeneration flows, fetch via `withLocalCopy(storage, sourceKey, fn)` first.
*/
async function generateThumbnail(imagePath, options = {}) {
const filename = path.basename(imagePath);
const thumbnailFilename = `thumb_${filename}`;
const thumbnailDir = getThumbnailPath();
const thumbnailPath = path.join(thumbnailDir, thumbnailFilename);
const thumbnailRelKey = path.posix.join('thumbnails', thumbnailFilename);
const storage = getStorage();
// Get thumbnail settings
const settings = await getThumbnailSettings();
// Ensure thumbnail directory exists
await fs.mkdir(thumbnailDir, { recursive: true });
// Check if we need to regenerate (for broken thumbnails)
// Force regeneration: drop the existing object before writing the new one
if (options.regenerate) {
try {
await fs.unlink(thumbnailPath);
logger.info(`Deleted broken thumbnail: ${thumbnailPath}`);
} catch (err) {
// File might not exist, that's okay
}
await storage.delete(thumbnailRelKey).catch(() => {});
}
try {
// First, verify the source image is complete and valid
const metadata = await sharp(imagePath).metadata();
if (!metadata.width || !metadata.height) {
throw new Error('Invalid image metadata - file may be incomplete');
}
// Create sharp instance with memory-efficient settings
let sharpInstance = sharp(imagePath, {
let sharpInstance = sharp(imagePath, {
limitInputPixels: 268402689, // ~16k x 16k max
sequentialRead: true, // More memory efficient for large images
failOnError: false // Don't fail on minor issues
sequentialRead: true,
failOnError: false
});
// Strip EXIF/metadata from thumbnails (privacy: prevent GPS leak etc.)
sharpInstance = sharpInstance.withMetadata(false);
// Apply resize with configured settings
// For square thumbnails with 'cover' fit, we crop to center
sharpInstance = sharpInstance.resize(settings.width, settings.height, {
withoutEnlargement: true,
fit: settings.fit, // 'cover' will crop to fill the exact dimensions
position: 'center' // Center the crop for better composition
fit: settings.fit,
position: 'center'
});
// Apply format-specific options
if (settings.format === 'jpeg') {
sharpInstance = sharpInstance.jpeg({
sharpInstance = sharpInstance.jpeg({
quality: settings.quality,
progressive: true, // Progressive JPEG for better loading
mozjpeg: true // Better compression
progressive: true,
mozjpeg: true
});
} else if (settings.format === 'png') {
sharpInstance = sharpInstance.png({
@@ -147,74 +154,87 @@ async function generateThumbnail(imagePath, options = {}) {
} else if (settings.format === 'webp') {
sharpInstance = sharpInstance.webp({
quality: settings.quality,
effort: 4 // Balance between speed and compression
effort: 4
});
}
// Save the thumbnail
await sharpInstance.toFile(thumbnailPath);
// Verify the thumbnail was created successfully
const stats = await fs.stat(thumbnailPath);
if (stats.size === 0) {
const buffer = await sharpInstance.toBuffer();
if (!buffer || buffer.length === 0) {
throw new Error('Generated thumbnail is empty');
}
return path.relative(getStoragePath(), thumbnailPath);
await storage.put(thumbnailRelKey, buffer, { contentType: contentTypeFor(settings.format) });
return thumbnailRelKey;
} catch (error) {
const msg = (error && error.message) ? error.message : String(error);
logger.error(`Failed to generate thumbnail for ${filename}: ${msg}`);
// Clean up any partially created file
try {
await fs.unlink(thumbnailPath);
} catch (unlinkErr) {
// Ignore unlink errors
}
// Return null if thumbnail generation fails, don't fail the whole upload
// Clean up any partially uploaded object
await storage.delete(thumbnailRelKey).catch(() => {});
return null;
}
}
/**
* Check if a thumbnail exists and is valid
* Check if a thumbnail exists and is valid. For local-fs storage we open the
* file with sharp to confirm it parses; for S3 we trust the byte-integrity
* checks built into the protocol and only verify size > 0.
*/
async function isThumbnailValid(thumbnailPath) {
const storage = getStorage();
try {
const fullPath = path.join(getStoragePath(), thumbnailPath);
const stats = await fs.stat(fullPath);
// Check if file exists and has content
if (stats.size === 0) {
const stat = await storage.stat(thumbnailPath);
if (!stat || stat.size === 0) {
return false;
}
// Try to read metadata to ensure it's a valid image
await sharp(fullPath).metadata();
if (storage.kind() === 'local') {
const localPath = storage.resolveLocalPath(thumbnailPath);
await sharp(localPath).metadata();
}
return true;
} catch (error) {
return false;
}
}
/**
* Wraps a callback that needs the source image as a local file. In local-fs
* mode the storage path is used directly (no copy); in S3 mode the object is
* streamed to a tmp file which is removed afterwards.
*/
async function withLocalCopy(sourceKey, fn) {
const storage = getStorage();
if (storage.kind() === 'local') {
return fn(storage.resolveLocalPath(sourceKey));
}
const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'picpeak-src-'));
const tmpPath = path.join(tmpDir, `${crypto.randomBytes(4).toString('hex')}_${path.basename(sourceKey)}`);
try {
await storage.getToFile(sourceKey, tmpPath);
return await fn(tmpPath);
} finally {
await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
}
}
/**
* Regenerate thumbnail if it's broken or missing
*/
async function ensureThumbnail(photo) {
const { db } = require('../database/db');
const { resolvePhotoFilePath } = require('./photoResolver');
let originalPath;
const { resolvePhotoStorageKey } = require('./photoResolver');
let sourceKey;
try {
const event = await db('events').where('id', photo.event_id).first();
originalPath = resolvePhotoFilePath(event, photo);
logger.info(`Ensuring thumbnail for photo ${photo.id} from source: ${originalPath}`);
sourceKey = resolvePhotoStorageKey(event, photo);
logger.info(`Ensuring thumbnail for photo ${photo.id} from key: ${sourceKey}`);
} catch (e) {
const msg = (e && e.message) ? e.message : String(e);
logger.error(`Failed to resolve original path for thumbnail (photo ${photo.id}): ${msg}`);
logger.error(`Failed to resolve original key for thumbnail (photo ${photo.id}): ${msg}`);
return null;
}
// Check if thumbnail exists and is valid
if (photo.thumbnail_path) {
const isValid = await isThumbnailValid(photo.thumbnail_path);
@@ -223,45 +243,40 @@ async function ensureThumbnail(photo) {
}
logger.warn(`Invalid thumbnail detected for photo ${photo.id}, regenerating...`);
}
// Generate new thumbnail
const newThumbnailPath = await generateThumbnail(originalPath, { regenerate: true });
// Generate new thumbnail (sources via withLocalCopy so this works in S3 mode)
const newThumbnailPath = await withLocalCopy(sourceKey, (localPath) =>
generateThumbnail(localPath, { regenerate: true })
);
if (newThumbnailPath) {
// Update database with new thumbnail path
const { db } = require('../database/db');
await db('photos')
.where({ id: photo.id })
.update({ thumbnail_path: newThumbnailPath });
logger.info(`Regenerated thumbnail for photo ${photo.id}`);
return newThumbnailPath;
}
return null;
}
async function generateVideoPlaceholder(originalFilename, options = {}) {
const parsed = path.parse(originalFilename || '');
const baseName = parsed.name || 'video';
const thumbnailDir = getThumbnailPath();
const thumbnailFilename = `thumb_${baseName}.jpg`;
const thumbnailPath = path.join(thumbnailDir, thumbnailFilename);
const thumbnailRelKey = path.posix.join('thumbnails', thumbnailFilename);
const storage = getStorage();
const settings = await getThumbnailSettings();
const width = settings.width || DEFAULT_THUMBNAIL_WIDTH;
const height = settings.height || DEFAULT_THUMBNAIL_HEIGHT;
if (options.regenerate) {
try {
await fs.unlink(thumbnailPath);
} catch (_) {
// ignore if missing
}
await storage.delete(thumbnailRelKey).catch(() => {});
}
try {
await fs.mkdir(thumbnailDir, { recursive: true });
const svg = `
<svg width="${width}" height="${height}" viewBox="0 0 ${width} ${height}" xmlns="http://www.w3.org/2000/svg">
<defs>
@@ -279,26 +294,20 @@ async function generateVideoPlaceholder(originalFilename, options = {}) {
</svg>
`;
await sharp(Buffer.from(svg))
const buffer = await sharp(Buffer.from(svg))
.resize(width, height, { fit: 'cover' })
.jpeg({ quality: settings.quality || DEFAULT_THUMBNAIL_QUALITY })
.toFile(thumbnailPath);
.toBuffer();
return path.relative(getStoragePath(), thumbnailPath);
await storage.put(thumbnailRelKey, buffer, { contentType: 'image/jpeg' });
return thumbnailRelKey;
} catch (error) {
logger.error('Failed to generate video placeholder thumbnail:', error.message);
return null;
}
}
// Hero image settings - optimized for large displays
const DEFAULT_HERO_WIDTH = 1920;
const DEFAULT_HERO_HEIGHT = 1080;
const DEFAULT_HERO_QUALITY = 85;
const DEFAULT_HERO_FORMAT = 'jpeg';
const getHeroPath = () => path.join(getStoragePath(), 'heroes');
/**
* Generate a hero-optimized image for gallery headers
* Outputs a 1920x1080 image suitable for full-width hero sections
@@ -306,36 +315,24 @@ const getHeroPath = () => path.join(getStoragePath(), 'heroes');
async function generateHeroImage(imagePath, options = {}) {
const filename = path.basename(imagePath);
const heroFilename = `hero_${filename}`;
const heroDir = getHeroPath();
const heroPath = path.join(heroDir, heroFilename);
const heroRelKey = path.posix.join('heroes', heroFilename);
const storage = getStorage();
// Ensure hero directory exists
await fs.mkdir(heroDir, { recursive: true });
// Check if we need to regenerate
if (options.regenerate) {
try {
await fs.unlink(heroPath);
logger.info(`Deleted existing hero image: ${heroPath}`);
} catch (err) {
// File might not exist, that's okay
}
await storage.delete(heroRelKey).catch(() => {});
}
try {
// First, verify the source image is complete and valid
const metadata = await sharp(imagePath).metadata();
if (!metadata.width || !metadata.height) {
throw new Error('Invalid image metadata - file may be incomplete');
}
// Calculate dimensions to maintain aspect ratio while fitting within hero bounds
const heroWidth = options.width || DEFAULT_HERO_WIDTH;
const heroHeight = options.height || DEFAULT_HERO_HEIGHT;
const quality = options.quality || DEFAULT_HERO_QUALITY;
// Create sharp instance with memory-efficient settings
let sharpInstance = sharp(imagePath, {
limitInputPixels: 268402689,
sequentialRead: true,
@@ -345,43 +342,31 @@ async function generateHeroImage(imagePath, options = {}) {
// Strip EXIF/metadata from hero images (privacy: prevent GPS leak etc.)
sharpInstance = sharpInstance.withMetadata(false);
// Resize to fit hero dimensions while maintaining aspect ratio
// Use 'cover' to fill the hero area (crops if needed)
sharpInstance = sharpInstance.resize(heroWidth, heroHeight, {
withoutEnlargement: false, // Allow upscaling for small images
withoutEnlargement: false,
fit: 'cover',
position: 'center'
});
// Apply JPEG format with high quality
sharpInstance = sharpInstance.jpeg({
quality: quality,
progressive: true,
mozjpeg: true
});
// Save the hero image
await sharpInstance.toFile(heroPath);
// Verify the hero image was created successfully
const stats = await fs.stat(heroPath);
if (stats.size === 0) {
const buffer = await sharpInstance.toBuffer();
if (!buffer || buffer.length === 0) {
throw new Error('Generated hero image is empty');
}
logger.info(`Generated hero image for ${filename}: ${heroPath}`);
return path.relative(getStoragePath(), heroPath);
await storage.put(heroRelKey, buffer, { contentType: 'image/jpeg' });
logger.info(`Generated hero image for ${filename} → ${heroRelKey}`);
return heroRelKey;
} catch (error) {
const msg = (error && error.message) ? error.message : String(error);
logger.error(`Failed to generate hero image for ${filename}: ${msg}`);
// Clean up any partially created file
try {
await fs.unlink(heroPath);
} catch (unlinkErr) {
// Ignore unlink errors
}
await storage.delete(heroRelKey).catch(() => {});
return null;
}
}
@@ -390,16 +375,16 @@ async function generateHeroImage(imagePath, options = {}) {
* Check if a hero image exists and is valid
*/
async function isHeroValid(heroPath) {
const storage = getStorage();
try {
const fullPath = path.join(getStoragePath(), heroPath);
const stats = await fs.stat(fullPath);
if (stats.size === 0) {
const stat = await storage.stat(heroPath);
if (!stat || stat.size === 0) {
return false;
}
// Try to read metadata to ensure it's a valid image
await sharp(fullPath).metadata();
if (storage.kind() === 'local') {
const localPath = storage.resolveLocalPath(heroPath);
await sharp(localPath).metadata();
}
return true;
} catch (error) {
return false;
@@ -410,21 +395,19 @@ async function isHeroValid(heroPath) {
* Ensure a hero image exists for a photo, regenerate if needed
*/
async function ensureHeroImage(photo) {
const { db } = require('../database/db');
const { resolvePhotoFilePath } = require('./photoResolver');
const { resolvePhotoStorageKey } = require('./photoResolver');
let originalPath;
let sourceKey;
try {
const event = await db('events').where('id', photo.event_id).first();
originalPath = resolvePhotoFilePath(event, photo);
logger.info(`Ensuring hero image for photo ${photo.id} from source: ${originalPath}`);
sourceKey = resolvePhotoStorageKey(event, photo);
logger.info(`Ensuring hero image for photo ${photo.id} from key: ${sourceKey}`);
} catch (e) {
const msg = (e && e.message) ? e.message : String(e);
logger.error(`Failed to resolve original path for hero image (photo ${photo.id}): ${msg}`);
logger.error(`Failed to resolve original key for hero image (photo ${photo.id}): ${msg}`);
return null;
}
// Check if hero image exists and is valid
if (photo.hero_path) {
const isValid = await isHeroValid(photo.hero_path);
if (isValid) {
@@ -433,11 +416,11 @@ async function ensureHeroImage(photo) {
logger.warn(`Invalid hero image detected for photo ${photo.id}, regenerating...`);
}
// Generate new hero image
const newHeroPath = await generateHeroImage(originalPath, { regenerate: true });
const newHeroPath = await withLocalCopy(sourceKey, (localPath) =>
generateHeroImage(localPath, { regenerate: true })
);
if (newHeroPath) {
// Update database with new hero path
await db('photos')
.where({ id: photo.id })
.update({ hero_path: newHeroPath });
@@ -451,12 +434,9 @@ async function ensureHeroImage(photo) {
/**
* Extract capture date from EXIF metadata
* @param {string} imagePath - Path to the image file
* @returns {Date|null} - The capture date or null if not available
*/
async function extractCaptureDate(imagePath) {
try {
// Parse EXIF data, looking for common date fields
const exif = await exifr.parse(imagePath, {
pick: ['DateTimeOriginal', 'CreateDate', 'DateTimeDigitized', 'ModifyDate']
});
@@ -465,23 +445,19 @@ async function extractCaptureDate(imagePath) {
return null;
}
// Priority order: DateTimeOriginal > CreateDate > DateTimeDigitized > ModifyDate
const captureDate = exif.DateTimeOriginal ||
exif.CreateDate ||
exif.DateTimeDigitized ||
exif.ModifyDate;
if (captureDate) {
// exifr returns Date objects directly when parsing dates
if (captureDate instanceof Date) {
// Validate the date is reasonable (not in the future, not before 1990)
const now = new Date();
const minDate = new Date('1990-01-01');
if (captureDate > minDate && captureDate <= now) {
return captureDate;
}
}
// Handle string dates if necessary
if (typeof captureDate === 'string') {
const parsed = new Date(captureDate);
if (!isNaN(parsed.getTime())) {
@@ -492,7 +468,6 @@ async function extractCaptureDate(imagePath) {
return null;
} catch (error) {
// Log only as debug - many images don't have EXIF data
logger.debug(`Could not extract EXIF date from ${path.basename(imagePath)}:`, error.message);
return null;
}
@@ -506,5 +481,6 @@ module.exports = {
generateHeroImage,
isHeroValid,
ensureHeroImage,
extractCaptureDate
extractCaptureDate,
withLocalCopy,
};
backend/src/services/photoProcessor.js
+56 -51
View File
@@ -4,9 +4,7 @@ const { db } = require('../database/db');
const { generateThumbnail } = require('./imageProcessor');
const { generatePhotoFilename } = require('../utils/filenameSanitizer');
const { processUploadedVideo, isVideoMimeType } = require('./videoProcessor');
// Get storage path from environment or default
const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
const { getStorage } = require('./storage');
function normalizeFiles(files) {
// Handle null, undefined, or falsy values
@@ -99,11 +97,6 @@ async function processUploadedPhotos(files, eventId, uploadedBy = 'admin', categ
extension
);
// Move file to event folder
const destPath = path.join(getStoragePath(), 'events/active', event.slug);
await fs.mkdir(destPath, { recursive: true });
const newPath = path.join(destPath, newFilename);
const tempPath = file?.path || file?.filepath || file?.tempFilePath;
if (!tempPath) {
@@ -116,7 +109,7 @@ async function processUploadedPhotos(files, eventId, uploadedBy = 'admin', categ
throw new Error(`Uploaded file is missing a temporary path. File info: ${fileInfo}`);
}
// Verify temp file exists before copying
// Verify temp file exists before processing
try {
await fs.access(tempPath);
} catch (accessErr) {
@@ -127,56 +120,33 @@ async function processUploadedPhotos(files, eventId, uploadedBy = 'admin', categ
throw new Error(`Uploaded file not found at temporary location: ${tempPath}`);
}
// Use copyFile and unlink instead of rename to avoid cross-device issues
try {
await fs.copyFile(tempPath, newPath);
console.log(`Successfully copied ${file.originalname} to ${newPath}`);
} catch (copyErr) {
console.error(`Failed to copy file from ${tempPath} to ${newPath}:`, copyErr);
throw new Error(`Failed to copy uploaded file: ${copyErr.message}`);
} finally {
// Clean up temp file with better error handling
try {
await fs.unlink(tempPath);
console.log(`Cleaned up temp file: ${tempPath}`);
} catch (unlinkErr) {
// Only warn if file exists but couldn't be deleted
// ENOENT means file was already deleted, which is fine
if (unlinkErr?.code !== 'ENOENT') {
console.warn(`Failed to clean up temp upload ${tempPath}:`, {
error: unlinkErr.message,
code: unlinkErr.code
});
}
}
}
// Final storage key under events/active/{slug}/{newFilename}.
const relativePath = path.posix.join(event.slug, newFilename);
const finalKey = path.posix.join('events/active', relativePath);
// Determine if this is a video or image
const isVideo = isVideoMimeType(file.mimetype);
const mediaType = isVideo ? 'video' : 'image';
// Generate thumbnail and extract metadata
// Generate thumbnail and extract metadata FROM the temp file (still on
// local disk) before uploading the original.
let thumbnailPath;
let videoMetadata = null;
let imageMetadata = null;
if (isVideo) {
// Process video: extract metadata and generate thumbnail
const thumbnailDir = path.join(getStoragePath(), 'thumbnails');
await fs.mkdir(thumbnailDir, { recursive: true });
const videoThumbnailPath = path.join(thumbnailDir, `thumb_${newFilename.replace(/\.[^.]+$/, '.jpg')}`);
const result = await processUploadedVideo(newPath, videoThumbnailPath);
const videoThumbnailKey = path.posix.join(
'thumbnails',
`thumb_${newFilename.replace(/\.[^.]+$/, '.jpg')}`
);
const result = await processUploadedVideo(tempPath, videoThumbnailKey);
videoMetadata = result.metadata;
thumbnailPath = path.relative(getStoragePath(), videoThumbnailPath);
thumbnailPath = result.thumbnailKey;
} else {
// Process image: generate thumbnail and extract dimensions
thumbnailPath = await generateThumbnail(newPath);
// Extract image dimensions using sharp
thumbnailPath = await generateThumbnail(tempPath);
try {
const sharp = require('sharp');
const metadata = await sharp(newPath).metadata();
const metadata = await sharp(tempPath).metadata();
if (metadata.width && metadata.height) {
imageMetadata = {
width: metadata.width,
@@ -188,10 +158,29 @@ async function processUploadedPhotos(files, eventId, uploadedBy = 'admin', categ
}
}
// Calculate relative paths
const storagePath = getStoragePath();
const relativePath = path.relative(path.join(storagePath, 'events/active'), newPath);
const relativeThumbPath = thumbnailPath; // thumbnailPath is already relative to storage root
// Now upload the original through the storage backend and remove the
// local temp copy.
try {
await getStorage().putFromFile(finalKey, tempPath, {
contentType: file.mimetype,
});
} catch (uploadErr) {
console.error(`Failed to upload ${file.originalname} → ${finalKey}:`, uploadErr);
throw new Error(`Failed to upload to storage: ${uploadErr.message}`);
} finally {
try {
await fs.unlink(tempPath);
} catch (unlinkErr) {
if (unlinkErr?.code !== 'ENOENT') {
console.warn(`Failed to clean up temp upload ${tempPath}:`, {
error: unlinkErr.message,
code: unlinkErr.code
});
}
}
}
const relativeThumbPath = thumbnailPath;
// Add to database with uploaded_by field and media metadata
let insertResult;
@@ -247,7 +236,23 @@ async function processUploadedPhotos(files, eventId, uploadedBy = 'admin', categ
// Commit transaction
await trx.commit();
// Webhook (#327) — fires for every entry path that lands in this
// service: guest upload + auto-import + admin upload via API.
try {
const webhookService = require('./webhookService');
await webhookService.fire('photo.uploaded', {
event: { id: event.id, slug: event.slug, event_name: event.event_name },
photo: {
id: photoId,
filename: newFilename,
original_filename: file.originalname,
size_bytes: file.size,
uploaded_by: uploadedBy,
},
});
} catch (e) { /* non-fatal */ }
uploadedPhotos.push({
id: photoId,
filename: newFilename,
backend/src/services/photoReplacementService.js
+31 -38
View File
@@ -13,10 +13,10 @@ const { db } = require('../database/db');
const { generateThumbnail, extractCaptureDate } = require('./imageProcessor');
const { generatePhotoFilename } = require('../utils/filenameSanitizer');
const watermarkGeneratorService = require('./watermarkGeneratorService');
const { getStorage } = require('./storage');
const { resolvePhotoStorageKey } = require('./photoResolver');
const logger = require('../utils/logger');
const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
/**
* Find a replacement candidate by matching original_filename (case-insensitive).
* Returns the photo row if exactly one match, { ambiguous: true, count } if multiple, or null.
@@ -42,46 +42,21 @@ async function findReplacementCandidate(eventId, originalFilename) {
* @returns {{ success: boolean, photo?: Object, error?: string }}
*/
async function replacePhoto(existingPhoto, newFileTempPath, { originalFilename, mimeType, event }) {
const eventDir = path.join(getStoragePath(), 'events', 'active', event.slug);
const categorySlug = existingPhoto.type === 'collage' ? 'collages' : 'individual';
const targetDir = path.join(eventDir, categorySlug);
try {
// Generate new filename
// Generate new filename + storage key
const ext = path.extname(originalFilename);
const newFilename = generatePhotoFilename(event.event_name, categorySlug, Date.now(), ext);
const tempTargetPath = path.join(targetDir, `_replacing_${Date.now()}_${newFilename}`);
const finalPath = path.join(targetDir, newFilename);
const relativePath = path.join(event.slug, categorySlug, newFilename);
const relativePath = path.posix.join(event.slug, categorySlug, newFilename);
const finalKey = path.posix.join('events/active', relativePath);
const storage = getStorage();
// Write new file to temp name in target directory
await fsp.mkdir(targetDir, { recursive: true });
await fsp.copyFile(newFileTempPath, tempTargetPath);
// Delete old physical file
const oldFilePath = path.join(getStoragePath(), 'events', 'active', existingPhoto.path);
await fsp.unlink(oldFilePath).catch(() => {});
// Delete old thumbnail
if (existingPhoto.thumbnail_path) {
const oldThumbPath = path.join(getStoragePath(), existingPhoto.thumbnail_path);
await fsp.unlink(oldThumbPath).catch(() => {});
}
// Delete old watermark cache
try {
await watermarkGeneratorService.deleteForPhoto(existingPhoto.id);
} catch {
// Ignore — watermark may not exist
}
// Rename temp → final
await fsp.rename(tempTargetPath, finalPath);
// Extract metadata from new file
// Sharp/EXIF need a local file. The temp file from multer still satisfies
// that — we read metadata before uploading the original to storage.
let capturedAt = null;
try {
capturedAt = await extractCaptureDate(finalPath);
capturedAt = await extractCaptureDate(newFileTempPath);
} catch {
// No EXIF — keep null
}
@@ -89,23 +64,41 @@ async function replacePhoto(existingPhoto, newFileTempPath, { originalFilename,
let width = null;
let height = null;
try {
const metadata = await sharp(finalPath).metadata();
const metadata = await sharp(newFileTempPath).metadata();
width = metadata.width || null;
height = metadata.height || null;
} catch {
// Non-image or corrupt
}
const stats = await fsp.stat(finalPath);
const stats = await fsp.stat(newFileTempPath);
// Generate new thumbnail
// Generate new thumbnail FROM the local temp before uploading the original.
let thumbnailPath = null;
try {
thumbnailPath = await generateThumbnail(finalPath);
thumbnailPath = await generateThumbnail(newFileTempPath);
} catch {
logger.warn('Failed to generate thumbnail for replaced photo', { photoId: existingPhoto.id });
}
// Delete old assets BEFORE uploading the new key — if they share the path
// (rare but possible if filename collision), we want the new content.
const oldOriginalKey = resolvePhotoStorageKey(event, existingPhoto);
if (oldOriginalKey && oldOriginalKey !== finalKey) {
await storage.delete(oldOriginalKey).catch(() => {});
}
if (existingPhoto.thumbnail_path && existingPhoto.thumbnail_path !== thumbnailPath) {
await storage.delete(existingPhoto.thumbnail_path).catch(() => {});
}
try {
await watermarkGeneratorService.deleteForPhoto(existingPhoto.id);
} catch {
// Ignore — watermark may not exist
}
// Upload the new original.
await storage.putFromFile(finalKey, newFileTempPath, { contentType: mimeType });
// Update DB record — preserve id, event_id, category_id, type, visibility,
// uploaded_at, sort_order, feedback counts, view/download counts
const updates = {
backend/src/services/photoResolver.js
+41
View File
@@ -4,6 +4,39 @@ const { safePathJoin } = require('../utils/fileSecurityUtils');
const getStoragePath = () => process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
/**
* Resolve a managed photo's relative key under the storage backend.
* Returns null for external-mode photos (those never live in the managed
* storage backend; callers should fall back to resolvePhotoFilePath for
* external references on local disk).
*
* Storage layout (relative to STORAGE_PATH or S3 bucket prefix):
* events/active/{slug}/individual/{filename}
* events/active/{slug}/collages/{filename}
*
* Legacy `photo.path` values may already include `events/active/` — we
* normalize so the returned key always has it exactly once.
*/
function resolvePhotoStorageKey(event, photo) {
if (!event || !photo) throw new Error('resolvePhotoStorageKey requires event and photo');
const mode = (photo.source_origin || event.source_mode || 'managed');
if (mode === 'reference' || mode === 'external') {
// External photos don't live in the managed backend.
return null;
}
const rel = photo.path ? photo.path.replace(/\\/g, '/').replace(/^\/+/, '') : '';
if (!rel) {
throw new Error(`resolvePhotoStorageKey: photo.path is empty for photo ${photo.id}`);
}
// Already prefixed (legacy uploads from a previous code revision).
if (rel.startsWith('events/active/')) return rel;
return path.posix.join('events/active', rel);
}
/**
* Resolve absolute photo file path based on event + photo origin
* Managed: storage/events/active + photo.path (legacy variants supported)
@@ -19,6 +52,13 @@ function resolvePhotoFilePath(event, photo) {
const mode = (photo.source_origin || event.source_mode || 'managed');
if (mode === 'reference' || mode === 'external') {
if (!photo.external_relpath) {
// Mixed-source events: a reference-mode event can also hold managed
// (uploaded) photos. If we have a regular `path` and no
// external_relpath, treat this row as managed instead of throwing.
if (photo.path && !photo.source_origin) {
const relativeSegment = photo.path.replace(/^\/+/, '');
return safePathJoin(path.join(getStoragePath(), 'events/active'), relativeSegment);
}
throw new Error('Missing external_relpath for external photo');
}
// Normalize duplicate leaf segments (e.g., event.external_path ends with 'individual'
@@ -51,4 +91,5 @@ function resolvePhotoFilePath(event, photo) {
module.exports = {
resolvePhotoFilePath,
resolvePhotoStorageKey,
};
backend/src/services/storage/LocalFsStorage.js
+166
View File
@@ -0,0 +1,166 @@
const fs = require('fs');
const fsp = require('fs').promises;
const path = require('path');
const { pipeline } = require('stream/promises');
const crypto = require('crypto');
const logger = require('../../utils/logger');
/**
* Filesystem-backed implementation of the StorageBackend interface.
* All keys are relative to `root` (typically process.env.STORAGE_PATH).
*
* Path traversal protection: every key is normalized to POSIX form and rejected
* if it tries to escape the root via "..". Callers should not need to think
* about this — but if a key arrives via user input it must still be filtered.
*/
class LocalFsStorage {
constructor({ root }) {
if (!root) throw new Error('LocalFsStorage requires a `root` directory');
this.root = path.resolve(root);
}
kind() {
return 'local';
}
async init() {
await fsp.mkdir(this.root, { recursive: true });
// Sanity check: must be writable.
const probe = path.join(this.root, '.storage-write-probe');
await fsp.writeFile(probe, '');
await fsp.unlink(probe);
logger.info(`[storage] LocalFsStorage initialized at ${this.root}`);
}
_resolve(relPath) {
if (!relPath || typeof relPath !== 'string') {
throw new Error(`LocalFsStorage: invalid relative path: ${relPath}`);
}
const normalized = path.posix.normalize(relPath.replace(/\\/g, '/'));
if (normalized.startsWith('..') || normalized.includes('/../') || normalized === '..') {
throw new Error(`LocalFsStorage: path traversal rejected: ${relPath}`);
}
return path.join(this.root, normalized);
}
async put(relPath, body, _options = {}) {
const abs = this._resolve(relPath);
await fsp.mkdir(path.dirname(abs), { recursive: true });
// Write to a sibling tmp file first then rename for crash safety.
const tmp = `${abs}.tmp.${process.pid}.${crypto.randomBytes(4).toString('hex')}`;
try {
if (Buffer.isBuffer(body)) {
await fsp.writeFile(tmp, body);
} else if (body && typeof body.pipe === 'function') {
await pipeline(body, fs.createWriteStream(tmp));
} else {
throw new Error('LocalFsStorage.put: body must be a Buffer or Readable stream');
}
await fsp.rename(tmp, abs);
} catch (err) {
await fsp.unlink(tmp).catch(() => {});
throw err;
}
}
async putFromFile(relPath, localPath, _options = {}) {
const abs = this._resolve(relPath);
await fsp.mkdir(path.dirname(abs), { recursive: true });
// copyFile is atomic from the destination's perspective on POSIX.
await fsp.copyFile(localPath, abs);
}
async get(relPath) {
const abs = this._resolve(relPath);
return fs.createReadStream(abs);
}
async getToFile(relPath, localPath) {
const abs = this._resolve(relPath);
await fsp.mkdir(path.dirname(localPath), { recursive: true });
await fsp.copyFile(abs, localPath);
}
async exists(relPath) {
try {
await fsp.access(this._resolve(relPath), fs.constants.F_OK);
return true;
} catch {
return false;
}
}
async stat(relPath) {
try {
const s = await fsp.stat(this._resolve(relPath));
return { size: s.size, mtime: s.mtime };
} catch (err) {
if (err.code === 'ENOENT') return null;
throw err;
}
}
async delete(relPath) {
try {
await fsp.unlink(this._resolve(relPath));
} catch (err) {
if (err.code !== 'ENOENT') throw err;
}
}
async list(prefix) {
const absPrefix = this._resolve(prefix || '.');
const entries = [];
async function walk(dir, relBase) {
let dirents;
try {
dirents = await fsp.readdir(dir, { withFileTypes: true });
} catch (err) {
if (err.code === 'ENOENT') return;
throw err;
}
for (const ent of dirents) {
const childAbs = path.join(dir, ent.name);
const childRel = relBase ? `${relBase}/${ent.name}` : ent.name;
if (ent.isDirectory()) {
await walk(childAbs, childRel);
} else if (ent.isFile()) {
const s = await fsp.stat(childAbs);
entries.push({ key: childRel, size: s.size, mtime: s.mtime });
}
}
}
const baseRel = prefix && prefix !== '.' ? prefix.replace(/\\/g, '/') : '';
await walk(absPrefix, baseRel);
return entries;
}
async rename(srcRelPath, dstRelPath) {
const src = this._resolve(srcRelPath);
const dst = this._resolve(dstRelPath);
await fsp.mkdir(path.dirname(dst), { recursive: true });
await fsp.rename(src, dst);
}
async copy(srcRelPath, dstRelPath) {
const src = this._resolve(srcRelPath);
const dst = this._resolve(dstRelPath);
await fsp.mkdir(path.dirname(dst), { recursive: true });
await fsp.copyFile(src, dst);
}
async signedUrl(_relPath, _ttlSeconds = 300) {
throw new Error('LocalFsStorage does not support signedUrl. Set STORAGE_BACKEND=s3 to use presigned URLs.');
}
// Escape hatch for callers that genuinely need a filesystem path
// (e.g. ffmpeg, archiver — anything that takes a path argument rather
// than a stream). S3Storage exposes the same method but returns null,
// forcing callers to use the streaming API instead.
resolveLocalPath(relPath) {
return this._resolve(relPath);
}
}
module.exports = LocalFsStorage;
backend/src/services/storage/S3StorageBackend.js
+165
View File
@@ -0,0 +1,165 @@
const path = require('path');
const fs = require('fs');
const fsp = require('fs').promises;
const { HeadObjectCommand } = require('@aws-sdk/client-s3');
const S3StorageAdapter = require('./s3Storage');
const logger = require('../../utils/logger');
/**
* StorageBackend wrapper around the existing S3StorageAdapter.
*
* S3StorageAdapter was originally written for the backup service and exposes
* upload/download/uploadStream/etc. This thin layer maps that surface onto the
* canonical put/get/exists/delete/list/rename/copy/signedUrl interface used by
* the rest of the codebase, and applies an optional `prefix` so a single bucket
* can host multiple deployments without collisions.
*
* Atomicity: S3 has no rename. `rename()` is implemented as `copy()` + `delete()`.
* If the process crashes between the two, the source object remains until the
* next list-and-prune sweep — see `cleanupAbandonedTempUploads()` callers.
*/
class S3StorageBackend {
constructor(config) {
if (!config || !config.bucket) {
throw new Error('S3StorageBackend requires a bucket name');
}
this.adapter = new S3StorageAdapter(config);
this.prefix = (config.prefix || '').replace(/^\/+|\/+$/g, '');
}
kind() {
return 's3';
}
_key(relPath) {
if (!relPath || typeof relPath !== 'string') {
throw new Error(`S3StorageBackend: invalid relative path: ${relPath}`);
}
const normalized = relPath.replace(/\\/g, '/').replace(/^\.?\/+/, '');
if (normalized.startsWith('..') || normalized.includes('/../')) {
throw new Error(`S3StorageBackend: path traversal rejected: ${relPath}`);
}
return this.prefix ? `${this.prefix}/${normalized}` : normalized;
}
async init() {
await this.adapter.testConnection();
logger.info(`[storage] S3StorageBackend initialized bucket=${this.adapter.bucket} prefix=${this.prefix || '(none)'}`);
}
async put(relPath, body, options = {}) {
const key = this._key(relPath);
if (Buffer.isBuffer(body)) {
const { Readable } = require('stream');
const stream = Readable.from(body);
await this.adapter.uploadStream(stream, key, {
contentType: options.contentType,
cacheControl: options.cacheControl,
});
return;
}
if (body && typeof body.pipe === 'function') {
await this.adapter.uploadStream(body, key, {
contentType: options.contentType,
cacheControl: options.cacheControl,
});
return;
}
throw new Error('S3StorageBackend.put: body must be a Buffer or Readable stream');
}
async putFromFile(relPath, localPath, options = {}) {
await this.adapter.upload(localPath, this._key(relPath), {
contentType: options.contentType,
cacheControl: options.cacheControl,
});
}
async get(relPath) {
return this.adapter.downloadStream(this._key(relPath));
}
async getToFile(relPath, localPath) {
await fsp.mkdir(path.dirname(localPath), { recursive: true });
await this.adapter.download(this._key(relPath), localPath);
}
async exists(relPath) {
return this.adapter.exists(this._key(relPath));
}
async stat(relPath) {
try {
const head = await this.adapter.s3Client.send(
new HeadObjectCommand({ Bucket: this.adapter.bucket, Key: this._key(relPath) })
);
return {
size: head.ContentLength,
mtime: head.LastModified,
};
} catch (err) {
if (err.name === 'NotFound' || err.$metadata?.httpStatusCode === 404) return null;
throw err;
}
}
async delete(relPath) {
try {
await this.adapter.delete(this._key(relPath));
} catch (err) {
if (err.name === 'NoSuchKey' || err.$metadata?.httpStatusCode === 404) return;
throw err;
}
}
async list(prefix) {
const fullPrefix = this._key(prefix || '.');
const entries = [];
let continuationToken;
do {
const result = await this.adapter.list(fullPrefix, { continuationToken });
for (const obj of result.Contents || []) {
const stripped = this.prefix && obj.Key.startsWith(`${this.prefix}/`)
? obj.Key.slice(this.prefix.length + 1)
: obj.Key;
entries.push({ key: stripped, size: obj.Size, mtime: obj.LastModified });
}
continuationToken = result.NextContinuationToken;
} while (continuationToken);
return entries;
}
async copy(srcRelPath, dstRelPath) {
await this.adapter.copy(this._key(srcRelPath), this._key(dstRelPath));
}
async rename(srcRelPath, dstRelPath) {
await this.copy(srcRelPath, dstRelPath);
await this.delete(srcRelPath);
}
async signedUrl(relPath, ttlSeconds = 300) {
return this.adapter.getSignedUrl('getObject', this._key(relPath), { expiresIn: ttlSeconds });
}
// S3 has no local path; consumers that need one must use getToFile to a
// temp location first. Returning null here makes the contract explicit so
// legacy code using `storage.resolveLocalPath` fails fast instead of
// silently constructing a bad path.
resolveLocalPath(_relPath) {
return null;
}
// Expose the underlying adapter so backupService keeps working.
// New code should prefer the canonical interface above.
get rawAdapter() {
return this.adapter;
}
static fileStreamFromPath(localPath) {
return fs.createReadStream(localPath);
}
}
module.exports = S3StorageBackend;
backend/src/services/storage/StorageBackend.js
+42
View File
@@ -0,0 +1,42 @@
/**
* Storage backend interface that LocalFsStorage and S3Storage implement.
*
* All paths are POSIX-style relative keys under the deployment's storage root
* (e.g. "events/active/wedding-smith/individual/IMG_0001.jpg"). Concrete adapters
* resolve the absolute filesystem path or S3 key internally so callers never deal
* with the difference between local and remote storage.
*
* Concurrency: methods are safe to call in parallel; ordering is the caller's
* responsibility. `put` is best-effort atomic (LocalFs writes to a temp file
* then renames; S3 returns only after the multipart upload is finalized).
*
* @typedef {Object} PutOptions
* @property {string} [contentType] - MIME type stored in object metadata.
* @property {string} [cacheControl] - Cache-Control header (S3 only).
*
* @typedef {Object} StatResult
* @property {number} size - Size in bytes.
* @property {Date} [mtime] - Last modified timestamp (best-effort; S3 uses LastModified).
*
* @typedef {Object} ListEntry
* @property {string} key - Relative path under the storage root.
* @property {number} size - Size in bytes.
* @property {Date} [mtime] - Last modified timestamp.
*
* @typedef {Object} StorageBackend
* @property {() => string} kind - Returns 'local' or 's3'.
* @property {() => Promise<void>} init - Validates configuration and reachability. Called once at startup.
* @property {(relPath: string, body: NodeJS.ReadableStream | Buffer, options?: PutOptions) => Promise<void>} put
* @property {(relPath: string, localPath: string, options?: PutOptions) => Promise<void>} putFromFile
* @property {(relPath: string) => Promise<NodeJS.ReadableStream>} get - Returns a readable stream of the object body.
* @property {(relPath: string, localPath: string) => Promise<void>} getToFile - Streams the object to a local path (creates parent dirs).
* @property {(relPath: string) => Promise<boolean>} exists
* @property {(relPath: string) => Promise<StatResult|null>} stat - Null if missing.
* @property {(relPath: string) => Promise<void>} delete - No-op if missing.
* @property {(prefix: string) => Promise<ListEntry[]>} list
* @property {(srcRelPath: string, dstRelPath: string) => Promise<void>} rename - Atomic on local fs; copy+delete on S3.
* @property {(srcRelPath: string, dstRelPath: string) => Promise<void>} copy
* @property {(relPath: string, ttlSeconds?: number) => Promise<string>} signedUrl - Presigned download URL (S3 only; LocalFs throws).
*/
module.exports = {};
backend/src/services/storage/index.js
+102
View File
@@ -0,0 +1,102 @@
const LocalFsStorage = require('./LocalFsStorage');
const S3StorageBackend = require('./S3StorageBackend');
const { getStoragePath } = require('../../config/storage');
const logger = require('../../utils/logger');
let instance = null;
/**
* Build the storage backend selected by STORAGE_BACKEND env var.
*
* STORAGE_BACKEND=local (default)
* Uses STORAGE_PATH on the local filesystem. Backwards compatible with every
* existing deployment.
*
* STORAGE_BACKEND=s3
* Reads STORAGE_S3_* vars. Compatible with AWS S3 and any S3-compatible
* service (MinIO, R2, Backblaze, Wasabi, DigitalOcean Spaces, etc.) by
* pointing STORAGE_S3_ENDPOINT at the alternate host.
*
* Required S3 vars:
* STORAGE_S3_BUCKET
* STORAGE_S3_REGION (default us-east-1)
* STORAGE_S3_ACCESS_KEY
* STORAGE_S3_SECRET_KEY
* Optional S3 vars:
* STORAGE_S3_ENDPOINT — custom endpoint URL (MinIO/R2/etc.)
* STORAGE_S3_PREFIX — namespace prefix inside the bucket
* STORAGE_S3_FORCE_PATH_STYLE=true|false (default: auto when endpoint set)
* STORAGE_S3_SSL=true|false (default: true)
*/
function buildStorage() {
const backend = (process.env.STORAGE_BACKEND || 'local').toLowerCase();
if (backend === 's3') {
const required = ['STORAGE_S3_BUCKET', 'STORAGE_S3_ACCESS_KEY', 'STORAGE_S3_SECRET_KEY'];
const missing = required.filter((v) => !process.env[v]);
if (missing.length) {
throw new Error(
`STORAGE_BACKEND=s3 but missing required env vars: ${missing.join(', ')}`
);
}
return new S3StorageBackend({
bucket: process.env.STORAGE_S3_BUCKET,
region: process.env.STORAGE_S3_REGION || 'us-east-1',
endpoint: process.env.STORAGE_S3_ENDPOINT,
accessKeyId: process.env.STORAGE_S3_ACCESS_KEY,
secretAccessKey: process.env.STORAGE_S3_SECRET_KEY,
prefix: process.env.STORAGE_S3_PREFIX,
forcePathStyle: process.env.STORAGE_S3_FORCE_PATH_STYLE === 'true' ? true : undefined,
sslEnabled: process.env.STORAGE_S3_SSL !== 'false',
});
}
if (backend !== 'local') {
throw new Error(`Unknown STORAGE_BACKEND: ${backend}. Expected 'local' or 's3'.`);
}
return new LocalFsStorage({ root: getStoragePath() });
}
/**
* Lazily build + memoize the storage backend. Tests can pass an injected
* instance via `setStorageForTesting` to bypass env-var configuration.
*/
function getStorage() {
if (!instance) {
instance = buildStorage();
}
return instance;
}
/** @internal */
function setStorageForTesting(stub) {
instance = stub;
}
/** @internal — clear the memoized instance so the next call re-reads env. */
function resetStorage() {
instance = null;
}
/**
* Initialize the configured backend. Call once at server startup so config errors
* surface before any request comes in.
*/
async function initStorage() {
const storage = getStorage();
try {
await storage.init();
} catch (err) {
logger.error(`[storage] init failed for backend=${storage.kind()}: ${err.message}`);
throw err;
}
return storage;
}
module.exports = {
getStorage,
initStorage,
setStorageForTesting,
resetStorage,
};
backend/src/services/videoProcessor.js
+55 -43
View File
@@ -2,7 +2,11 @@ const ffmpeg = require('fluent-ffmpeg');
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
const path = require('path');
const fs = require('fs').promises;
const fsSync = require('fs');
const os = require('os');
const crypto = require('crypto');
const logger = require('../utils/logger');
const { getStorage } = require('./storage');
// Set FFmpeg path
ffmpeg.setFfmpegPath(ffmpegPath);
@@ -45,36 +49,48 @@ async function extractVideoMetadata(videoPath) {
}
/**
* Generate thumbnail from video
* @param {string} videoPath - Path to the video file
* @param {string} outputPath - Path for the output thumbnail
* @param {Object} options - Thumbnail options
* @returns {Promise<string>} - Path to generated thumbnail
* Generate a video thumbnail and persist it via the storage backend.
*
* @param {string} videoPath - Local path to the video file (ffmpeg needs a real fs path).
* @param {string} thumbnailKey - Relative storage key the thumbnail will be saved under
* (e.g. "thumbnails/thumb_video.jpg").
* @param {Object} options
* @returns {Promise<string>} The thumbnail's relative storage key.
*/
async function generateVideoThumbnail(videoPath, outputPath, options = {}) {
async function generateVideoThumbnail(videoPath, thumbnailKey, options = {}) {
const {
timeOffset = '00:00:01', // Take screenshot at 1 second
size = '300x300',
quality = 2 // 1-31, lower is better quality
timeOffset = '00:00:01',
size = '300x300'
} = options;
return new Promise((resolve, reject) => {
ffmpeg(videoPath)
.screenshots({
timestamps: [timeOffset],
filename: path.basename(outputPath),
folder: path.dirname(outputPath),
size: size
})
.on('end', () => {
logger.info('Video thumbnail generated', { videoPath, outputPath });
resolve(outputPath);
})
.on('error', (err) => {
logger.error('Error generating video thumbnail', { error: err.message, videoPath });
reject(err);
});
});
const storage = getStorage();
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'picpeak-vidthumb-'));
const tmpFilename = `${crypto.randomBytes(4).toString('hex')}_${path.basename(thumbnailKey)}`;
const tmpPath = path.join(tmpDir, tmpFilename);
try {
await new Promise((resolve, reject) => {
ffmpeg(videoPath)
.screenshots({
timestamps: [timeOffset],
filename: tmpFilename,
folder: tmpDir,
size: size
})
.on('end', () => resolve())
.on('error', (err) => reject(err));
});
if (!fsSync.existsSync(tmpPath)) {
throw new Error('ffmpeg did not produce a thumbnail file');
}
await storage.putFromFile(thumbnailKey, tmpPath, { contentType: 'image/jpeg' });
logger.info('Video thumbnail generated', { videoPath, thumbnailKey });
return thumbnailKey;
} finally {
await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
}
}
/**
@@ -108,37 +124,33 @@ async function getVideoDuration(videoPath) {
}
/**
* Process uploaded video - extract metadata and generate thumbnail
* @param {string} videoPath - Path to the video file
* @param {string} thumbnailPath - Path for the thumbnail
* @param {Object} options - Processing options
* @returns {Promise<Object>} - Video metadata and processing result
* Process an uploaded video: extract metadata and produce a thumbnail through
* the storage backend.
*
* @param {string} videoPath - Local path to the source video (ffmpeg requires fs).
* @param {string} thumbnailKey - Relative storage key for the thumbnail.
* @returns {Promise<{success: boolean, metadata: Object, thumbnailKey: string}>}
*/
async function processUploadedVideo(videoPath, thumbnailPath, options = {}) {
async function processUploadedVideo(videoPath, thumbnailKey, options = {}) {
try {
// Validate video
const isValid = await isValidVideo(videoPath);
if (!isValid) {
throw new Error('Invalid video file');
}
// Extract metadata
const metadata = await extractVideoMetadata(videoPath);
await generateVideoThumbnail(videoPath, thumbnailKey, options);
// Generate thumbnail
await generateVideoThumbnail(videoPath, thumbnailPath, options);
// Verify thumbnail was created
try {
await fs.access(thumbnailPath);
} catch (err) {
throw new Error('Thumbnail generation failed');
const storage = getStorage();
const exists = await storage.exists(thumbnailKey);
if (!exists) {
throw new Error('Thumbnail generation failed (not in storage)');
}
return {
success: true,
metadata,
thumbnailPath
thumbnailKey
};
} catch (error) {
logger.error('Error processing video', { error: error.message, videoPath });
backend/src/services/watermarkGeneratorService.js
+18 -41
View File
@@ -8,10 +8,10 @@
* - Tracking regeneration progress
*/
const path = require('path');
const { db } = require('../database/db');
const watermarkService = require('./watermarkService');
const { getStoragePath } = require('../config/storage');
const { resolvePhotoStorageKey, resolvePhotoFilePath } = require('./photoResolver');
const { withLocalCopy } = require('./imageProcessor');
class WatermarkGeneratorService {
constructor() {
@@ -57,14 +57,15 @@ class WatermarkGeneratorService {
return { success: false, error: 'Watermarking is disabled' };
}
// Resolve the original file path
const originalPath = this.resolvePhotoPath(photo);
if (!originalPath) {
return { success: false, error: 'Could not resolve photo path' };
}
// Generate and save watermark
const result = await watermarkService.generateAndSaveWatermark(photo, originalPath, settings);
// Resolve the source via the storage backend (managed) or local disk
// (external reference mode). watermarkService needs a local file path.
const event = { slug: photo.slug, source_mode: photo.source_mode, external_path: photo.external_path };
const storageKey = resolvePhotoStorageKey(event, photo);
const result = storageKey
? await withLocalCopy(storageKey, (lp) =>
watermarkService.generateAndSaveWatermark(photo, lp, settings)
)
: await watermarkService.generateAndSaveWatermark(photo, resolvePhotoFilePath(event, photo), settings);
if (result.success) {
// Update database with watermark path
@@ -83,31 +84,6 @@ class WatermarkGeneratorService {
}
}
/**
* Resolve the full file path for a photo
*/
resolvePhotoPath(photo) {
const storagePath = getStoragePath();
// Handle external/reference mode
if (photo.source_mode === 'reference' && photo.external_relpath) {
const externalRoot = process.env.EXTERNAL_MEDIA_PATH || path.join(storagePath, 'external');
return path.join(externalRoot, photo.external_path || '', photo.external_relpath);
}
// Standard managed mode
if (photo.file_path) {
// file_path might be absolute or relative
if (path.isAbsolute(photo.file_path)) {
return photo.file_path;
}
return path.join(storagePath, photo.file_path);
}
// Fallback to constructing path from slug and filename
return path.join(storagePath, 'events', 'active', photo.slug, photo.filename);
}
/**
* Generate watermarks for all photos in an event
* @param {number} eventId - The event ID
@@ -189,12 +165,13 @@ class WatermarkGeneratorService {
*/
async processPhotoWatermark(photo, settings) {
try {
const originalPath = this.resolvePhotoPath(photo);
if (!originalPath) {
return { success: false, photoId: photo.id, error: 'Could not resolve path' };
}
const result = await watermarkService.generateAndSaveWatermark(photo, originalPath, settings);
const event = { slug: photo.slug, source_mode: photo.source_mode, external_path: photo.external_path };
const storageKey = resolvePhotoStorageKey(event, photo);
const result = storageKey
? await withLocalCopy(storageKey, (lp) =>
watermarkService.generateAndSaveWatermark(photo, lp, settings)
)
: await watermarkService.generateAndSaveWatermark(photo, resolvePhotoFilePath(event, photo), settings);
if (result.success) {
await db('photos')
backend/src/services/watermarkService.js
+16 -37
View File
@@ -2,7 +2,7 @@ const sharp = require('sharp');
const path = require('path');
const fs = require('fs').promises;
const { db } = require('../database/db');
const { getStoragePath } = require('../config/storage');
const { getStorage } = require('./storage');
class WatermarkService {
constructor() {
@@ -234,19 +234,6 @@ class WatermarkService {
this.cache.clear();
}
/**
* Get the watermarks directory path, creating it if needed
*/
async getWatermarksDir() {
const watermarksDir = path.join(getStoragePath(), 'watermarks');
try {
await fs.access(watermarksDir);
} catch {
await fs.mkdir(watermarksDir, { recursive: true });
}
return watermarksDir;
}
/**
* Get the file extension from a filename
*/
@@ -258,46 +245,42 @@ class WatermarkService {
}
/**
* Generate watermarked version of a photo and save to disk
* Generate watermarked version of a photo and persist it through the
* storage backend. The source must be a local filesystem path because
* sharp doesn't take streams; callers in S3 mode should materialize a
* tmp local copy via imageProcessor.withLocalCopy first.
*
* @param {Object} photo - Photo object with id, filename, and path info
* @param {string} originalPath - Full path to the original image file
* @param {string} originalPath - Local path to the original image file
* @param {Object} settings - Watermark settings (optional, will fetch if not provided)
* @returns {Object} { success, watermarkPath, error }
*/
async generateAndSaveWatermark(photo, originalPath, settings = null) {
try {
// Get settings if not provided
if (!settings) {
settings = await this.getWatermarkSettings();
}
// If watermarking is disabled, return early
if (!settings || !settings.enabled) {
return { success: false, watermarkPath: null, error: 'Watermarking is disabled' };
}
// Verify original file exists
try {
await fs.access(originalPath);
} catch {
return { success: false, watermarkPath: null, error: 'Original file not found' };
}
// Generate watermarked buffer using existing method
const watermarkedBuffer = await this.applyWatermark(originalPath, settings);
// Determine output path
const watermarksDir = await this.getWatermarksDir();
const ext = this.getFileExtension(photo.filename);
const outputFilename = `${photo.id}_watermarked${ext}`;
const outputPath = path.join(watermarksDir, outputFilename);
// Write the watermarked image to disk
await fs.writeFile(outputPath, watermarkedBuffer);
// Return relative path for database storage
const relativePath = `watermarks/${outputFilename}`;
await getStorage().put(relativePath, watermarkedBuffer, {
contentType: ext === '.png' ? 'image/png' : ext === '.webp' ? 'image/webp' : 'image/jpeg',
});
return {
success: true,
watermarkPath: relativePath,
@@ -314,22 +297,18 @@ class WatermarkService {
}
/**
* Delete a pre-generated watermark file
* @param {string} watermarkPath - Relative path to the watermark file
* @returns {boolean} - True if deleted successfully
* Delete a pre-generated watermark file from the storage backend.
* @param {string} watermarkPath - Relative storage key (e.g. "watermarks/123_watermarked.jpg")
* @returns {boolean} - True if a delete was attempted (no-op if missing)
*/
async deleteWatermarkFile(watermarkPath) {
if (!watermarkPath) return false;
try {
const fullPath = path.join(getStoragePath(), watermarkPath);
await fs.unlink(fullPath);
await getStorage().delete(watermarkPath);
return true;
} catch (error) {
// File might not exist, which is fine
if (error.code !== 'ENOENT') {
console.error('Error deleting watermark file:', error);
}
console.error('Error deleting watermark file:', error);
return false;
}
}