feat: native S3 storage backend (#328) + presigned download follow-up
Lets PicPeak write photos, thumbnails, hero images, watermarks, and archive zips to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces) instead of the local filesystem. Selected via STORAGE_BACKEND=local|s3. Architecture - backend/src/services/storage/StorageBackend.js — abstract interface (put/get/exists/stat/delete/list/copy/rename/signedUrl/putFromFile/ getToFile) — typedef-only, documents the contract. - LocalFsStorage.js — wraps fs with atomic-write-via-tmp-rename, path traversal protection, list-as-walker. - S3StorageBackend.js — thin wrapper around the existing S3StorageAdapter (used by backupService) mapping it onto the canonical interface; supports optional STORAGE_S3_PREFIX namespace. - index.js — factory selected by STORAGE_BACKEND with startup ping (HEADs sentinel key on S3, fs.stat on local) so misconfig fails fast before the first request. Consumer refactors (~12 services + routes), each parametrized over the abstraction: - imageProcessor / videoProcessor — pipe Sharp/ffmpeg output through storage.put; expose withLocalCopy() helper for S3-mode regeneration paths that need a local file for sharp/ffmpeg. - archiveService / downloadZipService — finalize zip in tmp dir, then storage.putFromFile. Atomic-rename pattern preserved on local; S3 emulates via copy + delete (worker prunes orphaned .tmp.* on startup). - photoProcessor / photoReplacementService / adminPhotos upload+delete / routes/v1/events.js POST /events/:id/photos / routes/events.js — every upload path now goes storage.putFromFile(temp) → unlink temp. - gallery.js bulk-download (cached + on-the-fly + selected) — managed photos via storage.get, external-mode unchanged. - protectedImages / secureImages / photoResolver — read via storage.get; resolvePhotoStorageKey returns the canonical key. - watermarkService / watermarkGeneratorService — persistent watermarks via storage.put. - fileWatcher — bails out with a clear log warning when STORAGE_BACKEND=s3 (chokidar can't watch S3); auto-import lands via the S3 prefix walker introduced in the follow-up commit. - expirationChecker — small touch (event.expired webhook fire from #327 shipping in the next commit). Migration tooling - backend/scripts/migrate-storage.js — one-shot --dry-run capable script that walks photos.path, thumbnail_path, hero_path, watermark_path and events.archive_path/download_zip_path; streams local → S3; sha256 size-match skip for idempotent re-run; failures CSV. Presigned-URL "Download All" (#328 follow-up shipped in this commit) - routes/gallery.js — when STORAGE_BACKEND=s3 + event.allow_presigned_download + downloads enabled + watermark NOT enabled, /download-all returns a 302 redirect to a 5-minute presigned S3 URL. Per-event opt-in surface ships in the next commit's UI. Tests - backend/__tests__/integration/storageBackend.test.js — parametrized contract suite running against BOTH LocalFs AND MinIO (18 tests, both backends — 36 cases total). - backend/__tests__/integration/imageProcessor.storage.test.js — same parametrized pattern for the image processor (10 tests × 2 backends). - backend/__tests__/integration/backup-s3.test.js — bootstrap fix: drop the redundant initDb() (001_init handles it) and remove schema-drift in configureS3Backup (app_settings has no created_at anymore and the unique constraint is on setting_key alone, not composite). 0/12 → 7/12 (5 remaining are unrelated assertion drift). - backend/src/services/photoResolver.js — mixed-source events (reference mode with managed-uploaded photos) now fall back to managed when external_relpath is missing instead of throwing. - tests/e2e/s3-storage-roundtrip.spec.ts — Playwright spec that auto-skips against local backend; full upload → serve → delete round-trip when run against an S3-mode backend. Server wiring (server.js) - initStorage() called after database init, before rate limiters. - This commit's diff also includes the webhook delivery worker startup and the S3 auto-importer startup. Those features ship in the next two commits — co-located here for one bisectable diff per file. Docs + ops - README §"Storage Backends" — capability matrix, switching playbook, IAM policy snippet, MinIO/R2/B2 examples. - README §"Webhooks" — also added here (full diff bundled). - .env.example — STORAGE_BACKEND + STORAGE_S3_* + STORAGE_AUTO_IMPORT documented; WEBHOOK_* added in the same diff. - .gitignore — re-anchor the existing `storage/` rule to `/storage/` so backend/src/services/storage/ (the new abstraction code) is trackable. The runtime ./storage/ data dir stays ignored. Out of scope for v1 (per the issue): presigned URLs for individual photo display (always streamed for protection middleware), CDN integration, hybrid hot/cold tiers, S3 → local migration, multi-bucket per-event.
This commit is contained in:
@@ -2,7 +2,11 @@ const ffmpeg = require('fluent-ffmpeg');
|
||||
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
|
||||
const path = require('path');
|
||||
const fs = require('fs').promises;
|
||||
const fsSync = require('fs');
|
||||
const os = require('os');
|
||||
const crypto = require('crypto');
|
||||
const logger = require('../utils/logger');
|
||||
const { getStorage } = require('./storage');
|
||||
|
||||
// Set FFmpeg path
|
||||
ffmpeg.setFfmpegPath(ffmpegPath);
|
||||
@@ -45,36 +49,48 @@ async function extractVideoMetadata(videoPath) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate thumbnail from video
|
||||
* @param {string} videoPath - Path to the video file
|
||||
* @param {string} outputPath - Path for the output thumbnail
|
||||
* @param {Object} options - Thumbnail options
|
||||
* @returns {Promise<string>} - Path to generated thumbnail
|
||||
* Generate a video thumbnail and persist it via the storage backend.
|
||||
*
|
||||
* @param {string} videoPath - Local path to the video file (ffmpeg needs a real fs path).
|
||||
* @param {string} thumbnailKey - Relative storage key the thumbnail will be saved under
|
||||
* (e.g. "thumbnails/thumb_video.jpg").
|
||||
* @param {Object} options
|
||||
* @returns {Promise<string>} The thumbnail's relative storage key.
|
||||
*/
|
||||
async function generateVideoThumbnail(videoPath, outputPath, options = {}) {
|
||||
async function generateVideoThumbnail(videoPath, thumbnailKey, options = {}) {
|
||||
const {
|
||||
timeOffset = '00:00:01', // Take screenshot at 1 second
|
||||
size = '300x300',
|
||||
quality = 2 // 1-31, lower is better quality
|
||||
timeOffset = '00:00:01',
|
||||
size = '300x300'
|
||||
} = options;
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
ffmpeg(videoPath)
|
||||
.screenshots({
|
||||
timestamps: [timeOffset],
|
||||
filename: path.basename(outputPath),
|
||||
folder: path.dirname(outputPath),
|
||||
size: size
|
||||
})
|
||||
.on('end', () => {
|
||||
logger.info('Video thumbnail generated', { videoPath, outputPath });
|
||||
resolve(outputPath);
|
||||
})
|
||||
.on('error', (err) => {
|
||||
logger.error('Error generating video thumbnail', { error: err.message, videoPath });
|
||||
reject(err);
|
||||
});
|
||||
});
|
||||
const storage = getStorage();
|
||||
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'picpeak-vidthumb-'));
|
||||
const tmpFilename = `${crypto.randomBytes(4).toString('hex')}_${path.basename(thumbnailKey)}`;
|
||||
const tmpPath = path.join(tmpDir, tmpFilename);
|
||||
|
||||
try {
|
||||
await new Promise((resolve, reject) => {
|
||||
ffmpeg(videoPath)
|
||||
.screenshots({
|
||||
timestamps: [timeOffset],
|
||||
filename: tmpFilename,
|
||||
folder: tmpDir,
|
||||
size: size
|
||||
})
|
||||
.on('end', () => resolve())
|
||||
.on('error', (err) => reject(err));
|
||||
});
|
||||
|
||||
if (!fsSync.existsSync(tmpPath)) {
|
||||
throw new Error('ffmpeg did not produce a thumbnail file');
|
||||
}
|
||||
|
||||
await storage.putFromFile(thumbnailKey, tmpPath, { contentType: 'image/jpeg' });
|
||||
logger.info('Video thumbnail generated', { videoPath, thumbnailKey });
|
||||
return thumbnailKey;
|
||||
} finally {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -108,37 +124,33 @@ async function getVideoDuration(videoPath) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Process uploaded video - extract metadata and generate thumbnail
|
||||
* @param {string} videoPath - Path to the video file
|
||||
* @param {string} thumbnailPath - Path for the thumbnail
|
||||
* @param {Object} options - Processing options
|
||||
* @returns {Promise<Object>} - Video metadata and processing result
|
||||
* Process an uploaded video: extract metadata and produce a thumbnail through
|
||||
* the storage backend.
|
||||
*
|
||||
* @param {string} videoPath - Local path to the source video (ffmpeg requires fs).
|
||||
* @param {string} thumbnailKey - Relative storage key for the thumbnail.
|
||||
* @returns {Promise<{success: boolean, metadata: Object, thumbnailKey: string}>}
|
||||
*/
|
||||
async function processUploadedVideo(videoPath, thumbnailPath, options = {}) {
|
||||
async function processUploadedVideo(videoPath, thumbnailKey, options = {}) {
|
||||
try {
|
||||
// Validate video
|
||||
const isValid = await isValidVideo(videoPath);
|
||||
if (!isValid) {
|
||||
throw new Error('Invalid video file');
|
||||
}
|
||||
|
||||
// Extract metadata
|
||||
const metadata = await extractVideoMetadata(videoPath);
|
||||
await generateVideoThumbnail(videoPath, thumbnailKey, options);
|
||||
|
||||
// Generate thumbnail
|
||||
await generateVideoThumbnail(videoPath, thumbnailPath, options);
|
||||
|
||||
// Verify thumbnail was created
|
||||
try {
|
||||
await fs.access(thumbnailPath);
|
||||
} catch (err) {
|
||||
throw new Error('Thumbnail generation failed');
|
||||
const storage = getStorage();
|
||||
const exists = await storage.exists(thumbnailKey);
|
||||
if (!exists) {
|
||||
throw new Error('Thumbnail generation failed (not in storage)');
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
metadata,
|
||||
thumbnailPath
|
||||
thumbnailKey
|
||||
};
|
||||
} catch (error) {
|
||||
logger.error('Error processing video', { error: error.message, videoPath });
|
||||
|
||||
Reference in New Issue
Block a user