feat(upload): async photo processing — backend (PR-B part 1) · 851744c3c4 - picpeak

feat(upload): async photo processing — backend (PR-B part 1)

Move thumbnail / EXIF / dimensions / watermark / webhook work off the
upload request thread and into a background worker pool. Upload
requests now return 202 in seconds even on NFS-backed storage; the
worker(s) drain the pending queue independently and update each
photo's processing_status to 'complete' or 'failed' on its own.

Schema (migration 085_async_photo_processing.js):
  - photos.processing_status     enum default 'complete' (existing
                                 rows are already done)
  - photos.processing_error      populated on 'failed'
  - photos.processing_started_at timestamp for janitor recovery
  - photos.upload_id             groups all photos from one upload
                                 request so the frontend can poll
                                 status by group
  - indexes on processing_status and upload_id for queue lookups

services/photoProcessor.js
  - queueFilesForProcessing(files, options) — shared helper used by
    the admin and gallery upload routes. Moves files to final storage
    + inserts pending rows; returns { uploadId, photos, errors }.
  - processPhoto(photoId) — worker-mode: reads original from storage
    via withLocalCopy (transparent local/S3), generates thumbnail and
    EXIF/dimensions or video metadata, queues watermark, fires
    photo.uploaded webhook, marks 'complete'. Throws => caller marks
    'failed' with the error message.
  - processUploadedPhotos kept untouched — chunkedUploadService still
    uses the synchronous path.

services/backgroundProcessor.js (new)
  - N independent worker loops per backend instance (default 2,
    UPLOAD_PROCESSOR_CONCURRENCY env override).
  - Multi-pod safe: postgres SELECT FOR UPDATE SKIP LOCKED, sqlite
    UPDATE-with-status-guard. Pods race on rows, exactly one wins.
  - Janitor every minute resets photos stuck in 'processing' for >10
    minutes (worker died, pod restarted) back to 'pending'.
  - UPLOAD_PROCESSOR_DISABLED=true opt-out for CI/test.
  - Started from server.js after the other long-running workers.

routes/adminPhotos.js — POST /:eventId/upload
  - Replaced batch-of-25 sync processing loop with per-file
    move-to-storage + insert-pending. Response is now 202 with
    upload_id, count, photo_ids in addition to the legacy
    successCount / replacedCount fields the existing frontend reads.
  - Per-request temp directory cleanup is now a single idempotent
    handler on res.finish/res.close (was three inline blocks for
    error paths only, leaking dirs on success — original bug from
    contributor analysis).
  - GET /uploads/:upload_id/status — JSON snapshot of pending /
    processing / complete / failed counts plus per-photo state.
  - GET /uploads/:upload_id/stream — SSE upgrade. Polls internally
    every 1.5s, emits on snapshot change, ends when all photos
    reach a terminal state.
  - POST /photos/:photoId/retry — flips a 'failed' photo back to
    'pending' so the worker picks it up again.
  - GET /:eventId/thumbnail/:photoId now returns 503 with Retry-After
    while the photo is still pending/processing, and 422 on 'failed'.
    The admin grid renders placeholders accordingly.

routes/gallery.js — POST /:eventId/upload (guest)
  - Refactored to use queueFilesForProcessing instead of the synchronous
    processUploadedPhotos. Same 202 + upload_id shape.
  - GET /:slug/photos now filters processing_status to 'complete' (or
    NULL for pre-migration rows) so guests never see in-flight photos.

Side-effect timing change:
  - photo.uploaded webhook now fires from the worker after the photo
    is actually processed (thumbnail + dimensions populated) instead
    of from inside the upload request. Same payload fields. Worth a
    one-line note in the changelog.

This commit is contained in:

Paul Nothaft

2026-05-02 22:56:04 +02:00

parent 86dfcc4f11

commit 851744c3c4

6 changed files with 815 additions and 277 deletions

									
										backend/server.js
									
		+8
		-2
	
												View File
												
				@@ -23,6 +23,7 @@ const { startExpirationChecker } = require('./src/services/expirationChecker');

				const { initializeTransporter, startEmailQueueProcessor } = require('./src/services/emailProcessor');

				const { startBackupService } = require('./src/services/backupService');

				const { startScheduledBackups } = require('./src/services/databaseBackup');

				const backgroundProcessor = require('./src/services/backgroundProcessor');

				const { maintenanceMiddleware } = require('./src/middleware/maintenance');

				const { sessionTimeoutMiddleware } = require('./src/middleware/sessionTimeout');

				const { errorHandler, notFoundHandler } = require('./src/middleware/errorHandler');

				@@ -660,10 +661,15 @@ async function startServer() {

				    // Start backup service

				    await startBackupService();

				    // Start database backup service

				    await startScheduledBackups();

				    // Start the async photo-processing worker pool. Picks up

				    // photos in 'pending' state (from POST /upload) and runs the

				    // sharp/ffmpeg/EXIF pipeline off the request thread.

				    backgroundProcessor.start();

				    app.listen(PORT, () => {

				      logger.info(`Server running on port ${PORT}`);

				      logger.info(`Admin interface: ${process.env.ADMIN_URL || 'http://localhost:3000'}`);