From 714a9f6fb1f48ba1316cc240054d5128749581d8 Mon Sep 17 00:00:00 2001 From: Paul Nothaft Date: Wed, 17 Jun 2026 23:04:30 +0200 Subject: [PATCH] fix(upload): auto-throttle on low-memory hosts + correct documented RAM minimum (#628) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README claimed 2GB RAM as the minimum, but two background-processor worker loops × sharp.concurrency(2) means up to four libvips threads can decode full-resolution images in parallel — peak RSS lands at 1.5GB+ on a batch of 20MP+ photos. Add Postgres + Redis + Node baseline and one heavy batch on a 2GB VPS OOM-kills the backend, surfacing as 503s on thumbnails until restart:unless-stopped brings it back. Reported in #602, filed as #628. Three changes, smallest-surface-area each: 1. backgroundProcessor.js — on startup, when UPLOAD_PROCESSOR_CONCURRENCY is NOT set and os.totalmem() reports < 3GB, default to 1 instead of 2 and log a one-shot warning naming the override env var. Explicit env-var setters keep their value. os.totalmem() reports container memory under cgroup v2 so this works in Docker / k8s as well as bare metal. 2. README.md — bumped the documented minimum from 2GB to 4GB, kept 2GB only as a "Low-memory hosts" recipe pointing at UPLOAD_PROCESSOR_CONCURRENCY=1 with the throughput trade-off spelled out. Added the 503-on-OOM symptom so the next reporter finds it via search. 3. docker-compose.production.yml — commented mem_limit / memswap_limit example on the backend service. Off by default (don't surprise existing deployments) but visible to operators thinking about shared/multi-tenant hosts. restart:unless-stopped already on every service. No code path for memory-aware runtime throttling (Luca's option 4) — out of scope for a bug fix; tracked separately if #1-#3 don't close the case. --- README.md | 27 ++++++++++++++- backend/src/services/backgroundProcessor.js | 38 +++++++++++++++++++-- docker-compose.production.yml | 9 +++++ 3 files changed, 71 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e571ba24..38321678 100644 --- a/README.md +++ b/README.md @@ -299,7 +299,12 @@ For local development with a receiver on the same machine or docker network, set ### Minimum Requirements - **CPU**: 2 CPU cores -- **RAM**: 2GB minimum +- **RAM**: **4 GB minimum** for a normal photo-upload workload — sharp/libvips + decodes the full uncompressed frame before resize, and the default two + worker loops at sharp-concurrency 2 can push peak RSS past 1.5 GB on a + batch of 20-MP+ photos. On a 2 GB VPS that's enough to OOM-kill the + backend mid-batch (surfaces as 503s on thumbnails — see [Low-memory + hosts](#low-memory-hosts) below for the recipe to run on 2 GB). - **Storage**: 20GB minimum (plus photo storage needs) - **OS**: Linux (Ubuntu 20.04+), macOS, or Windows with WSL2 - **Node.js**: v18.0.0 or higher @@ -309,6 +314,26 @@ For local development with a receiver on the same machine or docker network, set - **Docker**: v20.10.0+ - **Docker Compose**: v2.0.0+ +### Low-memory hosts + +Running on 2 GB RAM (e.g. an entry-level VPS) is workable but requires +tuning the upload-processor concurrency down. The backend auto-detects +total RAM at startup via `os.totalmem()` — on a host that reports < 3 GB, +it defaults `UPLOAD_PROCESSOR_CONCURRENCY` to **1** instead of 2 and logs +a one-shot warning. You can pin the value explicitly in `.env`: + +```env +# Single worker loop — slower batch processing, lower peak RSS +UPLOAD_PROCESSOR_CONCURRENCY=1 +``` + +The trade-off is throughput: a single worker processes one photo at a +time, so a 100-photo batch takes ~2× as long but won't OOM. **Health-check +note**: if the backend dies under memory pressure, the gallery serves +`503 Service Unavailable` on thumbnails until Docker's +`restart: unless-stopped` brings the container back. Persistent 503s +during/after an upload batch on a low-memory host are almost always this. + ### Video Support Requirements When enabling video uploads, consider these additional resources: diff --git a/backend/src/services/backgroundProcessor.js b/backend/src/services/backgroundProcessor.js index 32148405..004c9c51 100644 --- a/backend/src/services/backgroundProcessor.js +++ b/backend/src/services/backgroundProcessor.js @@ -16,18 +16,52 @@ * is enough for the rare two-process case during dev). * * Tunables (env, all optional): - * UPLOAD_PROCESSOR_CONCURRENCY default 2 + * UPLOAD_PROCESSOR_CONCURRENCY default 2 on hosts with ≥3GB RAM, + * 1 on smaller hosts (auto-detected + * via os.totalmem() with one-shot + * warning, #628). Always honoured + * when set explicitly. * UPLOAD_PROCESSOR_POLL_MS default 1000 * UPLOAD_PROCESSOR_STUCK_TIMEOUT_MS default 600000 (10 minutes) * UPLOAD_PROCESSOR_DISABLED default false (set 'true' to opt out, e.g. in CI) */ +const os = require('os'); const { db } = require('../database/db'); const logger = require('../utils/logger'); const { processPhoto } = require('./photoProcessor'); const POLL_INTERVAL_MS = parseInt(process.env.UPLOAD_PROCESSOR_POLL_MS || '1000', 10); -const CONCURRENCY = Math.max(1, parseInt(process.env.UPLOAD_PROCESSOR_CONCURRENCY || '2', 10)); + +// Soft default: two worker loops × sharp.concurrency(2) means up to four +// libvips threads can decode full-resolution photos in parallel. Each decode +// holds the full uncompressed frame in RAM — a 24MP photo is ~96MB before +// resize. On a 2GB VPS (the documented but barely-viable minimum) one busy +// batch is enough to OOM-kill the backend and surface as 503s on thumbnails +// (#628). When the host reports < 3GB total memory AND the admin hasn't set +// an explicit override, drop the default to 1 and log a one-shot warning +// naming the override env var. Explicit env-var setters keep their value. +// +// os.totalmem() reports container memory under cgroup v2 (Docker / k8s) and +// host memory on bare metal — accurate enough for this decision in either +// deployment shape. +function pickDefaultConcurrency() { + if (process.env.UPLOAD_PROCESSOR_CONCURRENCY !== undefined) { + return parseInt(process.env.UPLOAD_PROCESSOR_CONCURRENCY, 10); + } + const totalRamGB = os.totalmem() / (1024 ** 3); + if (totalRamGB < 3) { + logger.warn?.( + `[backgroundProcessor] Detected ${totalRamGB.toFixed(1)}GB total RAM (< 3GB threshold). ` + + 'Defaulting UPLOAD_PROCESSOR_CONCURRENCY to 1 to avoid OOM on heavy upload batches. ' + + 'Set UPLOAD_PROCESSOR_CONCURRENCY=2 (or higher) explicitly to override.', + ); + return 1; + } + return 2; +} + +const CONCURRENCY = Math.max(1, pickDefaultConcurrency()); const STUCK_TIMEOUT_MS = parseInt(process.env.UPLOAD_PROCESSOR_STUCK_TIMEOUT_MS || '600000', 10); const JANITOR_INTERVAL_MS = 60 * 1000; diff --git a/docker-compose.production.yml b/docker-compose.production.yml index f6560438..77dbff71 100644 --- a/docker-compose.production.yml +++ b/docker-compose.production.yml @@ -69,6 +69,15 @@ services: redis: condition: service_healthy restart: unless-stopped + # Memory cap (optional, recommended on shared / multi-tenant hosts): + # uncomment to bound the backend's RSS. Sharp/libvips decodes the full + # uncompressed image before resize, so a multi-photo upload batch can + # spike memory. With a cap set, the kernel OOM-killer takes the + # container instead of the whole host; restart:unless-stopped brings + # it back. Match this to the RAM budget you've allocated for picpeak + # (`docker stats` shows the live usage). + # mem_limit: 3g + # memswap_limit: 3g healthcheck: # Backend exposes /health on internal port 3000. # The backend image only ships wget (Alpine base) — using curl