fix(upload): auto-throttle on low-memory hosts + correct documented RAM minimum (#628)

The README claimed 2GB RAM as the minimum, but two background-processor
worker loops × sharp.concurrency(2) means up to four libvips threads can
decode full-resolution images in parallel — peak RSS lands at 1.5GB+ on
a batch of 20MP+ photos. Add Postgres + Redis + Node baseline and one
heavy batch on a 2GB VPS OOM-kills the backend, surfacing as 503s on
thumbnails until restart:unless-stopped brings it back. Reported in #602,
filed as #628.

Three changes, smallest-surface-area each:

1. backgroundProcessor.js — on startup, when UPLOAD_PROCESSOR_CONCURRENCY
   is NOT set and os.totalmem() reports < 3GB, default to 1 instead of 2
   and log a one-shot warning naming the override env var. Explicit env-var
   setters keep their value. os.totalmem() reports container memory under
   cgroup v2 so this works in Docker / k8s as well as bare metal.

2. README.md — bumped the documented minimum from 2GB to 4GB, kept 2GB
   only as a "Low-memory hosts" recipe pointing at UPLOAD_PROCESSOR_CONCURRENCY=1
   with the throughput trade-off spelled out. Added the 503-on-OOM symptom
   so the next reporter finds it via search.

3. docker-compose.production.yml — commented mem_limit / memswap_limit
   example on the backend service. Off by default (don't surprise existing
   deployments) but visible to operators thinking about shared/multi-tenant
   hosts. restart:unless-stopped already on every service.

No code path for memory-aware runtime throttling (Luca's option 4) — out
of scope for a bug fix; tracked separately if #1-#3 don't close the case.
This commit is contained in:
Paul Nothaft
2026-06-17 23:04:30 +02:00
parent 83b568ee2d
commit 714a9f6fb1
3 changed files with 71 additions and 3 deletions
+36 -2
View File
@@ -16,18 +16,52 @@
* is enough for the rare two-process case during dev).
*
* Tunables (env, all optional):
* UPLOAD_PROCESSOR_CONCURRENCY default 2
* UPLOAD_PROCESSOR_CONCURRENCY default 2 on hosts with ≥3GB RAM,
* 1 on smaller hosts (auto-detected
* via os.totalmem() with one-shot
* warning, #628). Always honoured
* when set explicitly.
* UPLOAD_PROCESSOR_POLL_MS default 1000
* UPLOAD_PROCESSOR_STUCK_TIMEOUT_MS default 600000 (10 minutes)
* UPLOAD_PROCESSOR_DISABLED default false (set 'true' to opt out, e.g. in CI)
*/
const os = require('os');
const { db } = require('../database/db');
const logger = require('../utils/logger');
const { processPhoto } = require('./photoProcessor');
const POLL_INTERVAL_MS = parseInt(process.env.UPLOAD_PROCESSOR_POLL_MS || '1000', 10);
const CONCURRENCY = Math.max(1, parseInt(process.env.UPLOAD_PROCESSOR_CONCURRENCY || '2', 10));
// Soft default: two worker loops × sharp.concurrency(2) means up to four
// libvips threads can decode full-resolution photos in parallel. Each decode
// holds the full uncompressed frame in RAM — a 24MP photo is ~96MB before
// resize. On a 2GB VPS (the documented but barely-viable minimum) one busy
// batch is enough to OOM-kill the backend and surface as 503s on thumbnails
// (#628). When the host reports < 3GB total memory AND the admin hasn't set
// an explicit override, drop the default to 1 and log a one-shot warning
// naming the override env var. Explicit env-var setters keep their value.
//
// os.totalmem() reports container memory under cgroup v2 (Docker / k8s) and
// host memory on bare metal — accurate enough for this decision in either
// deployment shape.
function pickDefaultConcurrency() {
if (process.env.UPLOAD_PROCESSOR_CONCURRENCY !== undefined) {
return parseInt(process.env.UPLOAD_PROCESSOR_CONCURRENCY, 10);
}
const totalRamGB = os.totalmem() / (1024 ** 3);
if (totalRamGB < 3) {
logger.warn?.(
`[backgroundProcessor] Detected ${totalRamGB.toFixed(1)}GB total RAM (< 3GB threshold). ` +
'Defaulting UPLOAD_PROCESSOR_CONCURRENCY to 1 to avoid OOM on heavy upload batches. ' +
'Set UPLOAD_PROCESSOR_CONCURRENCY=2 (or higher) explicitly to override.',
);
return 1;
}
return 2;
}
const CONCURRENCY = Math.max(1, pickDefaultConcurrency());
const STUCK_TIMEOUT_MS = parseInt(process.env.UPLOAD_PROCESSOR_STUCK_TIMEOUT_MS || '600000', 10);
const JANITOR_INTERVAL_MS = 60 * 1000;