Compare commits

...

2 Commits

Author SHA1 Message Date
Paul Nothaft 82b412f718 fix(admin): keep header-style tiles from overflowing their cards (stable) (#1423)
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build and Push Docker Images / merge-backend (push) Blocked by required conditions
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Build and Push Docker Images / merge-frontend (push) Blocked by required conditions
Build and Push Docker Images / summary (push) Blocked by required conditions
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 11m9s
Fresh-install smoke / fresh-install (push) Failing after 7m12s
Release Please / release-please (push) Failing after 1m26s
Release Please / whatsnew (push) Waiting to run
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 9m46s
Tests / backend (push) Failing after 1s
Schema drift (#530) / upgrade-from-bootstrap (push) Failing after 7m13s
Tests / frontend (push) Failing after 8m32s
Apply the header-style overflow fix to stable: size the grids from their container width and wrap long translated labels within each card.
2026-09-11 15:37:39 +02:00
Paul Nothaft 222b144eed fix(gallery): bound the cached-zip builder's reads and cap rebuild concurrency (stable) (#1421)
Backport of the two halves that landed on main, which together are what the
issue asked for: a per-build read cap plus a separate build-concurrency cap.

Per build: the builder opened one storage read per photo and handed each to
archiver, which drains them one at a time — so every read past the one being
written parked an S3 socket holding unread bytes. Nothing reclaimed them:
archiver's abort() does not touch source streams, and the SDK arms its socket
timeout on a 3s delay then clears it as soon as response headers land, so a
fast response never gets one. Reads are now capped at two and destroyed on
every exit, including the invalidated one, which previously cleaned up nothing
at all. Invalidation also cancels an in-flight build directly rather than
leaving a note for the loop, which matters once the loop can be parked waiting
for a slot a stalled archive will never free.

Across builds: invalidateAll() invalidates every event holding a cached zip and
each invalidate() arms its own debounce timer in the same tick, so they all
fired together and every one started building at once. Background rebuilds now
run two at a time. The cap is on that path only — a foreground generateZip,
where a guest is waiting on the download, is never queued behind a burst.

Two deliberate differences from the main twins. The read cap uses the shared
archiveStreamGuard helper this branch already has rather than main's inline
copy — same contract, less duplicated code. And main's stop() drain has no
counterpart here because this branch has no stop(), so that machinery is left
out rather than carried as dead code.

Relates to issue 1399

Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
2026-09-11 13:52:32 +02:00
4 changed files with 415 additions and 10 deletions
@@ -0,0 +1,127 @@
/**
* Background zip rebuilds are capped (#1399).
*
* invalidateAll() invalidates every event holding a cached zip, and each
* invalidate() arms its own debounce timer in the same tick — so they all fire
* together. Every build opens its own storage reads, so a settings change
* across 25 events was enough to exhaust the S3 agent pool and stall uploads,
* thumbnails and gallery reads until the burst drained.
*
* The cap is on the BACKGROUND path only: a guest waiting on a download must
* not be queued behind a settings-change burst.
*/
jest.mock('../../src/database/db', () => ({ db: jest.fn() }));
jest.mock('../../src/utils/logger', () => ({
info: jest.fn(), warn: jest.fn(), error: jest.fn(), debug: jest.fn(),
}));
const { db } = require('../../src/database/db');
const service = require('../../src/services/downloadZipService');
const flush = () => new Promise((r) => setImmediate(r));
describe('downloadZipService background regen concurrency (#1399)', () => {
let peak;
let inFlight;
let release;
beforeEach(() => {
// setImmediate must stay real: the flush() helper below rides on it, and
// jest's modern fake timers mock it too.
jest.useFakeTimers({ doNotFake: ['setImmediate'] });
peak = 0;
inFlight = 0;
release = [];
service.regenActive = 0;
service.regenWaiters = [];
service.debounceTimers.clear();
service.activeBuilds.clear();
jest.spyOn(service, 'generateZip').mockImplementation(() => {
inFlight += 1;
peak = Math.max(peak, inFlight);
return new Promise((resolve) => {
release.push(() => { inFlight -= 1; resolve(); });
});
});
jest.spyOn(service, '_cleanup').mockResolvedValue(undefined);
});
afterEach(() => {
jest.useRealTimers();
jest.restoreAllMocks();
});
it('never runs more than two rebuilds at once, however many fire together', async () => {
const rows = Array.from({ length: 12 }, (_, i) => ({ id: i + 1 }));
db.mockReturnValue({
whereNotNull: () => ({ select: () => Promise.resolve(rows) }),
});
await service.invalidateAll();
// Every debounce timer was armed in the same tick — fire them all.
jest.runAllTimers();
await flush();
expect(peak).toBe(2);
expect(service.generateZip).toHaveBeenCalledTimes(2);
});
it('starts the next rebuild as each one finishes', async () => {
const rows = Array.from({ length: 5 }, (_, i) => ({ id: i + 1 }));
db.mockReturnValue({
whereNotNull: () => ({ select: () => Promise.resolve(rows) }),
});
await service.invalidateAll();
jest.runAllTimers();
await flush();
expect(service.generateZip).toHaveBeenCalledTimes(2);
release.shift()();
await flush();
expect(service.generateZip).toHaveBeenCalledTimes(3);
expect(peak).toBe(2);
while (release.length) { release.shift()(); await flush(); }
expect(service.generateZip).toHaveBeenCalledTimes(5);
expect(peak).toBe(2);
});
it('does not queue a foreground download behind the burst', async () => {
const rows = Array.from({ length: 6 }, (_, i) => ({ id: i + 1 }));
db.mockReturnValue({
whereNotNull: () => ({ select: () => Promise.resolve(rows) }),
});
await service.invalidateAll();
jest.runAllTimers();
await flush();
expect(service.generateZip).toHaveBeenCalledTimes(2);
// A guest asking for a zip right now calls generateZip directly. It must
// not park behind the two rebuilds already holding the slots.
service.generateZip(999);
await flush();
expect(service.generateZip).toHaveBeenCalledWith(999);
expect(inFlight).toBe(3);
});
it('leaves the queue empty once every rebuild has run', async () => {
const rows = Array.from({ length: 5 }, (_, i) => ({ id: i + 1 }));
db.mockReturnValue({
whereNotNull: () => ({ select: () => Promise.resolve(rows) }),
});
await service.invalidateAll();
jest.runAllTimers();
await flush();
expect(service.regenWaiters.length).toBeGreaterThan(0);
while (release.length) { release.shift()(); await flush(); }
// Nothing parked, nothing counted as running — no slot leaked on the way
// through, which is what would quietly wedge the next burst.
expect(service.regenWaiters).toHaveLength(0);
expect(service.regenActive).toBe(0);
});
});
@@ -0,0 +1,193 @@
/**
* A failed pre-zip build must not leave storage reads open.
*
* The builder opened one storage read per photo and handed the raw stream to
* archiver. archiver drains its queue one entry at a time, so on an S3 backend
* every photo beyond the one being written parked a socket with a full receive
* buffer, and the error path (a source stream dying, or a photo upload
* invalidating the build) walked away from all of them. archiver's abort()
* does not touch the source streams, and the AWS SDK arms its socket timeout
* on a 3s delay then clears it once the response headers arrive, so nothing
* ever reclaimed those sockets. On a live server 43 of the 50 pooled sockets
* ended up stuck for days and photo uploads stopped completing.
*/
const path = require('path');
const fs = require('fs');
const os = require('os');
process.env.NODE_ENV = 'test';
process.env.TEST_DATABASE_PATH = path.join(
fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-zipleak-')), 'db.sqlite',
);
process.env.JWT_SECRET = process.env.JWT_SECRET || 'zipleak-test-secret';
process.env.STORAGE_PATH = fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-zipleak-storage-'));
const { Readable } = require('stream');
const PHOTO_COUNT = 6;
const MAX_INFLIGHT_READS = 2;
// One storage read. It never ends on its own, which is what a large photo
// looks like to the builder: the bytes only move while archiver pulls them.
class StoredObject extends Readable {
constructor(key, failAfterReads, chunks) {
super();
this.key = key;
this.failAfterReads = failAfterReads;
this.chunks = chunks;
this.reads = 0;
}
_read() {
this.reads += 1;
if (this.failAfterReads && this.reads > this.failAfterReads) {
// What a dropped connection to S3 looks like in Node.
this.destroy(new Error('aborted'));
return;
}
this.push(this.reads > this.chunks ? null : Buffer.alloc(4096, 1));
}
}
const reads = { opened: [], live: 0, peak: 0 };
const failingKey = { value: null };
const onOpen = { fn: null };
// A read only finishes when the build pulls the whole object. Photos big
// enough to matter never finish inside one archiver turn, and a stream that
// ends on its own would be auto-destroyed and hide the leak.
const objectChunks = { value: Number.POSITIVE_INFINITY };
function openStoredObject(key) {
const stream = new StoredObject(key, key === failingKey.value ? 1 : 0, objectChunks.value);
reads.opened.push(stream);
reads.live += 1;
if (reads.live > reads.peak) reads.peak = reads.live;
let settled = false;
const settle = () => { if (!settled) { settled = true; reads.live -= 1; } };
stream.once('end', settle);
stream.once('close', settle);
if (onOpen.fn) onOpen.fn(reads.opened.length);
return stream;
}
const mockStorage = {
kind: () => 's3',
get: jest.fn(async (key) => openStoredObject(key)),
getToFile: jest.fn(async () => undefined),
putFromFile: jest.fn(async () => undefined),
stat: jest.fn(async () => ({ size: 1234, mtime: new Date() })),
delete: jest.fn(async () => undefined),
exists: jest.fn(async () => true),
};
jest.mock('../../src/services/storage', () => ({
getStorage: () => mockStorage,
initStorage: async () => mockStorage,
}));
// Nothing to watermark, so the builder takes the stream-from-storage branch,
// which is the one that holds sockets. (This branch has no rendition step —
// the resize/watermark split that main mocks out here does not exist yet.)
jest.mock('../../src/services/watermarkService', () => ({
getWatermarkSettings: jest.fn(async () => ({ enabled: false })),
applyWatermark: jest.fn(),
}));
const { bootCrmDb, seedMinimal } = require('../integration/helpers/crmDb');
const downloadZipService = require('../../src/services/downloadZipService');
describe('pre-zip build releases its storage reads', () => {
let db; let cleanup; let eventId;
beforeAll(async () => {
({ db, cleanup } = await bootCrmDb());
await seedMinimal(db);
const ev = await db('events').insert({
slug: 'zipleak',
event_type: 'wedding',
event_name: 'Zip Leak',
event_date: '2026-09-01',
host_email: 'h@example.com',
admin_email: 'a@example.com',
password_hash: 'x',
share_link: '/gallery/zipleak/s',
share_token: 'zipleak-share',
expires_at: new Date(Date.now() + 7 * 864e5).toISOString(),
is_active: 1,
is_archived: 0,
is_draft: 0,
require_password: 0,
allow_downloads: 1,
created_at: new Date().toISOString(),
}).returning('id');
eventId = ev[0]?.id ?? ev[0];
for (let i = 0; i < PHOTO_COUNT; i += 1) {
await db('photos').insert({
event_id: eventId,
filename: `photo-${i}.jpg`,
path: `zipleak/photo-${i}.jpg`,
type: 'individual',
source_origin: 'managed',
mime_type: 'image/jpeg',
visibility: 'visible',
uploaded_at: new Date(Date.now() - i * 1000).toISOString(),
});
}
}, 120000);
afterAll(async () => { if (cleanup) await cleanup(); });
beforeEach(() => {
reads.opened = [];
reads.live = 0;
reads.peak = 0;
failingKey.value = null;
onOpen.fn = null;
objectChunks.value = Number.POSITIVE_INFINITY;
mockStorage.get.mockClear();
downloadZipService.versions.clear();
downloadZipService.activeBuilds.clear();
});
it('destroys every open read when a source stream dies mid-build', async () => {
// The oldest photo is written first, so failing it strands the rest.
failingKey.value = 'events/active/zipleak/photo-0.jpg';
const result = await downloadZipService.generateZip(eventId);
expect(result.success).toBe(false);
expect(reads.opened.length).toBeGreaterThan(1);
const stranded = reads.opened.filter((s) => !s.destroyed);
expect(stranded.map((s) => s.key)).toEqual([]);
});
it('destroys every open read when an upload invalidates the build', async () => {
// What adminPhotos does on every upload, delete and bulk edit, landing
// while the archive is half built.
onOpen.fn = (count) => {
if (count !== 2) return;
downloadZipService.invalidate(eventId);
// invalidate() also schedules a rebuild; this test is not about that.
clearTimeout(downloadZipService.debounceTimers.get(eventId));
downloadZipService.debounceTimers.delete(eventId);
};
const result = await downloadZipService.generateZip(eventId);
expect(result).toEqual({ success: false, error: 'Build invalidated' });
expect(reads.opened.filter((s) => !s.destroyed).map((s) => s.key)).toEqual([]);
});
it('never holds more storage reads open than the build needs', async () => {
objectChunks.value = 8;
const result = await downloadZipService.generateZip(eventId);
expect(result.success).toBe(true);
expect(mockStorage.get).toHaveBeenCalledTimes(PHOTO_COUNT);
expect(reads.peak).toBeLessThanOrEqual(MAX_INFLIGHT_READS);
});
});
+75 -3
View File
@@ -24,15 +24,52 @@ const watermarkService = require('./watermarkService');
const { resolvePhotoStorageKey, resolvePhotoFilePath } = require('./photoResolver');
const { getStorage } = require('./storage');
const { getUseOriginalFilenames, getZipEntryNames } = require('./downloadFilenameService');
const { createArchiveStreamGuard } = require('../utils/archiveStreamGuard');
const logger = require('../utils/logger');
const DEBOUNCE_MS = 5000;
// How many cached zips may be REBUILT at once in the background.
//
// invalidateAll() invalidates every event that has a cached zip, and each
// invalidate() arms its own debounce timer in the same tick — so they all fire
// together and, before this, every one of them started building at once. Each
// build opens its own storage reads, so 25 events was enough to exhaust the S3
// agent pool and stall uploads, thumbnails and gallery reads until the burst
// finished.
//
// This caps the BACKGROUND path only. A foreground generateZip() — a guest
// actually waiting for a download — is never queued behind a rebuild.
const MAX_CONCURRENT_REGENS = 2;
class DownloadZipService {
constructor() {
this.activeBuilds = new Map(); // eventId -> { promise, version }
this.debounceTimers = new Map(); // eventId -> setTimeout handle
this.versions = new Map(); // eventId -> generation counter
this.buildCancellers = new Map(); // eventId -> abort the in-flight build
this.regenActive = 0; // background rebuilds running right now
this.regenWaiters = []; // resolvers parked waiting for a slot
}
/**
* Run a BACKGROUND rebuild under the concurrency cap. Foreground callers
* deliberately do not go through here: someone is waiting on that response,
* and making them queue behind a settings-change burst would trade one stall
* for another.
*/
async _withRegenSlot(fn) {
if (this.regenActive >= MAX_CONCURRENT_REGENS) {
await new Promise((resolve) => this.regenWaiters.push(resolve));
}
this.regenActive += 1;
try {
return await fn();
} finally {
this.regenActive -= 1;
const next = this.regenWaiters.shift();
if (next) next();
}
}
/**
@@ -107,6 +144,7 @@ class DownloadZipService {
async _build(eventId, version) {
const storage = getStorage();
let tmpDir;
let buildGuard = null;
try {
const event = await db('events').where({ id: eventId }).first();
@@ -152,8 +190,28 @@ class DownloadZipService {
const output = fs.createWriteStream(tmpPath);
const archive = archiver('zip', { zlib: { level: 0 } });
// Bound and reclaim the storage reads. archiver drains its sources one
// at a time, so appending one read per photo parks an S3 socket per
// photo holding unread bytes; nothing reclaims them, because
// archiver's abort() does not touch source streams and the SDK clears
// its socket timeout as soon as response headers land. An unbounded
// loop over a large event starves uploads, thumbnails and gallery
// reads for the duration of the build.
buildGuard = createArchiveStreamGuard({
onFatalError: (err) => { buildGuard.destroyAll(); archive.abort(); reject(err); },
});
// Invalidation cancels the build directly rather than leaving a note
// for the loop: with a read cap in place the loop can be parked waiting
// for a slot that a stalled archive will never free.
this.buildCancellers.set(eventId, () => {
buildGuard.destroyAll();
archive.abort();
reject(new Error('Build invalidated'));
});
output.on('close', resolve);
archive.on('error', reject);
archive.on('error', (err) => { buildGuard.destroyAll(); reject(err); });
archive.pipe(output);
const uniqueTypes = new Set(photos.map(p => p.type)).size;
@@ -164,6 +222,7 @@ class DownloadZipService {
const photo = photos[i];
// Check if build was invalidated
if (this.versions.get(eventId) !== version) {
buildGuard.destroyAll();
archive.abort();
return reject(new Error('Build invalidated'));
}
@@ -203,8 +262,9 @@ class DownloadZipService {
logger.warn('Skipping watermark in pre-zip', { photoId: photo.id, error: err.message });
}
} else if (storageKey) {
if (!await buildGuard.acquire()) return;
const stream = await storage.get(storageKey);
archive.append(stream, { name: archiveName });
archive.append(buildGuard.track(stream), { name: archiveName });
} else {
const filePath = resolvePhotoFilePath(event, photo);
archive.file(filePath, { name: archiveName });
@@ -243,6 +303,10 @@ class DownloadZipService {
logger.error('downloadZipService._build error', { eventId, error: err.message });
return { success: false, error: err.message };
} finally {
// Every exit path — success, invalidated, thrown — has to reclaim the
// reads, or they hold their sockets for the life of the process.
if (buildGuard) buildGuard.destroyAll();
this.buildCancellers.delete(eventId);
if (tmpDir) {
await fsp.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
}
@@ -261,6 +325,12 @@ class DownloadZipService {
const timer = this.debounceTimers.get(eventId);
if (timer) clearTimeout(timer);
// Cancel an in-flight build directly. Bumping the version only stops it the
// next time the loop looks, and with a read cap the loop can be parked
// waiting for a slot a stalled archive will never free.
const cancelBuild = this.buildCancellers.get(eventId);
if (cancelBuild) cancelBuild();
// Fire-and-forget cleanup
this._cleanup(eventId).catch(err =>
logger.warn('downloadZipService.invalidate cleanup error', { eventId, error: err.message })
@@ -269,7 +339,9 @@ class DownloadZipService {
// Debounce regeneration
const newTimer = setTimeout(() => {
this.debounceTimers.delete(eventId);
this.generateZip(eventId).catch(err =>
// Through the cap: invalidateAll arms every one of these in the same
// tick, so without it they all start building together.
this._withRegenSlot(() => this.generateZip(eventId)).catch(err =>
logger.warn('downloadZipService debounced regen error', { eventId, error: err.message })
);
}, DEBOUNCE_MS);
@@ -22,7 +22,13 @@ export const HeaderStyleCard: React.FC<HeaderStyleCardProps> = ({ localTheme, ha
<p className="text-sm text-neutral-600 dark:text-neutral-400 mb-4">
{t('branding.headerStyleDescription', 'Choose how the gallery header appears. The header style is independent of the photo layout.')}
</p>
<div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-4">
{/* auto-fit/minmax rather than viewport breakpoints (#1412): the
breakpoints size the columns off the WINDOW, but this card sits in a
settings panel that is far narrower, so `lg:grid-cols-3` produced
three ~85px columns no German string could fit in. A minimum track
width lets the column count follow the container instead, and drops
to fewer columns when there is no room. */}
<div className="grid grid-cols-[repeat(auto-fit,minmax(9rem,1fr))] gap-4">
{(Object.keys(headerStyleIcons) as HeaderStyleType[]).map((style) => (
<button
type="button"
@@ -34,14 +40,21 @@ export const HeaderStyleCard: React.FC<HeaderStyleCardProps> = ({ localTheme, ha
: 'border-neutral-200 dark:border-neutral-700 hover:border-neutral-300 dark:hover:border-neutral-600'
}`}
>
<div className="flex flex-col items-center text-center">
{/* min-w-0 + break-words: a grid item will not shrink below its
min-content width, and German compounds here are long enough to
exceed a narrow column — "Veranstaltungsinfo-Overlay" and
"Veranstaltungsdetails" spilled out of the card and over the
neighbouring one at three columns in a narrow panel (#1412).
Any translation can do this, so the constraint belongs on the
element rather than on the strings. */}
<div className="flex flex-col items-center text-center w-full min-w-0">
<div className="mb-2 text-neutral-700 dark:text-neutral-300">
{headerStyleIcons[style]}
</div>
<span className="font-medium text-sm capitalize text-neutral-900 dark:text-neutral-100">
<span className="w-full break-words font-medium text-sm capitalize text-neutral-900 dark:text-neutral-100">
{t(`branding.headerStyleOptions.${style}`, style)}
</span>
<span className="text-xs text-neutral-600 dark:text-neutral-400 mt-1">
<span className="w-full break-words text-xs text-neutral-600 dark:text-neutral-400 mt-1">
{t(`branding.headerStyleDescriptions.${style}`, '')}
</span>
</div>
@@ -61,7 +74,7 @@ export const HeaderStyleCard: React.FC<HeaderStyleCardProps> = ({ localTheme, ha
<p className="text-xs text-neutral-600 dark:text-neutral-400 mb-4">
{t('branding.heroDividerDescription', 'Choose how the transition between the hero image and gallery content looks.')}
</p>
<div className="grid grid-cols-2 sm:grid-cols-3 lg:grid-cols-5 gap-3">
<div className="grid grid-cols-[repeat(auto-fit,minmax(6rem,1fr))] gap-3">
{(Object.keys(dividerStylePreviews) as HeroDividerStyle[]).map((divider) => (
<button
type="button"
@@ -73,12 +86,12 @@ export const HeaderStyleCard: React.FC<HeaderStyleCardProps> = ({ localTheme, ha
: 'border-neutral-200 dark:border-neutral-700 hover:border-neutral-300 dark:hover:border-neutral-600'
}`}
>
<div className="flex flex-col items-center">
<div className="flex flex-col items-center w-full min-w-0">
<div className="w-full mb-2 bg-neutral-800 rounded-t overflow-hidden">
<div className="h-8"></div>
{dividerStylePreviews[divider]}
</div>
<span className="text-xs font-medium capitalize text-neutral-900 dark:text-neutral-100">
<span className="w-full break-words text-xs font-medium capitalize text-neutral-900 dark:text-neutral-100">
{t(`branding.dividerOptions.${divider}`, divider)}
</span>
</div>