Files
picpeak/backend/__tests__/routes/adminEventsCascadeStorage.test.js
T
peipeimo 202c553a08 fix(events): delete stored objects when cascading an event delete (#1051)
* fix(events): delete stored objects when cascading an event delete

* fix(events): sweep watermarks and the archive zip on cascade delete too

Two more objects in the same class as the originals: both are written
through the storage backend, both were only ever removed with fs.unlink,
so both outlive the event on S3.

- photo.watermark_path — a canonical key, deleted via getStorage() on the
  single-photo path (watermarkService.deleteWatermarkFile) and on archive
  (archiveService.js:227). The cascade neither selected nor removed it.
- event.archive_path — written by storage.putFromFile (archiveService.js:160)
  and typically the largest single object an event owns.

event.hero_logo_path is deliberately NOT included: multer writes logos to
local disk with diskStorage regardless of backend (adminEvents/logo.js:19-28),
so they are never bucket objects and the existing fs.unlink is correct.

Collect into a Set — an unresized gallery can carry one object in both
hero_path and preview_path, and the second delete would log a spurious
failure.

* fix(events): sweep the download caches, and delete objects concurrently

Both from an external review round on this PR.

The download caches are the subtle case: the pre-built "Download All" zip
(events.download_zip_path) and one zip per custom-resolution download job
(download_jobs.zip_path) both live under
events/active/{slug}/.download-cache/. On local disk the recursive fs.rm
already covered them, which is exactly why they were easy to miss — on S3
that prefix is not a directory, nothing covered them, and both are
gallery-sized. downloadZipService exposes a cleanup() documented as "used
on event deletion" that the cascade never called.

The job rows are read before the transaction for the same reason the photo
rows are: download_jobs.event_id is ON DELETE CASCADE, so on Postgres they
vanish with the event and take their keys with them. Guarded with hasTable
so a pre-#173 install doesn't abort the delete.

Deletes now run through a bounded pool instead of one await per key. A
400-photo gallery owns ~1600 objects once derived tiers are counted, and
that many sequential DeleteObject round trips runs to minutes — long enough
for a proxy to time the request out AFTER the commit, leaving the event
deleted and the sweep half-finished. A pool rather than Promise.all over
every key, so the fan-out can't exhaust the S3 client's connection pool.

* fix(events): never delete a derivative another gallery still uses

Round-2 findings from the external reviewer.

Canonical thumbnail/hero/preview keys are not event-scoped: the basename is
the photo's filename (imageProcessor passes no outputBasename for managed
photos, so the key is thumbnails/thumb_w300_<filename>), and filenames are
not unique across events — the responsive-tier code says so in as many
words, which is why THOSE keys carry a p{id}_ prefix. A legacy gallery can
therefore share a canonical derivative with a photo in another event, and
deleting it here blanked a surviving gallery's tile. Derived keys are now
checked against photos outside this event and anything still referenced is
left alone; if the check itself fails, every derivative is kept. An orphan
costs storage, a deleted derivative costs someone else's gallery. Originals
need no check — their keys embed the slug.

Also cancel any in-flight or debounced Download All build before snapshotting
paths. A builder that started before the delete would otherwise upload a
gallery-sized zip after the sweep and write its path onto a row that no
longer exists, orphaning it permanently. downloadZipService.cleanup() is the
service's own entry point for this and does all three things: bumps the
version so an in-flight build discards its result, clears the debounce so
nothing rebuilds for a deleted event, and removes the current object.

* revert(events): drop the Download All build cancellation

Reverted for the same reason as on the stable twin, where it was caught:
downloadZipService.cleanup() reaches getStorage() through _cleanup(), so
where the S3 backend is configured but unreachable every cascade delete pays
the adapter's retry backoff. On stable that took the backend CI job from ~2
minutes to past its 10-minute budget, twice, reproducibly. This branch's
suite happened not to trip it, but the same cost lands in the request path
of a real delete — and the twins have to carry the same code.

The race it addressed is narrow and costs one orphaned zip; documented as a
follow-up instead. The shared-derivative guard from the same review round
stays — that one prevented deleting a surviving gallery's thumbnail.

---------

Co-authored-by: Peifu Mo <peipeimo@Peifus-MacBook-Pro.local>
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
2026-09-01 08:17:04 +02:00

232 lines
8.3 KiB
JavaScript

/**
* Regression test: deleting an event must remove its stored objects.
*
* deleteEventCascade() cleaned up the local filesystem only (#608). On an
* S3/R2 storage backend that cleanup is a no-op, so every deleted gallery
* left its originals and derived tiers in the bucket — unreferenced,
* invisible in the UI, and billed forever. Measured on a v3.45.16 install
* against Cloudflare R2: deleting a 403-photo event changed the bucket
* object count by exactly zero.
*
* The keys must be collected BEFORE the transaction deletes the photo
* rows, because afterwards nothing knows which objects were this event's.
*/
const os = require('os');
const path = require('path');
// The cascade runs a real `fs.rm(..., { recursive: true })` over
// {STORAGE_PATH}/events/{active,archived}/{slug}. Point that at a throwaway
// directory before requiring the module under test — the default resolves
// into the working tree.
process.env.STORAGE_PATH = path.join(os.tmpdir(), 'picpeak-cascade-storage-test');
const mockStorage = { delete: jest.fn().mockResolvedValue(undefined) };
const mockEvent = {
id: 42,
slug: 'other-demo-2026-01-01',
event_name: 'Demo',
source_mode: 'managed',
// Written through the backend by archiveService, so it is a bucket object
// and the fs.unlink in the cascade never touched it on S3.
archive_path: 'archives/other-demo-2026-01-01.zip',
// The pre-built "Download All" zip. Lives under the event prefix, so the
// recursive fs.rm covers it on local disk and nothing covers it on S3.
download_zip_path: 'events/active/other-demo-2026-01-01/.download-cache/all.zip',
};
// One zip per custom-resolution download job, same prefix. The rows are
// ON DELETE CASCADE, so they must be read before the transaction.
const mockDownloadJobs = [
{ zip_path: 'events/active/other-demo-2026-01-01/.download-cache/job-abc123.zip' },
];
const mockPhotos = [
{
id: 1,
path: 'other-demo-2026-01-01/photo_one.jpg',
thumbnail_path: 'thumbnails/thumb_aaa_photo_one.jpg',
hero_path: null,
preview_path: 'previews/prev_aaa_photo_one.jpg',
watermark_path: 'watermarked/wm_aaa_photo_one.jpg',
source_origin: 'managed',
},
{
id: 2,
path: 'other-demo-2026-01-01/photo_two.jpg',
thumbnail_path: 'thumbnails/thumb_bbb_photo_two.jpg',
hero_path: null,
preview_path: null,
watermark_path: null,
source_origin: 'managed',
},
{
// External photos live outside the managed backend and must be left alone.
id: 3,
path: 'ignored.jpg',
thumbnail_path: null,
hero_path: null,
preview_path: null,
watermark_path: null,
source_origin: 'external',
},
];
let mockPhotoRowsDeleted = false;
let mockJobRowsDeleted = false;
// Photos in OTHER events that share a canonical derivative key with this one.
let mockSharedDerivatives = [];
// The shared-derivative probe: db('photos').whereNot(...).where(cb).select(...)
const sharedProbe = {
where: () => sharedProbe,
whereIn: () => sharedProbe,
orWhereIn: () => sharedProbe,
select: async () => mockSharedDerivatives,
};
function mockMakeDb() {
const table = (name) => {
const chain = {
where: () => chain,
first: async () => (name === 'events' ? mockEvent : undefined),
whereNotNull: () => chain,
whereNot: () => sharedProbe,
orWhereIn: () => chain,
whereIn: () => chain,
select: async () => {
if (name === 'download_jobs') {
return mockJobRowsDeleted ? [] : mockDownloadJobs;
}
if (name === 'photos') {
// The whole point: if this runs after the transaction, the rows
// are gone and we would collect nothing.
return mockPhotoRowsDeleted ? [] : mockPhotos;
}
return [];
},
del: async () => {
if (name === 'photos') mockPhotoRowsDeleted = true;
if (name === 'download_jobs') mockJobRowsDeleted = true;
return 1;
},
};
return chain;
};
// #1132 guards the merge-dismissals delete behind a hasTable check.
table.schema = { hasTable: async (t) => t === 'download_jobs' };
table.transaction = async (cb) => cb(table);
return table;
}
jest.mock('../../src/database/db', () => ({
db: mockMakeDb(),
logActivity: jest.fn().mockResolvedValue(undefined),
}));
jest.mock('../../src/services/storage', () => ({
getStorage: () => mockStorage,
}));
const { deleteEventCascade } = require('../../src/routes/adminEvents/helpers');
describe('deleteEventCascade — storage cleanup', () => {
beforeEach(() => {
mockStorage.delete.mockClear();
mockPhotoRowsDeleted = false;
mockJobRowsDeleted = false;
mockSharedDerivatives = [];
});
it('deletes originals and every derived tier from the storage backend', async () => {
await deleteEventCascade(42, { id: 1, username: 'admin' });
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
expect(deleted).toEqual(expect.arrayContaining([
'events/active/other-demo-2026-01-01/photo_one.jpg',
'events/active/other-demo-2026-01-01/photo_two.jpg',
'thumbnails/thumb_aaa_photo_one.jpg',
'thumbnails/thumb_bbb_photo_two.jpg',
'previews/prev_aaa_photo_one.jpg',
]));
});
it('deletes pre-generated watermarks and the archive zip', async () => {
await deleteEventCascade(42, { id: 1, username: 'admin' });
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
// Both are storage-backend objects that only fs.unlink ever touched, so
// both survived an event delete on S3.
expect(deleted).toEqual(expect.arrayContaining([
'watermarked/wm_aaa_photo_one.jpg',
'archives/other-demo-2026-01-01.zip',
]));
});
it('deletes the download caches, which only fs.rm ever covered', async () => {
await deleteEventCascade(42, { id: 1, username: 'admin' });
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
// Both sit under events/active/{slug}/.download-cache/ — swept by the
// recursive fs.rm on local disk, invisible to it on S3 where the prefix
// is not a directory. Both are gallery-sized.
expect(deleted).toEqual(expect.arrayContaining([
'events/active/other-demo-2026-01-01/.download-cache/all.zip',
'events/active/other-demo-2026-01-01/.download-cache/job-abc123.zip',
]));
});
it('leaves a derivative alone when another event still points at it', async () => {
// Canonical thumbnail/hero/preview keys are not event-scoped — the
// basename is the photo's filename, and filenames are not unique across
// events. Deleting one a surviving gallery still references would blank
// its tile.
mockSharedDerivatives = [{
thumbnail_path: 'thumbnails/thumb_aaa_photo_one.jpg',
hero_path: null,
preview_path: null,
watermark_path: null,
}];
await deleteEventCascade(42, { id: 1, username: 'admin' });
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
expect(deleted).not.toContain('thumbnails/thumb_aaa_photo_one.jpg');
// The originals are slug-scoped and must still go.
expect(deleted).toContain('events/active/other-demo-2026-01-01/photo_one.jpg');
// So must a derivative nobody else claims.
expect(deleted).toContain('thumbnails/thumb_bbb_photo_two.jpg');
});
it('never asks the backend to delete the same key twice', async () => {
await deleteEventCascade(42, { id: 1, username: 'admin' });
const managed = mockStorage.delete.mock.calls
.map(([key]) => key)
.filter((key) => !key.startsWith('thumbnails/thumb_w') && !key.startsWith('previews/preview_w'));
expect(managed).toEqual([...new Set(managed)]);
});
it('leaves external/reference photos in place', async () => {
await deleteEventCascade(42, { id: 1, username: 'admin' });
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
expect(deleted).not.toEqual(expect.arrayContaining(['ignored.jpg']));
expect(deleted).not.toEqual(expect.arrayContaining(['events/active/ignored.jpg']));
});
it('still completes the delete when the storage backend throws', async () => {
mockStorage.delete.mockRejectedValue(new Error('bucket unreachable'));
await expect(deleteEventCascade(42, { id: 1, username: 'admin' }))
.resolves.toEqual({ id: 42, name: 'Demo' });
mockStorage.delete.mockResolvedValue(undefined);
});
});