202c553a08
* fix(events): delete stored objects when cascading an event delete
* fix(events): sweep watermarks and the archive zip on cascade delete too
Two more objects in the same class as the originals: both are written
through the storage backend, both were only ever removed with fs.unlink,
so both outlive the event on S3.
- photo.watermark_path — a canonical key, deleted via getStorage() on the
single-photo path (watermarkService.deleteWatermarkFile) and on archive
(archiveService.js:227). The cascade neither selected nor removed it.
- event.archive_path — written by storage.putFromFile (archiveService.js:160)
and typically the largest single object an event owns.
event.hero_logo_path is deliberately NOT included: multer writes logos to
local disk with diskStorage regardless of backend (adminEvents/logo.js:19-28),
so they are never bucket objects and the existing fs.unlink is correct.
Collect into a Set — an unresized gallery can carry one object in both
hero_path and preview_path, and the second delete would log a spurious
failure.
* fix(events): sweep the download caches, and delete objects concurrently
Both from an external review round on this PR.
The download caches are the subtle case: the pre-built "Download All" zip
(events.download_zip_path) and one zip per custom-resolution download job
(download_jobs.zip_path) both live under
events/active/{slug}/.download-cache/. On local disk the recursive fs.rm
already covered them, which is exactly why they were easy to miss — on S3
that prefix is not a directory, nothing covered them, and both are
gallery-sized. downloadZipService exposes a cleanup() documented as "used
on event deletion" that the cascade never called.
The job rows are read before the transaction for the same reason the photo
rows are: download_jobs.event_id is ON DELETE CASCADE, so on Postgres they
vanish with the event and take their keys with them. Guarded with hasTable
so a pre-#173 install doesn't abort the delete.
Deletes now run through a bounded pool instead of one await per key. A
400-photo gallery owns ~1600 objects once derived tiers are counted, and
that many sequential DeleteObject round trips runs to minutes — long enough
for a proxy to time the request out AFTER the commit, leaving the event
deleted and the sweep half-finished. A pool rather than Promise.all over
every key, so the fan-out can't exhaust the S3 client's connection pool.
* fix(events): never delete a derivative another gallery still uses
Round-2 findings from the external reviewer.
Canonical thumbnail/hero/preview keys are not event-scoped: the basename is
the photo's filename (imageProcessor passes no outputBasename for managed
photos, so the key is thumbnails/thumb_w300_<filename>), and filenames are
not unique across events — the responsive-tier code says so in as many
words, which is why THOSE keys carry a p{id}_ prefix. A legacy gallery can
therefore share a canonical derivative with a photo in another event, and
deleting it here blanked a surviving gallery's tile. Derived keys are now
checked against photos outside this event and anything still referenced is
left alone; if the check itself fails, every derivative is kept. An orphan
costs storage, a deleted derivative costs someone else's gallery. Originals
need no check — their keys embed the slug.
Also cancel any in-flight or debounced Download All build before snapshotting
paths. A builder that started before the delete would otherwise upload a
gallery-sized zip after the sweep and write its path onto a row that no
longer exists, orphaning it permanently. downloadZipService.cleanup() is the
service's own entry point for this and does all three things: bumps the
version so an in-flight build discards its result, clears the debounce so
nothing rebuilds for a deleted event, and removes the current object.
* revert(events): drop the Download All build cancellation
Reverted for the same reason as on the stable twin, where it was caught:
downloadZipService.cleanup() reaches getStorage() through _cleanup(), so
where the S3 backend is configured but unreachable every cascade delete pays
the adapter's retry backoff. On stable that took the backend CI job from ~2
minutes to past its 10-minute budget, twice, reproducibly. This branch's
suite happened not to trip it, but the same cost lands in the request path
of a real delete — and the twins have to carry the same code.
The race it addressed is narrow and costs one orphaned zip; documented as a
follow-up instead. The shared-derivative guard from the same review round
stays — that one prevented deleting a surviving gallery's thumbnail.
---------
Co-authored-by: Peifu Mo <peipeimo@Peifus-MacBook-Pro.local>
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
232 lines
8.3 KiB
JavaScript
232 lines
8.3 KiB
JavaScript
/**
|
|
* Regression test: deleting an event must remove its stored objects.
|
|
*
|
|
* deleteEventCascade() cleaned up the local filesystem only (#608). On an
|
|
* S3/R2 storage backend that cleanup is a no-op, so every deleted gallery
|
|
* left its originals and derived tiers in the bucket — unreferenced,
|
|
* invisible in the UI, and billed forever. Measured on a v3.45.16 install
|
|
* against Cloudflare R2: deleting a 403-photo event changed the bucket
|
|
* object count by exactly zero.
|
|
*
|
|
* The keys must be collected BEFORE the transaction deletes the photo
|
|
* rows, because afterwards nothing knows which objects were this event's.
|
|
*/
|
|
|
|
const os = require('os');
|
|
const path = require('path');
|
|
|
|
// The cascade runs a real `fs.rm(..., { recursive: true })` over
|
|
// {STORAGE_PATH}/events/{active,archived}/{slug}. Point that at a throwaway
|
|
// directory before requiring the module under test — the default resolves
|
|
// into the working tree.
|
|
process.env.STORAGE_PATH = path.join(os.tmpdir(), 'picpeak-cascade-storage-test');
|
|
|
|
const mockStorage = { delete: jest.fn().mockResolvedValue(undefined) };
|
|
|
|
const mockEvent = {
|
|
id: 42,
|
|
slug: 'other-demo-2026-01-01',
|
|
event_name: 'Demo',
|
|
source_mode: 'managed',
|
|
// Written through the backend by archiveService, so it is a bucket object
|
|
// and the fs.unlink in the cascade never touched it on S3.
|
|
archive_path: 'archives/other-demo-2026-01-01.zip',
|
|
// The pre-built "Download All" zip. Lives under the event prefix, so the
|
|
// recursive fs.rm covers it on local disk and nothing covers it on S3.
|
|
download_zip_path: 'events/active/other-demo-2026-01-01/.download-cache/all.zip',
|
|
};
|
|
|
|
// One zip per custom-resolution download job, same prefix. The rows are
|
|
// ON DELETE CASCADE, so they must be read before the transaction.
|
|
const mockDownloadJobs = [
|
|
{ zip_path: 'events/active/other-demo-2026-01-01/.download-cache/job-abc123.zip' },
|
|
];
|
|
const mockPhotos = [
|
|
{
|
|
id: 1,
|
|
path: 'other-demo-2026-01-01/photo_one.jpg',
|
|
thumbnail_path: 'thumbnails/thumb_aaa_photo_one.jpg',
|
|
hero_path: null,
|
|
preview_path: 'previews/prev_aaa_photo_one.jpg',
|
|
watermark_path: 'watermarked/wm_aaa_photo_one.jpg',
|
|
source_origin: 'managed',
|
|
},
|
|
{
|
|
id: 2,
|
|
path: 'other-demo-2026-01-01/photo_two.jpg',
|
|
thumbnail_path: 'thumbnails/thumb_bbb_photo_two.jpg',
|
|
hero_path: null,
|
|
preview_path: null,
|
|
watermark_path: null,
|
|
source_origin: 'managed',
|
|
},
|
|
{
|
|
// External photos live outside the managed backend and must be left alone.
|
|
id: 3,
|
|
path: 'ignored.jpg',
|
|
thumbnail_path: null,
|
|
hero_path: null,
|
|
preview_path: null,
|
|
watermark_path: null,
|
|
source_origin: 'external',
|
|
},
|
|
];
|
|
|
|
let mockPhotoRowsDeleted = false;
|
|
let mockJobRowsDeleted = false;
|
|
|
|
// Photos in OTHER events that share a canonical derivative key with this one.
|
|
let mockSharedDerivatives = [];
|
|
|
|
// The shared-derivative probe: db('photos').whereNot(...).where(cb).select(...)
|
|
const sharedProbe = {
|
|
where: () => sharedProbe,
|
|
whereIn: () => sharedProbe,
|
|
orWhereIn: () => sharedProbe,
|
|
select: async () => mockSharedDerivatives,
|
|
};
|
|
|
|
function mockMakeDb() {
|
|
const table = (name) => {
|
|
const chain = {
|
|
where: () => chain,
|
|
first: async () => (name === 'events' ? mockEvent : undefined),
|
|
whereNotNull: () => chain,
|
|
whereNot: () => sharedProbe,
|
|
orWhereIn: () => chain,
|
|
whereIn: () => chain,
|
|
select: async () => {
|
|
if (name === 'download_jobs') {
|
|
return mockJobRowsDeleted ? [] : mockDownloadJobs;
|
|
}
|
|
if (name === 'photos') {
|
|
// The whole point: if this runs after the transaction, the rows
|
|
// are gone and we would collect nothing.
|
|
return mockPhotoRowsDeleted ? [] : mockPhotos;
|
|
}
|
|
return [];
|
|
},
|
|
del: async () => {
|
|
if (name === 'photos') mockPhotoRowsDeleted = true;
|
|
if (name === 'download_jobs') mockJobRowsDeleted = true;
|
|
return 1;
|
|
},
|
|
};
|
|
return chain;
|
|
};
|
|
// #1132 guards the merge-dismissals delete behind a hasTable check.
|
|
table.schema = { hasTable: async (t) => t === 'download_jobs' };
|
|
table.transaction = async (cb) => cb(table);
|
|
return table;
|
|
}
|
|
|
|
jest.mock('../../src/database/db', () => ({
|
|
db: mockMakeDb(),
|
|
logActivity: jest.fn().mockResolvedValue(undefined),
|
|
}));
|
|
|
|
jest.mock('../../src/services/storage', () => ({
|
|
getStorage: () => mockStorage,
|
|
}));
|
|
|
|
const { deleteEventCascade } = require('../../src/routes/adminEvents/helpers');
|
|
|
|
describe('deleteEventCascade — storage cleanup', () => {
|
|
beforeEach(() => {
|
|
mockStorage.delete.mockClear();
|
|
mockPhotoRowsDeleted = false;
|
|
mockJobRowsDeleted = false;
|
|
mockSharedDerivatives = [];
|
|
});
|
|
|
|
it('deletes originals and every derived tier from the storage backend', async () => {
|
|
await deleteEventCascade(42, { id: 1, username: 'admin' });
|
|
|
|
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
|
|
|
|
expect(deleted).toEqual(expect.arrayContaining([
|
|
'events/active/other-demo-2026-01-01/photo_one.jpg',
|
|
'events/active/other-demo-2026-01-01/photo_two.jpg',
|
|
'thumbnails/thumb_aaa_photo_one.jpg',
|
|
'thumbnails/thumb_bbb_photo_two.jpg',
|
|
'previews/prev_aaa_photo_one.jpg',
|
|
]));
|
|
});
|
|
|
|
it('deletes pre-generated watermarks and the archive zip', async () => {
|
|
await deleteEventCascade(42, { id: 1, username: 'admin' });
|
|
|
|
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
|
|
|
|
// Both are storage-backend objects that only fs.unlink ever touched, so
|
|
// both survived an event delete on S3.
|
|
expect(deleted).toEqual(expect.arrayContaining([
|
|
'watermarked/wm_aaa_photo_one.jpg',
|
|
'archives/other-demo-2026-01-01.zip',
|
|
]));
|
|
});
|
|
|
|
it('deletes the download caches, which only fs.rm ever covered', async () => {
|
|
await deleteEventCascade(42, { id: 1, username: 'admin' });
|
|
|
|
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
|
|
|
|
// Both sit under events/active/{slug}/.download-cache/ — swept by the
|
|
// recursive fs.rm on local disk, invisible to it on S3 where the prefix
|
|
// is not a directory. Both are gallery-sized.
|
|
expect(deleted).toEqual(expect.arrayContaining([
|
|
'events/active/other-demo-2026-01-01/.download-cache/all.zip',
|
|
'events/active/other-demo-2026-01-01/.download-cache/job-abc123.zip',
|
|
]));
|
|
});
|
|
|
|
it('leaves a derivative alone when another event still points at it', async () => {
|
|
// Canonical thumbnail/hero/preview keys are not event-scoped — the
|
|
// basename is the photo's filename, and filenames are not unique across
|
|
// events. Deleting one a surviving gallery still references would blank
|
|
// its tile.
|
|
mockSharedDerivatives = [{
|
|
thumbnail_path: 'thumbnails/thumb_aaa_photo_one.jpg',
|
|
hero_path: null,
|
|
preview_path: null,
|
|
watermark_path: null,
|
|
}];
|
|
|
|
await deleteEventCascade(42, { id: 1, username: 'admin' });
|
|
|
|
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
|
|
expect(deleted).not.toContain('thumbnails/thumb_aaa_photo_one.jpg');
|
|
// The originals are slug-scoped and must still go.
|
|
expect(deleted).toContain('events/active/other-demo-2026-01-01/photo_one.jpg');
|
|
// So must a derivative nobody else claims.
|
|
expect(deleted).toContain('thumbnails/thumb_bbb_photo_two.jpg');
|
|
});
|
|
|
|
it('never asks the backend to delete the same key twice', async () => {
|
|
await deleteEventCascade(42, { id: 1, username: 'admin' });
|
|
|
|
const managed = mockStorage.delete.mock.calls
|
|
.map(([key]) => key)
|
|
.filter((key) => !key.startsWith('thumbnails/thumb_w') && !key.startsWith('previews/preview_w'));
|
|
|
|
expect(managed).toEqual([...new Set(managed)]);
|
|
});
|
|
|
|
it('leaves external/reference photos in place', async () => {
|
|
await deleteEventCascade(42, { id: 1, username: 'admin' });
|
|
|
|
const deleted = mockStorage.delete.mock.calls.map(([key]) => key);
|
|
expect(deleted).not.toEqual(expect.arrayContaining(['ignored.jpg']));
|
|
expect(deleted).not.toEqual(expect.arrayContaining(['events/active/ignored.jpg']));
|
|
});
|
|
|
|
it('still completes the delete when the storage backend throws', async () => {
|
|
mockStorage.delete.mockRejectedValue(new Error('bucket unreachable'));
|
|
|
|
await expect(deleteEventCascade(42, { id: 1, username: 'admin' }))
|
|
.resolves.toEqual({ id: 42, name: 'Demo' });
|
|
|
|
mockStorage.delete.mockResolvedValue(undefined);
|
|
});
|
|
});
|