Files
picpeak/backend/__tests__/integration/externalImportCaptureDate.test.js
T
Paul Nothaft 7f0ed23ea4 fix(external-media): record captured_at on import and add a backfill (stable) (#1183)
* fix(external-media): record captured_at on import and add a backfill (#1172)

Stable twin of #1179.

External media never went through photoProcessor, so captured_at stayed NULL
for every externally imported photo. The gallery's "Date Taken" sort then
degraded into import order through its own COALESCE fallback — a library
imported in two batches showed the first days of a trip after the last ones.

Ported whole:
- adminExternalMedia.js reads the capture date at import, off the file it has
  already opened for the dimensions. Best-effort, like the dimensions.
- A backfill endpoint for photos imported before this, so existing installs
  can fix historical rows rather than only new imports. Managed originals go
  through resolvePhotoStorageKey + withLocalCopy so S3 installs work; archived
  events are excluded because archiving deletes their originals; the run flag
  is claimed before the candidate query so two POSTs cannot both start.
- gallery.js carries photos.id as a tiebreaker on all three sorts. A bulk
  import writes hundreds of rows inside the same second, so uploaded_at ties
  are the normal case and the grid reshuffled between page loads.

One deliberate difference from main: the backfill is gated on settings.edit /
settings.view rather than system.manage / system.view, which do not exist on
this branch. They are what settings.edit was later split into, and main's
migration 175 projects every settings.edit holder forward onto system.manage,
so both branches let exactly the same people through.

* fix(external-media): gate the status card on the permission the button needs (#1172)

The built-in admin role holds settings.view but not settings.edit
(056_add_role_permissions_table.js:63), and StatusTab renders its card and
enabled button purely on a successful status payload. Gating the status
endpoint on settings.view therefore showed every admin a Backfill button whose
every click 403s with no error surfaced.

* fix(gallery): make the Date Taken sort correct on SQLite (#1172)

Same defect as the main twin: photos.captured_at holds three storage classes on
SQLite — an epoch-millisecond integer from managed uploads (photoProcessor.js:441
hands knex a Date), ISO text from external imports and the backfill, and null
falling through to uploaded_at's 'YYYY-MM-DD HH:MM:SS' text. SQLite sorts
INTEGER before TEXT unconditionally, so a 2027 capture came back before a 2020
one, and within the text values the 'T' separator outranked the space.

Normalised in the ORDER BY; Postgres keeps the plain COALESCE, its column being
a real timestamp. Regression tests drive the real gallery route on real SQLite.

* fix(gallery): normalise epoch-integer uploaded_at too, and stop polling a 403 (#1172)

Both follow-ups from the main twin's review, ported.

uploaded_at is not always text on SQLite: a .picpeak restore can carry epoch
milliseconds in from an install that stored them that way, and the fallback
branch read it with substr(), comparing '1830297600000' against
'2020-01-01 00:00:00' as text. Both columns now get the integer/real branch.

The status card also polled every ten seconds regardless of permission. On this
branch that hits every built-in admin — they hold settings.view but not
settings.edit — so each would have had a 403 and a logged denial every ten
seconds for a panel they were never shown.

* style: quote convention in the capture-sort test (#1172)

* fix(capture-dates): skip watcher-imported videos, and make the status counts consistent (#1172)

All three follow-ups from the main twin, ported: the three-marker video filter
(fileWatcher sets type/mime but not media_type, so those rows sat in the
backlog forever), the single-aggregate status counts (two queries could report
a negative backlog mid-import), and the card's render gated on settings.edit as
well as the cached payload.

---------

Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
2026-08-26 09:04:18 +02:00

162 lines
6.8 KiB
JavaScript

/**
* External imports must record captured_at (#1172).
*
* Managed uploads get it from photoProcessor, which external media never goes
* through — so every externally imported photo carried captured_at NULL, and
* the gallery's "Date Taken" sort fell back to uploaded_at through its
* COALESCE. On a library imported in two batches that ordered a 12-day trip by
* which folder was imported first: the reporter's first two days landed at
* positions 4204-5296 of 5555.
*
* Driven through the real route against real files carrying real EXIF, because
* the whole question is whether the import reads the file it already has open.
*/
const fs = require('fs');
const path = require('path');
const os = require('os');
const express = require('express');
const request = require('supertest');
const sharp = require('sharp');
describe('external import capture dates (#1172)', () => {
let tmpDir; let db; let app; let mediaRoot;
/**
* A real JPEG carrying DateTimeOriginal.
*
* IFD2, not IFD0 — DateTimeOriginal lives in the Exif IFD, and exifr does not
* see it anywhere else (IFD0 takes plain DateTime, which surfaces as
* ModifyDate instead).
*/
const writeJpegWithExif = async (rel, iso) => {
const full = path.join(mediaRoot, rel);
await fs.promises.mkdir(path.dirname(full), { recursive: true });
const d = new Date(iso);
const pad = (n) => String(n).padStart(2, '0');
const exifDate = `${d.getUTCFullYear()}:${pad(d.getUTCMonth() + 1)}:${pad(d.getUTCDate())} `
+ `${pad(d.getUTCHours())}:${pad(d.getUTCMinutes())}:${pad(d.getUTCSeconds())}`;
await sharp({ create: { width: 60, height: 40, channels: 3, background: { r: 10, g: 20, b: 30 } } })
.withExif({ IFD2: { DateTimeOriginal: exifDate } })
.jpeg()
.toFile(full);
return full;
};
const writeJpegNoExif = async (rel) => {
const full = path.join(mediaRoot, rel);
await fs.promises.mkdir(path.dirname(full), { recursive: true });
await sharp({ create: { width: 60, height: 40, channels: 3, background: { r: 200, g: 10, b: 10 } } })
.jpeg().toFile(full);
};
beforeAll(async () => {
tmpDir = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'picpeak-capdate-'));
mediaRoot = path.join(tmpDir, 'media');
await fs.promises.mkdir(mediaRoot, { recursive: true });
process.env.NODE_ENV = 'test';
process.env.TEST_DATABASE_PATH = path.join(tmpDir, 'data', 'db.sqlite');
await fs.promises.mkdir(path.dirname(process.env.TEST_DATABASE_PATH), { recursive: true });
process.env.STORAGE_PATH = path.join(tmpDir, 'storage');
process.env.EXTERNAL_MEDIA_ROOT = mediaRoot;
process.env.JWT_SECRET = process.env.JWT_SECRET || 'capdate-secret';
jest.resetModules();
jest.doMock('../../src/middleware/auth', () => ({
adminAuth: (req, _res, next) => { req.admin = { id: 1, username: 'tester', roleName: 'admin' }; next(); },
}));
jest.doMock('../../src/middleware/permissions', () => ({
requirePermission: () => (_req, _res, next) => next(),
}));
jest.doMock('../../src/middleware/ownership', () => ({
requireEventOwnership: (_req, _res, next) => next(),
}));
jest.doMock('../../src/services/imageProcessor', () => {
const actual = jest.requireActual('../../src/services/imageProcessor');
return { ...actual, generateThumbnail: jest.fn(async () => 'thumbnails/mock.jpg'), ensureThumbnail: jest.fn() };
});
jest.doMock('../../src/utils/logger', () => ({
debug: jest.fn(), info: jest.fn(), warn: jest.fn(), error: jest.fn(),
}));
({ db } = await require('./helpers/crmDb').bootCrmDb());
app = express();
app.use(express.json());
app.use('/api/admin/external-media', require('../../src/routes/adminExternalMedia'));
}, 180000);
afterAll(async () => {
if (db) await db.destroy?.();
await fs.promises.rm(tmpDir, { recursive: true, force: true }).catch(() => {});
});
async function seedEvent() {
await db('photos').del();
await db('events').del();
await fs.promises.rm(mediaRoot, { recursive: true, force: true });
await fs.promises.mkdir(mediaRoot, { recursive: true });
const [e] = await db('events').insert({
slug: `capdate-${Math.random().toString(36).slice(2, 8)}`,
event_type: 'wedding', event_name: 'capdate', event_date: '2026-01-01',
host_email: 'h@example.com', admin_email: 'a@example.com', password_hash: 'x',
share_link: `capdate-${Math.random()}`, expires_at: new Date().toISOString(),
source_mode: 'reference',
}).returning('id');
return typeof e === 'object' ? e.id : e;
}
const runImport = (eventId, external_path) => request(app)
.post(`/api/admin/external-media/events/${eventId}/import-external`)
.send({ external_path, recursive: true });
it('records the EXIF capture date on import', async () => {
const eventId = await seedEvent();
await writeJpegWithExif('trip/a.jpg', '2026-06-01T09:45:03Z');
await runImport(eventId, 'trip');
const photo = await db('photos').where({ event_id: eventId }).first();
expect(photo.captured_at).toBeTruthy();
// NOT asserted as an absolute instant. EXIF carries a naive wall-clock
// time and exifr resolves it against the HOST timezone, so the stored UTC
// value differs between a CEST developer machine and a UTC runner. What
// this fix is about is that the field is populated and orders correctly;
// that captured_at is not a true instant is a separate, pre-existing
// problem shared with managed uploads (#1172's own footnote).
expect(new Date(photo.captured_at).getUTCFullYear()).toBe(2026);
expect(new Date(photo.captured_at).getUTCMonth()).toBe(5); // June
});
it('imports a photo with no EXIF date rather than failing it', async () => {
// Plenty of sources carry none; that must stay an import, not an error.
const eventId = await seedEvent();
await writeJpegNoExif('trip/plain.jpg');
const res = await runImport(eventId, 'trip');
expect(res.body.imported).toBe(1);
const photo = await db('photos').where({ event_id: eventId }).first();
expect(photo.captured_at).toBeNull();
});
it('orders a two-batch import by capture time, not by batch', async () => {
// The reported shape: the FIRST days of the trip imported second. Sorting
// on COALESCE(captured_at, uploaded_at) put them after the last days,
// because uploaded_at is the import timestamp.
const eventId = await seedEvent();
await writeJpegWithExif('late/day12.jpg', '2026-06-12T10:00:00Z');
await runImport(eventId, 'late');
await writeJpegWithExif('early/day01.jpg', '2026-06-01T10:00:00Z');
await runImport(eventId, 'early');
const rows = await db('photos')
.where({ event_id: eventId })
.orderByRaw('COALESCE(captured_at, uploaded_at) asc')
.select('filename');
expect(rows.map((r) => r.filename)).toEqual(['day01.jpg', 'day12.jpg']);
});
});