paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(external-media): record captured_at on import and add a backfill (stable) (#1183)

Browse Source 
* fix(external-media): record captured_at on import and add a backfill (#1172)

Stable twin of #1179.

External media never went through photoProcessor, so captured_at stayed NULL
for every externally imported photo. The gallery's "Date Taken" sort then
degraded into import order through its own COALESCE fallback — a library
imported in two batches showed the first days of a trip after the last ones.

Ported whole:
- adminExternalMedia.js reads the capture date at import, off the file it has
  already opened for the dimensions. Best-effort, like the dimensions.
- A backfill endpoint for photos imported before this, so existing installs
  can fix historical rows rather than only new imports. Managed originals go
  through resolvePhotoStorageKey + withLocalCopy so S3 installs work; archived
  events are excluded because archiving deletes their originals; the run flag
  is claimed before the candidate query so two POSTs cannot both start.
- gallery.js carries photos.id as a tiebreaker on all three sorts. A bulk
  import writes hundreds of rows inside the same second, so uploaded_at ties
  are the normal case and the grid reshuffled between page loads.

One deliberate difference from main: the backfill is gated on settings.edit /
settings.view rather than system.manage / system.view, which do not exist on
this branch. They are what settings.edit was later split into, and main's
migration 175 projects every settings.edit holder forward onto system.manage,
so both branches let exactly the same people through.

* fix(external-media): gate the status card on the permission the button needs (#1172)

The built-in admin role holds settings.view but not settings.edit
(056_add_role_permissions_table.js:63), and StatusTab renders its card and
enabled button purely on a successful status payload. Gating the status
endpoint on settings.view therefore showed every admin a Backfill button whose
every click 403s with no error surfaced.

* fix(gallery): make the Date Taken sort correct on SQLite (#1172)

Same defect as the main twin: photos.captured_at holds three storage classes on
SQLite — an epoch-millisecond integer from managed uploads (photoProcessor.js:441
hands knex a Date), ISO text from external imports and the backfill, and null
falling through to uploaded_at's 'YYYY-MM-DD HH:MM:SS' text. SQLite sorts
INTEGER before TEXT unconditionally, so a 2027 capture came back before a 2020
one, and within the text values the 'T' separator outranked the space.

Normalised in the ORDER BY; Postgres keeps the plain COALESCE, its column being
a real timestamp. Regression tests drive the real gallery route on real SQLite.

* fix(gallery): normalise epoch-integer uploaded_at too, and stop polling a 403 (#1172)

Both follow-ups from the main twin's review, ported.

uploaded_at is not always text on SQLite: a .picpeak restore can carry epoch
milliseconds in from an install that stored them that way, and the fallback
branch read it with substr(), comparing '1830297600000' against
'2020-01-01 00:00:00' as text. Both columns now get the integer/real branch.

The status card also polled every ten seconds regardless of permission. On this
branch that hits every built-in admin — they hold settings.view but not
settings.edit — so each would have had a 403 and a logged denial every ten
seconds for a panel they were never shown.

* style: quote convention in the capture-sort test (#1172)

* fix(capture-dates): skip watcher-imported videos, and make the status counts consistent (#1172)

All three follow-ups from the main twin, ported: the three-marker video filter
(fileWatcher sets type/mime but not media_type, so those rows sat in the
backlog forever), the single-aggregate status counts (two queries could report
a negative backlog mid-import), and the card's render gated on settings.edit as
well as the cached payload.

---------

Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
2026-08-26 09:04:18 +02:00
committed by GitHub
co-authored by Paul Nothaft
parent 2b1c3588ae
commit 7f0ed23ea4
11 changed files with 1005 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/adminExternalMedia.js
+28 -2
View File
@@ -8,7 +8,7 @@ const { list, resolveExternalPath, getExternalMediaRoot } = require('../services
const { db, logActivity } = require('../database/db');
const sharp = require('sharp');
const logger = require('../utils/logger');
const { generateThumbnail } = require('../services/imageProcessor');
const { generateThumbnail, extractCaptureDate } = require('../services/imageProcessor');
const { isUniqueViolation } = require('../utils/dbErrors');
const router = express.Router();
@@ -162,6 +162,27 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
logger.warn(`Could not extract dimensions for ${f.rel}: ${dimErr.message}`);
}
// Capture date from EXIF (#1172). Managed uploads get this from
// photoProcessor, which external media never goes through — so
// captured_at stayed NULL for every externally imported photo, and the
// gallery's "Date Taken" sort silently degraded into import order via
// its COALESCE fallback. On a library imported in two batches that put
// the first days of a trip after the last ones.
//
// Read here because the file is already open a few lines above for the
// dimensions, so this costs one more read of the same source rather
// than a second pass over the mount.
//
// Best-effort, exactly like the dimensions: a source without EXIF, or
// one Sharp/exifr cannot parse, imports with captured_at NULL and
// falls back to uploaded_at as before.
let capturedAt = null;
try {
capturedAt = await extractCaptureDate(f.full);
} catch (dateErr) {
logger.warn(`Could not extract capture date for ${f.rel}: ${dateErr.message}`);
}
let inserted;
try {
inserted = await db('photos')
@@ -176,7 +197,12 @@ router.post('/events/:id/import-external', adminAuth, requirePermission('photos.
width,
height,
source_origin: 'external',
external_relpath: relFromRoot
external_relpath: relFromRoot,
// .toISOString() rather than the Date: inside jest, Dates handed
// to the sqlite3 binding land as the literal string
// "[object Object]" (see CLAUDE.md). Strings round-trip on both
// engines.
captured_at: capturedAt ? capturedAt.toISOString() : null
})
.returning('id');
} catch (insertErr) {
backend/src/routes/adminPhotoDimensions.js
+245
View File
@@ -14,6 +14,14 @@ let repairProgress = {
lastResult: null
};
// Same shape, separate state: the two jobs walk the same photos but read
// different things out of them, and one running must not block or report for
// the other.
let captureDateProgress = {
isRunning: false,
lastResult: null
};
// Repair photo dimensions (background job)
router.post('/repair-dimensions', adminAuth, requirePermission('photos.edit'), async (req, res) => {
try {
@@ -152,4 +160,241 @@ router.get('/repair-dimensions/status', adminAuth, requirePermission('photos.vie
}
});
/**
* Backfill captured_at from EXIF (#1172).
*
* External imports never read EXIF before this release, so every
* externally-imported photo carries captured_at NULL — and the gallery's
* "Date Taken" sort falls back to uploaded_at, which on a bulk import is the
* import timestamp. A 5555-photo trip came back ordered by which folder was
* imported first.
*
* Deliberately an endpoint rather than a migration: the originals live on a
* mount that may be unavailable at upgrade time, reading 8000+ of them blocks
* the boot, and a run that found nothing needs to be repeatable once the mount
* is back. Same reasoning, and the same shape, as the dimension repair above —
* including resolvePhotoFilePath, which is what makes it work for external
* rows at all (the thumbnail regenerator resolves under
* storage/events/active/<path>, which never exists for them).
*/
// settings.edit, not photos.edit: this walks every event in the install and
// rewrites their metadata, which is a maintenance action rather than a photo
// edit. photos.edit is held by the `editor` role
// (056_add_role_permissions_table.js:73), which is scoped to contributing
// content, not to running an install-wide S3/NAS scan across other people's
// events. settings.edit is the restrictive one here — `admin` carries only
// settings.view (056:63).
//
// main gates the same endpoint on system.manage, which does not exist on this
// branch: it is one of the permissions settings.edit was later split into, and
// migration 175 there projects every settings.edit holder forward onto it. So
// this is the same gate under its older name, and the two branches let exactly
// the same people through.
router.post('/repair-capture-dates', adminAuth, requirePermission('settings.edit'), async (req, res) => {
try {
if (captureDateProgress.isRunning) {
return res.status(409).json({ error: 'Capture date backfill is already running' });
}
// Claimed here, not after the candidate query: that query is an await, and
// two POSTs arriving inside it would both read isRunning === false and both
// start a pass over the same rows. Every early exit below has to release it
// again, hence the try/catch around the query.
captureDateProgress.isRunning = true;
captureDateProgress.lastResult = null;
let photos;
try {
photos = await db('photos')
.join('events', 'photos.event_id', 'events.id')
.whereNull('photos.captured_at')
// Three markers, because no single one is reliable. fileWatcher's
// auto-import sets type='video' and a video/* mime but never
// media_type (fileWatcher.js:128-130), so those rows keep the 'image'
// default from migration 048 and a media_type-only filter queues them
// forever: extractCaptureDate returns null for a video, captured_at
// stays null, and every run picks it up again.
.where(function () {
this.where('photos.media_type', '!=', 'video').orWhereNull('photos.media_type');
})
.where(function () {
this.where('photos.type', '!=', 'video').orWhereNull('photos.type');
})
.where(function () {
this.whereNull('photos.mime_type').orWhere('photos.mime_type', 'not like', 'video/%');
})
// Archiving deletes the originals from storage but keeps the photos
// rows (archiveService.js:166,199). Those files are inside the zip and
// nothing here can read them, so including them would fail every row
// on every run and leave the button permanently lit.
.where(function () {
this.where('events.is_archived', false).orWhereNull('events.is_archived');
})
.select(
'photos.id', 'photos.path', 'photos.filename',
'photos.source_origin', 'photos.external_relpath', 'photos.event_id',
'events.source_mode', 'events.external_path', 'events.slug'
);
} catch (err) {
captureDateProgress.isRunning = false;
throw err;
}
if (photos.length === 0) {
captureDateProgress.isRunning = false;
return res.json({ message: 'No photos need a capture date', count: 0 });
}
res.json({
message: `Started backfilling capture dates for ${photos.length} photos`,
count: photos.length
});
setImmediate(async () => {
const { extractCaptureDate, withLocalCopy } = require('../services/imageProcessor');
const { resolvePhotoStorageKey } = require('../services/photoResolver');
let successCount = 0;
let missingCount = 0;
let errorCount = 0;
for (const photo of photos) {
try {
const event = { source_mode: photo.source_mode, external_path: photo.external_path, slug: photo.slug };
const isExternal = photo.source_origin === 'external' || photo.source_origin === 'reference';
// Two source shapes, same split the thumbnail regenerator uses
// (imageProcessor.js:391-420). External rows live on a local mount
// and are read directly; managed rows live behind the storage
// backend, which on an S3 install is not a filesystem at all — going
// through resolvePhotoFilePath there would build a STORAGE_PATH that
// holds nothing and fail every managed photo.
let captured;
if (isExternal) {
let fullPath;
try {
fullPath = resolvePhotoFilePath(event, photo);
} catch (err) {
logger.warn(`Photo ${photo.id} has no resolvable path, skipping capture date: ${err.message}`);
errorCount++;
continue;
}
try {
await fs.access(fullPath);
} catch (err) {
logger.warn(`File not found for photo ${photo.id}: ${fullPath}`);
errorCount++;
continue;
}
captured = await extractCaptureDate(fullPath);
} else {
let sourceKey;
try {
sourceKey = resolvePhotoStorageKey(event, photo);
} catch (err) {
logger.warn(`Photo ${photo.id} has no resolvable storage key, skipping capture date: ${err.message}`);
errorCount++;
continue;
}
// In local-fs mode withLocalCopy hands back the resolved path
// without checking it exists, so the access probe stays. In S3
// mode a missing object throws out of getToFile and lands in the
// outer catch — both end up counted as failures, which is what a
// missing original is.
captured = await withLocalCopy(sourceKey, async (localPath) => {
await fs.access(localPath);
return extractCaptureDate(localPath);
});
}
if (!captured) {
// No date recovered. Usually genuine — plenty of sources carry no
// EXIF — but extractCaptureDate also returns null when the file is
// unreadable as an image, so this bucket is "nothing to write",
// not "definitely has no EXIF". The failure counter above is the
// one that means the storage is broken.
missingCount++;
continue;
}
// whereNull, not a blanket set: the job can run for a long time on a
// large library, and an import or a replacement finishing meanwhile
// has already written a date this pass would otherwise overwrite
// with the same-or-worse value.
const updated = await db('photos')
.where({ id: photo.id })
.whereNull('captured_at')
.update({ captured_at: captured.toISOString() });
if (updated) successCount++;
if (successCount % 50 === 0 && successCount > 0) {
logger.info(`Capture date backfill progress: ${successCount} updated...`);
}
} catch (error) {
logger.error(`Error backfilling capture date for photo ${photo.id}:`, error);
errorCount++;
}
}
captureDateProgress.isRunning = false;
captureDateProgress.lastResult = { success: successCount, noExif: missingCount, failed: errorCount };
logger.info(`Capture date backfill complete: ${successCount} updated, ${missingCount} without EXIF, ${errorCount} errors`);
});
} catch (error) {
logger.error('Error starting capture date backfill:', error);
res.status(500).json({ error: 'Failed to start capture date backfill' });
}
});
// The same permission as the POST, not the read-only settings.view. The built-in
// `admin` role holds settings.view but not settings.edit
// (056_add_role_permissions_table.js:63), and StatusTab has no permission gate
// of its own — a successful status payload is what renders the card and its
// enabled button. Gating on settings.view therefore showed every admin a live
// backfill button whose every click 403s with no error surfaced.
router.get('/repair-capture-dates/status', adminAuth, requirePermission('settings.edit'), async (req, res) => {
try {
// Same scope as the job itself — counting archived photos here would show
// a permanent backlog the button can never clear.
const scoped = () => db('photos')
.join('events', 'photos.event_id', 'events.id')
.where(function () {
this.where('photos.media_type', '!=', 'video').orWhereNull('photos.media_type');
})
.where(function () {
this.where('photos.type', '!=', 'video').orWhereNull('photos.type');
})
.where(function () {
this.whereNull('photos.mime_type').orWhere('photos.mime_type', 'not like', 'video/%');
})
.where(function () {
this.where('events.is_archived', false).orWhereNull('events.is_archived');
});
// One query, two aggregates. As two separate counts an import committing a
// dated photo between them could be counted by the second and not the
// first, so withCaptureDate came out larger than total and the card showed
// a negative backlog — with the button enabled to "fix" it.
const counts = await scoped()
.count('photos.id as total')
.count({ dated: db.raw('CASE WHEN photos.captured_at IS NOT NULL THEN 1 END') })
.first();
const total = Number(counts.total);
const withCaptureDate = Number(counts.dated);
res.json({
total,
withCaptureDate,
withoutCaptureDate: total - withCaptureDate,
isRunning: captureDateProgress.isRunning,
lastResult: captureDateProgress.lastResult
});
} catch (error) {
logger.error('Error fetching capture date backfill status:', error);
res.status(500).json({ error: 'Failed to fetch capture date backfill status' });
}
});
module.exports = router;
backend/src/routes/gallery.js
+56 -5
View File
@@ -447,15 +447,66 @@ router.get('/:slug/photos', verifyGalleryAccess, resolveGuest, async (req, res)
});
}
// Apply sort option
// Apply sort option.
//
// Every branch carries photos.id as a tiebreaker (#1172). Without one the
// order within a tie is whatever the engine happens to return, and ties are
// the normal case rather than the exception: a bulk import writes hundreds
// of rows inside the same second, so uploaded_at collapses — and with
// captured_at NULL the COALESCE below collapses onto it too. The visible
// symptom is a grid that reshuffles between page loads. id is insertion
// order, so it also makes the fallback ordering meaningful rather than
// arbitrary.
if (sort === 'capture_date') {
// Sort by capture date, falling back to uploaded_at if capture date is null
photosQuery = photosQuery.orderByRaw('COALESCE(photos.captured_at, photos.uploaded_at) ' + sortOrder);
// Sort by capture date, falling back to uploaded_at if capture date is null.
//
// On SQLite that fallback cannot be a plain COALESCE, because the two
// columns do not hold one type. photos.captured_at ends up carrying three
// different storage classes:
//
// integer managed uploads — photoProcessor.js:441 writes a Date, which
// the sqlite3 binding stores as epoch milliseconds
// text external imports and the backfill, which write ISO-8601
// ('2026-06-03T01:15:00.000Z') per the CLAUDE.md rule that
// Dates must not be handed to the binding in tests
// null no capture date, so the sort falls through to uploaded_at —
// usually text in knex's 'YYYY-MM-DD HH:MM:SS' default shape,
// but epoch milliseconds on rows a .picpeak restore carried in
// from an install that stored them that way, so that column
// needs the same two branches
//
// SQLite orders INTEGER before TEXT unconditionally, so every managed
// photo carrying EXIF sorted ahead of every photo that did not, whatever
// the actual dates — a 2027 capture landing before a 2020 one. Among the
// text values the 'T' separator (0x54) also outranks the space (0x20), so
// a same-day ISO 01:15 sorted after a fallback 23:00.
//
// Normalising in the ORDER BY rather than rewriting the column: the data
// fix would have to touch every existing row and every writer, which is a
// much heavier change than the sort it is meant to correct. The cost here
// is that this sort stops using idx_photos_captured_at on SQLite — an
// acceptable trade on the fallback engine, where the alternative is an
// index-assisted wrong answer.
//
// Postgres is untouched: captured_at is a real timestamp there, so
// COALESCE already compares correctly.
if (db.client.config.client === 'pg') {
photosQuery = photosQuery
.orderByRaw('COALESCE(photos.captured_at, photos.uploaded_at) ' + sortOrder);
} else {
photosQuery = photosQuery.orderByRaw(`CASE
WHEN typeof(photos.captured_at) IN ('integer', 'real') THEN datetime(photos.captured_at / 1000, 'unixepoch')
WHEN photos.captured_at IS NOT NULL THEN replace(replace(substr(photos.captured_at, 1, 19), 'T', ' '), 'Z', '')
WHEN typeof(photos.uploaded_at) IN ('integer', 'real') THEN datetime(photos.uploaded_at / 1000, 'unixepoch')
ELSE substr(photos.uploaded_at, 1, 19)
END ${sortOrder}`);
}
photosQuery = photosQuery.orderBy('photos.id', sortOrder);
} else if (sort === 'filename') {
photosQuery = photosQuery.orderBy('photos.filename', sortOrder);
photosQuery = photosQuery.orderBy('photos.filename', sortOrder).orderBy('photos.id', sortOrder);
} else {
// Default: sort by upload date
photosQuery = photosQuery.orderBy('photos.uploaded_at', sortOrder);
photosQuery = photosQuery.orderBy('photos.uploaded_at', sortOrder).orderBy('photos.id', sortOrder);
}
// Execute the query