5db0a76cce94de03f86295ba2bd6ba526661d16d
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul Nothaft
|
cec8eff70c |
fix(images): fence the capture-date backfill on the file it read (#1201) (#1204)
The capture-date backfill committed its result keyed on the row id alone. It snapshots every candidate up front, then walks them one at a time reading originals off S3 or a NAS mount — a pass that can run for many minutes. replacePhoto, reachable from the replace_by_name upload path, swaps a NEW file under an existing row and rewrites path/filename. A replacement landing inside that window carries no date of its own, so captured_at was still NULL, the whereNull guard passed, and the previous file's EXIF date was written onto the new photo. Silent: nothing errored, the run reported it as a success, and the gallery just sorted that photo to the wrong place. Fenced on path and filename as well as the id — the same fence #1199 put on the orientation backfill for the same reason — so a replaced row matches zero rows and is skipped. The candidate query already selects both columns, so no query change. Knex renders a null value in the object form as `is null` on both the pg and sqlite3 clients, so a row with a NULL path still matches itself. Those skipped candidates are now counted rather than dropped. replacePhoto is not the only writer of path/filename — eventRenameService rewrites both on an event rename, which is not a content change — and another writer filling captured_at first lands in the same place. Without a counter they fell out of the run's arithmetic entirely: success + noExif + failed no longer added up to the count the operator was shown when they started the job, on the card as well as in the log. The card shows the count only when it is non-zero, the same shape the orientation job uses for staleTiers. The wording states what is known — changed by something else, not updated — rather than promising a retry: for the already-dated case there is nothing to retry, and the Missing Capture Date figure above is what says whether work is left. Locale coverage matches the staleTiers key (en, de, fr, sl), with the defaultValue carrying the rest. Regression test: a replacement landing mid-run leaves captured_at NULL and is not counted as updated. Verified to fail against the unfenced code. |
||
|
Paul Nothaft
|
410b8f8f6f |
fix(external-media): record capture dates on import, and backfill existing libraries (#1172) (#1179)
* fix(external-media): record capture dates on import, and backfill existing libraries (#1172) External imports never read EXIF, so photos.captured_at stayed NULL for every row they created. The gallery sorts "Date Taken" with COALESCE(captured_at, uploaded_at), which on a bulk import is the import timestamp — so the sort silently degraded into "order by import batch" with no error and nothing in the UI to say the sort key was missing. The reporter's 12-day trip came back with its first two days at positions 4204-5296 of 5555, because those folders happened to be imported second. - the import reads the capture date next to the sharp().metadata() call that already opens the file, so this costs one more read of the same source rather than a second pass over the mount. Best-effort like the dimensions: a source without EXIF imports with captured_at NULL, as before. - POST /api/admin/photos/repair-capture-dates backfills existing libraries, modelled on the dimension repair beside it — background pass, in-flight guard, status endpoint, and resolvePhotoFilePath, which is what reaches an external row at all. Not a migration: the originals sit on a mount that may be down at upgrade time, reading 8000+ of them would block the boot, and a run that found nothing has to be repeatable. - "no EXIF date" is counted separately from "could not read the file". An operator needs to tell "these files carry no date" from "the mount is broken" before deciding to re-run. - the update is guarded whereNull, so an import finishing mid-run is not overwritten by a slower pass. - every sort branch now carries photos.id as a tiebreaker, not just capture_date. A bulk import writes hundreds of rows inside one second, so uploaded_at and the COALESCE fallback both collapse and the grid reshuffles between loads. id is insertion order, which makes the fallback meaningful. Not addressed: extractCaptureDate reads no OffsetTimeOriginal, and exifr resolves a naive EXIF timestamp against the HOST timezone — so captured_at is not a true instant, and the same file imported on two machines yields two values. That predates this and applies to managed uploads equally; the tests here deliberately assert ordering rather than an absolute instant so they do not encode the bug. Worth its own issue. * fix(capture-dates): read managed originals through storage, skip archived, claim the run flag (#1172) Four holes in the backfill endpoint, all found in review: - Managed photos were resolved with resolvePhotoFilePath, which builds a STORAGE_PATH filesystem path. On an S3 install nothing is there, so every managed row failed. Now split the way the thumbnail regenerator does: external rows read from the mount directly, managed rows go through resolvePhotoStorageKey + withLocalCopy. - Archived events keep their photos rows but their originals are deleted on archive, so those rows failed every run and kept the button lit forever. Excluded from both the job and the status counts. - isRunning was claimed after the candidate query, so two concurrent POSTs could both pass the guard and start a pass. Claimed before the await, with every early exit releasing it. - The noExif comment promised a distinction extractCaptureDate does not make (it returns null for unreadable files too). Reworded to what it is. * chore: drop a stray node_modules symlink committed by mistake The .gitignore pattern is `node_modules/`, which matches a directory and not a symlink of the same name, so a local convenience link slipped past it. It pointed at an absolute path on one machine and would dangle everywhere else, breaking `cd backend && npm install`. * fix(capture-dates): gate the backfill as system maintenance, stop overstating the counters (#1172) The endpoint walks every event in the install and rewrites their metadata, but required only photos.edit — which the built-in team_photographer preset holds (175_granular_permissions_and_presets.js:106). That role exists for a contributing shooter, who should not be able to start a whole-library S3/NAS scan or touch another owner's photos. Now system.manage, with the status endpoint on system.view so the panel simply stays hidden for everyone else. The "without EXIF date" wording also promised a distinction the code does not draw: extractCaptureDate returns null for an unparseable file as well as for one that genuinely carries no date, so both land in that bucket. Reworded to "no date found" / "unreachable" in en, de and fr, which is what the two numbers actually separate. * docs: point the permission note at the follow-up PR (#1172) The dimension repair's matching gate landed in #1182, so the comment no longer needs to describe it as unaddressed. * fix(i18n): align the Slovenian capture-date wording with the other locales (#1172) sl was missed when the counters were reworded from 'without EXIF date' / 'unreadable' to what they actually measure. * fix(capture-dates): gate the status card on the permission the button needs (#1172) system.view and system.manage are independent grants, and StatusTab has no permission gate of its own — a successful status payload is what renders the card and its enabled button (StatusTab.tsx:637). Gating the status endpoint on system.view therefore handed a system.view-only role a live Backfill button whose every click 403s, with no error surfaced by the mutation. The comment above it already claimed this endpoint matched the POST. Now it does. * fix(gallery): make the Date Taken sort correct on SQLite (#1172) photos.captured_at does not hold one type on SQLite. Three writers put three different things in it: integer managed uploads — photoProcessor.js:488 hands knex a Date, which the sqlite3 binding stores as epoch milliseconds text external imports and the backfill, which write ISO-8601 null no capture date, so the sort falls through to uploaded_at, itself text in knex's 'YYYY-MM-DD HH:MM:SS' default shape A plain COALESCE over that is not an ordering. SQLite sorts INTEGER before TEXT unconditionally, so every managed photo carrying EXIF came back ahead of every photo that did not, whatever the dates said — a 2027 capture landing before a 2020 one. Among the text values 'T' (0x54) also outranks the space (0x20), so a same-day ISO 01:15 sorted behind a fallback 23:00. Both failures predate this branch — the first needs only two managed photos — but making that sort correct is what #1172 is about, so it is fixed here rather than left for the issue it belongs to. Normalised in the ORDER BY rather than by rewriting the column: the data fix would have to touch every existing row and every writer, which is a far heavier change than the sort it corrects. The cost is that this sort no longer uses idx_photos_captured_at on SQLite — an acceptable trade on the fallback engine, where the alternative is an index-assisted wrong answer. Postgres is untouched: captured_at is a real timestamp there and COALESCE already compares correctly. The regression tests drive the real gallery route on real SQLite. They write the epoch-millisecond integer directly, because the Date that produces it in production cannot be reproduced inside jest — there the binding's type dispatch misses sandbox Dates and stores "[object Object]" (CLAUDE.md). All four behavioural tests fail on the unfixed ORDER BY; verified by reverting it. * fix(gallery): normalise epoch-integer uploaded_at too, and stop polling a 403 (#1172) Two follow-ups from review. uploaded_at is not always text on SQLite either. A legacy archive restore leaves epoch milliseconds in it — there is a test pinning exactly that (__tests__/integration/sqliteEpochTimestamps.test.js) — and the fallback branch read it with substr(), so '1830297600000' was compared against '2020-01-01 00:00:00' as text and a 2028 upload sorted first. Both columns now get the integer/real branch. The status card also polled every ten seconds regardless of permission. With the endpoint correctly requiring system.manage, anyone who can open the Status tab but cannot run the job would have had a 403 and a logged denial every ten seconds for a panel they were never shown. The query is now gated on the same permission the endpoint requires, so it never starts. * style: quote convention in the capture-sort test (#1172) * fix(capture-dates): skip watcher-imported videos, and make the status counts consistent (#1172) Three follow-ups from review. fileWatcher.processNewPhoto sets type='video' and a video/* mime but never media_type (fileWatcher.js:128-130), so those rows keep the 'image' default from migration 048. Filtering on media_type alone queued every such video on every run — extractCaptureDate returns null for a video, captured_at stays null, and the backlog never cleared. Candidate query and status scope now check all three markers. The status counts were two separate queries, so an import committing a dated photo between them could be counted by the second and not the first: the card then showed withCaptureDate > total and a negative backlog, with the button enabled to "fix" it. One aggregate now. And the card's render checked only the cached payload. TanStack keeps that after `enabled` flips false, so a lower-privileged admin logging in behind a system.manage user inside the cache lifetime would still have seen the card and a button whose POST 403s. The permission is part of the render condition now. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |