4f352dec39634d35f2752707a9babbbfd2ecf921
268
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 peipeimoandPeifu Mo
|
4f352dec39 |
fix(auth): treat zxcvbn suggestions as advice, not blocking errors (#1050)
validatePassword() appended zxcvbn's feedback.suggestions to the errors array unconditionally, and validity is errors.length === 0 — so any password that merely earned a suggestion was rejected even when it satisfied every configured rule. The effective policy was stricter than the configured complexity level and invisible to the admin. Suggestions now surface only alongside a real strength failure. They stay available to callers in result.feedback.suggestions, so a UI can still show them as guidance while typing. The weak-password fixture is assembled from parts rather than inlined: an 8-char alphanumeric literal next to validatePassword( reads as a hardcoded credential to the required GitGuardian check. Both fixtures pin their zxcvbn score — the compliant one is load-bearing at exactly the moderate minimum (2), and a future zxcvbn bump promoting it to 3 would leave the test green while no longer covering the bug. Co-authored-by: Peifu Mo <[email protected]> |
||
|
MarianandPaul Nothaft
|
0d340f4e81 |
fix(archives): take the restored category from the manifest (#1240)
* fix(archives): take the restored category from the manifest The archive writer already persists `category_name` per photo in photos_manifest.json — that is why the manifest exists, and the comment above it says so: "(and category linkage) can't be derived from the extracted files alone". The restore route then read only `original_filename` out of it and kept deriving the category from the ZIP's first path segment. Archives store photos exactly as they sit on disk, so an event whose photos live in the gallery root produces a FLAT zip. `path.dirname()` is '.' for every entry, no category is resolved, and every restored photo lands with `category_id = null` — silently, behind a 200. Seen on a real restore: 596 photos back, 0 with a category, while the nine category rows sat untouched in the table. Now the manifest is the source of truth and the first path segment is the fallback, so foldered archives and legacy archives without a manifest behave exactly as before. The find-or-create is pulled into `resolveCategoryId` so both paths share it and each name is resolved once per restore. Tests: __tests__/integration/adminArchives.restoreCategories.test.js builds real ZIPs (flat with manifest, flat with an existing category row, foldered without manifest) and drives POST /:id/restore. Without this change the two manifest cases fail and the foldered one passes — the fallback is unchanged. * fix(archives): let the manifest be authoritative when it says "no category" Review follow-up on #1240, pushed with the author's agreement. The manifest won for "category X" but not for "none": an entry with a null category_name fell through to the directory fallback, so a photo the archive recorded as uncategorized came back filed under a category anyway. That matters because the directory is not a category. Archive entry names are the storage key minus `events/active/{slug}`, and that layout is `individual/{filename}` / `collages/{filename}` — categories have never been directories there. Reading the first path segment on a real archive invents categories literally named "individual" and "collages", so the fallback was overriding an accurate record with a junk one. The fallback is now confined to photos with NO manifest entry at all: archives written before the manifest existed, where the directory is the only signal left and inventing those names still beats losing every category. Tests: the legacy case now uses `individual/`, the shape a real archive actually has, instead of a category-shaped folder no archive produces — so it documents what the fallback really does. Plus a new case pinning that a manifest saying uncategorized leaves the photo uncategorized and creates no category row. It fails without this change; the legacy fallback keeps passing. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
44a8c416c7 |
refactor(email): stop reading the webhook response body at all (#1225) (#1239)
The transport carried 41 lines of bounded-read-with-deadline to recover a messageId a receiver MIGHT return. That value is only ever logged — nothing persists it, there is no email_queue.message_id column — and the code to get it produced two of the last four review findings: the size cap made a DELIVERED message retry (axios throws while reading), and the missing deadline let an unclosed stream hang the queue and resend. Not reading the body is how that whole class stops being reachable rather than defended against. responseType 'stream' still keeps axios from buffering; the stream is destroyed immediately and the id is synthesised as before. The status was always the delivery verdict, and it is known before any of this. An 'error' listener goes on before destroy(): destroy can emit on a socket-backed stream, and an unhandled 'error' on a stream throws — which would have turned a receiver's teardown into a failed send. Net: -45 lines of service code, one fewer constant, one fewer test seam, and three of the hardest cases in the suite replaced by two simpler ones. 23 tests, 63 across the email suites, eslint clean. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
6ca8baab23 |
fix(watcher): stop re-importing a photo whose file was replaced (#1226) (#1237)
The existence check matched on filename OR path. replacePhoto regenerates both — a fresh generated filename and a fresh managed path — so a watched-folder photo that had its file replaced stopped matching either arm. The original is still sitting in the watched folder, so the next sweep imported it again and the gallery ended up holding the delivered edit AND the untouched original: the same duplicate shape external_relpath prevents for reference galleries. source_filename is now a third arm. It is the stable key here — written once at ingest by this same path and preserved across a replace by design. Rows predating migration 193 are covered by its backfill: COALESCE(original_filename, filename), and this path never wrote original_filename, so for watcher rows that resolves to the basename being compared. The query is lifted into an exported findExistingPhoto() so the test drives it rather than a copy — the thing under test IS the query, so a query-builder mock would only assert that knex was called the way the test expects. Predates the Lightroom round-trip and applies to the admin replace path too; it became reachable when #1165 brought watcher galleries into round-trip scope. Six tests against a real SQLite database. The load-bearing one fails without the change, verified by removing the arm and re-running; the other five pin what must not move — filename and path matching, the pre-193 backfill shape, a genuinely new file, and event scoping. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
4105c099c9 |
test(external): make the fold-collision guard test the real code (#745 follow-up) (#1234)
The regression test added with #1165 re-implemented the claim ordering and the claim loop inside the test file and asserted against its own copy. It never required externalRelpathFold, so changing the real sort left it green — a guard against silently deleting a client's delivered edit that guarded nothing. The ordering is now a named, exported claimOrderFor() and the test drives it. Verified by sabotage: replacing the comparator with `return 0` fails the test, where before it passed. Three cases added while the seam existed: the managed row wins from BOTH input orders (the original bug was that the survivor was whichever came first, so one order proves nothing), the sort is stable for rows of the same kind, and it does not mutate the caller's array. No behaviour change — the comparator is byte-identical, only lifted out. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
0d41fe5bf1 |
fix(email): keep the webhook payload out of the logs, and bound the response read (#1225) (#1233)
Round 4 of external review, on the merged commit. Both findings are consequences of the round-3 streaming change, which is exactly why the round was worth running. An AxiosError carries the request it failed on: `config.data` is the ENTIRE serialised message, base64 attachments included, and `config.headers` holds the signature. emailProcessor logs the error object and winston serialises it, so a DNS blip or a refused connection wrote password-reset links, guest recovery codes and multi-megabyte invoices into combined.log — verified against axios rather than assumed. Every rejection is now caught and replaced with a message-and-code-only error, so nothing downstream can serialise the request back out of it. readBounded had no deadline. axios' `timeout` covers the response HEADERS, and with responseType 'stream' it has already resolved by the time the body is read — so a receiver that answered 2xx and never closed its body left the await hanging, the queue row stayed pending, and the next processor pass sent the same message again. An unclosed stream was duplicate email. There is now a 10s wall clock that destroys the stream, with the timer unref'd so a hung read cannot hold the process open at exit. 23 transport tests (2 new, both failing without these fixes), 63 across the email suites, eslint clean. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
d62407f431 |
feat(email): webhook transport as an alternative to SMTP (#1225) (#1231)
Setting EMAIL_WEBHOOK_URL makes PicPeak stop sending mail itself and POST each composed message as JSON instead, for something downstream (n8n, Make, a self-hosted relay) to deliver. Unset, every SMTP path is unchanged. Settles the four things #1225 left open: - SSRF: the URL goes through the same DNS-resolving check the outbound webhook worker uses, before every send. Private receivers are opt-in. - Transport security: https is required for anything leaving the machine. The HMAC proves who sent the body, not who can read it, and these bodies carry password-reset links and guest recovery codes. The private-network opt-in doubles as the plaintext opt-in. - Authentication: EMAIL_WEBHOOK_SECRET is required and signs the body as X-PicPeak-Signature, the same scheme as gallery webhooks. A URL without a secret leaves the transport OFF and says so once. - Attachments: carried as base64, not dropped. Oversized ones fail and stay queued rather than arriving without the invoice. Configuration is environment-only on purpose: this redirects every outbound message including password resets, so it must not be changeable from a compromised admin session. Three wiring details decide whether it works at all: docker-compose.yml declares an explicit environment block, so the vars had to be forwarded there; a fresh webhook-only install has no email_configs row (migration 001 seeds it only when SMTP_HOST is set), so the From identity falls back to EMAIL_FROM; and processEmailQueue used to return early when SMTP could not initialise, which would have left the queue permanently unprocessed. guestRecoveryService and the admin test-email endpoint were bypassing the transport — the first dereferenced a null transporter, the second told webhook-only admins to go configure SMTP. emailIntakeService deliberately stays on SMTP: it round-trips a specific mailbox's own credentials. Response handling is streamed and read bounded by hand rather than capped via axios: maxContentLength throws while reading, so a receiver that delivered the mail and then echoed a large body would have been recorded as failed and the message sent again. Note: docker-compose.dev.yml is gitignored and local-only, so the equivalent entries there are not part of this change. docker-compose.production.yml needs none — it passes .env through with env_file. Three rounds of external review; 21 transport tests, 61 across the email suites. |
||
|
Paul NothaftandPaul Nothaft
|
f4c054a661 |
fix(export): name the camera master in photo exports, not the delivered render (#1229) (#1230)
#1165 added photos.source_filename to this service's select, with a comment saying it was there so the Lightroom round-trip could still match after a re-upload — and then nothing read it. Every output path still used original_filename, which is overwritten the first time an edited render is uploaded over a proof (#745). So after a round-trip the exports named the render. Each of these formats exists to help a photographer find the master on disk, and the render's name does not. The XMP case is the sharpest: the sidecar is written next to a RAW master, so a wrongly-named one is never associated with it. Two helpers, because the sites want different things when nothing is known: cameraName() source_filename || original_filename || null cameraFilenameOrStored() the above, else the stored name The dedicated `original_filename` fields (CSV column, JSON key) keep reporting blank/null when unrecorded — echoing the sanitized stored name there would invite a match against a file that does not exist under it. The places that must emit some name (text list, CSV filename cell, XMP sidecar) fall back to the stored one, as they did before. filename_format='stored' is untouched, and rows with no source_filename still resolve to original_filename, so nothing moves for installs that have never run a replacement. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
4f684eb482 |
fix(feedback): name the camera original in the exports, not just the stored file (#1224) (#1228)
Both feedback exports carried only `photos.filename` — the sanitized stored name (`wedding-smith_individual_1755892345.jpg`). Acting on client picks means finding the master on disk, and that name matches nothing in a Lightroom catalog, so the export could not be joined to anything. Adds the camera-original name to the long and pivot shapes, and by extension to the archive's feedback_data.csv/.json, which reuse the same query. COALESCE(source_filename, original_filename), not original_filename alone: the latter is overwritten the first time an edited render is uploaded over a proof (#745), so an export taken after a round-trip would name the render rather than the master and silently stop matching. source_filename is written once at ingest and survives a replace by design (migration 193). That case is the load-bearing test. Aliased to `original_filename` — the name the sibling photo export already uses for this column, and the question the reader is asking. Left empty when neither is known rather than echoing the stored name: blank reads as "no match possible", where repeating the sanitized name invites a match attempt against a file that does not exist under it. The column is added, not swapped: `filename` is untouched, so anything reading the old column keeps working. Reported by the 8digit/picpeak fork, which has carried a narrower version of this patch (original_filename only) across rebases. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Luca
|
8db8527f9e |
feat(api): Lightroom round-trip — read proofing marks, put finished edits back (#745) (#1165)
* feat(api): Lightroom round-trip — read marks, put edits back (#745) Gets a client's proofing verdict into a desktop catalogue and a finished edit back over its proof, without anyone re-matching files by hand. Three parts: **Keep the camera filename.** photos.original_filename is the only carrier of `IMG_1234.JPG` — the stored filename is rewritten by generatePhotoFilename. But replacePhoto() overwrites original_filename with whatever name the new file arrives under, so the first re-upload of a renamed render destroys the key the NEXT round-trip needs. Migration 185 adds photos.source_filename, written once at ingest and never touched by a replace, backfilled from original_filename so existing galleries can still match on their first pass. The backfill sits outside the column guard and keys on whereNull, so a run that dies partway self-heals instead of leaving half the rows empty forever. **Read the marks.** GET /api/v1/events/:id/photos returns each photo with its client colour tallies, the caller's own marks, and a merged colour + rating. Guards copied from the sibling upload route (apiTokenAuth + read scope + photos.view + requireEventOwnership). Filters: marked_only, mark_source, color_labels, my_color_labels, min_rating, my_min_rating. The route filters to a page of ids with PhotoFilterBuilder, then enriches just those through photoExportService.getPhotosWithFeedback — the two halves already existed and neither does both, and going id-first keeps the per-colour tally query bounded by page size. services/markMerge.js decides how three possible opinions (guest colours, guest star average, the photographer's own row in photo_admin_marks) collapse into the one colour and one rating Lightroom has room for. Colour goes to the photographer on a tie — one deliberate choice beats an aggregate a tie-break already had to guess at. Rating takes the max, because a rating is a magnitude and losing the higher one quietly demotes a photo somebody rated highly. Its roundRating matches XmpGenerator.mapRating exactly so the API and an XMP sidecar can never disagree about how many stars a photo has. **Put the edit back.** POST /api/v1/events/:id/photos accepts an optional replaces_photo_id and routes to the existing replacePhoto(), preserving the photo's id, feedback, comments and position. The plugin stores the picpeak id on the catalogue photo, so the id survives the editor renaming the render — which makes it the reliable key, not the filename. Scoped to the event in the URL: a token inherits its owner's powers across every event they can see, so an unscoped id would let one gallery overwrite another's photo. For renders whose RAW never went through the plugin, findReplacementCandidate gains an opt-in number_token mode matching on the trailing digit run. Deliberately the LONGEST run and never a fixed last-N slice: multi-camera shoots disambiguate by prefixing the camera index into the number (cam11234.jpg / cam21234.jpg), and a last-4 slice reads 1234 from both bodies and reintroduces exactly the collision the prefix removes. Ambiguity is refused, never guessed. Also drops the multer temp file on the two new early returns — this route only unlinks in its catch block. * refactor(api): one rating-rounding rule, and apply match_mode where it counts Three things the pre-review pass turned up on the round-trip work: - `match_mode` reached the photo-cap pre-count but not the loop that actually picks the replacement target, so asking for `number_token` would have been counted and then quietly ignored. Both call sites now take it. - `number_token` matching read `select('*')` over every photo in the event to compare one digit run. It now reads the three columns the match needs and re-reads the single winner in full, so a 5000-photo event doesn't pull 5000 full rows through memory to answer one question. - `XmpGenerator.mapRating` and `markMerge.roundRating` were the same five thresholds written twice — the second way to do one thing that drifts the moment either is touched. The thresholds now live in markMerge and the generator delegates, which is what keeps a sidecar and the v1 API from ever disagreeing about a photo's star count. * fix(api): keep the new route in the generated OpenAPI spec The `color_labels` description carried an inline JSON example. In an unquoted YAML scalar `{ "green": 2 }` parses as a flow mapping, so swagger-jsdoc threw YAMLSemanticError and dropped the WHOLE route from the spec — visible only as a warning on boot, with the route still serving normally, which is exactly the kind of failure that survives to release. Found by booting a real instance rather than by reading the diff. * fix(api): close the four blockers from review on #1165 1. Replacing an external photo silently kept serving the old file. resolvePhotoStorageKey gives photo.source_origin precedence and returns null for 'reference'/'external', so the edit was uploaded, the row updated and 200 returned while every viewer kept getting the untouched NAS original and the upload sat orphaned. replacePhoto now repoints the row to managed and clears external_relpath. The file on the share is never touched — this moves the pointer, not the data. 2. Every replacement leaked its temp file. putFromFile COPIES on local and uploads on S3; neither consumes the source, and replacePhoto never unlinked it — while the v1 route had disabled its own cleanup on the belief that replacePhoto moved the file. Cleanup now lives in replacePhoto, which closes the admin path too (adminPhotos only unlinks in its new-files branch, so replaced files leaked there as well). The v1 route also unlinks on the FAILURE path, which returned before any cleanup ran. 3. The download-all ZIP is invalidated after a replacement, as adminPhotos.js already does. Without it guests kept downloading the pre-edit photo indefinitely, which defeats the point of the feature. 4. The round-trip could not see reference or watcher galleries at all. fileWatcher and adminExternalMedia never set original_filename — the camera name lives in `filename` for those rows — so the backfill and the GET fallback both produced NULL for exactly the galleries most likely to be driven from Lightroom. The backfill now COALESCEs, both ingest paths set source_filename, and the GET falls back to filename. Concerns: - number_token no longer reads every photo row in the event per file. A LIKE on the digit run narrows the candidate set in SQL first; the exact trailing-run check still decides, so semantics are unchanged. The token is a regex-extracted digit run, so it cannot carry a wildcard. - The replacement's activity entry is scoped to event.id instead of null. The dashboard feed excludes NULL-event rows for scoped callers (GHSA-jhcf), so it was vanishing from the audit trail of the photographer who owns the event. Nit: dropped the unused higherPriorityColor export from markMerge. Three regression tests cover the external repoint, the temp cleanup and the COALESCE backfill. 21/21 pass. * chore(migrations): renumber 185 -> 193 after gallery-folders landed 185_add_category_is_folder.js merged to main while this was in review, so the number the PR reserved is taken and main is now at 192. Knex keys on filename rather than the prefix, so both would have run — but picpeakImportService guards restores with migrationOrder(), which parses that prefix, and two files answering 185 make the forward-only check pass a backup onto a schema missing its columns. Renumbered with every reference: the header comment, the test that requires the path, and the four call-site comments that cite it. The 'migration 182' reference inside it is the colour-labels migration and is unrelated; gallery.js:1134 cites upstream's 185 and is untouched. * fix(api): keep external_relpath when a replacement converts the row The external-photo blocker fix cleared external_relpath along with flipping source_origin, which closed one hole and opened another. adminExternalMedia dedupes a re-scan on (event_id, external_relpath) — routes/adminExternalMedia.js:195 — and migration 186 puts a unique index on exactly that pair. With the column nulled, the next scan of the share would not recognise the NAS original as already imported and would insert it again, so the gallery would end up holding both the edit and a fresh copy of the file it replaced. Only source_origin needs to change: it is what resolvePhotoStorageKey keys on, and every other consumer of external_relpath reads the two together and lets source_origin decide. The stale relpath on a managed row is inert for resolution and still correct as a dedupe key. Test updated to assert the value is kept rather than cleared. * fix(uploads): say when exiftool is missing instead of blaming the RAW A server without exiftool reported `No usable embedded preview in RAW file X.CR3: spawn exiftool ENOENT` for every RAW upload. The headline describes a corrupt photo; the actual cause is a package that was never installed, demoted to a trailing detail. It sends people hunting through their camera files. Hit while testing the Lightroom round-trip (#745): an export of RAW originals failed 11 times with that message, and the file was fine. RAW upload is the only feature that needs exiftool, so an install can be missing it indefinitely and only find out when someone uploads a CR3 — which makes the wording the whole diagnosis. ENOENT now produces a message naming the dependency and the install command for Debian/Alpine/macOS, and breaks out of the tag loop instead of spawning the same missing binary twice more to report the last failure as if it described the photo. A genuinely preview-less RAW still gets the original message. Verified both paths by making exiftool unreachable via PATH rather than mocking: missing tool and unreadable file now report differently. * fix(external): a delivered edit must win a relpath-fold collision Follow-up to keeping external_relpath on a replaced photo. Keeping it is what lets adminExternalMedia still dedupe the folder re-scan, but it also leaves the row inside externalRelpathFold's sweep — and that sweep does not merely rewrite paths, it DELETES collision losers via externalPhotoDedupe. The survivor was whichever row happened to be claimed first, which is iteration order. So a replaced photo — source_origin 'managed', holding the edit the photographer just delivered — could be deleted in favour of the untouched camera original sitting next to it on the share. Managed rows now claim first and therefore survive. The external row that loses is the recoverable one: it is still on the share and a re-scan re-imports it. The edit is not recoverable. Note this is deliberately NOT the "skip managed rows in the fold" shape suggested in review. Skipping would leave those rows holding a base-relative path while every other row moved to root-relative, so the scanner — which computes root-relative — would stop matching them and import the camera original again as a duplicate. That is the exact bug keeping external_relpath exists to prevent, reintroduced through a different door. Rebasing them and protecting them from deletion keeps both properties. |
||
|
Paul NothaftandPaul Nothaft
|
5c85e0c0e4 |
fix(guests): surface duplicate guest registrations, and stop making so many (#1210) (#1216)
* fix(guests): surface duplicate guest registrations, and stop making so many (#1210) Guest registration always inserts. A client whose token expired — or who opens the gallery on a second device — becomes a new gallery_guests row, and their likes and favourites split across the copies. The photographer's 'final selection' is then only trustworthy if somebody notices two Tinas with half the picks each. Two halves, neither of which touches the registration path. **Say which rows are the same person.** Merging already worked, endpoint and UI both; nothing said WHICH rows to merge. The guests list now marks each row with the others sharing its email and returns a count for the banner, and the admin list offers the group straight to the merge mode that already exists. Case-folded and trimmed, because the same person types Tina@ one day and tina@ the next and both read as distinct rows. Email only — two guests called Anna are not evidence of anything, and rows without an email are not grouped at all since require_name_email is off by default and a shared link produces plenty of them. It preselects rather than merges: which row survives decides the name and verification state the merged guest keeps, and that is the admin's call. **Create fewer of them.** The guest token was 24h and every call site took that default, so even the same browser lost its identity after a day of inactivity. Now 30 days, GUEST_TOKEN_TTL to override. A guest token is scoped to one event, carries no admin capability, and the gallery is already behind whatever protects it — 30 days is the shape of a real proofing cycle. Deliberately NOT done: reusing a guest row when a typed email matches, which the report suggests first. It would let anyone who knows an address inherit that person's identity and selections, and answering differently for a known email would leak which addresses are in the gallery — the thing /guest/recover already goes out of its way to avoid. Prevention at the entry path needs the verification round-trip, which is a separate decision about friction. 13 tests; 8 of the 9 backend ones fail without the change. The frontend ones caught a real bug while being written — the new useMemo sat after the loading early-return, so the hook count changed between renders. * fix(guests): merge must not strand a pending invite (#1210) Three findings from external review of #1216. **A merge could kill an emailed invite link.** Creating an invite inserts a real gallery_guests row, so an admin who pre-mints one and then sees the guest self-register has two rows sharing an email — which this feature now points out and offers to merge. Redemption resolves guest_invites.guest_id with is_deleted: false, so merging soft-deleted the row the link pointed at: the client got 404 guest_missing while the invite dialog still showed the invite as Pending. Nothing anywhere said the link was dead. Unredeemed, unrevoked invites now move to the survivor first. Spent ones stay put — a redeemed invite records who redeemed what, and retargeting it would rewrite that. **The preselection silently chose the survivor.** performMerge keeps mergeSelection[0], and the group was handed over in API order, which is newest-first — so Review then Merge discarded an older, email-verified row holding most of the picks in favour of a fresh re-registration. The proposal is now ordered deliberately: verified first, then whoever holds the most feedback, then the oldest. Still only a proposal, and the confirmation now names the survivor by email as well as name, because duplicates share a name and 'Merge 2 guests into Tina?' said nothing. **duplicate_of was quadratic.** Every row carried the other n-1 ids, so a group of n serialised n² of them — and nothing consumed the list: the UI asked only whether a row was in a group, then regrouped by email itself. Replaced with duplicate_group, the normalised email, which keeps the payload linear and the case/whitespace folding in one place instead of reimplemented on the client. Two new backend tests for the invite paths, one frontend test asserting the merge call keeps the verified row. The invite test fails against the un-fixed code. * fix(guests): keep guest-controlled input out of who survives a merge (#1210) Round 2 of external review on #1216. **The survivor ranking used an attacker-controlled signal.** Preferring whoever holds the most feedback looked like the obvious tiebreak and is exactly the wrong one: registration does not verify the address, so anyone who knows a guest's email can register with it, mark enough photos to out-rank the real person, and be preselected as the survivor. An admin accepting a confirmation between two rows with the same name and email would then move the victim's picks onto an identity whose token the visitor still holds. distinct_photos is guest-controlled and has no business deciding this. The ranking is now email_verified_at then created_at — both server-set. **A merge could make the survivor unrecoverable.** Rows are grouped with case and whitespace folded out, so a merge can be proposed between [email protected] and [email protected]. /guest/recover lowercases what the guest types and then matches on equality, so a survivor left holding the raw value can never be recovered by email again. The kept row's address is now canonicalised during the merge. Both write paths normalise today, so this covers rows that predate that — which are exactly the rows case-folded grouping surfaces. Two more backend tests. The residual, stated plainly: an admin can still merge two unverified rows in either order. What is gone is the tool ranking them by something a visitor controls. * fix(compose): pass GUEST_TOKEN_TTL through to the backend (#1210) The override was documented in .env.example and could never take effect: the backend service takes an explicit environment list, so a variable not named there never reaches the container. An operator following the documentation would have shortened the guest session and seen nothing change. docker-compose.production.yml uses env_file: .env and already passed it through; docker-compose.dev.yml is gitignored, so only this file needs it. * fix(guests): the admin picks the merge survivor, the tool does not (#1210) Fourth review round on the same point, and the right conclusion is that there is no correct automatic answer. Every rule tried was wrong somewhere. Most-feedback is guest-controlled — the address is never verified at registration, so anyone who knows it can register and mark photos until they out-rank the real person. Oldest-first, the replacement, is worse for the ordinary case: when a token expires the OLD row is the dead identity and the new one is the visitor's live session, so keeping the oldest deletes the identity they are actually using, and the frontend holds that deleted guest in sessionStorage without clearing it on a 401. Registration timing is visitor-controlled too. The data does not say which row is really the person. So the UI asks: merge mode gains a Keep column, the button stays disabled until a row is nominated, and only rows included in the merge can be nominated. The group is still preselected — finding the duplicates was always the point — but nothing about who survives is decided by sort order any more. This also makes the claim in the PR description true. It said the admin decides which row survives; until now the preselection quietly decided it for them. Two rewritten frontend tests: the merge is blocked until a survivor is chosen and then keeps exactly that row, and a row outside the group cannot be nominated. The test i18n mock now interpolates, so aria-labels are queryable by their rendered text. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
22e00f80b6 |
feat(feedback): a third identity mode with one shared colour tag per photo (#1197) (#1208)
* feat(feedback): a third identity mode with one shared colour tag per photo (#1197) Split out of #1178, where @boergu asked for a colour tag with no identity dimension at all: not everyone sharing a device's state, but everyone — on any device — sharing the PHOTO's state. Guest A marks it green, guest B later marks it orange, and the tag simply becomes orange. One collaboratively-agreed verdict per photo instead of per-person tallies. identity_mode gains 'shared'. The mode is scoped to the colour tag: likes, ratings, comments, favourites and reactions stay per-visitor exactly as in 'simple', because that is what was asked for and widening it would change what every other control means. Stored as an ordinary photo_feedback row under a reserved identifier rather than as a column on photos. That is what keeps the rest of the system working untouched — the per-colour tally simply has exactly one entry, so dominant_color_label, color_label_count, the admin colour filter and the XMP/CSV export that #745 reads all keep their existing shapes, and no consumer has to learn a second one. The identifier cannot be claimed: real ones are sha256 hashes or server-minted UUIDs, and the per-guest write path rejects it outright. Last write wins, inside a transaction that locks the photo row. Without the lock two guests tapping different colours in the same instant both read 'no tag', both insert, and the photo ends up carrying two shared tags — the per-guest tally this mode exists to remove. Re-sending the colour already on a photo clears it, from any guest: the same toggle every other colour path uses, and the only way to remove a tag without inventing a second control. Switching modes is non-destructive. Existing per-guest labels are left alone and simply not read while shared is on; the shared tag starts empty rather than collapsing marks nobody agreed on, and switching back restores every original exactly. An event can hold both sets, only one of which is live. The tag stays visible with show_feedback_to_guests off — it arrives through the per-viewer channel, being the photo's own state rather than someone else's opinion — while the per-colour tallies stay hidden. The colour filters answer from it for the same reason, so a gallery with sharing off cannot show colours on tiles that no filter can find. Attribution is gone by design, and the settings panel says so before an operator picks the mode. Decisions (1), (4) and (5) from the issue were settled up front, as it asked. Decision (3) turned out not to need anything: guest colour filters already read my_color_label, and the admin's my_color_labels filters photo_admin_marks (#1183), not guest identity — so nothing collapses on either side. * fix(feedback): shared mode saves on Postgres, and dormant labels stay dormant (#1197) Three findings from external review, all confirmed against source before fixing. **The mode could not be saved on Postgres at all.** Migration 078 created identity_mode with a CHECK constraint pinned to ('simple','guest'), guarded on `client === 'pg'` — so SQLite never has it and no SQLite test can see it, while the database every default production install runs rejects the new value outright. Migration 192 drops and re-adds the constraint with 'shared' included; its down() resets any event using the mode to 'simple' first, or the narrower constraint could not be restored. Verified against a real Postgres on a scratch database: the insert fails before, succeeds after, up() is re-runnable, and down() puts the old constraint back. **Dormant labels were still being read.** Switching modes is deliberately non-destructive, which leaves both sets of colour labels in the table with only one live — and every read that did not say which set it meant kept counting the other. The per-colour tallies, color_label_count, the admin grid badge, the XMP/CSV export, both admin colour filters and the guest colour filter all saw labels the mode does not show; switching back exposed the shared row as an anonymous other guest's dot. The settings panel promises these are 'kept but not shown', and that has to mean every surface, not just the badge. Scoped at the source — the two count helpers resolve the mode themselves — so the admin grid and the export are fixed without touching either. **The create form's identity mode was dropped.** CreateEventPage has always rendered the chooser and the create route never read it, so a gallery created as 'guest' came out 'simple' and had to be set again on the event afterwards. A pre-existing bug that adding a third option made worse; threaded through now, which fixes it for all three modes. Six regression tests, each verified to fail against the un-fixed code. * fix(feedback): keep every colour surface consistent across a mode change (#1197) Second review round, four findings, all confirmed in source first. **Stored counters went stale on a mode switch.** photos.color_label_count is denormalized and recomputed on feedback writes, so changing identity_mode — which changes nothing about the rows, only which of them are live — left the old mode's totals on the tiles, the admin grid and the filter summary until each photo happened to be touched again. On a finished gallery that is never. Recounted for the event when the mode actually changes, as two statements rather than a per-photo recompute: four of the five counters cannot have moved. **Duplicating an event dropped the mode**, the same shape as the create-form bug from the last round — a gallery cloned to reuse its proofing setup came back in 'simple'. **The event feedback summary counted dormant labels**, inflating total_feedback in the admin analytics and the guest /feedback-summary while every other surface hid them. **The swatch trusted its optimistic guess over the server.** In shared mode the tag belongs to the photo, so another guest can move it between this viewer's last read and their click: a viewer still showing green clicks green, the server sets green because the tag had become red meanwhile, and the optimistic 'same colour, so clear' blanked the swatch against a server that holds one. The response already says which happened, so it is used. The per-guest modes are unaffected — only the guest can move their own label, so guess and answer always agreed there. Three regression tests, each verified to fail against the un-fixed code. * fix(feedback): shared tag is not a participant, and the keyboard path reconciles too (#1197) Third review round, two findings. **feedback_count counted the shared tag as a guest.** It is COUNT(DISTINCT guest identity) across all feedback types, and the reserved identifier looked like a person: a photo with one rating and a shared tag reported two. The column is exported as rating_count (photoExportService), so merely tagging a photo inflated its rating count in the CSV and JSON exports. **The lightbox keyboard path still trusted its own guess.** The reconciliation from the last round covered clicks through PhotoColorLabels, but the proofing shortcuts call PhotoLightbox.submitColorLabel directly and set local state from a locally computed toggle. That is the path a proofing client actually uses, so it had the divergence the previous fix was for: another guest moves the tag, this viewer presses the key, the server sets a colour and the swatch blanks. Both branches now read the outcome off the response. One regression test, verified to fail against the un-fixed code. * fix(feedback): identity-mode lookup must survive a migration-time caller (#1197) updatePhotoFeedbackStats is called from migrations as well as from the request path — migration 186's duplicate-photo dedupe (#1162) recomputes the survivor's totals — and a migration runs against a half-built schema where event_feedback_settings need not exist yet. The new inner join threw there, which took the whole stats update down with it, so the reparented rows were never counted and eight assertions in the 186 suite failed. Falls back to 'simple', which is the right answer rather than merely a safe one: an install with no feedback settings table has no event in shared mode, so the non-shared scope is exactly correct. Caught by CI, not by me — I had been running affected suites rather than the full one after each review round. * fix(feedback): atomic shared-tag write, scoped feedback list, safe PG fallback (#1197) Round 4 of external review, and one of the three is about the fix I made for the CI failure two rounds ago. **The identity-mode fallback could poison a Postgres transaction.** The join was wrapped in try/catch so a migration-time caller with a half-built schema would fall back to 'simple'. On Postgres a failed statement aborts the entire transaction, so catching it and carrying on left the caller's trx poisoned and the aggregate that follows failed with 'current transaction is aborted' — defeating the very compatibility the fallback was added for. It now asks whether the table exists before issuing the join, which is safe to ask and aborts nothing. Memoised once true, since a table does not un-create itself and this sits on the feedback write path. **The shared-tag stats were recomputed after the commit.** A failure there returned 500 for a tag that had already been written, so the client reverted its swatch and the next tap on the same colour toggled the committed tag off instead of setting it. Two concurrent writers could also race their aggregate updates. Recomputed inside the transaction now, while the photo row is still locked. **The raw feedback list still carried both label sets.** Only the tallies and my_feedback had been scoped, so a dormant per-guest label was still visible to anyone reading the list — and with sharing off it came back flagged is_mine. getPhotoFeedback now filters colour labels to the active set. One test for the list; the migration suite that caught the original CI regression still passes. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
696c69a6d0 |
fix(setup): put the setup token where a NAS user can find it (#1218) (#1219)
* fix(setup): put the setup token where a NAS user can find it (#1218) The token file was never missing — it was in a subdirectory nobody opens. The all-in-one image points DATA_DIR at /data/db, so the file lands beside the database inside the single volume; someone browsing that volume from a NAS container UI sees db/, storage/, logs/, backup/ and gives up. There is no shell on those boxes to run the documented `docker exec … cat` with, and the token value is deliberately kept out of the logs, so the install looked like it had swallowed its own bootstrap credential. When DATA_ROOT names a different directory, the token is now written there too — /data/SETUP_TOKEN, the first thing visible on opening the volume. The compose stack sets no DATA_ROOT and keeps exactly one file, so nothing changes there. Each copy is written independently: the canonical one failing while the volume-root copy succeeds still leaves a readable token, and only a run where every write failed falls back to logging the value. The startup banner names every copy rather than just the first, which is what sent people into db/. Both copies are 0600 and both are removed the moment setup completes. That is what makes a second copy of a single-use bootstrap secret acceptable rather than careless — and writing the test for it turned up that the burn path had TWO independent unlinks, one in clearSetupToken and one at the end of createInitialAdmin. Only the first had been updated, so the volume-root copy survived the burn: a live-looking token that no longer works, which is worse than no token at all. Docs for the same issue are already out (PicPeak/docs#15); .env.example now names the AIO paths too. * fix(setup): enforce 0600 on a token file that already exists (#1218) External review. fs.writeFileSync's `mode` applies only when the file is created — writing over an existing inode truncates it and leaves its permissions untouched. A SETUP_TOKEN someone had copied to the volume root by hand at 0644 would keep that mode, so the first-admin bootstrap credential sat group- and world-readable on a shared NAS mount while this code claimed 0600. Unlink then create, rather than chmod after write: recreating gives a fresh inode with the right mode and no window where the credential is on disk under the wrong one. The chmod stays as a fallback for an unlink that failed for a reason other than the file being absent. Test fails against the un-fixed code. * fix(setup): drop a token copy that cannot be made private (#1218) Round 2 of external review. Asking for 0600 is not the same as getting it: a CIFS/SMB mount — which is what a NAS commonly offers — carries no Unix modes, so chmod is a silent no-op and the file keeps whatever file_mode= the mount forces, typically 0644. This feature targets exactly those hosts, so it now verifies the resulting mode instead of assuming the request took. A copy that cannot be made private is removed rather than left lying there, and it does not count as written — so an install where neither copy can be protected falls through to the existing log fallback, which reaches the operator alone. Previously a chmod that threw after a successful write left the credential on disk, and a success on the other path cleared the error, so nothing reported the exposed copy at all. Test simulates the mode-less mount with chmod as a no-op and stat reporting 0644; it fails against the un-fixed code. * fix(setup): never write the token through a foreign inode, or into the logs (#1218) Round 3 of external review, two findings, both about the credential ending up readable by someone else on exactly the shared mounts this feature targets. **The log fallback defeated the point.** When no copy can be made private, the old branch logged the token at warn — and logger.js writes warnings to combined.log under LOG_DIR, which in the all-in-one image sits on the same mount as the token file. The credential moved from a file we had just refused to leave, into another file just as readable, that outlives setup. The warning no longer carries the token; server.js already prints it on stdout when no file was written, which reaches `docker logs` without touching the shared volume. **A file that could not be deleted was written through anyway.** The pre-write unlink swallowed every error, so a 0666 SETUP_TOKEN owned by another user in a sticky or ACL-controlled directory — still writable — received the live token into its existing inode. Only ENOENT is ignored now. And when the mode check finds an exposed copy it cannot remove, that is recorded separately and reported at error level: a success on the other path clears writeError, and an exposed credential must not be silenced by an unrelated success. Two tests, both failing against the un-fixed code. * fix(setup): fail closed on an exposed token, and refuse a raced symlink (#1218) Round 4 of external review. **An exposed copy left the token valid.** A directory that permits creation and denies deletion — ACL-backed or CIFS — could keep a group/world-readable file holding a live setup token, and /setup/admin went on accepting it: anyone able to read the mount could take the first super-admin account. Reporting that was not enough. The token is now revoked when a readable copy cannot be removed, which turns what is left on disk into a dead string. Private copies are removed with it, since they hold the same value. The next boot mints a fresh one and skips the undeletable file rather than rewriting it, so this converges instead of looping on the same exposure. **The write followed a raced symlink.** On a group-writable mount another local user could drop a symlink at the path between the unlink and the write, and the default 'w' flag would follow it — putting the live token in a file they own. Now created with 'wx' (O_CREAT|O_EXCL), which neither overwrites nor follows a link; having just unlinked, anything present again is that race. The mode check uses lstat for the same reason: it must describe the file, not a link target. **A verification that threw left the file behind.** writeFileSync succeeding and lstat then failing — plausible on the network filesystems this targets — left an unverified live copy on disk, and a success on the other path cleared the error so nothing said so. Cleanup is now keyed on 'did this iteration create a file', so every post-creation failure removes it. Three tests, one new; the new one fails against the un-fixed code. Full backend suite at the known baseline. * fix(setup): report the written token path again, so the banner stays quiet (#1218) A regression I introduced one commit ago. Rewriting the write loop dropped the three lines after it that publish the result, so writtenTokenFile stayed null even on a completely successful write. server.js prints the token itself only when no file was written. With this reporting nothing, the banner took that failure branch on every fresh install and put the live super-admin setup token into stdout and `docker logs` — beside a perfectly good 0600 file. That is the exact leak this path was built to close, reopened by a refactor that touched none of the logic around it. Found by external review, not by the suite: nothing asserted the accessor, only the files on disk. Now guarded — the new test fails against the regression. * fix(setup): survive a worker race, and revoke a copy that predates this run (#1218) Round 6 of external review. **A pre-existing exposed copy was invisible to the revocation.** A restart reuses the token from the database, so an old file holding that value is a live credential. If it had become group-readable and could not be deleted, nothing tracked it — created was false, so the fail-closed path never fired and /setup/admin kept accepting what was in that file. An undeletable file at the token path is now treated as live and triggers the same revocation. **A losing worker printed the token.** The shipped PM2 cluster config runs several workers against one DATA_DIR. Both pass the unlink, one wins the exclusive create, and the loser's wx write threw EEXIST — so it recorded nothing and its banner printed the live token into its own log while a perfectly good 0600 file already existed. EEXIST now checks the file: private, regular, and holding the same token counts as this loop's work already done. **A write that created the file and then threw left it behind.** ENOSPC, a short write, a delayed close on a network mount — writeFileSync can populate the inode before failing, and cleanup keyed on the call returning skipped it. Keyed on the write being attempted now, with an existence check. Two tests, both failing against the un-fixed code. Full backend suite at the known baseline (2342 passing). * refactor(setup): drop the volume-root token copy, keep the hardening (#1218) The second copy was for discoverability: DATA_DIR points into /data/db on the all-in-one image, and a NAS user browsing the volume does not open a folder called db. Six review rounds later it had earned a second inode to race, to verify, to clean up and to revoke — a symlink guard, an exclusive create, an lstat check, cluster-race handling and fail-closed revocation, nearly all of it load-bearing only because there were two files instead of one. That is a lot of attack surface for a convenience the documentation covers better. PicPeak/docs#15 now points NAS users at ADMIN_PASSWORD, which creates the admin on first boot and needs no file at all, and names the db/ subdirectory for anyone who does want the token. Neither needs a second copy. So: one file in DATA_DIR again, as before. Everything the review turned up stays, because none of it was about the second copy — the token is created with O_CREAT|O_EXCL so a raced symlink cannot capture it, its mode is verified with lstat rather than assumed, a copy that cannot be made private is removed, one that cannot be removed revokes the token instead of being logged about, a partial write is cleaned up, a concurrent worker's good file is accepted rather than triggering the log fallback, and the token never reaches the log files. setupTokenFilePaths and writtenSetupTokenFiles are gone with their tests; the hardening tests remain and still fail against unfixed code. * fix(setup): publish the token atomically instead of racing over one inode (#1218) Round 7 of external review found a race in the exclusive-create approach: two PM2 workers reaching the write together, the loser sees the winner's file after the inode exists but before its content lands, judges it wrong, and deletes it — after which the winner's own verification fails too, both report nothing written, and both print the live token into their logs. Rather than teach the loser to wait, the shared inode is gone. The token is written to a per-process temporary file, verified there, and published with rename(2). That is atomic: the file never appears at the published path with the wrong mode or half its content, a symlink sitting at that path is replaced rather than followed, and concurrent workers simply publish the same value one after another. The unlink-then-create dance, the EEXIST handling and the cross-worker deletion all disappear with it. Verifying the mode BEFORE the rename is the stronger order too: a credential that cannot be made private on a mode-less mount now never reaches the published path at all, instead of being written and then cleaned up. If publishing fails and something is still sitting at the token path, it is treated as a live credential we could not replace, and the token is revoked — unchanged in intent from the previous round, simpler in mechanism. * fix(setup): drop a dead assignment and an unused import (#1218) Both flagged by the code-quality review on #1219. `createdTmp = false` after rename(2) is never read — rename consumes the temp file, so the catch has nothing left to clean up either way. `os` was never used in the test. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
a490b64954 |
fix(admin): the "Uncategorized" photo filter returns every photo (#1211) (#1214)
The dropdown offered the filter and it never worked. It rendered as
`value="0"`, and adminPhotos.js skips '0' outright:
if (category_id !== undefined && category_id !== '' && category_id !== '0') {
so no category condition was applied and the whole event came back. Four lines
below that guard sits the branch that does the work, keyed on the literal
'uncategorized' — which nothing was sending. The two ends have never agreed on
the wire value, and neither is wrong on its own.
It fails silently, which is why it went unnoticed: a full list reads as 'the
filter found nothing to narrow' rather than 'the filter did not run'.
Send what the backend already understands rather than teaching it a second
spelling. The onChange passes non-numeric values through unchanged, so the
string arrives intact.
Reported in #1209 by someone re-categorising a few thousand photos imported
without a category — the filter is the first step of filter, Select All, bulk
assign, so its failure takes the whole path with it.
Tests both ends of the contract, since the bug was the pairing rather than
either half: the frontend emits 'uncategorized', and the endpoint answers it
with only the null-category rows. The backend test also pins that 0 means no
filter, so a future change there has to be a decision rather than an accident.
Co-authored-by: Paul Nothaft <[email protected]>
|
||
|
Paul NothaftandPaul Nothaft
|
0a36ca6056 |
feat(gallery): folders that contain photos instead of filtering them (#1160) (#1161)
* feat(gallery): folders that contain photos instead of filtering them (#1160) A category has always been a filter: its photos stay in the root grid and picking the category narrows that grid. D#1086 asked for the opposite — put the selects in a bucket and get them OUT of the main grid, so the client sees the 40 finals and clicks through for the other 200. `photo_categories.is_folder` makes that a per-category choice. One column is enough because the neighbouring features already built the substrate: hero_photo_id (#163) is the folder cover, allow_downloads (#640) is per-folder download rules, display_order + event_category_order (#782) is folder ordering, and photos.category_id being single-valued is already folder semantics. Deliberately no parent_id. "Root -> Selects folder" is depth one, i.e. plain containment; folders-inside-folders waits until someone asks. Containment lands in the one useMemo where the category filter was already applied, and the tiles render above the grid rather than inside a layout, so all eight gallery layouts inherit folders without eight implementations. Scope drives the counts too, so root reports 40 photos and not 240. `?folder=<slug>` carries the open folder, preserving token and admin_preview, so a folder is linkable and Back walks out of it instead of leaving the gallery. Defaults to false, so every existing gallery keeps filtering exactly as before. Folders are organisational, not access control: a foldered photo is served by the same per-photo auth as any other. A test pins that, so nobody later mistakes containment for a security boundary. * feat(gallery): download a folder on its own, and label folders when moving photos (#1160) Downloads now cover both halves of the requirement: - the gallery-wide "download all" keeps zipping every photo including the foldered ones (verified: 62 files), so a folder never quietly removes photos from the client's one-click download; - inside a folder there is a "Download folder (n)" button that zips only that folder, once (verified: 20 files). It reuses /download-selected, so there is no new endpoint and no second zip-building path. The button honours the per-category opt-out (#640) both ways: a folder with allow_downloads = false renders no button, and individual photos that opted out are excluded from the id list rather than silently 403-ing mid-zip. Moving photos into a folder already worked — a folder IS a category, so the existing bulk "move to category" flow does it. What was missing is that a folder and a filter category looked identical in that dropdown while having very different consequences, so folder options now read "<name> (folder — hidden from the main grid)". Threading is_folder through to the dialog needed the admin category prop types widened; the data was already on the wire. * fix(gallery): folders were unreachable in the full-bleed layouts (#1160) Containment comes from `filteredPhotos`, which BOTH layout branches use, but the tiles were only rendered in one. On a Premium or Story gallery the foldered photos therefore disappeared from the grid with no tile to click — moving 200 selects into a folder effectively deleted them from the client's view. The folder nav is now built once and rendered by both branches, so a branch can't hide photos without also offering the way in. Those two layouts are edge-to-edge by design, and a block of cover cards above the hero wrecks the opening they exist for, so they get a compact chip row (`Folders [icon Selects 20]`) instead. It only renders when the gallery actually has folders, leaving every existing full-bleed gallery byte-identical. Also scopes the people strip to the photos on screen. `face_count` comes from /people and spans the whole event, which contradicted the grid in two ways: inside a folder a face read "12 photos" but filtered down to the handful in that folder, and at root a person whose photos ALL lived in a folder showed up and filtered to nothing — a dead chip. Recounted from `photo.person_ids`, which is already what the filter itself uses, and zero-count people are dropped. No backend change; the ids were on the wire already. Verified in the running app: Premium renders the chip and navigates; the lightbox counter inside a folder reads "1 / 20", not 1 / 62; the people strip inside the folder drops from 12/11/4 to the one face actually present. * fix(gallery): folder edge cases found in external review (#1160) Seven issues, all verified against the code before fixing. Unreachable folders (the serious one). `adminCategories` derives a slug with `[^\w\s-]` stripping and `\w` is ASCII-only, so a valid name in a non-Latin script slugs to the empty string — `Избранное` and `日本語` both do. Keying the URL on the slug meant such a folder wrote no param and resolved to nothing: its photos left the root grid with no way back to them. This repo ships ru and sl locales, so that is a reachable state, not a hypothetical. Folders are now keyed by `folderKey()` — slug when there is one, id otherwise. Stale selection across a scope change. The grid clears its selection when `categoryId` changes, which is already null at root, so a selection made outside a folder survived into it and the toolbar would offer to download (or a client to hide) photos no longer on screen. Cleared on both explicit navigation and popstate. Dead category chips inside a folder. The filter branch ignores `selectedCategoryId` while a folder is open, so the chips did nothing when clicked. They are no longer offered there. "No photos found" beside folder tiles. A gallery whose photos all live in folders rendered the tiles and then the grid's empty state directly under them, claiming the gallery was empty while pointing at its contents. Counts that contradicted the grid. The filter bar and both people surfaces were still counting over every event photo, so a chip could advertise a total the scoped grid would never produce. All now count over `scopedPhotos`. `!!` on a validated boolean. express-validator's isBoolean() accepts the STRINGS "false" and "0", and `!!'false'` is true — a form-encoded caller asking for a filter would have silently got a folder. Uses the existing parseBooleanInput. Duplicate-event dropped folder-ness. The category clone selected only name, slug and is_global, so every folder in a duplicated gallery came back as a filter. * fix(gallery): folder scoping gaps from external review round 2 (#1160) Cache mutation, introduced by this branch. `photosInScope` returned the caller's own array on the no-folders fast path, and `filteredPhotos` sorts in place — so every gallery WITHOUT folders was reordering the React Query cache for every other consumer of `data.photos`. The pre-branch code cloned; now it always does. Colliding folder keys. UNIQUE is (slug, event_id), so a global folder and an event folder can share a slug, and the gallery merges both scopes. Keying on the slug alone meant the second folder resolved to the first and its photos could not be opened. The id is now always part of the key. "Download folder" downloaded a subset. Search, feedback, media and people filters stay active when entering a folder, and the ids came from `filteredPhotos` — so the button promised the folder and delivered whatever the filter had left, or vanished when it matched nothing. Built from `scopedPhotos`. Folder-only root misdetected. `rootIsFoldersOnly` tested `filteredPhotos`, so a search matching none of the loose root photos looked folder-only and swallowed the no-results message. Tests the unfiltered scope instead. Empty state in the full-bleed layouts. The Premium/Story branch was missing the folder-only guard the standard branch got, so a folder-only gallery printed "no photos found" under its own folder chips. Filter metadata still event-wide. `availableMediaTypes` and `colorLabelCounts` counted over every photo, so the sidebar could offer a Video or colour chip for something that only exists in another scope — always filtering to nothing. Both derive from `scopedPhotos`, which moved above them for that reason. * fix(gallery): honest folder downloads and scoped totals (#1160) Silent truncation. /download-selected slices the id list to 500 server-side (gallery.js:1776), so a folder larger than that delivered a truncated archive under a button promising the whole thing. The limit is now mirrored client-side: the request carries only what the server will honour and the label says "Download first 500 of 620" instead of claiming the folder. Gallery shell was being unmounted. Suppressing the folder-only empty state by skipping PhotoGridWithLayouts took the hero, event title, logout and download controls with it in the full-bleed layouts, since those render from inside that component — a folder-only Premium gallery collapsed to a bare chip row. Replaced with a suppressEmptyState prop so only the message goes. Two more counts that could contradict the grid: the sidebar's total and the people match-count denominator ("42 of 62" at a root that holds 42). Both scoped. The client-access visible/total stat is deliberately left event-wide — that one is a photographer-facing statistic about the gallery, not a filter affordance. Stale admin cache. EventDetailsPage caches the same category rows under 'admin-event-categories' and hands them to the Photos tab's move dialog, so toggling a folder left that dialog labelling it a plain category until remount. Both keys are invalidated now. Not changed, after challenging the review: select-all in the full-bleed layouts stays scoped to the displayed photos. Wiring it to the full event would select photos that are not on screen, contradicting containment and reviving the stale selection bug. The reviewer withdrew the finding on that basis. The residual UX gap — no one-click "everything" in Premium/Story once folders exist — is real and noted on the PR. * feat(gallery): one-click download-everything in the full-bleed layouts (#1160) Premium and Story have no header download button — their only gallery-wide download is select-all followed by download-selected, and select-all is correctly scoped to what is on screen. Once folders exist that left no single way to get the whole gallery. The folder strip now carries an event-wide "Download all photos" that hits /download-all (which has always included foldered photos), shown at the root only, since inside a folder the breadcrumb already offers that folder's download. Also lands the capped folder label that was written but never actually applied in the previous commit — the edit silently didn't match, so a 510-photo folder still advertised "Download folder (510)" while the request was capped to 500. Caught by building a real 510-photo folder rather than trusting the reasoning: it now reads "Download first 500 of 510". A unit test pins the client constant to the backend's cap so the two can't drift apart unnoticed. * fix(gallery): remount layouts on folder change, and stop scoped counts leaking into event-wide controls (#1160) Carousel crash. Layout state is only meaningful for the photo set it was built against, but the layout instance was reused across a folder change. In carousel mode an index valid at root (31 of 42) indexes past the end of a smaller folder, and CarouselGalleryLayout does `photos[currentIndex]` unguarded. The grid is now keyed by the open folder, so a scope change remounts: verified live, 31/42 at root becomes 1/20 on entering the folder instead of dereferencing undefined. The key also avoids driving one instance between the empty and non-empty render paths, which matters because that component's `photos.length === 0` early return sits ABOVE four useState calls — a pre-existing conditional-hook hazard this feature would otherwise have made reachable. Nested empty state. suppressEmptyState only silenced PhotoGridWithLayouts' own early return; the Premium and Story layouts have their own noPhotosFound return, so a folder-only root still printed "no photos" under the tiles proving otherwise. The flag is forwarded to them. Download All was labelled from the wrong number. The sidebar's total is now the folder scope (correct for the category list), but the same value labelled and disabled Download All — which fetches the event-wide archive. On a folder-only root that showed 0 and refused a valid download. Split into a separate downloadAllTotal. Feedback chip counts. likeCount, favoriteCount and ratedCount still counted over every event photo while clicking them filters the scope, so a chip could promise matches from another folder and deliver none. * fix(gallery): premium crash, story Download All, and empty-mount hazard (#1160) ReferenceError blanking the Premium gallery — my own bug from the previous commit. The suppressEmptyState prop landed on the nested PhotoCard instead of GalleryPremiumLayout (both destructure `allowDownloads = true`, and the patch hit the first one), so the layout's guard referenced an identifier that was not in its scope. A folder-only Premium root threw instead of rendering. Now declared and destructured on the layout, and exercised: 62 photos all foldered renders the tile, the hero and the download button with no message and no throw. Story's footer "Download All Photos" built its id list from the `photos` prop, which is now the folder scope — so it silently omitted every foldered photo while still calling itself Download All. Layouts now receive an event-wide downloadAllIds and prefer it. Premium's equivalent control is a select-all, not a download, and stays scoped by the same reasoning as before. Empty-array mounts. Suppressing the empty state meant the layout got mounted with photos=[], and CarouselGalleryLayout returns before four of its useState calls — driving one instance between empty and non-empty changes its hook count and React throws. Only the full-bleed layouts, which own the hero and logout chrome, are now mounted empty; every other layout renders nothing instead. * fix(gallery): keep the shell and drop dead controls on folder-only roots (#1160) Skipping the empty layout took the hero and welcome message with it. The early return sat above both, so a gallery whose photos all live in folders lost its configured hero and welcome copy at the root and only regained them after opening a folder. Only the layout child is skipped now; the surrounding shell renders as it always did. The filter bar was gated on the event-wide photo count, so a folder-only root still rendered search, sort and the feedback chips with nothing in scope for them to act on — the same empty filter row discussion #317 asked us to remove. Gated on the current scope. Story's download toast counted `photos` while the request now carries the event-wide id list, so it could announce "Downloading 0 photos" and then fetch the whole gallery. Counts the ids it actually sends. * fix(gallery): clear the person filter on scope change, and fix two folder-only shell details (#1160) A person selected in one scope can have no photos in the next. peopleInScope drops them from the strip, so the filter stayed active with nothing left to clear it — and the full-bleed layouts have no people UI at all, leaving a guest staring at an empty grid with a reload as the only way out. Cleared on both folder navigation and popstate, alongside the category selection and the photo selection already reset there. Story's hero announced "0 Photos" on a folder-only root, since it derives that stat from the scope it renders and the scope is empty by definition there. Falls back to the event-wide count. Premium's integrated Download All is a select-all over the current scope, so on a folder-only root it was a visible control that did nothing when clicked. It is hidden while the scope is empty rather than left dead. * fix(gallery): uncapped Story download, protected folder covers, scoped people order (#1160) The event-wide id list I added for Story's "Download All Photos" made it worse, not better: /download-selected caps at 500 ids server-side, so a gallery larger than that silently shipped a partial archive under a button promising all of it. Replaced with an onDownloadEverything callback that runs the whole-gallery /download-all path, which has no cap. eventPhotoCount now carries the number Story needs for its hero stat, so no id list crosses the boundary at all. Folder covers bypassed image protection. A cover is a real gallery photo, but it was rendered through AuthenticatedImage's defaults while every photo tile passes the gallery's protection settings — so on a gallery configured for canvas rendering or maximum protection, each cover was an ordinary blob-backed <img>. The tiles now receive and apply the same props as the grid. People kept /people's event-wide ordering after their counts were rescoped, so a folder's most-photographed person could sort behind someone with a single match — and PeopleStrip only shows the first twelve inline. Sorted by the recomputed count, with a test. * fix(gallery): folder covers honour maximum protection (#1160) Maximum protection implies canvas rendering even when the independent use_canvas_rendering toggle is off, which is its default — every other gallery image path spells that out as `useCanvasRendering || protectionLevel === 'maximum'` (PhotoGrid, PhotoLightbox, HeroHeader, JustifiedGalleryLayout). The folder cover forwarded the raw toggle, so on a maximum-protection gallery with the toggle untouched the cover fell back to a blob-backed <img>. Matches the convention now. * fix(gallery): don't let download-everything bypass a category opt-out, and keep folder links alive across renames (#1160) The whole-gallery route serves a prebuilt zip containing EVERY event photo with no per-category filter — gallery.js says so itself, next to bumpEventDownloadCounts, as a known pre-existing gap. Wiring Story's footer to that route therefore converted a path that DID enforce the #640 opt-out into one that doesn't, and because the callback was supplied unconditionally it affected Story galleries with no folders at all. The same reasoning applies to the download-everything button this branch added to the full-bleed folder strip: it routes there too, so on a gallery with a restricted category it would have handed over exactly the photos the opt-out withholds. Both are now withheld whenever any category opts out; those galleries keep the per-folder download, which enforces it. Verified both ways — the control disappears with a restricted category present and returns once the restriction is lifted. Folder links also survived a rename badly: the key embeds the slug for readability, and renaming a category rewrites that slug, so a URL already sent to a client stopped matching and silently opened the gallery root. Resolution now keys on the trailing category id, which does not move. * fix(gallery): make folder navigation clickable in the Story layout (#1160) Story renders `.story-nav` as `position: fixed` across the top of the viewport at z-index 50, and the folder strip sits in exactly that band — so the nav swallowed every click on the chips and the breadcrumb. A Story gallery whose photos all live in folders had no way to reach them at all. Confirmed with elementFromPoint at the chip's centre returning NAV.story-nav; the strip now carries its own stacking context above it and the same probe returns the chip. Story's footer download could also be offered with nothing to send: on a folder-only root of a gallery that has a category download opt-out, the parent deliberately withholds the whole-gallery callback and the scope is empty, so the button would have posted an empty id list and taken a 400. It is only rendered when one of the two actually exists. * fix(gallery): stop the Story folder strip from blocking the layout's own nav (#1160) The previous commit raised the whole folder strip above `.story-nav` so the chips could be clicked, and thereby traded the bug for its mirror image: the strip is mostly empty space, so as a solid z-60 container it swallowed the clicks for Story's own search, favourites and logout sitting underneath. The container no longer takes hits at all; only the chips, breadcrumb and download button opt back in. The download button also loses its ml-auto, since being pushed to the right put it physically on top of the nav's controls rather than merely above them in stacking order. Verified by hit-testing all three at once — folder chip, download button, and Story's nav control each resolve to themselves under elementFromPoint, so none is covering another. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
cec8eff70c |
fix(images): fence the capture-date backfill on the file it read (#1201) (#1204)
The capture-date backfill committed its result keyed on the row id alone. It snapshots every candidate up front, then walks them one at a time reading originals off S3 or a NAS mount — a pass that can run for many minutes. replacePhoto, reachable from the replace_by_name upload path, swaps a NEW file under an existing row and rewrites path/filename. A replacement landing inside that window carries no date of its own, so captured_at was still NULL, the whereNull guard passed, and the previous file's EXIF date was written onto the new photo. Silent: nothing errored, the run reported it as a success, and the gallery just sorted that photo to the wrong place. Fenced on path and filename as well as the id — the same fence #1199 put on the orientation backfill for the same reason — so a replaced row matches zero rows and is skipped. The candidate query already selects both columns, so no query change. Knex renders a null value in the object form as `is null` on both the pg and sqlite3 clients, so a row with a NULL path still matches itself. Those skipped candidates are now counted rather than dropped. replacePhoto is not the only writer of path/filename — eventRenameService rewrites both on an event rename, which is not a content change — and another writer filling captured_at first lands in the same place. Without a counter they fell out of the run's arithmetic entirely: success + noExif + failed no longer added up to the count the operator was shown when they started the job, on the card as well as in the log. The card shows the count only when it is non-zero, the same shape the orientation job uses for staleTiers. The wording states what is known — changed by something else, not updated — rather than promising a retry: for the already-dated case there is nothing to retry, and the Missing Capture Date figure above is what says whether work is left. Locale coverage matches the staleTiers key (en, de, fr, sl), with the defaultValue carrying the rest. Regression test: a replacement landing mid-run leaves captured_at NULL and is not counted as updated. Verified to fail against the unfenced code. |
||
|
Paul NothaftandPaul Nothaft
|
d3e9a7cf0d |
feat(auth): make the admin "Remember me" checkbox actually do something (#1186) (#1195)
The checkbox had no `checked`, no `onChange`, and no place in the login request; `rememberMe` existed only as an i18n label. On the backend establishAdminSession hardcoded `expiresIn: '24h'` and the cookie always got DEFAULT_MAX_AGE_MS, so there was nothing to receive it anyway. Wired end to end: state on the page, `remember_me` in the login body, and a 30-day JWT plus a matching 30-day cookie when it is set. Opt-in on purpose. An absent or malformed value means "no", so a client that never sends it keeps exactly the 24h session it always had, and a stolen cookie is still worth a day by default. The JWT and the cookie take their lifetime from the same flag. If they can disagree the session either dies early (long cookie, short token) or outlives what the user consented to, so the tests assert them against each other. Review found the feature was non-functional as written, which is the important part: sessionTimeoutMiddleware and isSessionExpired enforce security_session_timeout_minutes — 60 minutes by default — against a session's idle time regardless of how long its token lives, so a remembered admin was logged out within the hour with a 30-day token sitting unused. rememberMe now travels in the JWT payload and both checks exempt a remembered session from the IDLE timeout. Not from expiry: the token still dies on its own 30-day exp, and revocation, deactivation and password-change invalidation are untouched. Also: /api/admin/auth/change-password reissued a hardcoded 24h token without the flag, so a remembered admin dropped back to 24h the moment they changed their password — which is mandatory for new and reset accounts. It now inherits the choice from the session it replaces, carried on req.admin.rememberMe. Through MFA the choice rides inside the signed mfa_pending token rather than being resent, so the second leg cannot ask for longer than the first agreed to. The tests drive POST /api/auth/admin/login and read the real Set-Cookie and token rather than minting a local clone of the ternary they are meant to be checking, boot one database per file before anything reads it, and generate their credential per run so no literal that looks like a password lands in the repository. No visual change — the checkbox was uncontrolled, so it already toggled on click; it just did nothing. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
edef4d7365 |
fix(images): backfill orientation for libraries that predate the fix (#1199)
* fix(images): backfill orientation for libraries that predate the fix (#1198) #1194 corrected the generators and every ingest path, but did nothing for photos already in the database. Those rows end up worse than untouched ones: before the fix a rotated photo was CONSISTENTLY wrong — a sideways image in a tile shaped to match — and afterwards the regenerated thumbnail is correct while photos.width/height still describe the raw sensor order, so masonry and justified size a portrait photo with a landscape ratio. The dimension repair cannot reach them: it only selects rows with a NULL dimension, and an affected row has both, just transposed. Its own job rather than a mode of that one. They look alike but are not the same operation: the repair FILLS missing values and touches nothing else, while this RECOMPUTES and invalidates the derived data generated against the old orientation. Sharing a lease would also mean one blocks the other. A first attempt at this was reverted from #1194 after review found five problems. All five are addressed here: - Originals are read through resolvePhotoStorageKey + withLocalCopy + withProcessableImage, so the job works on S3 installs and on RAW/DNG. The dimension repair's direct fs read does neither, which stops being an edge case in a job that walks the whole library. - The canonical preview is cleared BEFORE faces are requeued. ensurePreviewImage returns a cached preview whenever it is still a valid image, and a pre-fix unrotated one is perfectly valid — so requeueing alone made the rescan read unrotated pixels and scale those boxes by the corrected dimensions, which is worse than leaving the data alone. - Invalidation keys off the EXIF transform, not a dimension delta. Orientations 2, 3 and 4 move every pixel while leaving width and height unchanged, as does 5-8 on a square image; a delta check skips exactly those rows. - Archived events are excluded — archiving deletes the originals and keeps the rows, so every one of them would fail its read. - The dimension write and the invalidation share a transaction. Split, a failure between them leaves stale face data that no retry can fix, because the retry computes "already correct". Tier deletion stays outside the transaction on purpose: it touches storage, and a failed object delete must not roll back a correct database write. A leftover tier regenerates on next read; a rolled-back write is silent corruption. * fix(images): invalidate every stale rendition, fence the writes, and give the job a button (#1198) Three things from review, one of which mattered a lot. The invalidation was too narrow. Clearing only preview_path fixed the face data and left the gallery worse off: ensureThumbnail and ensureHeroImage return their cached file whenever it is merely VALID, and a pre-fix sideways thumbnail is perfectly valid — so a corrected row rendered the old sideways image inside a newly-corrected portrait tile. All three canonical renditions are cleared now, their stored objects deleted, and both responsive tier sets with them. The responsive tiers also needed handling rather than a hopeful catch. Their helpers swallow delete errors, and ensurePreviewImageAtWidth treats storage.stat(key) as a cache hit — so a tier that survived deletion keeps serving unrotated forever and never regenerates. The keys are re-checked after deletion and survivors are counted into the result, so a run that could not clear them does not report itself as clean. Writes are fenced on the identity that was measured, not just the id. replacePhoto swaps a new file under an existing row and rewrites path/filename, and it IS reachable — from the replace_by_name upload path in adminPhotos.js. A replacement landing while this job read the old original would otherwise have had the previous file's dimensions written over it and its fresh renditions cleared. And the job had no way to start it: the endpoint existed with no caller, so an upgrade would have left every affected library untouched unless an operator found the API themselves. It gets a Status card like its two neighbours, with strings in en/de/fr/sl. No backlog counter, because unlike the other two it cannot know how many rows need it without doing the work. * fix(images): make the backfill idempotent, and stop it lying about what it did (#1198) Six things from review round 2. The job was not idempotent, and the way it failed was expensive. Its trigger is the EXIF tag on the ORIGINAL, which correcting a photo never changes — so every re-run threw away the renditions it had just regenerated and requeued every completed face scan. On a face-enabled install, running it twice meant re-detecting the whole library for nothing. Migration 191 adds photos.orientation_checked_at, written in the same transaction as the work it records, with `force` as the escape hatch for an interrupted run. The candidate query selected preview_path but not thumbnail_path or hero_path, which the deletion loop reads — so those two pointers were cleared in the database while the objects stayed in storage, still reachable through previously issued URLs. watermark_path was missed entirely. gallery.js serves it ahead of the original when branding watermarking is on, which makes it the most visible rendition of the lot. (Its generator needed rotating too — that went into #1185, where the other three live.) storage.stat() RESOLVES with null for a missing key rather than rejecting, so counting "the promise settled" marked every deleted — and every never-created — tier as a survivor. A perfectly clean run told the operator to re-run. Now a null means gone, and a rejection counts as stuck, since a storage error is not proof the object went away. Face data is invalidated whenever the stored dimensions change, not only when the change came from rotation: boxes are scaled by photo.width at read time, so any dimension change strands them. And `corrected` now comes from the affected-row count. If the fence rejected the write because the file was replaced mid-run, the photo was not corrected and the run must not claim it was. * fix(images): stop the backfill doing unnecessary work, and make its retry advice true (#1198) Round 3, four points, all narrower than the last two rounds. It re-processed photos that were already correct. A 5-8 rotation changes the dimensions, so a tagged photo whose stored dimensions are ALREADY oriented must have been ingested after #1185 — its renditions are fine and clearing them deletes valid files and rescans a completed face detection for nothing. Those are now skipped and simply marked. Orientations 2, 3 and 4 (and 5-8 on a square image) leave the dimensions identical either way, so they carry no such evidence and are still done once. The retry advice was impossible to follow. When a responsive tier could not be deleted the row was still marked, so the ordinary re-run the UI recommends found nothing and the stale tier kept serving unrotated forever. The marker is withheld when a tier survives, which is what makes that message honest. Storage cleanup now only runs when a fenced write actually landed. If the file was replaced mid-run every update matched zero rows, but the deletion went ahead anyway and could destroy renditions belonging to the REPLACEMENT — watermarks especially, which are keyed by photo id and alias straight onto the new file. And the full-photo ETag includes the backfill's timestamp. It was built from the ORIGINAL's mtime plus the watermark settings hash, neither of which this job touches — so a guest holding a pre-fix ETag would go on getting 304 and their cached sideways image no matter how many times the backfill succeeded. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
c18f54ede0 |
fix(images): respect EXIF orientation in thumbnails, heroes and previews (#1194)
* fix(images): respect EXIF orientation in thumbnails, heroes and previews (#1185) generateThumbnail, generateHeroImage and generatePreviewImage went straight from sharp(imagePath) to .resize(), so a photo whose Orientation tag is not 1 — routine for portrait shots on bodies that tag rather than rotate the sensor data — was resized from the raw frame and came out sideways. The same pipelines then call .withMetadata(false), stripping the tag from the output, so nothing downstream could correct it either. The download path already had this right: resizeToBox calls probe.rotate() for stills, which is why the same photo looked correct on download and rotated in the gallery. All three generators now do the same, guarded to stills for the reason resizeToBox already documents — .rotate() flattens a multi-frame source. The reporter also spotted the half that compounds it: photos.width/height were stored from sharp's metadata, which reports pixels as STORED, not as displayed. For orientation 5-8 those are swapped, so a portrait photo landed in the database as landscape and masonry/justified sized its tile with the wrong aspect ratio on top of the image being unrotated. A shared orientedDimensions() helper now does that conversion at all four capture sites — managed upload, background processing, external import and the dimension repair — so the stored numbers describe the rotated result the generators now produce. Existing rows keep their pre-rotation dimensions until the photo is reprocessed; the images themselves correct on the next thumbnail/preview regeneration. Tests fail on the unfixed generators — verified by reverting the rotate calls and the swap, which fails 4 of the 7. * fix(images): orient dimensions on every ingest path, and stop guarding rotate where it protects nothing (#1185) Review found the first cut covered four of eight dimension-capture sites. The filesystem watcher, the S3 auto-importer, the v1 upload API and replace-by-name all still persisted raw metadata.width/height, so an orientation 5-8 photo arriving that way got a correctly rotated thumbnail and a database row describing it as landscape — the same aspect-ratio mismatch this PR set out to remove, just on the paths I had not grepped. (I searched for `metadata.width` and the v1 route aliases it to `meta`.) The animated guard was also wrong in two of the three generators. generateThumbnail and generateHeroImage never pass `animated: true`, so they already flatten a multi-frame source to its first frame — skipping .rotate() there protected an animation that was being discarded anyway, while leaving the output in raw orientation against swapped stored dimensions. Both now rotate unconditionally. generatePreviewImage keeps the guard, because it genuinely does open animated sources as animated and .rotate() would flatten them. That leaves one corner unsolved rather than papered over: a multi-frame source that also carries an orientation tag keeps its raw orientation in the preview while the thumbnail and stored dimensions describe the rotated one. GIF has no EXIF and animated WebP effectively never sets it, so it is a real gap but not a common one, and closing it means rotating frame by frame rather than quietly dropping the animation. Documented at the guard. * fix(images): add a recompute mode so existing libraries get corrected too (#1185) The orientation fix only helped new photos. A row affected by the bug has BOTH dimensions stored — just in the raw order — so the repair job's NULL filter could never reach exactly the rows that needed it. Worse, once their thumbnails regenerated rotated, those rows went from consistently-wrong (sideways image in a matching tile) to inconsistent: correct image, wrong-shaped tile. `recompute` widens the candidate set to every image row. Opt-in, because it re-reads every original. It also has to deal with the consequence for faces. Detection runs against the preview and stores boxes in ORIGINAL pixel space, scaled by `photo.width / previewMeta.width` (faceProcessor.js:220-224) — so a photo whose stored dimensions change has face data recorded against a coordinate system that no longer exists, and the overlays crop the wrong region. Photos whose dimensions actually change are requeued for scanning; ones that were already correct are not, or a routine repair would rescan the whole library. Rows with face_status NULL are left alone so installs that never enabled the feature don't start scanning because of a dimension repair. Writing the test for that last rule caught a real bug in it: the candidate query never selected photos.width/height, so `photo.width` was undefined and every row compared as changed. Both columns are selected now. * Revert "fix(images): add a recompute mode so existing libraries get corrected too (#1185)" This reverts commit cb771d08. Review round 3 found five problems, all of them in this addition rather than in the orientation fix itself, and one of them an own-goal: requeueing face scanning makes processPhotoFaces call ensurePreviewImage, which returns the CACHED pre-fix preview when it is still a valid image — so the rescan reads unrotated pixels and scales those boxes by the newly corrected dimensions. That is worse than leaving the data alone. The rest need work this PR should not be carrying: the dimension repair reads originals through resolvePhotoFilePath and plain sharp, so it does nothing on an S3 install and rejects RAW/DNG; recompute pulls archived rows whose originals were deleted on archive; orientation 2, 3 and 4 change the pixels without changing width or height, so a dimension-delta test never notices them; and the dimension write and the face invalidation are not atomic, so a failure between them leaves a row that no retry will ever requeue. Split out so it can be designed and reviewed on its own. The orientation fix — .rotate() in the three generators and orientedDimensions() at all eight ingest sites — is unaffected and stays. * fix(images): the watermarked rendition needs orienting too (#1185) A fourth generator with the same bug, found while reviewing the backfill that builds on this. watermarkService composites and re-encodes through its own sharp pipeline with no .rotate(), and gallery.js serves photos.watermark_path ahead of the original when branding watermarking is on — so on a watermarked gallery the sideways image is precisely what a guest sees. Two details this needed beyond the .rotate() itself: metadata() is read from a separate, unrotated handle. .rotate() does not change what metadata() reports — a 400x200 source tagged orientation 6 still reads 400x200 — and every use of those numbers here is positioning: watermark scale, font size, composite extent. They have to be the DISPLAYED dimensions or the mark is placed against the wrong axis, so they go through orientedDimensions. The composite offsets are floored. getPositionCoordinates derives from the SVG's estimated text extent and returns fractional pixels; sharp rejects a non-integer offset and applyWatermark catches its own error and returns the image unwatermarked. Landing on a whole pixel was luck, and changing the dimensions it is computed from ran out of it — the test surfaced a real "Expected integer for left but received 92.8". --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
849a5807b7 |
fix(admin): make "Storage used" report storage used (#1164) (#1170)
* fix(admin): make "Storage used" report storage used (#1164) The tile summed photos.size_bytes — the catalogued size of the ORIGINALS, which has no relationship to the disk PicPeak runs on. In reference mode those files are never copied and sit on the NAS; duplicate rows counted the same file twice (#1162); and it ignored everything PicPeak genuinely does write locally: thumbnails, previews, hero renditions, watermarks and the per-event download cache. The reporter's tile read ~80 GB against 21 GB of real usage. Worse than the label: the same number drove the storage soft-limit warning bar and, via /storage/info, the recommended soft limit — so a reference-mode install got a disk-capacity recommendation computed from bytes that are not on the disk. - new localStorageUsage service walks the storage root and reports the total plus a breakdown. Walking rather than summing DB columns is the point: thumbnail/preview/hero rows record a key and never a byte count, and orphans from a deleted event or an interrupted import are real bytes. Symlinks are not followed, so a link into the media mount cannot put the NAS back in the total. Cached for 5 minutes, since the dashboard polls. - the dashboard tile and /storage/info now report that, with the catalogued figure kept and labelled as such next to it. A failed measurement reads as "unavailable" rather than substituting a number that means something else. On the local rig: 64.37 MB used against 15.75 MB catalogued, of which 27.9 MB is watermarks and 6.8 MB is download cache — none of which the old figure could see. Not addressed here: `.download-cache/all.zip` still has no TTL or size cap. It is now at least visible in the breakdown, which is what makes the case for capping it. * fix(admin): exclude the media share from local storage usage (#1164) External review found the walk could reintroduce the exact over-count it replaces. EXTERNAL_MEDIA_ROOT's compose default is `<storage>/external-media`, where the NAS is bind-mounted. That is a plain directory, not a symlink, so the symlink guard did not cover it and the walk descended into the share — putting every referenced original back into a figure whose whole purpose is to leave them out, and comparing NAS bytes against statfs() of the local disk. On the reference-mode installs this issue is about, that is the failure mode reappearing inside its own fix. The configured root is now skipped when it lies inside the storage root, and the result reports which path was excluded. A directory that merely shares the name is still counted, because those really are local bytes. Also from the review: - concurrent cold-cache callers now share one walk. /dashboard/stats, /storage/info and the sidebar are routinely requested together, and each was starting its own stat-per-file traversal of the whole library. - storage_partial is surfaced in the StorageInfo type and the sidebar tile, not just the dashboard and analytics cards. An unreadable subtree makes the total a floor, and a floor silently compared against a soft limit reads as "safely under". * fix(admin): do not report a disk walk on an S3 backend (#1164) Second review round. S3 installs were regressed. With STORAGE_BACKEND=s3 the originals, renditions, archives and download caches are objects in the bucket and STORAGE_PATH holds only incidental local files — so the walk reported near-zero and the soft-limit recommendation was derived from it. Those installs now keep the catalogued figure, which is the approximation they had before this PR, and the response says which measurement it is (`storage_measurement: 'disk' | 'catalog'`) so the UI labels it instead of implying a disk measurement that never happened. The Settings → Status storage card ignored storage_partial, formatting a lower bound as exact and deriving the limit percentage from it — so an unreadable subtree could read as safely under the limit. It now carries the same `+` marker as the sidebar and dashboard. * fix(admin): stop rendering an absent measurement as zero usage (#1164) Third review round, two findings. The analytics storage bar coerced a null measurement to 0, drawing an empty bar labelled "0% of limit" and suppressing the over-limit state — reading as plenty of room at exactly the moment nothing is known. It now shows the catalogued figure on S3, where that IS the available answer, and says "no measurement available" rather than inventing a percentage when there is none. /storage/info walked the filesystem before checking the backend and then threw the result away on S3. The sidebar polls that endpoint, so a migrated install still holding a large local tree paid a full stat-per-file traversal on every cold cache for nothing. Gated before the walk, as the dashboard route already was. * fix(admin): tell "no disk to measure" apart from "the measurement failed" (#1164) External review of the stable twin. Both were reported as `storage_measurement: 'catalog'`, so a failed local walk made the dashboard claim the objects live in S3. They are different things — one is a fact about the install, the other is a fault — and there is now an `unavailable` state for the second. The analytics percentage could reach the billions. `safeSoftLimit` fell back to `storageUsed || 1`, and on S3 that is null → 1, while the figure beside it came from `catalogedBytes`. An editor or viewer holds `analytics.view` but not `settings.view`, so `/storage/info` 403s for them and `storageInfo` is undefined — which is exactly when that fallback fires. It now falls back to the measured figure, and suppresses the percentage entirely when there is no real limit rather than dividing usage by itself and always reading 100%. Also lands the AnalyticsPage half of the previous round, which the commit message claimed but the commit did not contain — only its backend counterpart was staged. The stable twin has carried it since it was written, so this is the parity gap in the unusual direction. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
1366d6d14c |
fix(previews): preserve alpha and animation in the preview tier (#1171)
* fix(gallery): stop the lightbox loading originals to display a photo (#1166) The lightbox read `preview_url`, which the server only emits once an admin has flipped lightbox_preview_enabled — off by default. So a stock install fell straight through to `url`, the untouched original: a reporter measured 16.5 MB for a photo whose preview is 345 KB. The lightbox renders its neighbours too, so opening one photo pulled three originals. `slideshow_url` is the same /preview/:id URL, watermark query included, and has been emitted unconditionally for images since #1015 — the slideshow never had a fallback worth taking. Preferring it fixes every existing install with no migration and no admin action, and `url` still backstops videos, where both derivative URLs are null. Verified on the local rig with the toggle off, so the photos API returns preview_url: null exactly as filed. Opening one photo: before GET /photo/82, /photo/81, /photo/21 (3 originals) after GET /preview/82?w=1280, /preview/81, /preview/21 397 KB -> 23 KB per image on that gallery's test photos. The toggle no longer decides whether the lightbox uses previews, so its copy said something untrue; it now describes what it still does, which is pre-generate rather than wait for the first guest to open a photo. Updated in en/de/fr/sl, the locales that carry those keys. * fix(gallery): cover the layouts the lightbox fix missed (#1166) External review found the fix was incomplete, and the review of it found one more. Premium galleries were untouched. PhotoGridWithLayouts returns early for gallery-premium, which builds its own yet-another-react-lightbox slides with `src: photo.url` — so those galleries kept pulling full originals and the reported bandwidth problem remained. They now use lightboxImageUrl for the display source; `download` deliberately stays on photo.url, because what a guest saves must be the original. The Story layout was worse, and neither the issue nor the review caught it: StoryPhotoCard rendered the full original as its GRID TILE, at object-cover in a small card. That is the one place where "hundreds of megabytes for a gallery" was literally true. It now uses the per-device thumbnail tier like PhotoCard, and its PhotoSwipe source uses the preview tier. Animated GIFs keep the original. generatePreviewImage always encodes JPEG, so routing an animated source through the preview tier would have replaced the animation with its first frame — a regression the toggle-off default never had. Animated WebP has the same problem and cannot be distinguished by MIME alone; that needs the backend to report it (Sharp's `metadata.pages > 1`) and is left rather than costing every static-WebP gallery the bandwidth fix. The settings copy claimed too much. "Pre-generate lightbox previews" does not generate anything on save — it unlocks the regenerate button and keeps preview_url emitted. Reworded to say that, in en/de/fr/sl. Not changed: the review's P1 said this bypassed the secure-image route on enhanced/maximum galleries. It does not. AuthenticatedImage collects requiresToken and secureUrlTemplate into an explicitly-voided unusedProps and never substitutes {{token}}, so on those protection levels photo.url was a literal `.../secure/82/{{token}}` that returns 400 — the lightbox was falling back to the 300px thumbnail, not to a protected image. Verified against a live maximum-protection gallery. Codex withdrew the finding on that evidence. * fix(gallery): keep premium downloads working and story framing intact (#1166) Second review round, three findings — two of them regressions this PR introduced. Premium Download became a no-op. handleDownloadFromLightbox recovered the photo with `filteredPhotos.find(p => p.url === slide.src)`, and slide.src is a derivative now, so the lookup found nothing and the button silently did nothing. The slide carries the photo id and the handler resolves by that; what Download hands over is still the original. Story cards were reframed. thumbnail_fit is seeded to 'cover' on every install, so thumbnails are square centre-crops — and story cards are not square (400x500 in the carousel, fixed-height in the desktop grid), so the card's own object-cover cropped them a second time and every photo shifted. They now use the preview tier, which is fit:'inside' and therefore the whole frame: the card looks exactly as it did before, without pulling an original. APNG joins the animated-format guard. It declares image/apng and the preview route would serve a static frame. Animated WebP still cannot be detected from MIME and remains the documented gap. * fix(gallery): keep PNG on the original, alpha and all (#1166) Third review round. generatePreviewImage encodes JPEG, which drops ALPHA as well as animation — a transparent PNG came back flattened against a solid background. And an APNG is normally reported as image/png, so the image/apng check alone missed the common upload path. PNG now stays on the original: it is where transparency is the norm, and rare enough in an event gallery that the bandwidth given up is small. Animated or alpha WebP still cannot be detected from MIME and remains the documented gap; it needs the backend to report Sharp's `pages`/`hasAlpha`. Two further findings are acknowledged and deferred rather than fixed here: - Story cards now request /preview on mount, so a cold gallery generates its previews in one burst. That is a new CPU cost, not a regression — those cards previously fetched full ORIGINALS on mount, which is strictly worse. Doing it properly means viewport-gating AuthenticatedImage, which is a change to a component every gallery surface uses and belongs in its own PR. - The premium layout memoizes slide URLs, so rotating the device before opening the lightbox can leave a photo on the tier chosen for the old geometry. The result is a slightly undersized image, and the fix is a resize subscription this PR does not otherwise need. * fix(gallery): load Story images on approach, and give the hero its own tier (#1166) Every card in a Story gallery mounts at page load — `whileInView` gates the animation, not the render — and AuthenticatedImage fetches from an effect on mount, so all of them requested at once. That was tolerable while they pointed at photo.url, because nothing was generated; pointing them at the preview tier meant a gallery with cold previews would Sharp-decode every original in one burst. The image now waits until the card is within 200px of the viewport, using framer-motion's useInView — the same observer the entrance animation already relies on — with `once` so a card never unloads on scroll-away. Verified on a 62-photo Story gallery: 3 images fetched at load, growing to 15 as you scroll, where all 62 would have fired before. While confirming that, the hero turned out to be doing the same thing the cards were. StoryHero rendered photo.url as a full-bleed object-cover background — a full original on the critical path for first paint of every Story gallery — when hero_url exists for exactly this and is a 1920x1080 cover crop emitted unconditionally for every photo (gallery.js:1139). That gallery now issues no /photo/ request at all: hero_url for the hero, the preview tier for the cards, and only as they come into range. * fix(previews): preserve alpha and animation in the preview tier Follow-up to #1166, which had to bypass the preview tier for GIF, APNG and PNG to avoid a visible regression. This removes the cause. generatePreviewImage encoded JPEG unconditionally. JPEG has no alpha channel and no second frame, so a transparent PNG came back flattened onto a solid background and an animated GIF came back as its first frame — for every consumer of this tier, not just the lightbox: the slideshow (#1015), admin previews, and the face avatars that read it as a whole-frame rendition. It was only invisible by default because the lightbox served originals. Sources with alpha, or more than one page, are now encoded as WebP, which carries both and is still far smaller than the original. Ordinary photos stay JPEG — the common path pays nothing. Two things had to move with it: - The output extension now matches what was written. A PNG source previously produced `preview_foo.png` holding JPEG bytes; harmless while the route hard-coded image/jpeg, wrong once the encoding varies. Existing keys keep working — they are still JPEG and still served as such. - The preview route derives Content-Type from the key. With `nosniff` set, mislabelling would show a broken image rather than being silently corrected. The watermark branch re-encodes to JPEG, so it labels itself explicitly; preserving animation through the watermark compositor is a separate problem. The frontend guess-by-MIME goes away entirely — including the case it could never get right, since a still and an animated WebP declare the same type. Verified on the local rig: a transparent PNG round-trips as `Content-Type: image/webp`, `hasAlpha: true`, 8.3 KB; an ordinary photo still serves `image/jpeg` from a `.jpg` key. * fix(previews): retire the legacy preview keys, and stop mislabelling watermarked ones External review of the stable twin found two defects, both on this branch too. Legacy keys collide with the new naming. The old generator kept the SOURCE basename verbatim while always writing JPEG, so a `.webp` upload produced `previews/preview_shot.webp` holding a JPEG. My PR body claimed "pre-existing keys have no .webp suffix and are JPEG" — that was simply wrong. The route now derives Content-Type from the key and the response carries nosniff, so every photo uploaded as WebP would have rendered as a broken image in the lightbox. Legacy `.png` keys are wrong the other way: flattened JPEGs of what may have been transparent sources, which isPreviewValid would have let stand forever. Migration 188 clears photos.preview_path outright — all of it, not just the suspicious extensions, because a `.jpg` key can equally be a flattened rendition and nothing in the key says so. Previews regenerate lazily on next view under the new encoder, so the cost is one regeneration per photo actually viewed. Storage is untouched, as elsewhere. The watermark branch mislabelled its output. applyWatermark PRESERVES the source format (watermarkService.js:200-211: png stays png, webp stays webp), and its input is the preview — so the output already matches the key the header was derived from. Forcing image/jpeg mislabelled every watermarked WebP preview, and nosniff means the browser would not correct it. The override is gone; the animation loss through the compositor is documented where it happens. * fix(gallery): make the Story hero fix actually work on external galleries (#1166) External review of the stable twin, both applying here too. hero_url was inert for external media. ensureHeroImage only ever called resolvePhotoStorageKey, which returns null for external/reference photos by design — and that null was handed straight to withLocalCopy, which throws, so the hero route caught it and redirected to the full ORIGINAL. #1078 fixed exactly this shape for ensurePreviewImage and nobody carried it across. It stayed invisible until this PR pointed the Story hero at hero_url: on a managed gallery that is a real saving, on a reference-mode gallery it quietly changed nothing. ensureHeroImage now has the same external branch ensurePreviewImage does — direct fs read, per-photo output basename — and returns null instead of throwing for a reference-mode row with no source_origin. The format bypass trusted mime_type, which is not trustworthy here. Migration 039 backfilled every pre-existing photo to image/jpeg regardless of what it was, and adminExternalMedia inserts rows with no mime_type at all — so a mislabelled PNG sailed past the guard and came back flattened. It now checks the filename extension as well. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
77953c15c1 |
fix(gallery): stop the lightbox loading originals to display a photo (#1166) (#1169)
* fix(gallery): stop the lightbox loading originals to display a photo (#1166) The lightbox read `preview_url`, which the server only emits once an admin has flipped lightbox_preview_enabled — off by default. So a stock install fell straight through to `url`, the untouched original: a reporter measured 16.5 MB for a photo whose preview is 345 KB. The lightbox renders its neighbours too, so opening one photo pulled three originals. `slideshow_url` is the same /preview/:id URL, watermark query included, and has been emitted unconditionally for images since #1015 — the slideshow never had a fallback worth taking. Preferring it fixes every existing install with no migration and no admin action, and `url` still backstops videos, where both derivative URLs are null. Verified on the local rig with the toggle off, so the photos API returns preview_url: null exactly as filed. Opening one photo: before GET /photo/82, /photo/81, /photo/21 (3 originals) after GET /preview/82?w=1280, /preview/81, /preview/21 397 KB -> 23 KB per image on that gallery's test photos. The toggle no longer decides whether the lightbox uses previews, so its copy said something untrue; it now describes what it still does, which is pre-generate rather than wait for the first guest to open a photo. Updated in en/de/fr/sl, the locales that carry those keys. * fix(gallery): cover the layouts the lightbox fix missed (#1166) External review found the fix was incomplete, and the review of it found one more. Premium galleries were untouched. PhotoGridWithLayouts returns early for gallery-premium, which builds its own yet-another-react-lightbox slides with `src: photo.url` — so those galleries kept pulling full originals and the reported bandwidth problem remained. They now use lightboxImageUrl for the display source; `download` deliberately stays on photo.url, because what a guest saves must be the original. The Story layout was worse, and neither the issue nor the review caught it: StoryPhotoCard rendered the full original as its GRID TILE, at object-cover in a small card. That is the one place where "hundreds of megabytes for a gallery" was literally true. It now uses the per-device thumbnail tier like PhotoCard, and its PhotoSwipe source uses the preview tier. Animated GIFs keep the original. generatePreviewImage always encodes JPEG, so routing an animated source through the preview tier would have replaced the animation with its first frame — a regression the toggle-off default never had. Animated WebP has the same problem and cannot be distinguished by MIME alone; that needs the backend to report it (Sharp's `metadata.pages > 1`) and is left rather than costing every static-WebP gallery the bandwidth fix. The settings copy claimed too much. "Pre-generate lightbox previews" does not generate anything on save — it unlocks the regenerate button and keeps preview_url emitted. Reworded to say that, in en/de/fr/sl. Not changed: the review's P1 said this bypassed the secure-image route on enhanced/maximum galleries. It does not. AuthenticatedImage collects requiresToken and secureUrlTemplate into an explicitly-voided unusedProps and never substitutes {{token}}, so on those protection levels photo.url was a literal `.../secure/82/{{token}}` that returns 400 — the lightbox was falling back to the 300px thumbnail, not to a protected image. Verified against a live maximum-protection gallery. Codex withdrew the finding on that evidence. * fix(gallery): keep premium downloads working and story framing intact (#1166) Second review round, three findings — two of them regressions this PR introduced. Premium Download became a no-op. handleDownloadFromLightbox recovered the photo with `filteredPhotos.find(p => p.url === slide.src)`, and slide.src is a derivative now, so the lookup found nothing and the button silently did nothing. The slide carries the photo id and the handler resolves by that; what Download hands over is still the original. Story cards were reframed. thumbnail_fit is seeded to 'cover' on every install, so thumbnails are square centre-crops — and story cards are not square (400x500 in the carousel, fixed-height in the desktop grid), so the card's own object-cover cropped them a second time and every photo shifted. They now use the preview tier, which is fit:'inside' and therefore the whole frame: the card looks exactly as it did before, without pulling an original. APNG joins the animated-format guard. It declares image/apng and the preview route would serve a static frame. Animated WebP still cannot be detected from MIME and remains the documented gap. * fix(gallery): keep PNG on the original, alpha and all (#1166) Third review round. generatePreviewImage encodes JPEG, which drops ALPHA as well as animation — a transparent PNG came back flattened against a solid background. And an APNG is normally reported as image/png, so the image/apng check alone missed the common upload path. PNG now stays on the original: it is where transparency is the norm, and rare enough in an event gallery that the bandwidth given up is small. Animated or alpha WebP still cannot be detected from MIME and remains the documented gap; it needs the backend to report Sharp's `pages`/`hasAlpha`. Two further findings are acknowledged and deferred rather than fixed here: - Story cards now request /preview on mount, so a cold gallery generates its previews in one burst. That is a new CPU cost, not a regression — those cards previously fetched full ORIGINALS on mount, which is strictly worse. Doing it properly means viewport-gating AuthenticatedImage, which is a change to a component every gallery surface uses and belongs in its own PR. - The premium layout memoizes slide URLs, so rotating the device before opening the lightbox can leave a photo on the tier chosen for the old geometry. The result is a slightly undersized image, and the fix is a resize subscription this PR does not otherwise need. * fix(gallery): load Story images on approach, and give the hero its own tier (#1166) Every card in a Story gallery mounts at page load — `whileInView` gates the animation, not the render — and AuthenticatedImage fetches from an effect on mount, so all of them requested at once. That was tolerable while they pointed at photo.url, because nothing was generated; pointing them at the preview tier meant a gallery with cold previews would Sharp-decode every original in one burst. The image now waits until the card is within 200px of the viewport, using framer-motion's useInView — the same observer the entrance animation already relies on — with `once` so a card never unloads on scroll-away. Verified on a 62-photo Story gallery: 3 images fetched at load, growing to 15 as you scroll, where all 62 would have fired before. While confirming that, the hero turned out to be doing the same thing the cards were. StoryHero rendered photo.url as a full-bleed object-cover background — a full original on the critical path for first paint of every Story gallery — when hero_url exists for exactly this and is a 1920x1080 cover crop emitted unconditionally for every photo (gallery.js:1139). That gallery now issues no /photo/ request at all: hero_url for the hero, the preview tier for the cards, and only as they come into range. * fix(gallery): make the Story hero fix actually work on external galleries (#1166) External review of the stable twin, both applying here too. hero_url was inert for external media. ensureHeroImage only ever called resolvePhotoStorageKey, which returns null for external/reference photos by design — and that null was handed straight to withLocalCopy, which throws, so the hero route caught it and redirected to the full ORIGINAL. #1078 fixed exactly this shape for ensurePreviewImage and nobody carried it across. It stayed invisible until this PR pointed the Story hero at hero_url: on a managed gallery that is a real saving, on a reference-mode gallery it quietly changed nothing. ensureHeroImage now has the same external branch ensurePreviewImage does — direct fs read, per-photo output basename — and returns null instead of throwing for a reference-mode row with no source_origin. The format bypass trusted mime_type, which is not trustworthy here. Migration 039 backfilled every pre-existing photo to image/jpeg regardless of what it was, and adminExternalMedia inserts rows with no mime_type at all — so a mislabelled PNG sailed past the guard and came back flattened. It now checks the filename extension as well. * test(gallery): the hero fixture follows the root-relative relpath contract (#1166) external_relpath has been resolved from EXTERNAL_MEDIA_ROOT rather than from event.external_path since #1163 landed. This fixture still carried the base-relative form — its own comment noted the change was 'a separate stack' — so the two tests stopped resolving and ensureHeroImage returned null the moment that stack merged. The production path was never affected. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
05e23ef1a1 |
fix(admin): move the maintenance sweeps' run state into the database (#1181) (#1184)
Both photo sweeps tracked whether they were running in a module-level variable. Correct on one replica, wrong behind a load balancer: the status poll answers from whichever process it reaches, so an idle replica reports isRunning false while another is mid-run, the UI re-enables the button, and the next POST lands elsewhere and starts a second pass over the whole library. The .whereNull() guards mean nothing is corrupted; the cost is duplicated S3/NAS I/O and an operator who cannot tell whether a job is running. Migration 189 adds one row per job. The claim is a conditional UPDATE whose affected-row count is the answer — the shape backgroundProcessor already uses to hand a photo to exactly one worker — so two replicas cannot both match. The lease is fenced on a per-claim token: taking over a stale claim does not stop the old runner, so without fencing a superseded runner finishing late cleared the new owner's flag and overwrote its result. heartbeat() reports renewal failure and the loops stop on it. Renewal runs on a timer spanning the claim through release, including the candidate query, because one hung NAS read can outlast the stale window inside a single iteration. maintenance_jobs is excluded from .picpeak archives — an archive taken mid-sweep would otherwise restore a live lease with no runner to release it. The importer filters the same set, so older archives are skipped too. Response shape is unchanged, so the frontend needs no change. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
410b8f8f6f |
fix(external-media): record capture dates on import, and backfill existing libraries (#1172) (#1179)
* fix(external-media): record capture dates on import, and backfill existing libraries (#1172) External imports never read EXIF, so photos.captured_at stayed NULL for every row they created. The gallery sorts "Date Taken" with COALESCE(captured_at, uploaded_at), which on a bulk import is the import timestamp — so the sort silently degraded into "order by import batch" with no error and nothing in the UI to say the sort key was missing. The reporter's 12-day trip came back with its first two days at positions 4204-5296 of 5555, because those folders happened to be imported second. - the import reads the capture date next to the sharp().metadata() call that already opens the file, so this costs one more read of the same source rather than a second pass over the mount. Best-effort like the dimensions: a source without EXIF imports with captured_at NULL, as before. - POST /api/admin/photos/repair-capture-dates backfills existing libraries, modelled on the dimension repair beside it — background pass, in-flight guard, status endpoint, and resolvePhotoFilePath, which is what reaches an external row at all. Not a migration: the originals sit on a mount that may be down at upgrade time, reading 8000+ of them would block the boot, and a run that found nothing has to be repeatable. - "no EXIF date" is counted separately from "could not read the file". An operator needs to tell "these files carry no date" from "the mount is broken" before deciding to re-run. - the update is guarded whereNull, so an import finishing mid-run is not overwritten by a slower pass. - every sort branch now carries photos.id as a tiebreaker, not just capture_date. A bulk import writes hundreds of rows inside one second, so uploaded_at and the COALESCE fallback both collapse and the grid reshuffles between loads. id is insertion order, which makes the fallback meaningful. Not addressed: extractCaptureDate reads no OffsetTimeOriginal, and exifr resolves a naive EXIF timestamp against the HOST timezone — so captured_at is not a true instant, and the same file imported on two machines yields two values. That predates this and applies to managed uploads equally; the tests here deliberately assert ordering rather than an absolute instant so they do not encode the bug. Worth its own issue. * fix(capture-dates): read managed originals through storage, skip archived, claim the run flag (#1172) Four holes in the backfill endpoint, all found in review: - Managed photos were resolved with resolvePhotoFilePath, which builds a STORAGE_PATH filesystem path. On an S3 install nothing is there, so every managed row failed. Now split the way the thumbnail regenerator does: external rows read from the mount directly, managed rows go through resolvePhotoStorageKey + withLocalCopy. - Archived events keep their photos rows but their originals are deleted on archive, so those rows failed every run and kept the button lit forever. Excluded from both the job and the status counts. - isRunning was claimed after the candidate query, so two concurrent POSTs could both pass the guard and start a pass. Claimed before the await, with every early exit releasing it. - The noExif comment promised a distinction extractCaptureDate does not make (it returns null for unreadable files too). Reworded to what it is. * chore: drop a stray node_modules symlink committed by mistake The .gitignore pattern is `node_modules/`, which matches a directory and not a symlink of the same name, so a local convenience link slipped past it. It pointed at an absolute path on one machine and would dangle everywhere else, breaking `cd backend && npm install`. * fix(capture-dates): gate the backfill as system maintenance, stop overstating the counters (#1172) The endpoint walks every event in the install and rewrites their metadata, but required only photos.edit — which the built-in team_photographer preset holds (175_granular_permissions_and_presets.js:106). That role exists for a contributing shooter, who should not be able to start a whole-library S3/NAS scan or touch another owner's photos. Now system.manage, with the status endpoint on system.view so the panel simply stays hidden for everyone else. The "without EXIF date" wording also promised a distinction the code does not draw: extractCaptureDate returns null for an unparseable file as well as for one that genuinely carries no date, so both land in that bucket. Reworded to "no date found" / "unreachable" in en, de and fr, which is what the two numbers actually separate. * docs: point the permission note at the follow-up PR (#1172) The dimension repair's matching gate landed in #1182, so the comment no longer needs to describe it as unaddressed. * fix(i18n): align the Slovenian capture-date wording with the other locales (#1172) sl was missed when the counters were reworded from 'without EXIF date' / 'unreadable' to what they actually measure. * fix(capture-dates): gate the status card on the permission the button needs (#1172) system.view and system.manage are independent grants, and StatusTab has no permission gate of its own — a successful status payload is what renders the card and its enabled button (StatusTab.tsx:637). Gating the status endpoint on system.view therefore handed a system.view-only role a live Backfill button whose every click 403s, with no error surfaced by the mutation. The comment above it already claimed this endpoint matched the POST. Now it does. * fix(gallery): make the Date Taken sort correct on SQLite (#1172) photos.captured_at does not hold one type on SQLite. Three writers put three different things in it: integer managed uploads — photoProcessor.js:488 hands knex a Date, which the sqlite3 binding stores as epoch milliseconds text external imports and the backfill, which write ISO-8601 null no capture date, so the sort falls through to uploaded_at, itself text in knex's 'YYYY-MM-DD HH:MM:SS' default shape A plain COALESCE over that is not an ordering. SQLite sorts INTEGER before TEXT unconditionally, so every managed photo carrying EXIF came back ahead of every photo that did not, whatever the dates said — a 2027 capture landing before a 2020 one. Among the text values 'T' (0x54) also outranks the space (0x20), so a same-day ISO 01:15 sorted behind a fallback 23:00. Both failures predate this branch — the first needs only two managed photos — but making that sort correct is what #1172 is about, so it is fixed here rather than left for the issue it belongs to. Normalised in the ORDER BY rather than by rewriting the column: the data fix would have to touch every existing row and every writer, which is a far heavier change than the sort it corrects. The cost is that this sort no longer uses idx_photos_captured_at on SQLite — an acceptable trade on the fallback engine, where the alternative is an index-assisted wrong answer. Postgres is untouched: captured_at is a real timestamp there and COALESCE already compares correctly. The regression tests drive the real gallery route on real SQLite. They write the epoch-millisecond integer directly, because the Date that produces it in production cannot be reproduced inside jest — there the binding's type dispatch misses sandbox Dates and stores "[object Object]" (CLAUDE.md). All four behavioural tests fail on the unfixed ORDER BY; verified by reverting it. * fix(gallery): normalise epoch-integer uploaded_at too, and stop polling a 403 (#1172) Two follow-ups from review. uploaded_at is not always text on SQLite either. A legacy archive restore leaves epoch milliseconds in it — there is a test pinning exactly that (__tests__/integration/sqliteEpochTimestamps.test.js) — and the fallback branch read it with substr(), so '1830297600000' was compared against '2020-01-01 00:00:00' as text and a 2028 upload sorted first. Both columns now get the integer/real branch. The status card also polled every ten seconds regardless of permission. With the endpoint correctly requiring system.manage, anyone who can open the Status tab but cannot run the job would have had a 403 and a logged denial every ten seconds for a panel they were never shown. The query is now gated on the same permission the endpoint requires, so it never starts. * style: quote convention in the capture-sort test (#1172) * fix(capture-dates): skip watcher-imported videos, and make the status counts consistent (#1172) Three follow-ups from review. fileWatcher.processNewPhoto sets type='video' and a video/* mime but never media_type (fileWatcher.js:128-130), so those rows keep the 'image' default from migration 048. Filtering on media_type alone queued every such video on every run — extractCaptureDate returns null for a video, captured_at stays null, and the backlog never cleared. Candidate query and status scope now check all three markers. The status counts were two separate queries, so an import committing a dated photo between them could be counted by the second and not the first: the card then showed withCaptureDate > total and a negative backlog, with the button enabled to "fix" it. One aggregate now. And the card's render checked only the cached payload. TanStack keeps that after `enabled` flips false, so a lower-privileged admin logging in behind a system.manage user inside the cache lifetime would still have seen the card and a button whose POST 403s. The permission is part of the render condition now. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
a7b74bcd87 |
fix(external-media): store external paths from the media root (#1163) (#1168)
* fix(external-media): store external paths from the media root (#1163) Importing a second folder into an event silently invalidated every photo already in it. photos.external_relpath was stored relative to events.external_path, and every import overwrites that column — so the older rows were rebased onto the new folder and their originals resolved to paths that do not exist. Nothing errored, and the grid still looked intact: thumbnails are written to local storage during the import while the base path is still correct. Only what needs the original broke — preview generation, the lightbox, downloads — which presents as a gallery that looks slow rather than one that is broken. The reporter had 7547 of 8004 rows pointing into the void and spent a while chasing it as a CPU problem. - external_relpath is now relative to EXTERNAL_MEDIA_ROOT, so a row is self-describing and nothing an admin does to the event afterwards can move an already-imported photo. - migration 187 folds each event's base path into its rows. Where the current resolution is missing on disk it walks up the base path for an ancestor under which the file IS there — the already-rebased case — and where it finds nothing it leaves the row resolving exactly where it resolves today. Skipped entirely when the media root is unmounted, since every file looks missing then. - the fold also runs after a .picpeak restore: knex_migrations is excluded from the archive, so a pre-#1163 backup would otherwise land base-relative rows on a migrated instance. - drops the duplicate-leaf-segment guess in photoResolver. It papered over this same double-prefixing and actively corrupts a root-relative path whose first segment legitimately repeats (base 'Trip', row 'Trip/x.jpg'). * fix(external-media): verify provenance and fold atomically (#1163) External review found four real defects in the fold. Repair could adopt the wrong file. Existence alone was accepted as proof that an ancestor candidate was the row's original — so a row whose file an admin simply deleted would adopt any same-named file one directory up (base `Trip/Sub`, relpath `photo.jpg`, an unrelated `Trip/photo.jpg`), and downloads would then serve a different photo. Worse than a dead link. An ancestor must now also match photos.size_bytes, which the import recorded from the very file the row describes; rows carrying no size are never repaired from an ancestor. The CURRENT base is still accepted on existence alone, because nothing is being inferred there — that is where the row already resolves. The fold was not atomic. Every UPDATE committed independently and the marker came last, so a process killed mid-fold left converted and unconverted rows with no marker — and the next run folded the converted ones a second time, putting every original one directory deeper with no undo. Probing is now a read-only first phase (so a slow cold NAS does not hold a write transaction open), and every rewrite plus the marker commit together. Failed rewrites certified a partial conversion. The per-row catch counted any error as a collision, carried on, and wrote the marker anyway — leaving that row in the old format for a resolver that now reads it differently. It also could not tell a genuine duplicate from a SQLite lock or I/O fault. Target collisions are now resolved in the planning phase, where they can be identified honestly, and a write that fails rolls the whole fold back. Restore ordering. The fold ran after the face requeue, with the worker live — so a worker could claim an external row while it was still base-relative, resolve it against the wrong path, and burn it to 'failed', a state only an explicit Re-scan clears. The fold now runs first, for the same reason the requeue already sat after restoreFiles. * fix(external-media): close the fold's remaining stranding paths (#1163) Second review round, three findings. A collision loser was left stranded. When an event imported one file through both `Trip` and `Trip/Sub`, two rows folded to the same path and the loser was skipped — keeping a base-relative value that the root-only resolver then reads as `<root>/<relpath>`, permanently wrong, with the marker claiming conversion was complete. It is a duplicate by construction, so it now goes through migration 186's deleteDuplicatePhotos, which reparents its feedback and marks and reconciles the face clusters instead of orphaning them. This branch is rebased onto #1162 for that helper. The other restore path had the same face-ordering bug. restoreService queued face scans in step 6, before step 7c runs pending migrations — so a pre-187 full or database restore handed the live worker rows whose paths were still event-relative, and it burned them to 'failed', a state the later fold does not clear. The requeue now happens after the migrations, where the files already are. A failed conversion was reported as a clean restore. The fold is transactional, so a failure leaves every external path in the old format under a resolver that reads from the media root — every original unreachable. It was logged as a warning and the restore returned success. It now returns externalPathsConverted/externalPathError, and suppresses the face requeue, which would otherwise mark those photos failed on top. * fix(external-media): make the fold safe against its own intermediate states (#1163) Third review round, four findings. A one-pass rewrite could collide with itself. Every FINAL path is distinct, but a final value can equal another row's CURRENT one — `photo.jpg` repairing to `Trip/photo.jpg` while the row already holding `Trip/photo.jpg` folds deeper — so the update violated migration 186's unique index halfway through. On Postgres that surfaces as 23505, which run-migrations-safe.js mistakes for "schema already exists" and records 187 as applied after the rollback, leaving every path unconverted with nothing to retry. Rows now park on a per-row staging value first, and migration 187 re-throws without the driver's code so the runner cannot misread it. The bulk update targeted rows the plan never saw. Phase 1 probes outside the transaction and can run for minutes; an import finishing in that window inserts an already root-relative row, and `where event_id` prefixed it again with the stale base. It now updates by the ids phase 1 captured. The restore UI never showed a conversion failure. The API carried externalPathsConverted, but PicpeakBackupCard neither declared nor read it and showed a green success either way — so an admin whose external originals were all unreachable was told the restore worked. restoreService requeued faces even when the migrations failed. The step 7c catch is deliberately non-fatal, so a pre-187 backup whose fold never ran still handed the live worker event-relative paths to burn to 'failed'. * fix(external-media): the fold's staging value must be storable on Postgres (#1163) External review of the stable twin caught this, and it was on both branches. The two-pass rewrite parks each row on a temporary value, and that value was written with a leading NUL. SQLite stores NUL in TEXT without complaint; Postgres rejects it outright — "invalid byte sequence for encoding UTF8: 0x00" — so migration 187 rolled back on exactly the installs that need the two-pass repair, and only on the engine most of them run. Restores hit the same wall and reported the conversion as failed. The prefix is ordinary text now. It still cannot collide with a real relative path and is still obviously wrong if a crash leaves one behind. Adds a gated Postgres test alongside the existing picpeakRestorePg one, because a SQLite-only suite structurally cannot catch this class: restoring the NUL makes exactly the two-pass repair case fail with that error, and nothing else. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
06da1b9f7e |
fix(external-media): one row per external file per event (#1162) (#1167)
* fix(external-media): one row per external file per event (#1162) Two overlapping import-external runs against the same event inserted every file twice. The route checked for an existing external_relpath and then inserted, with an fs.stat and a sharp().metadata() read sitting in between — a window wide enough for both runs to see "not there". A reporter's event held 8004 rows for 6012 distinct paths. Nothing at the storage layer stopped it: migration 041 created only a NON-unique (event_id, source_origin) index. - migration 186 removes the duplicates that already exist and adds a partial unique index on (event_id, external_relpath). The survivor is the lowest id that has a thumbnail, so a half-finished import does not cost a grid tile, and hero references are repointed first because the FK is SET NULL. - the route treats a unique violation as a skip and carries on, so a second writer this process cannot see (another replica) converges instead of duplicating or 500ing. - a second import while one is already running now gets a 409 rather than walking the whole tree to have every insert bounce. The duplicates' thumbnail files are left behind as unreferenced bytes — a migration is the wrong place to reach into storage, which may be S3. * fix(external-media): keep dependent rows and legacy restores intact (#1162) External review found two real defects in the dedupe half of this fix. Dangling rows on SQLite. Every FK into photos declares ON DELETE CASCADE, but PicPeak never sets `PRAGMA foreign_keys = ON` — the codebase says so where it deletes an event (adminEvents/helpers.js:245) — so on every SQLite install the cascade is inert and deleting a duplicate photo left its face embeddings, guest feedback and admin marks behind, pointing at an id that no longer exists. Biometric data outliving its photo is exactly the invariant the event delete goes out of its way to hold. Dependents are now handled explicitly, and moved rather than discarded where they can be: the duplicates were separate tiles in the grid, so a guest's comment or an admin's rating could legitimately be on either, and dropping it inside a fix for silent data loss would be its own bug. Where the target already holds an equivalent row — the same guest's like, the same admin's mark, the same transfer's entry — the loser is dropped, because those tables mean one row per (photo, actor). photo_faces is the deliberate exception: both rows were scanned, so moving would duplicate every embedding and split the person clusters built from them. Legacy restores. Suspending FK enforcement does not suspend a UNIQUE index on either engine, so a .picpeak backup taken before migration 186 — carrying exactly the duplicates it removes — would hit the new index mid-batchInsert and roll the whole restore back, after every table had already been emptied. The restore now drops the index for the load and rebuilds it after running the same dedupe. Also: a failed CREATE INDEX is no longer swallowed. Recording the migration as applied without it leaves the install permanently racy, with nothing to trigger a retry. The shared work moves to services/externalPhotoDedupe.js, which the migration and the restore both call. * fix(external-media): reconcile derived state around the dedupe (#1162) Second review round, four more real findings. The index throw did not actually stop anything. run-migrations-safe.js treats 23505 as "schema already exists" and marks the migration applied (run-migrations-safe.js:138) — and a CREATE UNIQUE INDEX that finds duplicate rows raises exactly 23505 on Postgres. A replica inserting one between the dedupe and the index lock is a real rolling-deploy shape, and the outcome was the thing the throw was added to prevent. The index is now verified against the catalog afterwards, and failure raises a code-less error the runner cannot mistake for idempotence. Two people sharing a device were treated as one. photo_feedback carries both guest_identifier (per device) and guest_id (per person, migration 078), and feedbackService scopes by guest_id when present. Keying equivalence on the identifier alone deleted one of two different people's ratings. It now uses the same COALESCE rule the service does. Deleting faces raw left ghost people. event_people counts and centroids are derived from the photo_faces rows being removed, and #1132's separation snapshots hold a copy of each side's centroid — which is why faceProcessor exposes purgePhotoFaces and says it is "called from every photo-deletion path". The dedupe now goes through it. Reparenting feedback left the survivor's totals stale. photos carries denormalized feedback_count / like_count / average_rating / favorite_count and the later reaction and colour counts, so a survivor that now owns feedback kept rendering zero. updatePhotoFeedbackStats takes a trx so the dedupe can recompute on its own connection. Also: the equivalence-key delimiter was a literal NUL byte, which made git classify the whole file as binary and hide its diff. Escaped. * fix(external-media): stop the dedupe discarding half-states (#1162) Third review round. Five findings, four applied. - is_hidden joins the feedback equivalence key. feedbackService lets a moderator-hidden row coexist with the guest's visible replacement and counts only the visible one, so ignoring it deleted the visible row as redundant. - admin marks merge instead of dropping. rating and color_label are written independently, so the same admin can have rated one tile and coloured the other; the loser now hands over any field the winner has no value for. - a survivor that loses the only completed scan is requeued. Otherwise the purge takes the sole embeddings and nothing re-queues it — the photo just silently stops having a face. - view_count and download_count are carried over. Those are real interactions recorded per row, and dropping them quietly lowered the engagement the admin grid shows. Not applied: repointing a category hero can in principle land on a survivor in another category. It needs the two duplicate rows to have been re-categorised apart after the racing import, and the result is a cosmetic hero mismatch that the admin category routes already guard on write. Not worth the extra branch in a data migration. * fix(external-media): invalidate the download zip when duplicates are removed (#1162) External review of the stable twin. Applies to both branches. The pre-built "download everything" archive still contained the duplicate rows the dedupe had just deleted, so guests kept receiving them until something else happened to invalidate it. Every ordinary photo-deletion path calls downloadZipService.invalidate for exactly this reason. The columns are cleared rather than the service being called: that service carries debounce timers and a regeneration queue, which is not something a migration should start. getZipInfo already treats a cleared record as a cache miss and rebuilds on the next request, so this is the durable half of what invalidate does. The stale object is left in storage for the same reason the duplicates' thumbnails are — a migration is the wrong place to reach into a backend that may be S3. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
c305ad4146 |
feat(faces): make "not the same person" survive a re-scan (#1132) (#1145)
A separation — an explicit dismissal, or the implicit one a Split records — was stored as a pair of event_people.id. Those ids do not survive re-derivation: recluster() deletes every person, and a full re-scan replaces a photo's faces outright, so face ids die too. The only thing that survives both is the embedding, so the decision is keyed on the two centroids the pair had when the photographer separated them. It binds while both sides still look like the clusters that were separated, and lapses once they have drifted past recognition. The constraint is now honoured at assignment time as well as in consolidate(), which is what makes it hold across a re-scan rather than being reformed before any later pass could object. Six review rounds shaped the matching itself: each candidate must resolve to the OPPOSITE side rather than merely matching something (a split leaves two similar halves, and the loose test fragmented the person the split was not even about); assignment judges both sides at the ordinary match threshold, since a single face — or a cluster of one part-way through a recluster — cannot resemble a settled centroid; separations carry their own model_version; and the projections are hoisted out of the innermost loop, which took a 2000-photo scan from ~15s of dot products to 0.23s. Lifecycle closed three ways: purgePhotoFaces re-anchors each side onto the live cluster it still describes and drops rows that describe nothing left, deleteEventCascade and the permanent archive delete clear the table (which deliberately has no event FK), and a later manual merge drops the separations it reverses. All of it matched on vectors rather than ids, since a row that has outlived a recluster names people who no longer exist. Merged with admin privileges: the author cannot self-approve. |
||
|
Paul Nothaft
|
2c81888eaf |
fix(gallery): a guest's own hidden feedback is hidden from them too (#1150) (#1153)
Everything in the system treats a hidden row as absent — getPhotoFeedback drops it even for the guest's own feedback, and updatePhotoFeedbackStats does not count it. The per-viewer is_liked heart and my_color_label badge read the row without looking at is_hidden, so a like the photographer had hidden still showed as liked on a photo whose like_count was zero. Making those agree exposes why it had not been fixed: the duplicate check behind like/favorite toggling did not skip hidden rows either, so the now-empty heart, when clicked, found the hidden row and toggled it OFF — the click did nothing visible and the moderation was silently undone. Skipping hidden rows there makes the click create a fresh, visible row. Review found four more surfaces still treating a hidden row as present: the per-guest caps (an at-cap guest with one hidden met their own click with limit_reached), /my-feedback (which drives the Liked/Favorited/Rated chips in guest identity mode), getEventFeedbackSummary (disagreeing with the photo counters in the same response), and unhide (leaving two visible rows for one guest). The rating-clear and single-value delete scopes are visible-only now, so a follow-up mutation no longer destroys the admin's hidden record, and the unhide collapse is skipped when there is no stable identity to scope by — that fallback was 'guest_identifier IS NULL', i.e. other visitors' rows. Not taken: refusing to hide non-comment feedback, which the issue recommended. #839 and #1044 both ship hiding for reactions and colour labels with tests asserting a hidden one stops counting; only the admin UI's Hide button is comment-only. Merged with admin privileges: the author cannot self-approve. |
||
|
Paul Nothaft
|
00b20b2d72 |
fix(gallery): guest filters respect show_feedback_to_guests, and marks survive a mid-write clear (#1147)
Two follow-ups from the review of #1137. Filters were a second way to read hidden feedback. Every token on /photos is an OR of two halves: what THIS viewer marked, and what ANYONE marked. The response fields built from the second half — like_count, comment_count, color_label_count — are all gated on show_feedback_to_guests. The filter was not, so with the setting off a guest could still send ?filter=liked and get back exactly the photos other people liked, across all five tokens. Reachable by a direct API caller holding a gallery token; the frontend never sends filter to this endpoint. The half it left standing was also the wrong half. It read guest_identifier from the guest_id QUERY PARAMETER, which never matched anything — the frontend invents that string in localStorage and never sends it when submitting feedback, while submissions store generateGuestIdentifier(req). So gating the aggregate would have emptied these filters rather than narrowing them to 'mine', and accepting a caller-supplied identifier was a way back through the gate. Resolved from the request now, hidden rows excluded to match what the viewer can see. A mark whose row is cleared mid-write lost its value. #1137 fixed two calls both writing; this is one clearing while another sets. The clear empties the row, the row is deleted for being empty, and the setter's update matches nothing — the caller told 'no mark'. A zero-row update now reports itself and the caller re-reads, bounded at three passes, throwing rather than reporting a success that did not happen. Merged with admin privileges: the author cannot self-approve. |
||
|
Paul Nothaft
|
e4a8be8e7e |
fix(gallery): no Logout button on galleries that don't require a password (#1149) (#1152)
showLogout was hard-coded true, so a gallery with no password showed a Logout button. Logging out of it is meaningless — no credential to drop, nothing to return to — and it stranded the visitor: GalleryPage's auto-login is a one-shot latch, so clearing the session left the page on its skeleton until a manual reload. That is the 'turns blank' in the report. The button is gated on requiresPassword || isClient || viaCustomer at both call sites. The full-page layouts render it on the callback being present rather than on a flag, so withholding the callback is how the gate reaches them. Session kind now comes from /auth/session rather than sessionStorage, which is per-tab while the cookie is per-browser: a gallery reopened in a second tab lost 'client' while the backend kept serving it as one. viaCustomer marks a portal token, which bypasses reveal mode and so is a credential that does not look like one. The public-gallery branch no longer returns the skeleton unconditionally — once auto-login has run and left us unauthenticated it shows the reason and a Retry. That state was otherwise unrecoverable, and it also swallowed loginError entirely. Merged with admin privileges: the author cannot self-approve. |
||
|
Paul Nothaft
|
b581267031 |
fix(scripts): regenerate-thumbnails resolves external sources through ensureThumbnail (#1148) (#1151)
The CLI fallback carried the defect #1129 fixed in the admin route: it computed `storage/events/active/<photo.path>` and fs.access'd it, a location that does not exist for external or reference rows. Every one failed the check and was counted as an error, so on an external-media install the script was inert while reporting one error per photo. Resolution now goes through ensureThumbnail, which already branches on source_origin and owns the per-photo ext<id>_ output name — sharing it is what stops the script and the route drifting apart again. Also: videos skipped on every marker they can carry (fileWatcher writes type and mime_type but never media_type), responsive tiers backfilled alongside the canonical rendition, skip-vs-generate asked from isThumbnailValid rather than inferred from an unchanged path, tier failures counted rather than swallowed, and a nonzero exit when the backfill was incomplete. The script is now importable with the CLI behind a require.main guard; it previously ran on require and called process.exit, so it could not be tested at all — which is why this survived #1129. Merged with admin privileges: the author cannot self-approve. |
||
|
Luca
|
e2844d1909 |
feat(gallery): colour labels for client proofing, and one global default per feedback type (#1044) (#1137)
Colour labels for client proofing, plus the photographer's own stars and colours in the admin grid. - Guest colour labels alongside likes/reactions, opt-in per event (defaults off so live galleries do not change mid-proofing), with 'colors' and 'lightroom' keybind schemes. - One global default per feedback type, replacing the per-type scatter. - Admin marks live in their own table (photo_admin_marks) so they can never reach a guest-facing surface. - XMP export prefers a real label, keeping the rating-derived mapping as a fallback. Review: concurrent-write loss on the mark update path, migration index idempotency and error classification all fixed in 7139bcae; migrations renumbered to 182/183 in 8fecdfae after 180/181 were taken on main. Merged with admin privileges: bypass-size-gate is a required check that fails on size alone for review-bypass authors and never re-evaluates on review, which is its designed behaviour once a maintainer has approved. |
||
|
Paul Nothaft
|
3583c924da |
feat(faces): consolidate look-alike clusters after a scan, and suggest the rest (#1107)
consolidate() has existed since #1074 and described this exact symptom in its own comment, but its only caller was recluster() — i.e. when an admin pressed Re-group people. After a normal background scan the centroids converged and nobody looked, so a gallery settled with 14 people that should have been 8. It now runs when a scan drains. There is no scan-finished event to hook, so an idle worker asks whether the events it touched have actually drained — 'a worker went idle' is deliberately not treated as sufficient, because with concurrency above one the others may still be working. The uncertain band asks instead of acting: pairs between the assignment threshold and the stricter auto-merge one surface as accept/dismiss suggestions, with sticky dismissals. Nothing merges silently — a pass that merged anything reports it and points at Split. Review rounds hardened it against overruling explicit decisions: it no longer absorbs ignored clusters (mergePeople ORs is_ignored onto the survivor, which would have hidden a real person), no longer merges dismissed pairs, no longer undoes a manual Split (which now records a separation), and no longer runs after detection is switched off. The dismissal read fails closed, a failed pass is retried with backoff rather than lost or hot-looped, and the new table follows event_people out of exports and backups. Name autocomplete needs no endpoint — the people list already open is the source, and it is event-scoped on purpose. Known limitation, tracked in #1132: separations are keyed on person ids, so a full re-scan loses them. Reported by @BraynArts. |
||
|
Paul Nothaft
|
87115b28e8 |
fix(gallery): give masonry tiles their real shape back (#1130, #1131)
Two independent causes of the same symptom — an aspect-ratio layout that does not lay anything out. gallery-premium discarded the tile height MasonryPhotoAlbum computed from photos.width/height and set height:auto on both card and image, so the rendered shape came from the intrinsic ratio of whatever rendition was served. With thumbnail_fit seeded 'cover' by migration 040 every rendition is square, so the layout drew identical squares and was indistinguishable from grid. The card now uses the height it is given and the stylesheet's existing height:100% applies. The bundled CSS templates pinned images to a fixed pixel height, which has specificity (0,1,1) and beats the .h-full utility (0,1,0) six of the seven layouts use. Elegant Dark is seeded is_default, so that was the out-of-the-box result for any layout other than grid/timeline. Migrations 052/053 corrected for fresh installs; 181 repairs the rows already seeded. Whitespace-tolerant because sanitizeCSS strips newlines from any template ever saved through the editor — an exact-text migration would have silently no-opped on most real installs. The height property is matched with a lookbehind so line-height/max-height/min-height are untouched, grouped selectors are handled, and nested rules are skipped rather than mis-rewritten. Both reported, measured in the live DOM, by @BraynArts. |
||
|
Paul Nothaft
|
97d92f8428 |
fix(thumbnails): regenerate external photos instead of dropping their tiers (#1129)
POST /admin/thumbnails/regenerate resolved every source as storage/events/active/<photo.path> and fs.access'd it. External and reference rows are not there, so every one failed and was counted as an error — and because the tier deletion runs first, the endpoint dropped every ?w= tier and rebuilt nothing, leaving the library worse than before it ran. The UI reported success either way. Now routed through ensureThumbnail, which resolves both source kinds, uses the per-photo ext<id>_ output name, and writes thumbnail_path back itself. Review rounds also removed both destructive deletes in generateThumbnail: the pre-delete ran before sharp opened the source, so an unreadable source left the previous rendition gone and the database pointing at it — across a bulk run, the whole gallery. Neither delete was needed, since put stages to a temp file and renames atomically and is the last statement in the try. Videos are filtered out, and the superseded rendition is removed only when the storage key actually moved, compared through the same canonicalisation the backends apply so a legacy backslash path is not mistaken for a different object. Reported by @BraynArts, who also identified the fix. |
||
|
Paul Nothaft
|
f735d26422 |
fix(gallery): a missing thumbnail tier must not take the backend down (#1128)
The first load of a gallery whose ?w= tiers do not exist yet could exit the Node process — not 500 one tile, kill the backend. Two defects stacked. The reader: LocalFsStorage.get() returns a lazy fs.createReadStream, so an ENOENT arrives after the await returned and outside the route's try/catch. An unhandled 'error' event is a process-level throw. pipeStreamToResponse attaches the handler the routes were missing — 404 for a vanished source, connection destroyed if bytes are already on the wire, file headers cleared so the JSON error is not served as image/jpeg or cached as a broken tile for an hour. Applied to all nine streaming responses in gallery.js. The writer: ensureThumbnailAtWidth passed regenerate:true, whose first act is to DELETE the target — on a path only reached when the tier is absent. A grid fires one request per tile, so one request unlinked the file another had just published and handed to a reader. Without the flag the write is an atomic rename. Generation is now also deduped per tier key: 8 concurrent requests ran 5 Sharp passes before, 1 after. Reported with a full diagnosis by @BraynArts. |
||
|
Paul Nothaft
|
bbce3cd2a2 |
feat(faces): let the photographer choose which photo represents a person (#1119)
Phase 1 of #1096. Clustering picks the cover, and its idea of a good one and a human's do not always agree. A cluster whose avatar is turned away or softer than the rest stays that way in the guest-facing people strip too, and nothing in the UI could change it. A picker reachable from each person row, reusing the face list the split dialog already loads — same query, same grid, different action on a click. Making the choice actually stick took four changes --------------------------------------------------------------------------- event_people.cover_face_id has existed since migration 177 and the PATCH already accepted it, so the first version of this was frontend-only. It was also a no-op: - facePeopleService.listPeople SELECTED cover_face_id and then discarded it, recomputing the cover as the best-scoring VISIBLE face on every read. The picker saved, said so, and the avatar reverted immediately. It now prefers the stored pick whenever this audience can see it, and falls back to the score-ordered choice otherwise — so visibility scoping still wins, and a guest is never handed a crop of a photo they cannot open. - recomputeCentroid overwrote cover_face_id unconditionally. It runs on rescan and on photo replacement, so any reprocessing silently undid a deliberate choice. It now keeps the chosen face while it is still a member of the cluster. - The face list is cached per person, and split/merge move faces between people. Until now the only reader closed itself after acting, so nobody saw the stale copy; the picker is a second reader of the same key. - cover_face_id meant two things. assignFaces seeded it with whichever face opened the cluster and recomputeCentroid overwrote it with the highest scoring one, so an automatic guess was indistinguishable from a deliberate choice — and honouring it would have pinned every UNCURATED person to that guess, which is worse than the fallback it replaced (the fallback is computed per audience and skips photos a guest cannot open). Both writers are gone, migration 179 clears the stored guesses, and the column now means one thing. That also removes the need to defend the choice against rescans: nothing overwrites it, and a dangling id self-heals to the derived cover. Clearing existing values is safe rather than destructive: no install has ever been able to SET a cover, so every stored value is an automatic guess by construction. Also fixes a PostgreSQL-only 500 --------------------------------------------------------------------------- GET /admin/events/:id/people/:personId/faces joined `photos` but did not table-qualify its WHERE, and photo_faces and photos BOTH have an event_id: column reference "event_id" is ambiguous Postgres refuses it, so the endpoint 500s and the Split dialog — its only consumer until now — has been broken on every PostgreSQL install since the join was added. SQLite resolves the ambiguity silently, which is why the suite stayed green. Reproduced against a real Postgres before and after. The query is now a named builder the route calls and the test imports, rather than a copy: an earlier version of that test re-declared the query, so the route could regress to the bare form while the assertions kept passing. Merge and recluster preserve the choice as well. Both already carried labels and privacy flags across; the chosen cover is human state of the same kind, so it now rides along — through a merge when the target has none, and through a recluster by following its FACE into whichever cluster ends up holding it, rather than the majority-descendant rule the label uses. The picker and the endpoint disagree past 500 faces, so the picker now says when it is showing a capped list rather than presenting it as exhaustive. Frontend suite 178 passing, backend 23 across the touched suites, build clean, no new type errors. Mutation-checked twice: dropping the cover preference fails the new listPeople test while the visibility-scoping test still passes, and restoring the auto-seed in assignFaces fails it too. |
||
|
Paul NothaftandPaul Nothaft
|
887bdbe6e5 |
feat(gallery): responsive grid thumbnails (#1095) (#1109)
* feat(gallery): responsive grid thumbnails (#1095) The half of #1095 that #1099 deliberately left out. Grid tiles are ~175 CSS px at the mobile 2-column default — about 530 device px on a DPR-3 phone — so the 300px thumbnail is upscaled ~1.8x and faces visibly mush. Backend mirrors the preview tiers exactly: ?w= on the gallery thumbnail route, whitelisted to 300/600/900, cached by width in storage, never written to photos.thumbnail_path, and keyed by photo id for every source type — basenames are not unique across events and a tier is served from a cache hit without re-reading the source, which is how the preview tiers nearly leaked one gallery's photo into another. The tier is in the ETag, or a client holding the 300px file gets a 304 for its 600px request. Cleanup and regenerate invalidation are wired the same way. generateThumbnail now takes width/height overrides; it keeps the configured `fit`, because the grid renders with object-cover and tiers that were framed differently would visibly jump as the viewport changes. The srcset only advertises tiers the SOURCE can fill. Thumbnails are generated withoutEnlargement, so a 400px original asked for 900 comes back at 400 — advertising "900w" would have the browser pick that candidate and upscale it, which is the reported softness made worse. That exact trap is why this was held back from #1099; the photo's own dimensions are now the guard, measured on the SHORT edge because thumbnails are square and a 4000x600 panorama can still only fill a 600 tile. A source that clears only one tier gets no srcset at all rather than a single pointless candidate. Two things this surfaced, both worth knowing separately: `npx tsc --noEmit` type-checks NOTHING in this project — the root tsconfig is `files: []` with project references, so the real command is `tsc -b`, which is what build:check runs. Under tsc -b the repo has 43 files with pre-existing type errors; this branch adds none, and the one error in a file I touched (PeopleManagerModal:91) is on main already and unrelated to the line I changed. * fix(gallery): wire grid tiers into the component that actually renders The srcSet landed in PhotoGrid.tsx, which nothing imports — GalleryView renders PhotoGridWithLayouts, and every grid layout funnels its tile through the shared PhotoCard. The frontend half of #1095 shipped nothing. Moved to PhotoCard, and switched from srcSet to a single sized URL, the same shape PhotoLightbox already uses for preview tiers. AuthenticatedImage fetches its src with the gallery bearer token and renders the blob; an <img> carrying a w-descriptor srcSet ignores src entirely, so that fetch would have been discarded and the browser would have issued its own — unauthenticated, and resolved against the page origin rather than the configured API host. One URL keeps the auth path and halves the requests. The tier comes from the tile's measured width via the IntersectionObserver entry, read on the same render that reveals the image so nothing is fetched twice. Column counts differ per layout and shift again with thumbnailScale, so the breakpoint table is only a fallback. Also closes what the tier cache leaked or served stale: - ensureThumbnailAtWidth short-circuits videos. Their thumbnail is a poster frame, so the tier path handed the video file to Sharp — after downloading it in full on S3, uncached, once per request. - The ETag names the tier actually served, not the one requested. A fallback to the canonical thumbnail was caching a 300px image under a 900px key. - Tier height scales from the configured aspect ratio instead of forcing a square; with fit:'cover' a 300x200 canonical and a 600x600 tier are two different crops and the photo reframed between tiers. - The canonical short-circuit compares against the configured thumbnail_width, not the 300 default, so a 600px install stops generating duplicate tiers. - Tier invalidation on /admin/thumbnails/regenerate, above the local-file check that skips S3 and external rows. - Tier cleanup in replacePhoto and deleteEventCascade. Both derive keys from the photo row, so the rows have to be read before they change or vanish. Preview tiers had the same two holes and are swept alongside. The clamp no longer drops a tier when the source falls between them: a 400px short edge asked for 600 returns all 400 pixels, where clamping to 300 threw 100 of them away. Backend 18 tier tests, frontend 22. Full suites green: 293 backend across the touched areas, 185 frontend, build clean, no new type errors. * fix(gallery): measure the tile, and stop regenerating the w300 tier Follow-up to the review of #1095. Closes the three items left open there, plus a defect the previous commit introduced. **The w300 tier regenerated on every request.** Decoupling the canonical short-circuit from the hardcoded 300 left generateThumbnail still tagging against DEFAULT_THUMBNAIL_WIDTH. On an install with thumbnail_width=600 a w=300 request wrote `thumb_<name>` while the caller probed for `thumb_w300_<name>`: the cache never hit, so every request re-downloaded the original and ran Sharp, and the file it left behind was in no cleanup list. The tag now follows the configured width, and thumbnailTierKeys lists all three widths — which one is canonical is a setting, so excluding 300 stranded exactly the file a 600-configured install generates. **The tier is chosen from the tile's measured width.** The observer entry only exists for `lazy` cards, and Mosaic, Masonry and Timeline don't pass it — Mosaic is 1-up on mobile where Grid is 2-up, so they are the layouts a breakpoint guess gets most wrong. Measured in a layout effect and gated: the image is not rendered until the width is known, so AuthenticatedImage never mounts with a src it has to replace. Attaching the observer ref unconditionally instead refetches every tile, since React flushes passive effects before the sync re-render a layout effect triggers — removing the gate makes the new single-request test fail, which is how that was confirmed rather than assumed. **Gallery Premium has its own card** and never reached the shared one, so its tiles kept pulling the canonical thumbnail. MasonryPhotoAlbum already hands the laid-out width to the render prop, so it needed no measurement. **Event rename orphaned tiers.** The key embeds the basename, so the DB update is the point past which the old keys cannot be derived. Dropped inside the filename-changed branch, not the loop body: unconditional would fire four storage deletes per photo on every rename, 20k calls against S3 for a 5,000-photo event that merely had its slug adjusted. Preview tiers had the same hole and are swept alongside. Carousel is the seventh layout and deliberately gets no tiering: its filmstrip thumbs are 80 CSS px, under the canonical 300 even at DPR 3. Tests: first PhotoCard suite (6), backend tier suite 21. Both new behaviours mutation-checked — reverting the width tag, the render gate, the measurement, or the rename sweep each fails a test. Full suites green: 298 backend across the touched areas, 191 frontend, build clean, no new type or lint findings. * fix(gallery): mount masonry cards once, into a measured layout Found while capturing screenshots for this PR, by attributing every thumbnail request to a photo id rather than eyeballing the grid. Masonry columns mode starts at 3 columns and runs its greedy distribution off a hardcoded 300px estimate until the container has been measured. Cards mounted into that guess are torn down when it settles — photos move to a different parent column, so React unmounts them — and since #1095 each mount picks its tier from its own width, the two mounts request two DIFFERENT urls. Measured on a 1440px desktop, production build, 62 photos: before 45 photos fetched at canonical AND w600, 17 stuck on w600 107 requests after 62 photos, canonical only, 62 requests Mobile was already landing on one tier either way, so both mounts produced the same url and the second was a cache hit — which is why it looked clean and the desktop case did not. The fix is the gate the rows/justified mode in this same file already applies for the same reason (line 346): hold the cards back until containerWidth is known. Only columns mode was missing it. Grid and Justified take their column counts from CSS breakpoints, so they have no transient measured value to discard and are unaffected. Worth noting this was NOT visible on main: without tiering both mounts request the same url, so the browser cache absorbs the duplicate. Tiering is what turns a harmless remount into a second download — the regression is this PR's, which is why it is fixed here rather than deferred. Frontend suite 194 passed (3 new). Mutation-checked: removing the gate fails the mount-once and placeholder tests. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
00776234fd |
fix(security): let cors() own Access-Control-Allow-Origin on protected images (#1118)
Closes #1116. secureImageMiddleware set its own Access-Control-Allow-Origin, overwriting the one cors(corsOptions) had already computed. server.js:247 mounts cors() on all of /api with credentials:true, so by the time the route handler ran the correct header was already there — and the local assignment replaced it with a worse answer in BOTH directions: unresolved -> '*'. Combined with the credentials:true that cors() sets, that is an invalid pair browsers reject outright. Unreachable on Docker until #1104 stopped compose injecting FRONTEND_URL; reachable on a fresh install from then until the wizard stores general_site_url. resolved -> the frontend origin, even when the request legitimately came from the allowlisted ADMIN_URL. A split admin host got a header naming the wrong origin and the browser rejected a request cors() had allowed. Deleting the line fixes both. cors() already validates the request Origin against the allowlist, sets Vary: Origin, omits the header entirely for a disallowed or absent Origin, and pairs correctly with credentials. Methods, Headers and Max-Age stay here: they are route-specific and cors() does not contradict them. Observed against a running instance before and after: allowlisted Origin ACAO: <that origin> + Vary: Origin + credentials:true disallowed Origin no ACAO no Origin header no ACAO Six tests, mounted on a real Express app with server.js's middleware order. Deliberately NOT a unit test against a response double: the first version of this fix was a guarded assignment that looked correct in isolation and still overwrote cors() whenever an origin resolved. A double cannot see middleware composition, which is exactly how that slipped through. Mutation-checked both ways — restoring the original `|| '*'` fails 5 of 6, and restoring the guarded assignment fails 3 of 6 including the admin-origin case. |
||
|
Paul NothaftandPaul Nothaft
|
011f6ae7ec |
feat(gallery): sized preview tiers so phones stop pulling 1920px (#1095) (#1099)
* feat(gallery): sized preview tiers so phones stop pulling 1920px (#1095) A phone can display ~1170px at most, but the preview tier is a single 1920px JPEG with no size parameter — so every lightbox swipe ships roughly twice the bytes it can use, and the slide track preloads neighbours, which multiplies it. On the reporter's all-external install a null preview_url falls back to the untouched NAS original, which makes it worse again. Backend: ?w= on the gallery preview route, whitelisted to 640/1280/1920. A whitelist rather than a free-form width because every distinct value is a permanent rendition on disk — an open parameter is an invitation to fill the volume. Unrecognised or absent values fall through to the canonical 1920 preview, so old clients and hand-typed URLs behave exactly as today. Extra tiers are cache, not state: ensurePreviewImageAtWidth keys them by width, looks them up in storage and generates on miss, and never writes photos.preview_path. That column owns the canonical rendition, and threading a width through it would mean the last size anyone requested silently becomes "the" preview. Requesting 1920 resolves to the existing preview rather than a w1920 duplicate, so no install grows a second copy of every preview it already has. The tier is part of the ETag. Without it a client holding the 1920 rendition gets a 304 for its 640 request and renders the wrong size, which is this feature inverted. Frontend: the lightbox picks a tier from innerWidth x devicePixelRatio, capped at DPR 3 — uncapped, a DPR-10 device asks for 3900px and lands straight back on the desktop rendition. At the top tier the URL is left byte-identical so existing caches and ETags stay valid and desktop sees no change at all. saveData and a 2g/3g effectiveType drop one tier; both are Chromium-only, so they are a bonus rather than the mechanism. Grid thumbnails are NOT tiered here, deliberately. generateThumbnail resolves its width from admin settings rather than an argument, so tiering it is a separate change — and shipping a srcset whose candidates the server ignores would be worse than shipping none: the browser would take the "600w" candidate, receive the 300px image and upscale it, which is the reported softness made slightly worse. That half of #1095 lands separately. * fix(gallery): scope tier keys per photo, size by long edge, clean up tiers External review. Three findings against the tier work, one a cross-gallery leak. The tier cache key was the photo's BASENAME. Managed uploads keep camera basenames, so two events can each hold an IMG_0001.jpg — and a tier is served straight from a cache hit without re-reading the source, so the second gallery gets the first gallery's photo. Keys are now scoped by photo id for every source type. The RAW branch passed proc.outputBasename, which would have dropped that scoping again; it now passes the scoped name. Tier selection used viewport WIDTH, but ?w= bounds the LONG edge (fit:'inside'). On a 390x844 phone at DPR 3 a 2:3 portrait is bound by height and renders ~1755 device px, so width-only picked 1280 and made portraits softer than today; landscape on the same phone needs ~1170. It now computes the rendered long edge from the photo's own dimensions and falls back to the top tier — today's behaviour — when they are unknown. Tiers live outside photos.preview_path, so nothing else knew they existed: delete, bulk-delete and archive left them orphaned in previews/ forever, and regenerate-previews refreshed only the canonical rendition while phones kept the stale copy. previewTierKeys derives them from the same deterministic scheme and all four paths clean up. Deliberately outside the preview_path guard — a tier can exist when the canonical rendition never did, so keying cleanup off preview_path would strand precisely the photos only ever viewed on a phone. The existing tier tests encoded the old width-only semantics and were updated rather than kept; that is a behaviour change, not a test fix. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
0b886ed942 |
fix(faces): defer on unreachable storage, and commit the import path first (#1097)
* fix(faces): defer on unreachable storage, and commit the import path first The two follow-ups left open by #1091, both consequences of external photos becoming scannable at all. **A dropped mount no longer burns the gallery.** ensurePreviewImage returns null for "this JPEG is corrupt" and "the NFS share is gone" alike, and faceProcessor marked both 'failed'. Nothing re-queues a failure automatically and the queue only ever claims 'pending', so a mount that blinked mid-scan cost the whole event a manual Re-scan — on external libraries, where network storage drops far more often than local disk, that is the common case rather than the corner one. faceProcessor now probes the containing DIRECTORY before failing, and throws TransientSourceError when it cannot be reached; faceQueue treats that exactly like SidecarUnavailableError — release to pending, back off, retry — with the warning rate-limited to one per five minutes, since an outage hits every photo in the event. The directory rather than the file is the whole point: a missing file inside a healthy directory is a broken photo and should still fail, and it still does. Anything that goes wrong deciding which case it is falls through to 'failed', because guessing 'transient' on an unknown condition would retry forever. **The import commits its path before inserting rows.** enqueueEvent accepts processing_status NULL (faceProcessor.js:243-246), which these inserts leave unset, so an admin hitting the toggle or Re-scan during a long import could queue partial rows while the event still resolved against the old directory — and burn them to 'failed'. Moving events.external_path ahead of the loop closes that, and fixes a pre-existing bug on the same line: an import that died at photo 500 of 1000 used to leave those 500 rows pointing into the new tree while the event still resolved against the old one, making every one of them unreadable. Safe to do first because the path is already validated above, and existing photos are unaffected — photo.source_origin takes precedence over event.source_mode in both resolvers and is NOT NULL defaulting to 'managed'. The #1090 test that asserted a missing source fails needed its setup corrected rather than its intent: it created no directory at all, which is now (correctly) a dropped mount. It now creates one, so it tests the case it always meant to — healthy storage, dead photo. Verified both fixes discriminate: removing the probe fails the defer test, and moving the event update back after the loop fails the ordering test. Face suite 59 passed across 8 suites; the full backend suite fails the same 10 pre-existing suites as unmodified main, no more. * fix(faces): stop a dead mount stalling the whole queue External review, and the first finding is one my own change created. Deferring by returning the row to 'pending' was a trap: claimNextPhoto orders by id ascending and the queue defaults to a single worker, so the same unreachable row becomes the oldest pending one after every backoff and the worker never reaches a higher id. One dead mount would have stalled face scanning for the entire install — unrelated events, fresh uploads, everything. Strictly worse than the permanent 'failed' this set out to replace. The row is now left parked in 'processing' with face_started_at intact. It is not claimable, so the worker moves straight on; the janitor that already exists returns it to 'pending' past STUCK_TIMEOUT_MS, which is the retry. No new column and no new timer. The sidecar branch still releases, because a down sidecar blocks every photo anyway — there is no other work to get on with. Second: probing existence was not enough. Unmounting an NFS or SMB share usually leaves the mountpoint behind as an ordinary empty directory, so fs.access succeeded on storage that was entirely gone and the photo was failed anyway — the exact case this was written for. An empty directory where the photo should live now counts as unreachable. The trade is deliberate and documented: a directory an admin genuinely emptied is retried rather than failed, which now costs one attempt per janitor sweep and nothing else. Third: a comment in adminExternalMedia claimed source_origin isolates existing photos from the early external_path update. That is true of managed rows and false of external ones — resolveExternalPath prefixes every external row with event.external_path, so importing folder B into an event referencing folder A rebases the A rows. Pre-existing rather than introduced here (the update always did this, just later), but the comment asserted otherwise, so it now says what actually happens and names the underlying single-base-path limitation. The deferral test initially passed against the blocking version too — database state alone cannot tell the fix from the bug. It now inspects the branch directly, the way the #596 contract tests do, and fails when releaseToPending is put back or the two branches are merged. * fix(faces): back off per event, and stop clobbering concurrent scans Round two of external review. Parking a row in 'processing' fixed the head-of-line block but not the cost: every janitor sweep handed the whole dead gallery back, and the worker walked all of it again — one stat per photo against storage that may be hard-mounted and slow to time out — before reaching any healthy event. Every one of those attempts also went through generatePreviewImage first, which logs an error per photo, so a down mount produced a recurring flood that the rate-limited warning did nothing about. So the backoff is now per EVENT and separate from the janitor: TransientSourceError carries the event id, the queue records a cooldown, and claimNextPhoto excludes those events while it lasts. The janitor keeps doing its own job, which is rescuing rows a crashed worker abandoned. Cooldown is in memory on purpose — a restart is usually what follows fixing a mount, so it should retry at once. Second: committing the event path before the loop means a toggle or Re-scan firing mid-import can now genuinely queue and finish some of those rows. The final bulk update was unconditional, so it dragged 'done' rows back to 'pending' for a duplicate sidecar scan and knocked 'processing' rows out from under the worker. It is now whereNull — only rows nothing has touched are ours to queue. Also corrected a comment of mine that had gone stale in the same file: it still described the enqueue as happening after the event path was written "below", which stopped being true when that update moved above the loop. * fix(faces): judge the mount, the path and the file separately Round three of external review. The probe was too coarse in both directions. It read any ENOENT on the photo's own directory as a mount-wide outage, so a deleted or renamed subfolder — individual/ gone while collages/ is healthy — deferred the entire event and starved every sibling folder, renewing the cooldown on each retry. It now judges the EVENT ROOT for that verdict: root missing, or present-but-empty, is an outage; anything below a populated root is a broken path and fails. And it read a listable directory as proof the photo was at fault, so EACCES on a reconnected share, EIO, or the classic NFS ESTALE handle were burnt as permanent failures. Only ENOENT now means genuinely gone; any other error opening the file defers. The event-wide backoff was also too broad. A reference event can hold managed uploads alongside imported external ones, and those live in local storage that is fine — excluding the whole event id left them unscanned for as long as external rows kept renewing the cooldown, which during a real outage is indefinitely. The exclusion is now scoped to external and reference rows. One of my own tests had modelled the unmount wrongly: it emptied the photo's subdirectory rather than the event root, which under the corrected logic is a populated mount with a missing folder — a failure, not an outage. It now empties the root, which is what an unmount actually leaves behind. Dropped the path require the first version of this probe needed; the event-root form does not. All three fixes mutation-checked: reverting each one fails the test written for it. Face suite 69 passed across 9 suites; full backend suite fails the same 10 pre-existing suites as main, no more. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
576924fa57 |
fix(faces): scan external/reference photos instead of skipping them (#1090) (#1091)
* fix(faces): scan external/reference photos instead of skipping them (#1090) faceProcessor short-circuited every photo with source_origin 'external' or 'reference' straight to 'skipped', before the sidecar was ever contacted. On an external-media install that is the entire library — the reporter's gallery sat at 0/3230 with every row skipped and no error, and a rescan changed nothing. The guard was correct when written: resolvePhotoStorageKey returns null for anything outside managed storage, so ensurePreviewImage could not build a preview and there was nothing to send. #1078 removed that limitation one release earlier — ensurePreviewImage now reads externals straight off the mount via resolvePhotoFilePath and writes the preview into managed storage, so the key faceProcessor already fetches through getStorage() is readable like any other. The guard outlived its reason. Photos whose source is genuinely gone still return a null preview key and land in the existing 'failed' branch, which is the honest outcome: that is a broken photo, not an unsupported one. The blanket skip was absorbing those too. No migration or manual reset needed — enqueueEvent already re-queues rows with face_status in (NULL, 'failed', 'skipped'), so previously skipped photos get picked up on the next scan. * fix(faces): queue external imports for scanning (#1090) The other half of the same bug, found by external review — and my first counter-argument against it was wrong. Managed uploads are enqueued by photoProcessor, which writes face_status 'pending' once a photo is processed (photoProcessor.js:573, commented as "the only correct place to enqueue"). External media never goes through photoProcessor at all: adminExternalMedia inserts rows directly, leaving face_status NULL. faceQueue.claimNextPhoto only claims 'pending' (faceQueue.js:64), so an import into an already-enabled event produced nothing until someone pressed Re-scan. Lifting the skip guard alone made external photos scannable but still not scanned — which looks like a complete fix right up until you import a photo. Resolved once per import rather than per file, since it is a per-event setting and the loop can run to a thousand files, and guarded on both the global flag and the per-event toggle exactly as photoProcessor guards it, so installs without the feature still never write a face_status. A failure to read the setting logs and imports anyway — the photos are the point. No video guard: walkDir only collects jpg/jpeg/png/webp, so nothing faceProcessor would skip as video can arrive through this route. * fix(faces): enqueue imports only after the event path is written External review caught a race I introduced in the previous commit. Marking rows 'pending' as they were inserted published claimable work while events.external_path still held the old value — or none at all, on a first import, since the route only writes it after the entire thumbnail loop. The face worker polls continuously, so on any import long enough to matter (the loop is ~100-300ms per photo, and the reporter's library is 6500+) it would claim those rows, resolve them against the wrong directory and mark them permanently 'failed' — a state only an explicit Re-scan clears. That is strictly worse than the unscanned photos this set out to fix. Ids are now collected during the loop and marked pending in one pass after the event path is written, chunked at 500 because SQLite caps a statement at 999 bound parameters. The test now drives the real route instead of re-implementing its logic, and observes the mid-loop state from inside the per-photo thumbnail call — the only hook that can see the window the race lived in. Verified it discriminates: deleting the enqueue fails two tests, and moving it back onto the insert fails the ordering test specifically. * fix(faces): read the face setting after the import, not before Third external-review round. The setting was captured before a loop that runs for many minutes on a large library, so an admin who enabled detection during an import left every photo imported after that moment at NULL forever — the toggle endpoint only queues rows that already existed when it fired. Ids are now collected unconditionally and the setting is evaluated immediately before the queue update, off a freshly read event row. The guard is unchanged in substance: both the global flag and the per-event toggle, so installs without the feature still never write a face_status. Test flips the toggle from inside the mocked per-photo thumbnail call, which is the same mid-loop hook the ordering test uses. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
af7970b069 |
fix(preview): generate lightbox previews for external/reference photos (#1078) (#1079)
* fix(preview): generate lightbox previews for external/reference photos (#1078) ensurePreviewImage() resolved its source only via resolvePhotoStorageKey(), which returns null for external/reference photos by design — those live on a media mount outside the managed storage tree. The null went straight into withLocalCopy(), which throws ("LocalFsStorage: invalid relative path: null"), so the preview route fell back to redirecting at the full-size original. A gallery whose photos are all external got no benefit from the preview tier (#492) at all: guests paid 5-12 MB on every lightbox open, with nothing surfaced in the admin UI. Add the external branch ensureThumbnail() has had since #423: resolve via resolvePhotoFilePath() and feed the mount path to generatePreviewImage() directly, with an ext<id>_ output basename so two events referencing the same NAS filename can't clobber each other's preview. Also close the adjacent hole that made the failure a throw rather than the documented null: a row with no source_origin in a reference-mode event takes its mode from the event, so resolvePhotoStorageKey returns null for it too. Return null instead of handing that to withLocalCopy. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(preview): select the columns the external branch needs on bulk regenerate POST /api/admin/thumbnails/regenerate-previews selected only id, event_id, path, media_type, mime_type and preview_path, so photo.source_origin was undefined by the time ensurePreviewImage branched on it. Every external row in a reference gallery took the managed path, resolvePhotoStorageKey returned null for it, and the endpoint reported success while generating nothing. Add source_origin, external_relpath and filename to the select, plus a source-inspection test pinning the caller contract and a service-level test showing a column-starved row is indistinguishable from a managed one. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * style(test): single-quote the source-inspection needles Matches the repo eslint quotes rule (no avoidEscape) by dropping the nested quotes from the search strings rather than escaping them. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
b69dd134d0 |
feat(faces): People in this gallery — face recognition via an optional ML sidecar (#1074) (#1075)
* feat(ml): optional face-detection sidecar, opt-in and inert by default (#1074) First of four PRs for "People in this gallery". This one ships only the sidecar, its wiring and its CI — no schema, no backend code, no UI. Nothing in PicPeak calls it yet. picpeak-ml is a single FastAPI + onnxruntime container: three endpoints (/health, /info, /faces), no database, no volumes, no egress, no model download at runtime. Clustering, person identity and every privacy decision stay in the backend where the data already lives. Models are YuNet (detection) + FaceNet-512 (embedding), both MIT, both pinned by URL and SHA-256 and verified at build time. The licence analysis is in ml/LICENSES.md: the more accurate InsightFace weights are non-commercial-only and PicPeak's users are working photographers, so they are never baked into an image we publish. Two things worth review attention: - Alignment uses a least-squares similarity transform (Umeyama), NOT cv2.estimateAffinePartial2D. RANSAC and LMEDS exist to reject outliers among many correspondences; given five landmarks and no outliers they fit a three-point subset exactly and let the rest drift. Measured on a real off-frontal portrait: eyes and nose pinned to 0.11px, mouth corners 11.8px out on a 160px crop. Umeyama distributes it (max 6.5px, rms 5.1 vs 7.4). The failure mode is silent — a bad warp still yields 512 confident floats — so tests/test_pipeline.py pins it numerically. - FACENET_ONNX_URL has no default and the build fails loudly without it. deepface distributes FaceNet-512 as Keras .h5 only, so the ONNX is produced once by tools/convert_facenet.py and published as a release asset. Converting inside the build would drag TensorFlow through both architecture legs of every build to produce a byte-identical file. The CI jobs are gated on the FACENET_ONNX_URL repository variable and skip cleanly until it is set. Off by default, twice over: the sidecar is behind the `faces` compose profile, and the backend will gate on a `faces` feature flag that defaults to false. FACE_ML_URL defaults to http://picpeak-ml:8000 so the standard deployment needs no configuration — nothing dials that host while the flag is off, which is why a non-resolving default is harmless. Verified: 27 pytest tests green; YuNet loads and detects against a real portrait with its landmark order matching the alignment template index-for-index; both compose files validate and the faces profile is correctly excluded from a default `up`; workflow YAML parses and the job graph resolves. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(ml): pin the converter toolchain, verify parity, drop a false reproducibility claim (#1074) Ran the FaceNet-512 conversion for real and corrected what the previous commit assumed about it. The conversion works: 23,497,424 parameters, 89.6 MB ONNX, and the converted graph matches the Keras original to 2.086e-06 absolute / cosine 1.0000000000. That check is now part of the script rather than something I did once by hand — a subtly wrong graph still returns 512 plausible floats, so it refuses to leave the file on disk if parity fails. Also ran the full pipeline against both real models end to end. The embedding is L2-normalized to 1.000000, and the same face survives being re-rendered: half scale 0.973, double scale 0.984, JPEG q40 0.987, rotated 8 degrees 0.984, brightness +40 0.988. Scale invariance in particular is evidence the alignment warp is doing its job. Corrected claim: the conversion is NOT byte-reproducible. Two runs with the same pinned versions on the same machine gave different SHA-256s. The graphs are functionally identical — same 336 nodes, same 271 initializers, every weight matching to 0.000e+00 — but a few initializer names differ because tf2onnx's traced-op naming is not deterministic (Keras layer naming is deterministic; I checked). The previous commit message and README both claimed byte-identical output. They were wrong, and it matters: anyone re-running the conversion gets a different hash, and without this note that reads like tampering. The build-time SHA-256 pins one published artifact so its URL cannot start serving different bytes; validating a fresh conversion is the parity check's job. requirements-convert.txt now pins the exact set that produced the artifact, including transitive keras/protobuf/numpy, and documents that the converter needs Python 3.11 while the image runs 3.12. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): schema, queue, clustering and API for People in this gallery (#1074) Backend half of the feature. Migration 177, a face-detection queue, the clustering engine, the gallery and admin APIs, and the privacy wiring. No UI yet; nothing is reachable until the `faces` feature flag is on, which defaults to false. The flag is the gate, not FACE_ML_URL. That variable now has a working default (the compose service name), so its presence proves nothing about intent — if it were the gate, every install would poll a hostname that does not resolve. faceQueue re-checks the flag every tick, so turning it off stops the workers without a restart. Visibility scoping is the part worth reviewing closely. Face rows have no concept of photo visibility, but guests are restricted to photos.visibility='visible'. A raw count leaks how many hidden photos someone appears in, and an unscoped cover face renders a crop of a photo the guest may not open — with the best-scoring face being the likeliest pick, so it would happen often rather than rarely. facePeopleService recomputes both per request against the caller's own scope, and event_people.face_count_total is named to be conspicuous in a guest path. Six tests cover it, including the case where a person's photos are ALL hidden and they must vanish entirely. Face data is excluded from backups and .picpeak exports, per the decision in the thread: it is derived, so a restore re-scans rather than carrying biometrics between operators. Three separate mechanisms, because the engines cannot be filtered alike — EXCLUDED_TABLES for export, --exclude-table-data (not --exclude-table; the CREATE TABLE must survive or restore breaks on the first query) for Postgres, and DELETE + VACUUM on the temp copy for SQLite, which has no way to exclude a table from a whole-file .backup. The VACUUM is not cosmetic: without it the pages stay in the file and the claim is false on disk. Archiving now purges face data explicitly. photo_faces cascades off photos, but archive deletes neither the photo rows nor the event, so without this an archived gallery kept its biometrics indefinitely. Other decisions: clustering keeps names across a re-cluster by majority inheritance (without it, one button click silently discards every name the photographer typed); consolidation refuses to merge two people who were named differently; assignment never compares across model_version, since embeddings from two pipelines are not comparable; low-quality faces are stored but left unassigned so they show in "this photo contains" without spawning junk people. Migration is 177, not 174 — 174/175/176 landed on main while this branch was open. 29 tests green: 7 migration (idempotency, down(), cascade, and that installing it enqueues NOTHING), 11 clustering, 11 privacy/visibility. Lint clean; the pre-existing error counts in databaseBackup.js and server.js are unchanged. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): People strip, face filter and admin controls (#1074) Frontend half. Renders nothing anywhere unless the `faces` feature flag is on AND the photographer enabled detection for the gallery — the whole guest surface hangs off one boolean, event.people_enabled, which the server computes from the flag, the per-event toggle and the show-to-guests toggle together. Guest side: a People strip between the filter bar and the grid, circular crops from each person's cover face, an active-filter chip row, and a "Show all" bottom sheet. The face filter composes with category, search, media type and the liked/saved/rated filters in the same useMemo rather than replacing them, so "photos of Anna that I liked" works. Two people selected means AND by default — that is what picking a second face almost always asks for — with a toggle to OR that appears only once a second person is picked. Unnamed people show a photo count and never "Person 7". A number is honest about what the system knows; an invented name is not. There is a test asserting we don't do it. The strip renders nothing below two people, collapses to one line when dismissed (persisted per slug, so dismissing one gallery says nothing about the next), and appears mid-backfill with a progress line rather than blocking the gallery behind a spinner. Avatar crops are computed in ratios of the source dimensions so they survive whatever rendition the browser gets; without width/height they fall back to an uncropped thumbnail, since a wrongly-offset crop is worse than no crop. No new download endpoint: "download these N" rides the existing photoIds path, which already enforces access level and per-category permissions server-side. Adding a person_id selector would have been a second thing to authorize for no gain. Guest-facing copy never says "biometric" or "recognition" — those words describe our implementation, not the guest's experience. The sheet's footnote answers the first question every guest has (where does this go?) inline. The admin card, by contrast, is explicit: it states the controller obligation next to the toggle, and warns that scanning materializes the preview tier on galleries that never generated one, which is real CPU and disk an admin should know about before a 2,000-photo backfill. EN + DE translations. 140 frontend tests green (8 new), tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): measured match threshold, working build defaults, 89MB smaller image (#1074) Ran the Phase 0 spike that had been outstanding, published the model, and fixed what both turned up. THRESHOLD IS NOW MEASURED, NOT GUESSED. LFW's standard 1000-pair protocol run through this exact pipeline (YuNet -> Umeyama alignment -> FaceNet-512 ONNX), 100% detection on 2000 images: same person cosine 0.6958 +/- 0.1415 diff person cosine 0.0849 +/- 0.1674 separation 0.6109 peak accuracy 96.60% @ 0.405 So the pipeline separates people well — the thing I could not previously claim, since every earlier number was the same face re-rendered. Default moves 0.62 -> 0.50. The old value was a placeholder and a bad one: it gave 0% false merges but 22.4% false splits, i.e. roughly one in four same-person pairs failing to join, which fragments a gallery badly. 0.50 gives 1.0% false merge / 8.2% false split. Peak accuracy (0.405) is deliberately NOT chosen: for clustering the two errors do not cost the same. A false split is a duplicate row the photographer can merge away; a false merge puts a stranger into someone's "download my photos" — and until the Phase 2 merge/split UI ships, there is no way to undo one. So this sits on the conservative side of the optimum. The spike is committed as ml/tools/benchmark_threshold.py rather than thrown away, so "why 0.50?" has an answer in six months and a re-tune is one command. BUILD DEFAULTS. FACENET_ONNX_URL/_SHA256 now default to the published ml-models-v1 release asset, so `docker build ml/` and `docker compose --profile faces up` work with no arguments. Blanking either still fails loudly — a URL without a checksum is never acceptable, since the checksum is what makes the URL safe to trust. Found by running compose for real: it failed exactly as designed, which was correct behaviour and a bad out-of-box experience now that a canonical artifact exists. IMAGE SIZE. 389MB -> 300MB single-arch. `chown -R` after COPY rewrote every copied file into a fresh layer, duplicating the 90MB model for nothing; the user is now created before the copies and ownership set via COPY --chown. Also drops pip/setuptools from the runtime image. Measured RSS is 186MiB idle, and the container answers /faces end-to-end in well under the 80-150ms/photo the issue budgeted. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): threshold 0.50 -> 0.60 from real clustering, theme-aware People strip (#1074) Both fixes come from running the feature on an actual gallery — 61 photos, 5 real identities — rather than reasoning about it. THRESHOLD. The LFW pairwise sweep in the previous commit said 0.50, and it was wrong. On a real gallery at 0.50, three of six visible clusters were contaminated: two different people merged into one strip entry, which is the exact failure that puts a stranger into someone's "download my photos". Pairwise error rates do not predict cluster purity. Greedy assignment compounds — one wrong face drags the centroid toward the midpoint between two identities, making the next wrong face likelier. A 1% pairwise false-merge rate is not a 1% chance of a clean gallery, and no amount of staring at an ROC curve would have shown that. Sweep against ground truth (5 identities): 0.50 -> 6 clusters, 3 contaminated 0.56 -> 6 clusters, 0 contaminated 0.60 -> 5 clusters, 0 contaminated <- exactly right 0.64 -> 5 clusters, 0 contaminated, fewer faces assigned 0.60 recovers the right number of people with no contamination; higher only loses coverage. Migration 177 carries the full reasoning so the next person to touch this knows why the obvious pairwise answer is the wrong one. THEME. The People strip hardcoded `text-neutral-800` for named people. On a dark gallery — which the screenshot immediately showed — that renders a named person's label almost invisibly, while UNNAMED people stayed legible. Exactly backwards. Labels, headings, the collapsed summary, the scan line and the filter chip row now read the gallery's own theme tokens (--color-text / --color-muted-text / --color-accent / --color-surface-border) like the rest of the gallery surface. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): keep the mobile filter row inside the viewport (#1074) At 390px the photo count and Clear link were pushed against the right edge by ml-auto and clipped. Only apply it from the sm breakpoint up, where there is room; below that they flow after the chips. Found by screenshotting the real thing on an iPhone-sized viewport. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): complete Phase 1, add People management and auto-categories (#1074) Closes the two Phase 1 gaps, then builds Phase 2 and Phase 3. PHASE 1 GAPS. "Download these N" was specified, described as done in an earlier summary, and never actually built — I had verified the backend needed no new endpoint and let that stand as if the button existed. It now hands the filtered photo ids to the same path as a manual selection, so the server re-applies access level and per-category permissions on the way through. Photos in a downloads-disabled category are excluded client-side too, so the number on the button is the number the guest receives. Hidden entirely when downloads are off for the gallery. Lightbox person chips ("In this photo: Anna") are the second way into the face filter — a guest looking at a photo of themselves can act on it without scrolling back to the strip. Tapping one closes the lightbox and filters the grid behind it. PHASE 2. A People management modal over the endpoints that already existed and were already tested: rename inline, merge (multi-select, first pick is the target so the name a photographer typed survives), split via a face picker, hide, ignore. This matters more than it sounds — clustering deliberately errs toward splitting because a wrong merge puts a stranger into someone's download, and that trade only works if merging is easy. PHASE 3. Rule engine over face_count plus face-area ratio: 0 -> Details, 1 large -> Portraits, 2-5 -> Small groups, >5 -> Groups. The area ratio is what separates "a portrait of someone" from "someone is in this landscape". Three guarantees, all tested: it only ever fills an EMPTY category (enforced in the query AND re-checked in the UPDATE, so a photographer setting one mid-run still wins), everything it touches is marked auto_categorized so undo is exact, and it is a no-op unless separately enabled. Migration 178 adds the column — separate from 177, which has already run wherever this branch is deployed. Verified on the real gallery: 61 photos -> 48 portraits + 13 small groups, undo cleared exactly 61 and left the manual ones alone. Merge moved faces and removed the source. Both confirmed against the database, not just the UI. TWO BUGS THE BROWSER CAUGHT, both invisible to tsc: - The lightbox destructure never landed — my patch targeted a line that has a default value, matched nothing, and failed silently. `people` resolved to something else entirely and the chips would never have rendered. eslint's "outer scope value" warning is what surfaced it. - Admin face thumbnails 403'd because <AuthenticatedImage> attaches whatever gallery token is in session storage; an admin who has also opened one of their own galleries sends a type:"gallery" bearer to an admin route. Admin routes authenticate from the httpOnly cookie, which a plain same-origin <img> sends by itself. Worth noting AdminPhotoGrid has the same latent shape; not touched here. Also: the admin card now reports "N people (M shown to guests)" when those differ, so the settings page and the gallery stop disagreeing without explanation. 45 backend tests (8 new) and 140 frontend tests green; tsc and eslint clean. EN + DE for every new string. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * perf(faces): batch migration DDL and drop the face stack from server.js import (#1074) CI's backend job timed out at 10 minutes on the first run of this branch. Nothing failed — 132 of 182 suites passed and the wall clock ran out. Main does the same 182 in 124s, and where main has 12 suites slow enough for jest to print a duration, this branch had 77. Two changes, both worth making regardless of how much of the gap they close: - Migration 177 added its columns one ALTER TABLE at a time (four on photos, three on events, plus a separate index statement) and seeded settings with a SELECT and an INSERT per key. It now uses one alterTable per table and one SELECT plus one bulk INSERT. 178 folds its index into the same statement as its column. That chain replays in ~90 suites, so statement count there is multiplied by 90. - server.js required faceQueue at module scope, which pulls in axios and — through imageProcessor — sharp. Every supertest suite that imports server.js was paying for a module graph it never uses. Now required inside the startup block, next to the call that needs it. Honest about the evidence: locally the migration delta measures at zero (1.15s vs 1.13s for the same suite, three runs each), so batching alone does not explain an eight-minute regression. A fast local disk and many cores mask per-statement and per-import costs that a two-core runner with a shared disk does not. These are the two real costs this branch added to a path that runs in almost every suite; whether they are sufficient is a question for CI, not for another round of local speculation. 37 face tests still green after the change. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * i18n(faces): complete EN and DE coverage for the face feature (#1074) The admin card and the Features toggle were rendering entirely from inline English `defaultValue` fallbacks — 22 keys existed in no locale file at all, so a German admin saw an English consent notice, English toggles and English buttons. The gallery side was already translated; the admin side was not, and nothing in the toolchain flags this because a `defaultValue` always renders something. Adds the missing `admin.faces.*` (19), `settings.features.faces.*` (2) and shared `common.clear/saved/saveFailed` in both languages. Existing keys are left alone (setdefault, not overwrite), so the shared `common` strings other features rely on are untouched. Committed the audit as frontend/scripts/i18n-faces-audit.py rather than throwing it away: it extracts every t() key the face components actually use and diffs it against each locale, and it also reports German values that are byte-identical to English, which is the usual shape of an untranslated copy-paste. Currently: 69 keys in use, EN complete, DE complete, no identical pairs. Verified in the browser, not just in the JSON — the German card reads "61 / 61 Fotos durchsucht · 16 Personen (5 für Gäste sichtbar)" end to end. Also checked the components for hardcoded user-facing text (JSX nodes, title/aria-label/placeholder attributes) outside t(); there is none. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 13 defects from external review — coordinates, counts, erasure, races (#1074) Codex reviewed the branch against main. Thirteen findings, nine P1. I checked every one against the code and could not dismiss a single one as a false positive, so all thirteen are fixed here. THE WORST ONE: bounding boxes were stored in the wrong coordinate system. The sidecar reports coordinates in the space of the image it was HANDED — which is the ≤1920px preview, not the original — while every consumer compares them against photos.width/height, the original dimensions. A 6000px photo therefore produced boxes ~3x too small and areas ~9x too small: avatar crops landed in the wrong place and the Portraits rule could never fire. It is invisible on any photo already under 1920px, which is exactly why the demo gallery and every screenshot looked correct. Now scaled once in faceProcessor so everything downstream can assume original-image coordinates. ERASURE. The FK cascade on photo_faces is decorative on SQLite: PicPeak never enables `PRAGMA foreign_keys`, so deleting a photo left its embeddings behind. I first enabled the pragma globally and reverted it — six unrelated suites immediately failed on pre-existing dangling references, and switching it on would start rejecting inserts on every existing install. That is a real change worth making, but it is its own PR, not a rider on this one. Instead deletion purges explicitly: purgePhotoFaces in the photo paths (single, bulk, service) and photo_faces/event_people in deleteEventCascade. Tests assert this with the pragma explicitly OFF, so they can only pass if the code does the work. COUNTS. A re-scan deleted the old face rows without undoing their contribution to event_people, so counts inflated on every re-scan and ghost people survived. Now the affected people are recomputed before the replacements are assigned. My own "must not double its faces" test only checked photo_faces rows, which is why it passed throughout. RACES. A worker that finished after an admin purged the event committed its rows anyway — erasure reported success and the data reappeared. The commit is now conditional on the row still being 'processing'. And assignFaces is read-modify-write over an event's people, so two workers lost each other's updates; it is now serialised per event with an in-process mutex plus a Postgres advisory lock for the multi-pod case the queue advertises. METADATA LOSS. Merging discarded the source's name and suppression flags, so a merge could erase a typed name or un-hide someone. Reclustering remembered only people with a label, so an unnamed-but-hidden bystander came back guest-visible after one "Re-group people" — and suppression now propagates to every descendant cluster, not just the majority one. Also: export reset face_status so a restored gallery re-scans instead of claiming to be scanned forever; manual category edits clear auto_categorized so "undo automatic" cannot delete a photographer's own choice; external photos are skipped rather than failed (resolvePhotoStorageKey returns null for them by design); the gallery refetches photo memberships as a scan progresses so filtering is not stale; a failed VACUUM now fails the backup rather than publishing one that may retain biometric pages; and the ML Dockerfile's `|| true` is scoped to the uninstall — as written it was `(install && uninstall) || true`, so a failed dependency install produced a green layer and an image with no onnxruntime. Four new regression tests. Full backend suite failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 12 more defects from review round 2 — cross-event purge, leaks, lifecycle (#1074) Second Codex round on the same diff, now including round 1's fixes. Twelve findings, seven P1. Again none were false positives. SECURITY, AND MINE FROM ROUND 1: the bulk-delete face purge iterated the raw `photoIds` from the request instead of the event-scoped `photos` rows the handler had already validated. purgePhotoFaces has no event scope of its own, so an editor could pass another gallery's photo id and delete its face data — even though the photo deletion right below it was correctly scoped. Fixing one thing and introducing another is exactly why the second round was worth running. ANOTHER VISIBILITY LEAK, same class as the one round 1 fixed: /people returns scan progress, and getScanStatus counted every photo with a face_status — including hidden ones. Guests could read the hidden-photo count off the progress bar while the people list and covers beside it were properly scoped. Now scoped by the same predicate, with the caller passing its audience. RECLUSTER, ROUND 1'S FIX WAS INCOMPLETE. I made suppression follow every descendant but still copied the flags from the majority ANCESTOR. When reclustering merges a visible named person with a hidden one, the majority ancestor is often the visible one — republishing the hidden person's photos. Suppression is now OR-ed across every ancestor contributing faces. The name also now goes to the genuine largest descendant; the previous code took whichever cluster came first in map order, which the comment already claimed it did not. LIFECYCLE. Face data is excluded from backups and exports, but photos. face_status came across intact, so a restored install claimed every photo was scanned while holding no faces — and the worker only claims 'pending', so it stayed that way forever. Now: the SQLite backup requeues in the dump, restore requeues after the pool reinit (the Postgres path cannot rewrite rows inside pg_dump), the portable importer purges LOCAL face tables (they were excluded from the replace list, so another instance's embeddings survived an import with FK checks suspended) and requeues, and archiving disables detection so a restored archive is honestly off rather than enabled-and-empty. WRITE PATHS. Only processPhoto enqueued. The synchronous upload path (chunked-upload completion, watch-folder) left photos unscanned, and replacePhoto kept the OLD image's faces on a row now pointing at a different picture — stale identities shown on the new photo. FRONTEND. PeopleSheet and the admin manager rendered centred thumbnails and ignored the bbox, so on group photos the avatar showed whoever stood in the middle and two people from one photo were indistinguishable — in the manager whose entire job is telling faces apart. The crop maths is now one shared helper (faceCrop.ts) so the three surfaces cannot drift again. Full-page layouts (gallery-premium, gallery-story) render their own lightbox and never received the people props. Backend failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): round 3 — five of round 2's fixes were wrong or no-ops (#1074) Third and final Codex round. Eight findings, four P1 — and the important part is that FIVE of them are defects in round 2's fixes, not in the original code. - The sync-upload enqueue I added was a silent no-op. It queried through `trx` after the transaction had already been committed, which throws "Transaction query already complete" straight into the catch I had wrapped it in. Chunked uploads and watch-folder imports were still never scanned, and the code read as though they were. Uses `db` now. - The post-restore requeue ran BEFORE the files were restored, in both the portable importer and the native restore. The face worker is live during a restore, so it could claim those rows and scan the previous instance's files, or fail them for originals not yet on disk — with nothing to requeue them afterwards. Both now run after file restoration; the native one is extracted into requeueFaceScans() and called from the full and database-only paths. - The admin face crop mixed coordinate spaces: an original-pixel bbox scaled against the THUMBNAIL's natural size. The API now returns the source dimensions alongside the box, so there is one space to reason about. - Forwarding people props through layoutProps did not make them work — the full-page layouts never destructured them. GalleryStoryLayout now threads them to its own lightbox. Genuinely new findings, all in the same class as ones already fixed: - releaseToPending updated unconditionally, so a photo purged while its sidecar request was in flight came back as 'pending' and was rescanned — biometric rows reappearing after the purge reported success. Round 2 fixed exactly this on the COMMIT path and I did not carry it to the retry path. Now guarded on 'processing'. - purgePhotoFaces left face_status alone, so a worker mid-scan still satisfied its commit guard and could write fresh faces into a photo being deleted — orphans, since the FK cascade is inert on SQLite. It now clears the claim as part of the purge. - Phase 3 was unreachable: the migration seeds face_auto_categorize_enabled false and nothing could ever write it, so the rule engine and its undo endpoint returned "disabled" in every real flow. Added GET/PUT and a toggle on the admin card, EN + DE. NOT fixed, deliberately: GalleryPremiumLayout uses yet-another-react-lightbox rather than the shared PhotoLightbox, so person chips there are a real port rather than a prop forward. Recorded as open rather than bodged. Backend failure set identical to origin/main; 41 face tests and 140 frontend tests green; i18n audit reports EN and DE complete at 71 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): block face recognition on the all-in-one image (#1074, #1042) The single-container image cannot run this feature, so it is refused there rather than left to degrade. WHY, since the reason is not obvious from the code: the AIO image runs the backend, the frontend, SQLite and every background worker inside one container aimed at "one photographer plus guests browsing". It has no Redis, SQLite gives it a single writer, and it contains no ML sidecar to talk to. Face detection would add a second image-processing pipeline competing with Sharp for the same CPU and memory. That failure is not loud — the install just becomes slow and looks broken, which is the worst possible shape for a deployment whose whole promise is one container and no decisions. Gated on an explicit PICPEAK_SINGLE_CONTAINER marker, NOT inferred from SERVE_FRONTEND or a SQLite path: plenty of legitimate multi-container setups serve the frontend from the backend or run SQLite, and none of them should lose the feature by accident. Three layers, because the first is the only one that enforces: - faceSettings.isFeatureEnabled() returns false before consulting the flag, so a database restored from a full deployment with `faces` enabled still cannot switch it on here. - The feature-flag API forces `faces: false` in both directions, so the admin UI reflects reality instead of offering a switch that refuses to stay on. - The Features tab renders the card disabled with a plain-language reason, read from a new `single_container` field on /admin/system/version (an endpoint the admin UI already calls). Documented in ml/README.md and .env.example. Three tests pin the behaviour, including that the marker only accepts explicit truthy values. NOTE FOR PR #1068: this expects `Dockerfile.aio` to set `ENV PICPEAK_SINGLE_CONTAINER=true`. That one line lives on that branch and is not in this commit — until it lands, an AIO build would still offer the feature. Worth adding alongside the `Limits` section of docs/single-container.md. 44 face tests green; EN + DE complete at 72 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(faces): pin the bbox coordinate space with a real scale factor (#1074) The coordinate-space bug — boxes stored in preview space while every consumer reads them as original-image pixels — had no test, and could not have been caught by the ones that existed: every photo in the demo gallery is 750px, so the scale factor was always exactly 1.0 and the correction never executed. Verified by hand first, on a real 4000x3000 upload with the face placed off-centre so a wrong crop would be unmistakable. Before the fix the stored box was 1493,204 (preview space, face actually at x≈2850-3618); after, 3110,426 — a factor of 2.083, exactly 4000/1920, landing inside the face. The admin crop then resolved to left=-395px/top=-46px on a 64px window, which is the face centred. That verification is now a test rather than a memory. Three cases: a 4000px photo must scale by 4000/1920, a 1920px photo must NOT change (the case that hid the bug), and a row with no width must fall back to unscaled rather than storing NaN. Note for anyone extending these: jest hoists mock factories above the file, so anything they close over has to be `mock`-prefixed. Getting that wrong fails at transform time with a message that does not name the variable. 47 face tests green. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
0874a30ac9 |
feat(docker): all-in-one image (#1042) — my version of #1067 (#1068)
* feat(docker): add all-in-one image — backend + frontend in one container (#1042) One container, one Node process, SQLite by default: `docker run` with no compose file, no nginx, no supervisor, no bundled Postgres/Redis. - Dockerfile.aio (repo-root context): frontend build stage + backend deps stage + a runtime stage mirroring backend/Dockerfile's production stage, with the built SPA copied to /app/frontend/dist and SERVE_FRONTEND=true. DATABASE_CLIENT=sqlite3 and STORAGE_PATH=/app/storage are pinned explicitly — the storage fallback resolves to container-root /storage, which EACCESes after the su-exec drop. - server.js: the SERVE_FRONTEND block now does what the nginx image did — renders ${BRAND_TITLE}/${BRAND_DESCRIPTION} into index.html once at boot, serves that rendered shell on /index.html and every SPA route, caches hashed /assets/* immutably while the shell revalidates, and gzips the bundle via compression() mounted after all /api routers. express.static now runs with index:false so `/` keeps flowing to handlePublicSiteRequest — its default index option was shadowing the landing page on native installs. - wait-for-db.sh: skip the Postgres readiness wait when DATABASE_CLIENT is sqlite3. The engine resolver still runs, still logs, and still refuses the populated-both conflict (#1038). - .dockerignore: **/node_modules, so the root-context build can't pick up host deps from backend/ or frontend/. - docker-build.yml: build-aio / merge-aio follow the same per-arch build → digest-merge → per-version tag scheme as backend/frontend (GHCR only for now; the Docker Hub mirror is wired once the Hub repo exists), plus a smoke-aio job that boots the image on every PR and asserts /health, the SPA shell, the rendered brand title, immutable asset caching and the SQLite engine resolution. Pointing DB_HOST/DB_USER/DB_PASSWORD + DATABASE_CLIENT=pg at an external Postgres works exactly like the backend image. * fix(ci): correct three smoke-aio assertions that would fail a green image (#1042) Found by running the smoke job locally against a real build — the image passed every behavioral check, but three assertions were wrong: - `/` asserts 200, but handlePublicSiteRequest 302s to /admin/login while the public landing site is disabled, which is the state of the fresh install the smoke container always is. Assert the redirect target instead — that still proves express.static's index option is not shadowing the handler, which is the thing the check exists for. - The placeholder-leak grep matched index.html's explanatory comment, which mentions BRAND_TITLE in prose and survives into the built shell. Match the literal ${BRAND_TITLE}/${BRAND_DESCRIPTION} tokens with -F, and cover the description token too. - Add a gzip assertion, probing with GET: the compression middleware skips bodyless responses, so a HEAD probe reports no Content-Encoding even when compression is active. Verified locally on linux/arm64: image builds clean, boots to healthy in ~8s on the SQLite default, and 25/25 checks pass (SPA shell, rendered brand title, immutable+gzipped assets, no-store shell, SPA fallbacks, npm removed, su-exec drop to nodejs, no errors in the boot log). The DATABASE_CLIENT=pg override was exercised against a real Postgres too — the readiness wait still runs and the engine resolves to postgres. * fix(server): serve the SPA for every client route, not just /admin and /gallery (#1042) nginx did `try_files $uri $uri/ /index.html`, so behind compose every client-side route survived a direct hit or a refresh and the short `['/admin', '/admin/*', '/gallery/*']` list was never exercised. Without nginx that list is the whole contract, and everything outside it 404'd: /setup /customer /impressum /datenschutz /payment-check /quote/:token /contract/:token /invite/:token /transfer/:token /transfer-upload/:token /setup is the first URL a new install visits, so the all-in-one image was unusable from a cold start. The catch-all is registered after `app.use('/api', notFoundHandler)`, so an unknown /api route still answers JSON instead of being handed the HTML shell, and after the /s/:shortSlug resolver, so a typo'd short URL still 404s (#699). It is GET-only — a stray POST keeps 404ing rather than getting a 200 page back. The handler is hoisted out of the SERVE_FRONTEND block via `spaCatchAll` because that block runs before the API 404 handler is registered. Verified on the built image: all ten routes above now 200, /api/nope still returns JSON 404, /s/nonexistent still returns 404, / still 302s to /admin/login, and the smoke suite is 25/25. Both boundaries are now asserted in the smoke-aio job. * docs(readme): document the single-container install (#1042) The README had no mention of the all-in-one image, so the only way to discover it was reading the workflow file. Adds a Quick Start subsection with the one-line `docker run` and the `docker exec … cat SETUP_TOKEN` step, plus a row in the documentation table. Deliberately does not sell it as the default: the note says the compose stack is still the right choice for anything busier, gives the reason (SQLite takes one writer at a time), and points at the `.picpeak` restore as the way out, so nobody picks it and then finds themselves stuck. Full details live at docs.picpeak.app/deployment/single-container (PicPeak/docs#8). * feat(docker): fold #1067's items into the all-in-one image (#1042) Consolidating the two parallel AIO branches into this one. This PR's approach is kept wherever the two differed on design — in particular the in-process brand render, `index: false` (which fixes express.static shadowing handlePublicSiteRequest, a bug #1067 had), the compression middleware, and the smoke-aio job. What follows is what #1067 had that this branch did not. Layout — the issue asks for a single mountable root, and this moves to one: /data/db picpeak.db (+ -wal/-shm) and SETUP_TOKEN /data/storage originals, thumbnails, archives /data/logs application logs /data/backup built-in backup output; /backup symlinks here `-v picpeak:/data` and nothing else to remember. README and the smoke job's database-path assertion follow the new layout. Correctness items: - sqlite CLI. DatabaseBackupService SPAWNS `sqlite3` for `.backup` and PRAGMA integrity_check; the npm module does not ship that binary. backend/Dockerfile omits it because compose always runs Postgres — this image defaults to SQLite, so every database backup failed with ENOENT. - /backup wired in. Migrations 029 + 030 seed /backup/picpeak and /backup/database as the backup destinations; nothing created or mounted them, so backups had nowhere to write and anything written would die with the container. Symlinked into the volume, subdirectories created at startup (a bind mount hides the tree baked into the image), and adopted only when BACKUP_DIR is set so it never gates boot for compose deployments that do not mount it. - logger.js honours LOG_DIR. It hard-coded <backend>/logs, so logs could not leave the container. Unset keeps the old path for every existing install. - wait-for-db.sh derives its writable roots from STORAGE_PATH / DATA_DIR / LOG_DIR instead of hard-coded /app paths, and mkdir -p's them before chown — a bind-mounted /data hides the image's tree, and chown against a missing path reports "the filesystem rejects chown", which is both wrong and a dead end. - .dockerignore excludes backend/-prefixed runtime data. Docker reads only the root file, so the unprefixed data/*.db, logs/* and storage/* rules missed backend/data, backend/logs and backend/storage entirely; a checkout used to run PicPeak would bake its database, photos, logs and SETUP_TOKEN into a published layer. - HEALTHCHECK follows $PORT rather than a hard-coded 3000. - --max-http-header-size=32768 matches nginx's large_client_header_buffers 4 32k; Node's 16 KiB default would reject a guest carrying several per-gallery JWT cookies. docs/single-container.md is added as the in-repo reference the README links to. The smoke job gains four assertions for the above: the one-volume layout and writable backup destinations, the sqlite3 CLI, logs landing on the volume, and the image carrying no runtime data from the build context. Verified on a built image — named volume, bind mount and PORT=8080 all healthy; every existing smoke assertion still passes, including / -> 302 /admin/login, the rendered BRAND_TITLE, immutable assets, gzip and /s/<unknown> -> 404. Co-authored-by: Luca-Timo <[email protected]> * fix(docker): restore the SPA-fallback exclusions and close the build-context leak (#1042) Both found by external review of the consolidated branch. - The SPA catch-all had no backend-owned exclusions. This was a regression I introduced while merging: #1067 carried a BACKEND_OWNED prefix list, and taking this branch's server.js wholesale (correctly — its index:false and in-process brand render are the better design) dropped it. /photos, /thumbnails, /uploads and /fonts are static mounts whose middleware calls next() on a miss, so the catch-all was answering 200 text/html under image and font URLs instead of 404. nginx gave each of those its own location block, so try_files never applied to them. - backend/data is now excluded wholesale rather than by suffix. The suffix list (*.db, *.db-wal, *.db-shm, SETUP_TOKEN) let real secrets through: a used checkout carries ADMIN_CREDENTIALS.txt next to the database, plus -journal files and any DATABASE_PATH not ending in .db. Since Dockerfile.aio builds from the repository root and COPYs backend/ wholesale, any of those would be baked into a published layer. The directory holds only runtime state and is already gitignored in full. smoke-aio gains an assertion that the backend static routes still 404, so the exclusion cannot be dropped again silently. Verified on a built image: /photos, /thumbnails, /fonts and /uploads misses all 404; /setup, /impressum, /gallery/x, /admin/login still 200; / still 302s to /admin/login; /api/nope still answers JSON; /s/<unknown> still 404s; and the image carries no *.db, ADMIN_CREDENTIALS.txt, logs or storage from the context. * fix(aio): three failures that only surface outside a dev laptop (#1042) Backups aborted on SQLite. getTableChecksums() built its digest with `CAST(t.* AS TEXT)`, which is Postgres row-to-text syntax; SQLite parses `*` there as a syntax error, so every backup threw before reaching the .backup call. Since the all-in-one image ships SQLite by default, that is every AIO install. Enumerate the columns via columnInfo() and sum their lengths instead. The shared /data mount root was never adopted. wait-for-db.sh chowned the children it creates but not the mount point itself, so a host directory arriving as 0700 with a foreign owner stayed untraversable by UID 1001 after the su-exec drop. Docker Desktop's permissive bind mounts hide this completely, which is why local testing passed; a NAS share does not. DATA_ROOT is now adopted first. Maintenance mode locked the admin out of the box. The middleware runs at server.js:493, long before the static block at 891, and exempted the auth endpoints but not the page that calls them. With the backend serving the frontend, /admin/login and /assets/* returned 503 JSON, so an admin who enabled maintenance mode could never load the UI to turn it off. nginx serves those paths in the compose stack, which is why it never surfaced there. Guest and API surfaces stay gated. Verified on a built image: checksums compute across all 95 tables; a bind mount created 0700/4000:4000 boots healthy and ends up 1001:1001; with general_maintenance_mode=true, /admin/login, /admin and /assets/* return 200 while /gallery/* and /api/gallery/* return 503 — and 503 across all three once the exemption is removed again. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): stop leaking .env into the image, fix the broken checksum test (#1042) The Jest suite was red: mocking db.raw is no longer enough now that the SQLite checksum branch asks the query builder for its column list, so db(table) came back undefined and getTableChecksums failed on every PR. The production code is right; the fixture needed to know about the call. backend/.env was landing in the published layer. The root ignore file's `.env`, `.env.*` and `data/*.db` rules read as unanchored but Docker matches them from the context root, so they catch ./.env and never backend/.env — and `COPY backend/ .` then puts a real JWT_SECRET at /app/.env. Matched at any depth instead, the way **/node_modules in the same file already is. Confirmed by building from a checkout carrying a planted secret: before, `cat /app/.env` printed it back. Business documents wrote outside the volume. quoteService, invoice sending/reminders and contract signatures build paths from process.cwd()/storage and never read STORAGE_PATH; compose hides it by setting STORAGE_PATH=/app/storage with WORKDIR /app so the two are the same directory. Here they are not, and /app is root-owned, so a quote or invoice PDF failed to write as UID 1001 — and would not survive the container if it had. Symlinked /app/storage into the volume, matching the /backup symlink beside it. Teaching those services STORAGE_PATH is the real fix and wants its own change. Two smaller ones: the mount root is now chowned shallow rather than recursively, since every child below it is already walked recursively and a NAS-sized photo library should not be traversed twice on each restart; and /assets/ joins the backend-owned prefixes, so a stale hashed chunk requested by a tab left open across an upgrade gets a 404 instead of index.html served with 200 under a .js URL. Verified on a built image: planted backend/.env and backend/probe.db are absent; /app/storage resolves to /data/storage and a business-doc write as UID 1001 appears on the host; a 0700 bind mount owned by 4000:4000 boots healthy; a missing /assets chunk 404s while the real bundle still serves 200 as application/javascript. The databaseBackup suite is green again, and the branch adds no failing suite that origin/main does not already fail on the same machine. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(aio): teach the leak assertion about the storage symlink (#1042) The previous check listed /app/storage/events and treated a hit as a leak. That was true while /app/storage was either absent or a copied directory; now it is a symlink into the volume, so the check followed it and found the empty tree the image itself creates — a false positive on its own design. Check the shape instead: /app/storage must be a symlink pointing at /data/storage, and the volume's photo tree must contain no files on a fresh install. A real directory there now fails loudly, which is the condition the assertion was always trying to catch. Also extended the path list to /app/.env and loose database files, matching the .dockerignore rules added alongside. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): show the maintenance screen instead of raw JSON to guests (#1042) The previous commit exempted the admin shell so an admin could still reach the switch they had just flipped. Guests had the same problem for the same reason: with no nginx in front, /gallery/<slug> reaches this middleware long before the static block, so a visitor during maintenance got a 503 JSON body where every other deployment shows the branded maintenance screen the frontend already ships. Replaced the two path-specific exemptions with the rule they were both special cases of: a GET that is not an API call and not a backend-owned content mount is the SPA shell, and the shell is inert HTML — it boots, reads /api/public/settings (already exempt) and renders MaintenanceMode on its own. Everything that carries real data stays gated: /api/*, /photos/, /thumbnails/, /fonts/, and any non-GET. Compose is untouched by construction, since nginx answers those paths and they never arrive here. Verified on a built image with the flag on: /gallery/x, /customer/x, /admin and /admin/login return 200 text/html while /api/gallery/x/verify, /photos/x.jpg and /thumbnails/x.jpg return 503 and a POST to a public API still returns 503; with the flag off the same paths go back to 404. Added a middleware test over that exemption matrix — over-exemption is the real risk in this change, so it asserts the gated half too. It fails on five cases without the fix. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): stop the shell exemption from un-gating /og and the public CMS (#1042) The previous commit exempted "any GET that is not an API call". That negative rule reads as safe and is not: /og/gallery/<slug> and its /cover render the event name and the hero thumbnail, /s/<code> renders short-link previews, and `/` is handed to the public CMS. All four are proxy_passed to the backend by nginx, so they were gated before this PR in every deployment — the rule un-gated them, and for compose too, not just the new image. A site switched to maintenance would have kept publishing gallery metadata. Replaced the guess with the split nginx already defines: exempt what the frontend container answers itself, gate what it proxies. That is the same rule the all-in-one image needs by definition, since its whole job is to be both halves of that stack, and it now matches compose in both directions rather than only in the direction the last commit tested. Verified on a built image with the flag on: /admin/login, /gallery/<slug> and /customer/* return 200, while /, /og/gallery/x, /og/gallery/x/cover, /s/abc, /robots.txt, /api/* and /photos/* return 503; with the flag off all of them behave normally again. The middleware test grew the gated cases — it now covers 21, most of them asserting what must NOT be exempt. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): give the image a FRONTEND_URL default so share links are absolute (#1042) getFrontendBaseUrl() reads FRONTEND_URL, falls back to the general_site_url setting, and otherwise returns an empty string — which makes share_url come back as a bare "/gallery/<slug>/<token>". Compose defaults the variable to http://localhost:3000, but the documented one-liner for this image passes only JWT_SECRET, so every fresh single-container install handed out relative links in API responses, QR codes and emails. Defaulted to the same value compose uses; -e FRONTEND_URL=https://... overrides it, as does the site URL field in Settings. Found by pointing tests/e2e/local at a running AIO container: auth/06-api-tokens asserts share_url matches /^https?:\/\//, and it was the one spec that failed for a product reason rather than a harness one. It passes now, and the suite is 19/20 against the image — the remaining failure is smoke/02-auth-flow, whose seed helper shells out to a hard-coded `docker exec picpeak-backend`, so it cannot arrange its precondition against any other container. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(aio): mark the image so face recognition stays off (#1042, #1074) Face recognition needs a separate ML container this image does not contain, and enabling it here would add a second image-processing pipeline competing with Sharp for the CPU and memory of a container sized for one photographer plus guests browsing. The failure mode would not be a clear error — just a slow install that looks broken. The backend gate for this lands in #1075 and keys on PICPEAK_SINGLE_CONTAINER. Without this line the guard never triggers on an actual all-in-one build, so the two changes have to arrive together: whichever merges second completes the pair. Verified against this file's exact value — isFeatureEnabled() returns false with it set. An explicit marker rather than inferring from SERVE_FRONTEND or the SQLite path, because legitimate multi-container deployments do both of those and should keep the feature. Also adds it to the Limits section of docs/single-container.md, next to the SQLite and Redis constraints, since that is where someone will look before choosing this image. --------- Co-authored-by: Paul Nothaft <[email protected]> Co-authored-by: the-luap <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
f22999aba6 |
fix(storage): write business documents under STORAGE_PATH, not the cwd (#1070)
* fix(storage): write business documents under STORAGE_PATH, not the cwd persistDocPdf, the invoice sending and reminder writers and both contract signature writers built their target from `path.join(process.cwd(), 'storage', 'business-docs', ...)` and never consulted STORAGE_PATH. docker-compose.yml and docker-compose.production.yml both pin STORAGE_PATH=/app/storage and the image's WORKDIR is /app, so on a stock deployment the two expressions name the same directory and nothing looked wrong. Point STORAGE_PATH anywhere else and quotes, invoices, Mahnungen and contract PDFs land outside the configured storage root: missed by the backup walker, invisible to the storage accounting, and gone when the container is replaced. It also fails outright where the working directory is not writable by the runtime user. Routed all six writers through getStoragePath(), the resolver the rest of the app already uses. Two read-side sites of the same class came along: the custom PDF font lookup now checks the storage root before the legacy cwd path (a font under STORAGE_PATH/fonts was simply never found, and the document silently fell back to the built-in face), and the dev-test scratch directory follows the same root. Left alone deliberately: resolveLogoFile and adminBusinessProfile already try both roots, so their cwd reference is a legacy fallback rather than a miss. No migration needed — the persisted path is stored absolute, so rows written before this keep resolving to where those files actually are. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(storage): allow the configured contract root, and move signature images too Two holes in the previous commit, both found by review. Contract downloads would have broken. assertContractPdfPath() guards the admin unsigned/signed PDF routes and GET /api/public/contracts/:token/pdf, and it listed only <cwd>/storage/business-docs/contract. Moving the writers to STORAGE_PATH without moving that root meant every newly generated contract was refused with PATH_OUTSIDE_STORAGE — a worse failure than the bug being fixed, and only on the installs the fix was for. The configured root is now allowed alongside the cwd one, which stays for contracts written before the move; their absolute paths are in the database and still resolve. Note the sibling root on the next line already honoured STORAGE_PATH, so the helper was half-migrated already. persistSignatureImage() still wrote customer and admin signature PNGs under process.cwd(). It was missed because its path.join is spread over seven lines while the others are single-line — and the regression test compared against the single-line literal, so it reported green over a live bug. The test now collapses whitespace before matching, which is the only reason a formatting difference ever hid this. A sweep of the whole of src/ with the same normalisation confirms the remaining process.cwd()/storage references are all deliberate `STORAGE_PATH || cwd` fallbacks, not misses. Added a case that drives assertContractPdfPath against real files on disk — the guard realpaths both the file and its roots, so a test using imaginary paths proves nothing. It fails without the fix. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(storage): resolve the contract guard's root through the shared resolver The guard still built its own `STORAGE_PATH || <cwd>/storage`. That matches getStoragePath() only while STORAGE_PATH is set — with it unset the shared resolver falls back module-relative to <repo>/storage while this fell back to <cwd>/storage, and the backend is normally started from backend/, so the two name different directories. Writers and guard then disagreed about where contracts live and the download routes refused them, which is the same failure the previous commit fixed for the configured case, reappearing in the fallback case. One resolver on both sides now, which is the point of the whole change. Docblock updated to describe the three roots as they actually are. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(storage): make the fallback test safe, and align the backup diagnostics The test added in the previous commit was dangerous. To exercise the STORAGE_PATH-unset case it deleted process.env.STORAGE_PATH and then, in cleanup, recursively removed `<resolved root>/business-docs` — which with the variable unset resolves to the developer's real, gitignored <repo>/storage. Running `npm test` in a working checkout would have destroyed local business documents. This checkout has 65 MB there, including a populated business-docs tree. Rewritten to mock the shared resolver instead. That is both safe (every path stays in the tmpdir) and a sharper assertion: if the guard consumes getStoragePath() the mock moves its root, and if it went back to rolling its own expression the mock would have no effect and the test fails — which is exactly the regression being pinned. backupCoverageService and backupIntegrityService kept their own `STORAGE_PATH || cwd` roots. The backup walker itself already falls back module-relative, so with the variable unset the two diagnostics inspected a directory neither the walker nor the writers use and would report the business-docs tree as missing while it was in fact being backed up. Both now use the shared resolver. No regression: the same jest invocation over contract/quote/invoice/pdf/ backup suites gives an identical 11 failed, 24 passed before and after — those failures are a locally missing cron-parser dependency and reproduce on an unmodified tree. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
52db982661 |
fix(gallery): make per-event banner overrides actually work, both banners (#440, #932) (#1064)
* fix(gallery): make per-event banner overrides actually work, both banners (#440, #932) The promo banner shipped with a per-event inherit/custom/off override that never reached a guest. GalleryView reads promo_mode from the /photos payload, and /photos never sent it — so every gallery resolved to 'inherit'. Setting a gallery's promo banner to "Off" did nothing; the global banner kept rendering. The info banner (#932) mirrored that shape and inherited the same gaps. Four places dropped the fields; all four now carry both banners: 1. GET /gallery/:slug/photos — send promo_mode/promo_markdown alongside the info fields. This is the fix that makes "Off" mean off. 2. POST /admin/events — the validators accepted both banners and the insert discarded them, so an API client could POST info_mode:'off', get 201, and find the row on 'inherit'. Markdown is stored only for 'custom', matching the PUT rule. 3. POST /admin/events/:id/duplicate — copy both from the source row. The dialog promises the copy "inherits the branding, behaviour, feedback, and category configuration"; a muted gallery un-muting on duplication is the opposite of that. 4. PUT /admin/events/:id — resolve the effective mode from the STORED row when a partial update sends only the markdown. Previously updates.promo_mode was undefined on such a request and the text was parked on an inherit/off gallery, then resurfaced when someone later switched it to 'custom'. The lookup is lazy: one extra query, only on that path. The two normalisation blocks are now one loop over both banners, so the pair can't drift apart again. Verified in a browser, both directions against the same global banner: promo_mode='off' -> not rendered; 'inherit' -> rendered. The /photos payload went from promo_mode ABSENT to carrying the value. * fix(gallery): thread promo into the reveal view, drop stale markdown on duplicate External review, round 1 on this PR. Two gaps in the plumbing it introduced: - The reveal-hidden branch copied only the info fields from /photos. Now that /photos carries promo too, a reveal-hidden gallery with promo_mode 'off' still fell back to 'inherit' and showed the global banner on the first load after login. Thread both banners there. - The duplicate copied markdown verbatim. A row written before the PUT normalisation landed can hold text while its mode is 'inherit'/'off', so the copy inherited hidden text that would resurface the moment someone switched it to 'custom' — violating the very invariant this PR establishes. Copy markdown only when the source mode is 'custom'. Test covers the stale-markdown source explicitly. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
b48fa62eea |
feat(gallery): info banner above the photo grid (#932) (#1063)
* feat(gallery): info banner above the photo grid (#932) A short informational note rendered at the TOP of a gallery, above the photos. Distinct from the promotional banner (#440), which stays by the footer for marketing copy — the reporter's case is an onboarding hint ("use the menu button to filter"), which is useless below a gallery the guest has to scroll past first. Mirrors the promo feature's shape rather than inventing a second one: a global default in Settings → Branding (branding_info_markdown) plus a per-event inherit/custom/off override. Markdown via the existing MarkdownContent sanitiser — no raw HTML, no CSS injection. Empty global default means nothing renders, so upgrading changes nothing visible. Deliberately NOT included: an alignment knob (this is short helper copy, not marketing layout) and guest dismissal — the issue lists dismissal as a nice-to-have, and it needs per-guest persistence that is its own decision. Migration 176 is idempotent (hasColumn / existing-key guarded). Note on the payload plumbing: the per-event fields travel in the /photos response, not just /info. GalleryAuthContext seeds its cached event from the gallery LOGIN response — a small identity subset — so anything absent there is undefined right after a guest signs in. /photos is the payload that refreshes on every gallery load, which is why the fields were added there and why GalleryView reads them from `data.event`. Verified in a browser across all three modes; reading them from the context event instead silently collapsed every override back to 'inherit'. * fix(branding): map branding_info_markdown on read so saving can't wipe it (#932) External review caught this. BrandingSettings declared no info_markdown and formatBrandingSettings never mapped branding_info_markdown, so BrandingPage's hydration — setBrandingSettings(prev => ({ ...prev, ...formatted })) — kept the empty-string initializer instead of the persisted value. The form loaded blank and the next Save posted '' back, wiping a configured banner. Silently: the gallery keeps rendering the old copy until that save lands. This is the same bug the footer/promo fields hit in #441 + #440 / #460, which the read mapper still carries a comment about. Add the field to the interface and the mapper, and pin the round-trip for the whole editable branding set so the next field added is caught by a test rather than by a user losing copy. Verified: the new test fails 3/4 with the mapper line removed. * fix(gallery): honour the info-banner override in the reveal-hidden view (#932) External review, round 2. The hidden-until-reveal branch renders GalleryLayout with the context `event`, which is seeded from the gallery login response and carries no banner fields — so while a gallery was hidden, a per-event 'off' silently resolved to 'inherit' and the global banner appeared on a gallery the admin had muted. Resolve the fields there the same way the main render path does. The two full-page layouts (gallery-premium, gallery-story) are deliberately left alone: they return before GalleryLayout and render no header, footer or promo banner either — injecting a wrapper into layouts documented as having 'their own integrated UI' would be a design change, not a fix. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
3a11e6ebb5 |
fix(pdf): RFC 6266-encode Content-Disposition on quote/invoice PDFs (#1024) (#1055)
* fix(pdf): RFC 6266-encode Content-Disposition on quote/invoice PDFs (#1024) The six quote/invoice PDF endpoints interpolated buildPdfFilename()'s result straight into `inline; filename="${filename}"`. That result deliberately preserves non-ASCII (it doubles as the PDF's internal Title metadata), and HTTP header values are latin1 — so a customer label reaching the header directly failed in one of two ways: - U+0080-U+00FF (ä ö ü ß — every German umlaut): no throw. The raw byte goes out and the client reads back a mangled name. Silent corruption. - above U+00FF (Polish ł, Czech ř, Turkish ş, €, Cyrillic, CJK, emoji): Node's setHeader rejects it with ERR_INVALID_CHAR. The throw lands after the PDF buffer is already rendered, so the request 500s. Note this corrects the issue's diagnosis: it reported umlauts as the 500 case, but umlauts are inside latin1 and mangle rather than throw. Both symptoms share this root cause and both are fixed here. Route through buildContentDisposition() (utils/filenameSanitizer, already used by secureImages.js), which emits an ASCII fallback plus the RFC 5987 `filename*=UTF-8''…` form, so the unicode name survives in browsers and the header stays legal. Applied to all six sites: adminQuotes (persisted + preview), adminInvoices (persisted + preview), customer (quote + invoice). Also correct buildPdfFilename's docstring, which advertised the preserved non-ASCII as suitable for Content-Disposition — the exact misreading that produced these call sites. * test(pdf): pin the ASCII fallback for fully non-Latin customer names (#1024) A name written entirely in another script leaves the legacy filename= token with just the document number (Q-2026-0042_.pdf) — filename* carries the real name. That's the intended trade, but it's the token a client without RFC 5987 support actually saves, so assert it stays legal, non-empty and carries the document number rather than leaving it unpinned. * fix(pdf): don't split surrogate pairs when truncating the filename (#1024) Codex review caught this. sanitiseSegment caps each segment at 80 UTF-16 code units, so a cap landing inside an astral character (emoji, rarer CJK) left a dangling high surrogate. encodeURIComponent throws URIError: URI malformed on a lone surrogate, so buildContentDisposition — the helper this PR routes the six PDF endpoints through — 500'd for e.g. company_name = 'a'.repeat(79)+'🎉', well inside the 120-char validator limit. Same 500 the PR set out to remove, reached a different way. Drop the orphaned surrogate instead of widening the cap, so the byte budget the limit exists to protect is unchanged. Tests cover both boundary cases and assert the cap semantics; they fail against the previous slice(). --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
LucaandPaul Nothaft
|
8809564aad |
feat(backup): open sqlite → pg .picpeak restore as the supported upgrade direction (#1041) (#1043)
Reshaped onto main after #1039 landed the coercion engine (typedColumnsFor / epochToIso / coerceForTargetEngine) — this PR is now only the policy delta on top of it: - validateManifest: replace the CLI-only allowEngineSwitch opt-in with a direction rule — sqlite → pg allowed (upload UI and CLI alike), pg → sqlite refused with a message naming the supported direction - importFromPicpeak: derive crossEngine from the manifest's engine (absent field = target engine, the exact pre-change behavior), log it, return it; route passes it through - scripts/migrate-sqlite-to-postgres.js: rely on the shared gate, drop the flag - restore card: direction stated in the intro, cross-engine notice after a converting restore; both strings in en.json + de.json; removed the orphaned settings.backup.picpeak locale node (unreferenced, stale copy) - picpeakCrossEngine.test.js: direction policy, epochToIso (ms, seconds, numeric strings), coerceForTargetEngine units, plus PICPEAK_PG_TEST_URL-gated real-Postgres stored-value assertions Co-authored-by: Paul Nothaft <[email protected]> |