f92d4bb2d9c2ea84f59dd4cfaa3a4272f1eec56b
123
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul NothaftandPaul Nothaft
|
9f4b9bab46 |
fix(usage): explain and de-emphasize the pending-packet button lock (#1363)
* fix(usage): explain and de-emphasize the pending-packet button lock An admin whose report delivery is stuck (old schema, network issue, etc.) saw the v5-upgrade and portal buttons greyed out with no indication why, or what to do about it — the backend's single-packet- in-flight guard is correct, but silent. Add an inline note pointing at "Retry / send if due" when a pending packet is the actual cause. Also: "Review expanded usage.v5 scope" didn't read as an upgrade action — renamed to "Upgrade to usage.v5" / "Auf usage.v5 upgraden". "Open usage portal" is now a primary (green) button in both its signed-in and pre-participation forms, matching the visual weight of the other primary actions on this tab instead of blending in as a secondary outline button. * fix(usage): scope the pending-packet note to controls it actually gates The note added in the previous commit rendered whenever pending_action was truthy, regardless of participation status. Outside `active` (activation_pending, deletion_pending) the portal renders as a plain un-gated link and no v5-upgrade section exists at all, so the note named two controls that either weren't blocked or weren't on screen. And when active but already on the current schema, it wrongly implied a v5-upgrade button existed. Gate on `active` (nothing is actually blocked outside it), and choose between the existing two-control message and a new portal-only one based on whether consent_update_available — which the v5-upgrade section itself is gated on — is true. Four new regression tests cover each shape: activation_pending, deletion_pending, active+current-schema (portal-only), and active+outdated-schema (both, the original case). Found by code review. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
9168bdd504 | fix(setup): require the full usage reporting disclosure | ||
|
Paul NothaftandPaul Nothaft
|
fb9da72f14 |
feat(security): opt-in recoverable gallery passwords (#1341)
* feat(security): opt-in recoverable gallery passwords Gallery passwords are bcrypt hashes, so an admin who needs to hand a password to a client a second time has to reset it, which invalidates what the client already has. This adds a security setting, security_gallery_password_recoverable, off by default, that keeps an AES-256-GCM encrypted copy of each gallery password and client PIN next to the hash. The key is derived from GALLERY_PASSWORD_ENCRYPTION_KEY or JWT_SECRET. While the setting is on: - create, publish, send-later, edit, reset and the v1 API write the copy alongside the hash; turning a gallery's password requirement off clears it - GET /api/admin/events/:id/password returns the copy to admins with events.edit and ownership, and writes a gallery_password_viewed activity entry on every real reveal - resend-email uses the stored password instead of the "set at creation" sentinel, so the client receives what already works Switching the setting off purges every stored copy. Login and hash verification are untouched; the copy is never read on the gallery side. The Security tab carries the toggle with a warning that stays visible, and the event page shows "Show password" with copy buttons only while the setting is on and the gallery has a secret. Relates to issue 1271 * fix(security): close the write-versus-switch-off race in the password vault The recoverable setting is read while an event insert is assembled and the client-PIN hash awaits after that, so a settings request that switched the feature off and purged in that gap was overtaken by the insert. Every write site now re-reads the setting right after its statement and clears its own row when the setting is off; the settings writer flips the value before it purges, so either the purge or the re-check catches the row. * fix(security): resend carries the stored client PIN and link; deterministic tamper test The creation mail includes the client-access link and PIN; a resend only sent the gallery password even when a stored PIN was available. The ciphertext tamper assertion replaced the last two characters with a constant, which was a no-op roughly once in 4096 runs. * fix(security): drop the revealed password after Send gallery email The send-later route can replace the password; the share card keys its revealed copy on the event query's refetch time, so invalidate the event after the send like the other password-changing mutations do. * fix(security): purge leftovers before the setting write when turning recovery on Switching on wrote the setting first and purged after, so a password write that read the new "on" in between stored a copy the purge then deleted. Turning on now purges before the write; turning off keeps purging after it, which together with the write-site re-check leaves the vault holding exactly what was written while the setting was on. * chore(security): drop the duplicate rateLimitService import left by the rebase * chore(usage): register the password recovery routes in the v5 coverage inventory The inventory moved from v4 to v5 on main; the entry added by this branch followed it. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
5c1e38d921 |
feat(usage): distinguish real edits and template delivery with v5 consent (#1339)
* feat(usage): distinguish real edits and template delivery with v5 consent * fix(usage): exclude queued test messages and count reorders as edits - queueEmail carries usageEligible: false into email_data and the queue processor passes it on, so the dev tools' send-test-email no longer records email_template_delivery once the worker sends it. - event-types/reorder and categories/reorder-global compare the persisted order before and after and record the v5 edit markers only when it changed, matching the display_order edit already counted on PUT. - normalized() builds arrays with Array.from so a row array from the sqlite binding compares equal under Jest's separate realm. * fix(usage): cover per-gallery category order and workflow test runs - categories/reorder records category_editing when an event's override changes; reorder/:eventId records it when an override was actually removed. - send_email and the collections handoff pass usageEligible: false for a workflow test run (engine.testRun sets __test), so a non-dry test send is not counted as template delivery. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
8017370271 |
feat(settings): expose the API rate limiter in the Security tab (#1338)
* feat(settings): expose the API rate limiter in the Security tab The general per-IP limiter had six settings in app_settings and a backend route to write them, and no screen. Installs ran on the code fallback — 300 requests per 15 minutes per IP — with no way to see it, which is how issue 1287 played out: a 546-photo gallery exhausted the budget for one viewer and the operator learned about the setting from a grep of the backend log. Security tab: a card with the six settings, the validation ranges the route enforces, a one-line explanation per field, and a note that the unit is the client IP — an office or household behind one NAT shares a budget, and behind a proxy TRUST_PROXY has to cover the proxy or every visitor shares its address. The tab's Save button saves the limiter through its own route. The limiter values are checked against the route's ranges before anything is written and the limiter is written first, so a rejected value cannot leave the password/session settings half-saved behind a failure toast. Backend, three things the screen needed: - The settings read fills the six keys with the code defaults when they have no row, so the form shows the budget in force rather than an empty field; the defaults live in one exported constant the limiter itself reads. - The write route upserts instead of updating: on a fresh install, which has no rows, the old UPDATE matched nothing and the route answered 200 while changing nothing. - The live limiter instances move into rateLimitService and the write route rebuilds them. express-rate-limit fixes windowMs when an instance is built — max and skip re-read the settings per request, the window does not — so a saved window used to apply only after a restart. The gates in server.js resolve the instance per request through the service's getters. A rebuild starts fresh counters, which on a settings change is acceptable. The limiters get explicit MemoryStores and a rebuild shuts the superseded ones down, because a store keeps a cleanup interval alive for as long as it exists and dropping the reference alone would leak one timer per save. Tests: the read surfaces defaults and honours the key filter; the write creates rows on a fresh database, the limiter sees the values immediately and hands the gates a fresh instance; existing rows are updated not duplicated; out-of-range values are rejected. The tab renders the values, edits through the hook state, carries the ranges, saves with the tab's button, and the pre-write validation accepts the bounds and rejects outside them and cleared fields. Relates to issue 1337 * docs(security): point the rate limiter doc at the Security tab and the upsert --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
9d18868a07 |
fix(usage): stop WebKit collapsing the consent dialog to its header and footer
In Safari the product-usage consent dialog opened as a 302px box on a 714px viewport with the disclosure text squeezed into a 32px strip between header and footer. The dialog is a native <dialog> laid out as a flex column with only a max-height, so its own height is indefinite, and the disclosure region used `flex-1`, which is `flex: 1 1 0%`. WebKit resolves that 0% basis against the indefinite container height as zero: the region's hypothetical size is zero, the dialog sizes to header plus footer, and the max-height never comes into play. Chromium treats the same basis as `content` and was fine. An `auto` basis with min-height 0 sizes the region from its content and lets it shrink to the max-height: measured in iOS Safari, 641px dialog and a 371px scrolling region, footer on screen, matching Chromium. Only the <dialog> is affected. The div-based modals that use the same `flex-1 overflow-y-auto` pattern inside a max-height column were measured in the same WebKit and size correctly, so they stay as they are. A test pins the classes with the reasoning, since jsdom cannot see the layout. |
||
|
Paul Nothaft
|
0986f7f7ac |
Merge pull request #1322 from PicPeak/fix/canvas-lightbox-only
fix(gallery): keep canvas rendering in the lightbox, render tiles as <img> |
||
|
Paul Nothaft
|
a02fa08f69 |
Merge pull request #1329 from PicPeak/feat/usage-portal-signin-and-copy
feat(usage): open the portal signed in, and rewrite the German copy |
||
|
Paul Nothaft
|
a16ff855dd |
Merge pull request #1321 from PicPeak/fix/1317-usage-portal-copy
feat(usage): plain link to the public usage portal, German opt-in copy |
||
|
Paul Nothaft
|
f114f3e876 |
feat(usage): open the portal signed in, with the credential never in a served URL
The "open usage portal" button was a plain link, so an operator who wanted to see their own data had to copy the lookup hash out of the settings page and paste it into the portal. Next to it sat a second control, "connect to requests & voting", which minted a collector session and then showed a third thing, a link to open it. One button now. Before participation it stays the plain link: the portal is public and someone deciding whether to join should be able to look at it first. While participating, a click asks the backend for a collector session (a signed `session` command, so the collector knows which installation this is) and opens the portal with that token in the URL fragment. Fragments are never sent over the wire; the portal drops it from the address bar on load and keeps the session in memory only. The lookup hash itself never leaves the settings page, and no URL a server or an access log sees ever carries a credential. The tab is opened synchronously in the click handler and navigated once the session exists, because opening it after the await trips popup blockers. If the collector cannot be reached the session command is queued for retry and the tab falls back to the public portal, so the click still lands somewhere; a failed request closes the tab again. The separate connect button and its session link are gone, and so are their strings. |
||
|
Paul Nothaft
|
7e40579217 |
feat(usage): plain link to the public usage portal, German opt-in copy
The only way into the usage portal from the settings tab was the session-bound link behind "Connect", which needs an active participation and creates a 15-minute voting session. An operator deciding whether to join had no way to look at the portal first. The status card now carries a plain "Open usage portal" link to the collector base URL the status endpoint already reports, shown whenever that URL is valid, opening in a new tab with rel="noopener noreferrer". The session link stays as it is. German "Teilnahme prüfen" read as a technical check rather than reviewing the consent details; it is now "Details ansehen" in both places the key is used (notice link and opt-in button). The actual opt-in stays "Produktnutzung aktivieren". Relates to issue 1317 |
||
|
Paul Nothaft
|
aa6e5d613f |
chore(usage): make the capability catalog English-only
features.v2/v3/v4.json carried every name, description and definition twice, once in English and once in German, inside a file that is source code, is vendored byte-identically into the collector and is served as the consented catalog. Source stays English. The German strings already existed a second time in the frontend locale file, which is what the consent dialog actually renders (UsageCatalog reads productUsage.catalog.<key>, never the JSON), so the copy in the catalog was a duplicate that could only ever drift. The `de` fields are gone from all three catalogs, their frontend copies and the inventory definitions; the docs coverage file and FEATURE_COVERAGE.md list English only. Nothing on the wire changes: the report schema is derived from the keys, and the catalog's text is not part of any signature or consent version string. The coverage test now pins the catalog to the en locale verbatim and requires the de locale to cover every key and field, without dictating its wording. The collector holds the same catalog files and needs the same change to stay byte-identical, plus its German catalog strings moved into its own locale file; that is filed there. |
||
|
Paul Nothaft
|
ef8a52f02c | fix(usage): introduce consented v4 without changing historical reports | ||
|
Paul Nothaft
|
c75839d3ea |
fix(gallery): keep canvas rendering in the lightbox, render tiles as <img>
Every tile, the hero and the folder covers switched to a <canvas> when the per-event toggle was on or the protection level was `maximum`. A canvas pins a backing store of naturalWidth × naturalHeight × 4 bytes that the browser is not allowed to evict, and iOS Safari has a hard budget for canvas memory that fails silently when exceeded — blank tiles, no error, on exactly the browser the large-gallery report came from. A gallery is several hundred tiles and one lightbox image. What canvas buys on a thumbnail is a slightly harder right-click. What actually protects the images is server-side: the served file is watermarked and the download route refuses when downloads are off. The photographer who reported the large-gallery case, shipping to real clients, said the same and turned the global toggle off once it was about to reach their next gallery. So: tiles, hero and folder covers always render <img>. The lightbox keeps both the per-event toggle and the `maximum` implication — one image, where the calculus is different. The toggle is now wired to the lightbox for the first time; before this it reached only the tiles, so the label that said "canvas rendering" turned every grid into canvases and left the lightbox alone. Labels in all four locales now say where it applies. `protectionLevel` was destructured in seven tile components only to feed that OR; those props and their pass-throughs go with it. The shared layout props keep it, since the story layout still hands it to its lightbox. A source-level test pins that only PhotoLightbox passes useCanvasRendering to AuthenticatedImage or turns it on for `maximum`. Relates to issue 1287 |
||
|
Paul Nothaft
|
02b353e54f |
fix(usage): report restricted gallery downloads in v3 instead of an always-true signal
gallery_downloads.configured was true on every installation with a gallery. allow_downloads ships true — column default in migration 037 and the create route both set it — and the snapshot asked "at least one gallery has it on". The fleet value was ~100% by construction and could not separate a deliberate configuration from an untouched one. v2 consented to that key under that description, so v2 keeps sending it unchanged. v3 replaces it with gallery_downloads_restricted: at least one gallery has downloads switched off, which is the only state of that column anyone actually decides. Same catalog position, so the disclosed capability count stays at 86; the frontend copy, the EN/DE catalog strings, the coverage inventory and FEATURE_COVERAGE.md follow. Done in v3 rather than a v4 because v3 is on main and in no release yet, so nobody has consented to it. The collector carries the same catalog and has to take this change before the release that ships v3. One guard for the window in which :main / :beta images already carried the old v3 catalog. A report queued under it fails local validation on this build, and deliver() left a locally invalid report pending for good, blocking every operation behind it. A report's payload is derived state, so deliver() now rebuilds it from the current snapshot in place and sends that. Packet ID and sequence are kept — a re-signed retry has to reuse them so a lost acknowledgement does not duplicate data — and reports only: a stale registration, deletion or command is a genuine conflict and keeps the existing handling. Tests: the v3 snapshot counts a switched-off gallery and ignores enabled ones, v2 still reports the old key with the old meaning, and a stale queued report goes out rebuilt under the same packet id while a valid one is sent untouched. Relates to issue 1308 |
||
|
Paul Nothaft
|
c358bc65f7 | feat(usage): add consented beta capabilities and gallery photo totals | ||
|
Paul Nothaft
|
e40bc474bc |
fix(usage): let an operator clear a participation the collector never accepted
Probing the live collector to settle the delete-sequence question turned up something else: usage.picpeak.app answers a valid usage.v2 registration with INVALID_PACKET while the identical v1 flow is accepted. It does not speak v2 yet — which the deployment notes already require, but the consequence of getting that order wrong was worse than "reports do not send". Opting in to v2 against a v1-only collector left the installation stuck. Registration was refused, so nothing existed at the collector at all; the row sat in activation_pending, disable moved it to deletion_pending, retry was futile forever, and enable refused because the row was not `disabled`. The abandon hatch added earlier did not apply: it was gated on SIGNING_KEY_UNREADABLE. So the most harmless possible failure — nothing registered anywhere — was the one an operator could not clear. The gate is now the property that actually matters: a participation the collector has provably never accepted (sequence 0, no receipt) with a failing delivery can be discarded, from activation_pending as well as deletion_pending. Its receipt records `never-registered` rather than an unconfirmed deletion, because nothing remote exists to be unsure about. A participation the collector *did* accept keeps the old narrow gate and its explicit warning — clearing local state while the collector still holds reports must stay a deliberate, warned-about act. A collector that rejects a registration or a deletion outright now reports SCHEMA_NOT_ACCEPTED instead of DELIVERY_FAILED, and the settings page says the collector does not accept this report version yet. Retrying cannot fix that, and sending the operator to look for a network fault they do not have was wrong. Verified end to end against the live collector: v2 opt-in reports SCHEMA_NOT_ACCEPTED, the exit is offered immediately, the receipt says never-registered, and joining again on v1 registers, reports and withdraws with a collector-confirmed deletion. |
||
|
Paul Nothaft
|
c741dc22c5 |
docs(usage): state in the consent dialog that the connection only runs outwards
The dialog described what is sent and where it goes, but never said which way the connection runs. That is the part an operator is actually being asked to accept: opening an outbound path to someone else's service. PicPeak sends and never pulls. One place in the service reaches the network, it is a POST, and it requests exactly two paths — /api/envelopes, and /api/participant/lookup only when an operator asks for their own export. No scheduled job contacts the collector; the daily rollup is driven solely by an authenticated admin hitting /activity. There is no route the collector could call, and redirect: 'error' means it cannot even point a request somewhere else. From a reply only the acknowledgement for the packet just sent is read, with every field compared against that packet before it is accepted; the stored copy drops the session token and no read path hands it back to the UI. A requested export is streamed to the operator as a file and never interpreted. The consequence is why it belongs in the consent text and not only in the docs: this channel cannot deliver code, configuration or content into an installation, not even from a collector that has been taken over. It is a security property by design rather than by convention. usageOutboundOnly.test.js guards it by source inspection rather than behaviour, because a behavioural test only proves that today's calls behave. It fails the moment someone adds a second fetch, a poll for messages, a scheduled pull, or a public route touching the usage service — verified by injecting each of those. |
||
|
Paul Nothaft
|
1e8b6f1b0f |
fix(usage): close the QA findings on opt-in product usage
A QA exploration of this branch against an isolated rig — own stub collector, SQLite and PostgreSQL — turned up one dead end and a set of signals and controls that did not hold up. This closes all of them. Rotating JWT_SECRET, the documented response to a suspected compromise, made the signing key unreadable. That was already named and documented, but it left no way out: the delete packet can never be signed, so the row stays deletion_pending forever, and enable() refuses because it is not `disabled`. An operator who rotated precisely because the secret was compromised cannot restore it, so the feature was bricked with no control left. POST /usage/abandon is offered only in that state; it drops the local identity and records the receipt as `collector-unconfirmed` rather than claiming a deletion that did not happen. Every failed delivery was retried on the next admin request, and /activity is open to any authenticated admin while the settings ticker fires it every five minutes per open tab — 30 activity calls against a rejecting collector produced 30 outbound requests. Migration 206 adds attempts/next_attempt_at and the unattended sender honours the gate; Retry and opt-out still send immediately, and the tab names the time of the next automatic attempt. Feedback, votes and portal sessions now share an installation-wide budget of 30/hour. They are the only endpoints whose effect is outbound traffic carrying operator-written free text, and the general limiter skips authenticated requests by design. Reading status and withdrawing stay unthrottled. gallery_image_protection was true on a bare install with no galleries: PicPeak ships default_protection_level='standard' and enable_devtools_protection=true, so it reported fleet-wide 100% and could never separate a decision from an untouched default. It now reads only what deviates from the shipped defaults, and the devtools flag is not read at all — being on by default, its only informative state is off, which is the opposite of what the key claims. Also: - the export receipt counted every packet and called the total "usage reports"; reports and participant operations are now counted and named separately - GET /usage/preview no longer persists the custom_css marker, so the transparency view stops changing what will be sent - the feedback route requires every field the packet schema requires, so an API caller gets the missing field named instead of a bare INVALID_PACKET from inside signing - the German strings for this feature use "Sie" throughout, matching the rest of the admin UI; the ignore hint says what ignoring will do rather than stating it as already true - the consent dialog returns focus to the control that opened it - the long buttons wrap instead of running off a 390px viewport - a deletion receipt is labelled as belonging to an earlier participation while a new one is active Regression tests cover each of these, including the delete packet's reuse of the last accepted sequence, which was an unwritten assumption about the collector rather than a defect. |
||
|
Paul Nothaft
|
a7382591bf | feat: expand opt-in capability coverage with versioned consent | ||
|
Paul Nothaft
|
5d31b61c8d | Merge remote-tracking branch 'origin/main' into codex/1110-usage-coverage | ||
|
Paul Nothaft
|
e347f8f40f | fix(usage): minimize session receipts and clarify privacy controls | ||
|
Paul Nothaft
|
75ef137b7d |
fix(usage): drop the tinted block and stop the modal opening with a focus ring
Two things reported on the reformatted consent modal.
The green box is gone. Setting "what is never included" apart as a
tinted panel broke the rhythm of the sections and read as an arbitrary
highlight rather than emphasis. All six sections are uniform now; the
icon and heading are enough to tell them apart.
The green bars across the disclosure were a focus ring, not a border.
showModal() focuses the first focusable descendant, which since the
reformat was the scrollable region I had given tabIndex={0} — so its
inset ring was drawn for every user the moment the dialog opened, and
because the dialog clips its sides a full-width inset ring appears as
two coloured bars. Focus now goes to the dialog itself, which is also
the better screen-reader behaviour: the title is announced on open, and
the region's ring appears only when someone deliberately tabs to it. It
is a thinner, softer ring for that case. The dialog suppresses its own
ring, since that focus is programmatic rather than keyboard navigation.
The collector shown in the transport sentence was never wrong: it
interpolates the configured collector, and the screenshots showing
http://127.0.0.1:9 were taken on a rig deliberately pointed at a dead
loopback port so they could not reach production. Re-checked with
USAGE_COLLECTOR_URL unset: the sentence reads
https://usage.picpeak.app and both links resolve there.
Refs #1110
|
||
|
Paul Nothaft
|
a9e51d8fd7 |
fix(usage): reformat the consent modal so the disclosure can be read
It was seven anonymous paragraphs in one scrolling block, with the title and the buttons scrolling away with them. The scroll container is keyboard-focusable, and unstyled it drew a default focus ring, so the disclosure also looked like a giant textarea. Now: a fixed header carrying the icon, title and purpose; a scroll region with six labelled sections, each with a small heading and icon so the thing can be scanned rather than only read; and a fixed footer with the consent checkbox and the actions, which no longer scroll out of reach on a short screen. "What is never included" is set apart as a tinted block, since it is the part that answers the question an operator actually has. The focus ring is now a deliberate inset ring on a labelled region, which is correct for keyboard use instead of an accident that looked like a form field. Dark mode is fixed as part of this, and it was my own doing: the dialog used `bg-theme-surface`, which does not follow dark mode, and the section text I added carries dark: variants. Light surface plus near-white text is unreadable. The surface is class-driven now — neutral-800, which is what `.card` resolves to in dark and what the rest of the admin UI uses. Verified in both themes through the app's own theme toggle rather than by forcing the class, which is what produced the misleading half-state the first time I looked. Six section headings added in EN and DE. Refs #1110 |
||
|
Paul Nothaft
|
bb76ca5375 |
fix(usage): keep the settings tab usable on a bad collector URL, and report layouts and CSS accurately
Three items, one of which explains an error seen in the app. "The operation could not be completed" could come from a config typo. status() called collectorUrl() bare, and that throws on a bare hostname, a path, a query, or http in production. The settings page renders one generic failure when its status query errors, so a misconfigured USAGE_COLLECTOR_URL replaced the whole tab with that sentence — no cause, and no way to read the status or withdraw, because every control there sits behind that call. The URL is now reported as collector_error: 'INVALID_COLLECTOR_URL' beside the real state, the tab says what is wrong and how to fix it, and the links are only rendered when there is somewhere to point them. gallery_layouts reported grid for every preset-themed install. color_theme holds either a theme object or the NAME of a preset — the theme picker stores names, and eventTypeService seeds them (`theme_preset: 'corporateTimeline'`). Only reading value.galleryLayout made masonry, timeline, mosaic and the two gallery presets invisible. Names now resolve, and an event with no theme of its own resolves through the global one instead of being counted as grid. Only the name -> layout mapping is duplicated, not the presets; frontend/src/types/theme.types.ts stays the source of truth, and an unknown name reports `other` so a preset added later degrades to "something else" rather than quietly inflating the grid count. custom_css missed CSS applied through a template. An enabled css_templates row applied via events.css_template_id is gallery styling by the same definition as the settings fields — the Custom CSS tab is where both are authored — but neither the snapshot nor the middleware saw it, so those installs reported custom_css entirely false. Existence only; template contents are never read. Eleven tests. Reverting each fix in turn fails 3, 1 and 3 of them. Refs #1110 |
||
|
Paul Nothaft
|
22da018e1b |
fix(usage): close the remaining withdrawal races, reset per-item name consent
Follow-up review on the previous commit, including a hole in that commit's own fix. The cancellation flag became a counter. Clearing a boolean needed a write of its own, and a /disable landing between the lease and that write was erased — the same race one level down. enable() now records the counter it started with and claims only if it is unchanged, so no clearing write exists to lose. It also fixes the case a boolean could not express at all: a stale cancellation already set, and a fresh one arriving mid-activation, are indistinguishable as flags and obvious as counts. Migration 203, separate from 202 for the reason 202 was separate from 201 — knex will not re-run an applied migration. deliver() re-checks immediately before dispatch. The existing check ran before the binding lookup, which is asynchronous, so a withdrawal that COMPLETED during it still had its registration or report sent afterwards. Not an already-in-flight request — a new one started after the operator had withdrawn. The outbox writes in tick() and command() are conditional on still being active. /disable clears pending_packet without holding the lease, so an unconditional write put a report — or a feedback body and name — back into an outbox the withdrawal had just emptied, where deliver() would then leave it, since it declines to send anything but the delete. Per-item name consent resets with the item. `named` stayed checked after submitting, so the next item carried the previous name automatically, contradicting the anonymous-by-default promise the disclosure makes for each item. The remembered name stays in preferences; attaching it is decided again each time. Two of these tests were worthless when first written and are noted because the pattern keeps recurring: the pre-dispatch case passed without the guard because an empty report payload failed schema validation during signing, so nothing reached the collector for reasons unrelated to the check. With a valid payload it fails without the guard and passes with it. Same for the counter: dropping it from the claim fails two. Refs #1110 |
||
|
Paul Nothaft
|
83fbb63e13 |
fix(usage): protect a pending withdrawal, widen the backup signal, explain an unreadable key
Three of four findings from the follow-up review.
A withdrawal is no longer clobbered by the instance-copy check. That
update was unconditional, so an opt-out arriving while the binding
lookup was in flight was replaced by identity_conflict — and tick()
stops there, so the deletion the operator asked for was never sent. It
now carries the same whereNot('deletion_pending') guard the
collector-conflict handler beside it already had.
Scheduled database backups count as a configured backup. The middleware
records /backup/* and /database-backup/* under one capability, but
`configured` read only backup_enabled, so an install whose only backup
is the scheduled database one reported used: true, configured: false —
a contradiction in the dataset this feature exists to produce.
SIGNING_KEY_UNREADABLE gets its own message. Naming the error in the
previous commit was half the job: the settings page still showed the
generic retry/disable advice, and neither action can succeed without the
original encryption material. It now says what happened and what is
actually required, in EN and DE.
NOT fixed, and reported instead: /disable overlapping an in-flight
/enable. While activation is still doing its slow work the row still
reads `disabled`, so disable's conditional update matches nothing and
the lease conflict from its tick() is swallowed — the operator is told
participation is off while activation completes and leaves it on.
Closing it properly needs a persisted cancellation flag that enable
checks before finalising: taking the lease cannot help, since it either
conflicts immediately or would block the request for the 60s lease. That
is a schema and state-machine decision for the author, not something to
restructure underneath them.
Refs #1110
|
||
|
Paul Nothaft
|
c043897b0e |
fix(usage): name the unreadable-key failure, unpin the collector default, align the tab
Review follow-ups on #1304. SIGNING_KEY_UNREADABLE. USAGE_ENCRYPTION_KEY defaults to JWT_SECRET, so rotating JWT_SECRET — the correct response to a suspected compromise — makes the stored Ed25519 key undecryptable. That surfaced as a generic DELIVERY_FAILED which retried forever, and it silently blocks the DELETE packet too: an operator who withdraws has their local state cleared while the collector keeps its copy. decrypt() now tags its own failure and deliver() reports it under its own name, without flagging an identity conflict — an unreadable key is not evidence of a clone. The docs already warned that losing the key breaks deletion signing; they now name the trigger and the error. The collector default is no longer an inline string in the constructor. It is a declared DEFAULT_COLLECTOR_URL, since it is a deployment choice: self-hosters point USAGE_COLLECTOR_URL at their own collector and the UI already derives every link from whatever is configured. schema.cjs is deliberately untouched — it is vendored byte-identical with picpeak-usage, and its $id is a schema identity, not a delivery address. Links in the consent dialog. It named the collector inside prose but never linked it, so an operator deciding whether to opt in could not open the destination or the public schema without retyping a URL. Both are links now, built from the configured collector. UI standards. The tab hand-rolled its surfaces as `<section className="rounded-xl border border-theme …">` and imported Button from a deep path; every other settings tab uses `<Card padding="md">` from the components/common barrel. Converted, with the feedback <form> wrapped rather than replaced so its semantics survive, and headings given the same colour tokens as ImageSecurityTab. The barrel pulls ErrorBoundary -> i18n/config, so the tab's test needed the initReactI18next shim the FaceRecognitionCard test already uses. Not changed: the delete packet reusing the current sequence. The collector handles delete before any sequence check — "possession proof is sufficient for deletion, including when a restored backup has a stale sequence" (picpeak-usage server/collector.js) — so deletion is deliberately sequence-exempt and the client is correct as written. Refs #1110 |
||
|
Paul Nothaft
|
b53e5d97b4 | feat: add opt-in product usage and feedback integration (#1110) | ||
|
Paul Nothaft
|
967224c030 |
fix: remove the image-fragmentation surface
Closes #1300. Fragmentation was configurable, stored per event, served to the gallery client, and consumed by nothing. It was not unbuilt scaffolding — both halves exist and are individually coherent — but they were never connected, and they disagree: the server cut a fixed 3x3 grid while the client reassembled a 4x4 one, so wiring them together as they stood would have produced scrambled images rather than protection. Removed rather than finished, because finishing it buys nothing. The client fetches the whole image and then redraws it in pieces on a canvas, so the full original has already crossed the wire before any "protection" is applied — that is obfuscation, not a control. The per-fragment canvas work also lands on mobile, which is the memory profile under investigation in #1287. Goes: secureImageService.fragmentImageBuffer and its branch, the ?fragment=N delivery path and handleFragmentedImage in secureImages, the fragmented-JSON response in protectedImages, fragmentation_level in the gallery payload, the default_fragmentation_level setting, the PUT validator, the ProtectedImage fragment renderer, and the operator control with its strings in all eight locales. No migration. `events.fragmentation_level` and the app_settings row stay — dropping a column is irreversible and the stored values are harmless once nothing reads them. If they should go, that is a deliberate data decision and its own migration. `fragmentGrid` on AuthenticatedImage and the layouts is deliberately untouched: #1299 already removes it as part of the inert prop surface, and doing it here would only collide. |
||
|
Paul Nothaft
|
fc595409b4 |
feat(crm): newsletter campaigns behind a newsletters flag (#1264)
Part B of #1264. Flag off by default, so an install that never enables it gains no route, no nav entry and no way to mass-mail. A campaign is a body plus a recipient rule. Queueing one writes ordinary email_queue rows (email_type 'newsletter', origin 'campaign', new campaign_id), so retry, rendered_html, sent_at and error_message all come from the existing processor rather than a parallel sender. Throttling staggers scheduled_at; the processor loop is untouched. Two rules the service enforces: no raw HTML is ever stored (sanitized on write and again on render, idempotently), and opt-out is checked at queue time AND again at send time. Migration 199 adds email_campaigns, email_campaign_recipients, email_queue.campaign_id, customer_accounts.marketing_opt_out(_at), and the newsletters.view / newsletters.send permissions. Three rounds of external review are folded in, including several that would otherwise have shipped broken: - Campaign rows never came due on SQLite. queueEmail writes a Date, which the sqlite3 binding stores as epoch ms; ISO text in the same column compares as TEXT against an INTEGER, and SQLite orders every INTEGER below every TEXT. The feature silently sent nothing there. - The flag had no Settings card and no sidebar entry, so it could not be enabled through the UI at all. - Consent is per ADDRESS, not per row: two accounts sharing an inbox meant unsubscribing stopped one and not the other, at both queue and send time. - The unsubscribe GET mutated consent, so a mail-security scanner walking a campaign could have unsubscribed much of the list. GET now confirms, POST acts. - The rate ceiling is clamped to the queue's real throughput (10/min), so the composer's estimate stops being wrong by up to 12x. Closes #1264 |
||
|
Paul Nothaft
|
2d403f7fb2 |
fix(email): wire the settings status card, and cap-aware truncation
Codex review round 4 on #1273. Settings → Status rendered a green check for the email processor unconditionally, against an API field that was itself the literal 'active'. Both ends were lying and only one of them got fixed: adminSystem started reporting the real state in an earlier commit, but StatusTab never read it, so the second place an admin looks to find out why mail is not arriving still said everything was fine. It now shows stopped and degraded, with the reason. The truncation flag missed the case it most needed to cover. The loop broke on the 200-row report cap before the flag could be set, so 201+ overdue rows came back as exactly 200 with scanTruncated false -- a partial report presented as complete. It is now set whenever rows were left unexamined. The grace-window comment claimed the processor clears ~6000 rows inside the window. It clears on the order of 100: ten rows a pass, one pass a minute. The comment now says so, and says why the processor's own state is reported above the list rather than inferred from it -- "running, last pass sent 10" next to a backlog reads very differently from "not running" next to the same backlog. One round-4 finding is NOT fixed, deliberately, and is written up at the retry route. Clearing scheduled_at leaves created_at at the original enqueue time, so a retried old row appears in the waiting list immediately, looking overdue, until the processor sends it. Restarting that clock needs a timestamp written there and no shape works: a Date matches how queueEmail writes the column and how processEmailQueue compares it, but jest's sandbox Dates store as "[object Object]" (CLAUDE.md) so it cannot be tested; an ISO string tests fine but stores as TEXT, which SQLite then orders above the numeric bound in the processor's own pickup query, leaving the row unsendable. A requeued_at column would settle it. Cosmetic either way, and not worth risking a stuck row. 1 more test, failing before this commit. |
||
|
Paul Nothaft
|
34685505be |
fix(analytics): serve self-hosted trackers same-origin so CSP stops blocking
A self-hosted Umami/Rybbit domain configured in Settings could never load: the CSP script-src allowlist is static, and the earlier pass could only add an admin-visible warning because nginx.conf:58 strips helmet's header and location / serves the SPA document off disk via try_files -- so helmet can never govern it in Docker. Verified by reading the config, not inferred; that kills the "make helmet dynamic" option outright. Rather than templating the CSP, the tracker is now same-origin. The script and every endpoint it talks to are served from /api/analytics/tracker/* and proxied server-side to the configured instance, so script-src 'self' and connect-src 'self' already cover it. The CSP is unchanged: nothing to template, no env var, no restart -- it takes effect when Settings is saved. That also closes A3 structurally rather than by widening a directive. Endpoint mapping taken from vendor sources, not guessed: Umami's host || currentScript.src + /api/send, and Rybbit's documented /track, /site/tracking-config/<id>, /site/<id>/feature-flags/evaluate. data-host-url is set explicitly so a COLLECT_API_HOST-built Umami cannot bypass the proxy. Session replay is deliberately NOT proxied: replaying gallery pages would capture the share token (GHSA-7m6c). nginx still needed one line, for a non-obvious reason: the static-asset regex location outranks the plain /api prefix in nginx's matching order, so /api/analytics/tracker/script.js resolved as a static file. Confirmed empirically against a real nginx:alpine -- 404 before the ^~ block, 502 (proxied) after, with /assets/app.js and /api/public/settings unchanged. The native SERVE_FRONTEND install needed no change; helmet already has 'self' in both directives and the proxy mounts ahead of express.static. Security boundary, since this makes the server fetch an admin-supplied URL: closed per-provider path+method allowlist (4 paths), DNS-resolving isHostAllowed blocking private/internal/metadata addresses in production (matching the s3Storage prod-only precedent), base rebuilt as origin + pathname so userinfo/query/fragment cannot smuggle anything, redirect: 'error', cookie/authorization/referer/host never forwarded, an HTML upstream response re-served as application/octet-stream + nosniff, and 64KB request / 2MB response / 5s timeout / 120rpm caps. X-Forwarded-For and User-Agent are forwarded so geo and device attribution survive. Residual, stated plainly: an unauthenticated rate-limited relay to one admin-chosen public host on 4 paths, and TOCTOU DNS rebinding is unmitigated as it is elsewhere in the repo. The Umami and Rybbit panels now explain they are proxied; the Custom panel keeps a CSP warning -- it is the one mode with nothing to proxy -- naming both script-src and connect-src. Refs testplan REPORT.md A2, A3. |
||
|
Paul Nothaft
|
3acb452090 |
fix(settings): remove the duplicated section heading on 11 tabs
A generic shell heading stacked on top of each tab component's own internal
heading. The report named five tabs "at least"; auditing all 28 found 11:
downloads, sso, apiTokens, webhooks, businessProfile, crm, accounting,
whatsapp, slideshow, moderation, styling. On the first eight the two headings
resolve to the identical string -- sso and businessProfile literally render
the same key twice. The other three were near-identical stacked titles
("Moderation"/"Word Filters", "Custom CSS"/"Custom CSS Templates",
"CRM behaviour"/"CRM settings").
Clean, and left alone: general, events, categories, thumbnails, security, seo,
imageSecurity, status, analytics (its first heading is a genuine sub-section),
plus the eight already in TABS_WITH_OWN_HEADER.
Removed the component side and kept the shell heading: the shell heading is
the consistent one (icon + label + divider on ~20 tabs) and always matches the
nav item the admin clicked, and none of these components are mounted outside
SettingsPage, so nothing loses a title. Subtitles and intro copy preserved
throughout; orphaned icon imports removed.
The guard test was checked against the pre-fix blobs and does fail on them.
Refs testplan REPORT.md, "duplicate H2 section heading" warning.
|
||
|
Paul Nothaft
|
7c9baff751 |
fix(upload): scope category ids, stop temp-file leaks, split the video cap
Four related fixes on the admin upload/photo path. B5 -- PATCH /photos/:photoId and POST /photos/bulk-update took any parseInt(...) > 0 straight into the update with no existence or scope check, so a photo could be moved into another event's category. The upload route already validated `event_id = X OR is_global` per #500/#525; extracted that query as findScopedCategory() and used it on all three routes so the 400 body is byte-identical. 0/negative/'individual'/'collage'/null still clear without a lookup, so the clear path costs no extra query. B9 -- three distinct temp-file leaks, not one. The validator's size branch never unlinked; the cleanup lived in the final handler, unreachable on any 400; and multer's `destination` callback runs per file and overwrote req.tempUploadPath, so even the success path only ever removed the last file's directory. Now: discardUploadedFiles() runs on every 4xx and the 500 (ENOENT tolerated, and files are only dropped when the whole request is being rejected, so the passing path is untouched); cleanup registered before multer so it also covers multer's own LIMIT_FILE_SIZE return; one directory per request. B8 -- the admin uploader filtered on MIME only, so an oversized file was uploaded in full before the server's 400. Mirrors UserPhotoUpload's existing per-file toast-and-drop. C4 -- general_max_file_size_mb was a single cap for photos and videos, so the 50MB default meant admins could not upload ordinary video without also raising the photo limit. Adds general_max_video_size_mb (default 500MB, clamped by the same 10GB MAX_ALLOWED_FILE_SIZE_MB ceiling, read per request, 60s cache), editable in Settings -> General. Photo uploads are protected from regressing by keeping multer's type-blind limit at max(photoCap, videoCap) and moving the per-kind decision into validateUploadContent, where file.mimetype exists. It 400s with the existing message shape, so an oversized photo is still rejected with the identical body it produced when multer did the rejecting. Known gap: chunked-upload/init still applies the photo cap to video. Making it video-aware would change an existing assertion that pins a 200MB video init being rejected under a 1MB general cap. No component calls that path today and the direction is strict rather than a bypass, so it is left as-is. Guest video uploads still share the single cap in gallery.js. Refs testplan REPORT.md B5, B8, B9, C4. |
||
|
Paul Nothaft
|
d8bd0cd449 |
i18n: close the admin translation coverage gaps
Recurring pattern of components and strings shipped without translation coverage, found across unrelated feature areas. +212 keys each to en.json and de.json, provably additive (flattened-key diff: removed=0, changed=0; formatting round-trips byte-identically). Genuinely un-wired components (grep -c useTranslation == 0), now wired: BulkArchiveModal (8 strings, count-pluralised), WebhookDeliveriesPage (27), CMSEditor's TipTap toolbar/link dialog/status bar/help modal (64). Hardcoded strings fixed in code: ImageSecurityTab's 4 spinbutton hints, ProjectsListPage's unlocalized status enum. Keys-only (component already calls t() correctly): General "Time format", Branding Social Media + Promotional Banner, Quotes detail/editor, cms.showInFooter. Two corrections to the report's attribution: - BlockLibraryPage was NOT un-wired -- it calls t() on every string with English defaults; all 32 contracts.blocks.* keys were simply absent from both locale files, so everything fell back to the JSX default. Same for ContractsListPage, where the report cited 3 missing keys and there are actually 9 (all 5 table column headers plus the pagination line). - CustomerDetailPage has full t() coverage; its single English "Contracts" was a missing customer.nav.contracts key behind a dynamic labelKey. Locale convention followed: i18next.config.ts manages en/de/nl/pt/ru/fr, but only en and de are kept at parity (5198/5200 keys); the rest are ~50% partial and rely on fallbackLng 'en'. Added to en + de only rather than inventing 212x6 unreviewable translations. Also added the 25 missing businessProfile.* keys (PDF-letterhead section, bank-accounts QR disclaimer). That component already calls t(), so those strings localize as soon as the keys exist; no wiring needed. Refs testplan REPORT.md #15a. |
||
|
Paul Nothaft
|
c6cb01865e |
fix(settings): derive the sidebar preview from the real sidebar declaration
SidebarPreview kept its own hand-maintained 6-item array with only two gates wired (analytics, userManagement), so toggling e.g. Workflows changed nothing in the preview even though it does add a real sidebar entry once saved. Export AdminSidebar's `navigation` as `adminNavigation` (2 lines) and derive the preview from it, so every gate -- transfers, messaging, analytics, userManagement, clients incl. its featureFlagsAny set, accounting, workflows -- is covered and the two can't drift again. Note the report's item list was partly wrong: Quotes, Contracts, Invoices, Hours, Projects, Calendar and the CRM dev tools have no top-level sidebar entries at all -- they are sub-nav inside /admin/clients and surface in the preview through the CRM entry's featureFlagsAny. Permission filtering is deliberately not applied (unchanged): the preview answers "what do these flags do to the sidebar", not "what can this admin see". Refs testplan REPORT.md #20 (Part 3, J.14). |
||
|
Paul Nothaft
|
3489610cb8 |
fix(analytics): warn about the CSP allowlist on every tracker provider
A self-hosted Umami/Rybbit domain configured in Settings -> Analytics is always blocked by the static script-src allowlist, silently, with only a console error. The amber CSP warning that explains this already existed but was rendered only inside the "custom" provider panel -- not on the two providers where an admin actually types a self-hosted URL. Extract it to a local CspWarning and render it in the Umami and Rybbit panels too. Both translation keys already exist in en.json/de.json. Interpretation: the dynamic-CSP option was investigated and rejected as not reachable for the header that actually governs these documents. In the Docker deployment nginx.conf:58 does `proxy_hide_header Content-Security-Policy`, so helmet's CSP and the res.setHeader CSP at server.js:445 are stripped before they leave the stack -- nginx's static server-level CSP is the only one the browser sees for the SPA documents the tracker is injected into. nginx.conf is COPYied verbatim by the Dockerfile (only index.html goes through envsubst), and the tracker URL lives in the DB rather than the environment, so making it reflect the setting would need start-time templating plus a DB read. The CSP itself therefore still has to be edited by hand; the warning now says so where the admin can see it. Refs testplan REPORT.md #18 (Part 3, B.02). |
||
|
Paul Nothaft
|
cec8eff70c |
fix(images): fence the capture-date backfill on the file it read (#1201) (#1204)
The capture-date backfill committed its result keyed on the row id alone. It snapshots every candidate up front, then walks them one at a time reading originals off S3 or a NAS mount — a pass that can run for many minutes. replacePhoto, reachable from the replace_by_name upload path, swaps a NEW file under an existing row and rewrites path/filename. A replacement landing inside that window carries no date of its own, so captured_at was still NULL, the whereNull guard passed, and the previous file's EXIF date was written onto the new photo. Silent: nothing errored, the run reported it as a success, and the gallery just sorted that photo to the wrong place. Fenced on path and filename as well as the id — the same fence #1199 put on the orientation backfill for the same reason — so a replaced row matches zero rows and is skipped. The candidate query already selects both columns, so no query change. Knex renders a null value in the object form as `is null` on both the pg and sqlite3 clients, so a row with a NULL path still matches itself. Those skipped candidates are now counted rather than dropped. replacePhoto is not the only writer of path/filename — eventRenameService rewrites both on an event rename, which is not a content change — and another writer filling captured_at first lands in the same place. Without a counter they fell out of the run's arithmetic entirely: success + noExif + failed no longer added up to the count the operator was shown when they started the job, on the card as well as in the log. The card shows the count only when it is non-zero, the same shape the orientation job uses for staleTiers. The wording states what is known — changed by something else, not updated — rather than promising a retry: for the already-dated case there is nothing to retry, and the Missing Capture Date figure above is what says whether work is left. Locale coverage matches the staleTiers key (en, de, fr, sl), with the defaultValue carrying the rest. Regression test: a replacement landing mid-run leaves captured_at NULL and is not counted as updated. Verified to fail against the unfenced code. |
||
|
Paul NothaftandPaul Nothaft
|
edef4d7365 |
fix(images): backfill orientation for libraries that predate the fix (#1199)
* fix(images): backfill orientation for libraries that predate the fix (#1198) #1194 corrected the generators and every ingest path, but did nothing for photos already in the database. Those rows end up worse than untouched ones: before the fix a rotated photo was CONSISTENTLY wrong — a sideways image in a tile shaped to match — and afterwards the regenerated thumbnail is correct while photos.width/height still describe the raw sensor order, so masonry and justified size a portrait photo with a landscape ratio. The dimension repair cannot reach them: it only selects rows with a NULL dimension, and an affected row has both, just transposed. Its own job rather than a mode of that one. They look alike but are not the same operation: the repair FILLS missing values and touches nothing else, while this RECOMPUTES and invalidates the derived data generated against the old orientation. Sharing a lease would also mean one blocks the other. A first attempt at this was reverted from #1194 after review found five problems. All five are addressed here: - Originals are read through resolvePhotoStorageKey + withLocalCopy + withProcessableImage, so the job works on S3 installs and on RAW/DNG. The dimension repair's direct fs read does neither, which stops being an edge case in a job that walks the whole library. - The canonical preview is cleared BEFORE faces are requeued. ensurePreviewImage returns a cached preview whenever it is still a valid image, and a pre-fix unrotated one is perfectly valid — so requeueing alone made the rescan read unrotated pixels and scale those boxes by the corrected dimensions, which is worse than leaving the data alone. - Invalidation keys off the EXIF transform, not a dimension delta. Orientations 2, 3 and 4 move every pixel while leaving width and height unchanged, as does 5-8 on a square image; a delta check skips exactly those rows. - Archived events are excluded — archiving deletes the originals and keeps the rows, so every one of them would fail its read. - The dimension write and the invalidation share a transaction. Split, a failure between them leaves stale face data that no retry can fix, because the retry computes "already correct". Tier deletion stays outside the transaction on purpose: it touches storage, and a failed object delete must not roll back a correct database write. A leftover tier regenerates on next read; a rolled-back write is silent corruption. * fix(images): invalidate every stale rendition, fence the writes, and give the job a button (#1198) Three things from review, one of which mattered a lot. The invalidation was too narrow. Clearing only preview_path fixed the face data and left the gallery worse off: ensureThumbnail and ensureHeroImage return their cached file whenever it is merely VALID, and a pre-fix sideways thumbnail is perfectly valid — so a corrected row rendered the old sideways image inside a newly-corrected portrait tile. All three canonical renditions are cleared now, their stored objects deleted, and both responsive tier sets with them. The responsive tiers also needed handling rather than a hopeful catch. Their helpers swallow delete errors, and ensurePreviewImageAtWidth treats storage.stat(key) as a cache hit — so a tier that survived deletion keeps serving unrotated forever and never regenerates. The keys are re-checked after deletion and survivors are counted into the result, so a run that could not clear them does not report itself as clean. Writes are fenced on the identity that was measured, not just the id. replacePhoto swaps a new file under an existing row and rewrites path/filename, and it IS reachable — from the replace_by_name upload path in adminPhotos.js. A replacement landing while this job read the old original would otherwise have had the previous file's dimensions written over it and its fresh renditions cleared. And the job had no way to start it: the endpoint existed with no caller, so an upgrade would have left every affected library untouched unless an operator found the API themselves. It gets a Status card like its two neighbours, with strings in en/de/fr/sl. No backlog counter, because unlike the other two it cannot know how many rows need it without doing the work. * fix(images): make the backfill idempotent, and stop it lying about what it did (#1198) Six things from review round 2. The job was not idempotent, and the way it failed was expensive. Its trigger is the EXIF tag on the ORIGINAL, which correcting a photo never changes — so every re-run threw away the renditions it had just regenerated and requeued every completed face scan. On a face-enabled install, running it twice meant re-detecting the whole library for nothing. Migration 191 adds photos.orientation_checked_at, written in the same transaction as the work it records, with `force` as the escape hatch for an interrupted run. The candidate query selected preview_path but not thumbnail_path or hero_path, which the deletion loop reads — so those two pointers were cleared in the database while the objects stayed in storage, still reachable through previously issued URLs. watermark_path was missed entirely. gallery.js serves it ahead of the original when branding watermarking is on, which makes it the most visible rendition of the lot. (Its generator needed rotating too — that went into #1185, where the other three live.) storage.stat() RESOLVES with null for a missing key rather than rejecting, so counting "the promise settled" marked every deleted — and every never-created — tier as a survivor. A perfectly clean run told the operator to re-run. Now a null means gone, and a rejection counts as stuck, since a storage error is not proof the object went away. Face data is invalidated whenever the stored dimensions change, not only when the change came from rotation: boxes are scaled by photo.width at read time, so any dimension change strands them. And `corrected` now comes from the affected-row count. If the fence rejected the write because the file was replaced mid-run, the photo was not corrected and the run must not claim it was. * fix(images): stop the backfill doing unnecessary work, and make its retry advice true (#1198) Round 3, four points, all narrower than the last two rounds. It re-processed photos that were already correct. A 5-8 rotation changes the dimensions, so a tagged photo whose stored dimensions are ALREADY oriented must have been ingested after #1185 — its renditions are fine and clearing them deletes valid files and rescans a completed face detection for nothing. Those are now skipped and simply marked. Orientations 2, 3 and 4 (and 5-8 on a square image) leave the dimensions identical either way, so they carry no such evidence and are still done once. The retry advice was impossible to follow. When a responsive tier could not be deleted the row was still marked, so the ordinary re-run the UI recommends found nothing and the stale tier kept serving unrotated forever. The marker is withheld when a tier survives, which is what makes that message honest. Storage cleanup now only runs when a fenced write actually landed. If the file was replaced mid-run every update matched zero rows, but the deletion went ahead anyway and could destroy renditions belonging to the REPLACEMENT — watermarks especially, which are keyed by photo id and alias straight onto the new file. And the full-photo ETag includes the backfill's timestamp. It was built from the ORIGINAL's mtime plus the watermark settings hash, neither of which this job touches — so a guest holding a pre-fix ETag would go on getting 304 and their cached sideways image no matter how many times the backfill succeeded. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
3991dc3ccb |
fix(admin): gate the dimension repair as system maintenance (#1182)
* fix(admin): gate the dimension repair as system maintenance (#1181) The endpoint's candidate query is unscoped, so it walks every event in the install, reads every original off S3 or the NAS mount, and rewrites their metadata. It required only photos.edit, which the built-in team_photographer preset holds (175_granular_permissions_and_presets.js:106) — a role that exists for a contributing second shooter, not for someone who should be able to start a whole-library scan or touch another owner's events. Now system.manage, whose own description is "run system maintenance actions", with the status endpoint on system.view to match. Nobody who should have it loses it: super_admin is granted every permission, solo_photographer is 'ALL', and migration 175 already projects every settings.edit holder forward onto system.manage on upgrade. The capture-date sweep next to it was gated this way in #1179; this brings its older twin in line. * fix(admin): gate the dimension status card on the permission the button needs (#1181) Same mismatch as the capture-date card: system.view and system.manage are independent grants and StatusTab renders its card and enabled button purely on a successful status payload (StatusTab.tsx:558), so a system.view-only role got a live Repair button whose every click 403s. * fix(admin): stop the dimension status card polling a 403 (#1181) With the endpoint correctly requiring system.manage, anyone who can open the Status tab but lacks it would have had a 403 and a logged denial every ten seconds for a panel they were never shown. The query is now gated on the same permission the endpoint requires, so it never starts. * fix(admin): gate the dimension card's render on the permission too (#1181) TanStack keeps the cached status after `enabled` flips false, so checking only the payload would still show the card — and an enabled Repair button whose POST 403s — to a lower-privileged admin logging in behind a system.manage user inside the cache lifetime. * fix(admin): name the dimension-card permission flag for the card it gates (#1181) #1179 adds a second system.manage-gated card to this same component with the same flag name. Two identical declarations merge WITHOUT a conflict and then fail to compile — TS2451, cannot redeclare block-scoped variable — and since each PR is green on its own, nothing catches it until main's build breaks. Verified by trial-merging both into main: no conflict, two declarations, tsc fails on both lines. Naming this one for the card it gates removes the trap; once both have landed the two flags can collapse into one. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
849a5807b7 |
fix(admin): make "Storage used" report storage used (#1164) (#1170)
* fix(admin): make "Storage used" report storage used (#1164) The tile summed photos.size_bytes — the catalogued size of the ORIGINALS, which has no relationship to the disk PicPeak runs on. In reference mode those files are never copied and sit on the NAS; duplicate rows counted the same file twice (#1162); and it ignored everything PicPeak genuinely does write locally: thumbnails, previews, hero renditions, watermarks and the per-event download cache. The reporter's tile read ~80 GB against 21 GB of real usage. Worse than the label: the same number drove the storage soft-limit warning bar and, via /storage/info, the recommended soft limit — so a reference-mode install got a disk-capacity recommendation computed from bytes that are not on the disk. - new localStorageUsage service walks the storage root and reports the total plus a breakdown. Walking rather than summing DB columns is the point: thumbnail/preview/hero rows record a key and never a byte count, and orphans from a deleted event or an interrupted import are real bytes. Symlinks are not followed, so a link into the media mount cannot put the NAS back in the total. Cached for 5 minutes, since the dashboard polls. - the dashboard tile and /storage/info now report that, with the catalogued figure kept and labelled as such next to it. A failed measurement reads as "unavailable" rather than substituting a number that means something else. On the local rig: 64.37 MB used against 15.75 MB catalogued, of which 27.9 MB is watermarks and 6.8 MB is download cache — none of which the old figure could see. Not addressed here: `.download-cache/all.zip` still has no TTL or size cap. It is now at least visible in the breakdown, which is what makes the case for capping it. * fix(admin): exclude the media share from local storage usage (#1164) External review found the walk could reintroduce the exact over-count it replaces. EXTERNAL_MEDIA_ROOT's compose default is `<storage>/external-media`, where the NAS is bind-mounted. That is a plain directory, not a symlink, so the symlink guard did not cover it and the walk descended into the share — putting every referenced original back into a figure whose whole purpose is to leave them out, and comparing NAS bytes against statfs() of the local disk. On the reference-mode installs this issue is about, that is the failure mode reappearing inside its own fix. The configured root is now skipped when it lies inside the storage root, and the result reports which path was excluded. A directory that merely shares the name is still counted, because those really are local bytes. Also from the review: - concurrent cold-cache callers now share one walk. /dashboard/stats, /storage/info and the sidebar are routinely requested together, and each was starting its own stat-per-file traversal of the whole library. - storage_partial is surfaced in the StorageInfo type and the sidebar tile, not just the dashboard and analytics cards. An unreadable subtree makes the total a floor, and a floor silently compared against a soft limit reads as "safely under". * fix(admin): do not report a disk walk on an S3 backend (#1164) Second review round. S3 installs were regressed. With STORAGE_BACKEND=s3 the originals, renditions, archives and download caches are objects in the bucket and STORAGE_PATH holds only incidental local files — so the walk reported near-zero and the soft-limit recommendation was derived from it. Those installs now keep the catalogued figure, which is the approximation they had before this PR, and the response says which measurement it is (`storage_measurement: 'disk' | 'catalog'`) so the UI labels it instead of implying a disk measurement that never happened. The Settings → Status storage card ignored storage_partial, formatting a lower bound as exact and deriving the limit percentage from it — so an unreadable subtree could read as safely under the limit. It now carries the same `+` marker as the sidebar and dashboard. * fix(admin): stop rendering an absent measurement as zero usage (#1164) Third review round, two findings. The analytics storage bar coerced a null measurement to 0, drawing an empty bar labelled "0% of limit" and suppressing the over-limit state — reading as plenty of room at exactly the moment nothing is known. It now shows the catalogued figure on S3, where that IS the available answer, and says "no measurement available" rather than inventing a percentage when there is none. /storage/info walked the filesystem before checking the backend and then threw the result away on S3. The sidebar polls that endpoint, so a migrated install still holding a large local tree paid a full stat-per-file traversal on every cold cache for nothing. Gated before the walk, as the dashboard route already was. * fix(admin): tell "no disk to measure" apart from "the measurement failed" (#1164) External review of the stable twin. Both were reported as `storage_measurement: 'catalog'`, so a failed local walk made the dashboard claim the objects live in S3. They are different things — one is a fact about the install, the other is a fault — and there is now an `unavailable` state for the second. The analytics percentage could reach the billions. `safeSoftLimit` fell back to `storageUsed || 1`, and on S3 that is null → 1, while the figure beside it came from `catalogedBytes`. An editor or viewer holds `analytics.view` but not `settings.view`, so `/storage/info` 403s for them and `storageInfo` is undefined — which is exactly when that fallback fires. It now falls back to the measured figure, and suppresses the percentage entirely when there is no real limit rather than dividing usage by itself and always reading 100%. Also lands the AnalyticsPage half of the previous round, which the commit message claimed but the commit did not contain — only its backend counterpart was staged. The stable twin has carried it since it was written, so this is the parity gap in the unusual direction. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
77953c15c1 |
fix(gallery): stop the lightbox loading originals to display a photo (#1166) (#1169)
* fix(gallery): stop the lightbox loading originals to display a photo (#1166) The lightbox read `preview_url`, which the server only emits once an admin has flipped lightbox_preview_enabled — off by default. So a stock install fell straight through to `url`, the untouched original: a reporter measured 16.5 MB for a photo whose preview is 345 KB. The lightbox renders its neighbours too, so opening one photo pulled three originals. `slideshow_url` is the same /preview/:id URL, watermark query included, and has been emitted unconditionally for images since #1015 — the slideshow never had a fallback worth taking. Preferring it fixes every existing install with no migration and no admin action, and `url` still backstops videos, where both derivative URLs are null. Verified on the local rig with the toggle off, so the photos API returns preview_url: null exactly as filed. Opening one photo: before GET /photo/82, /photo/81, /photo/21 (3 originals) after GET /preview/82?w=1280, /preview/81, /preview/21 397 KB -> 23 KB per image on that gallery's test photos. The toggle no longer decides whether the lightbox uses previews, so its copy said something untrue; it now describes what it still does, which is pre-generate rather than wait for the first guest to open a photo. Updated in en/de/fr/sl, the locales that carry those keys. * fix(gallery): cover the layouts the lightbox fix missed (#1166) External review found the fix was incomplete, and the review of it found one more. Premium galleries were untouched. PhotoGridWithLayouts returns early for gallery-premium, which builds its own yet-another-react-lightbox slides with `src: photo.url` — so those galleries kept pulling full originals and the reported bandwidth problem remained. They now use lightboxImageUrl for the display source; `download` deliberately stays on photo.url, because what a guest saves must be the original. The Story layout was worse, and neither the issue nor the review caught it: StoryPhotoCard rendered the full original as its GRID TILE, at object-cover in a small card. That is the one place where "hundreds of megabytes for a gallery" was literally true. It now uses the per-device thumbnail tier like PhotoCard, and its PhotoSwipe source uses the preview tier. Animated GIFs keep the original. generatePreviewImage always encodes JPEG, so routing an animated source through the preview tier would have replaced the animation with its first frame — a regression the toggle-off default never had. Animated WebP has the same problem and cannot be distinguished by MIME alone; that needs the backend to report it (Sharp's `metadata.pages > 1`) and is left rather than costing every static-WebP gallery the bandwidth fix. The settings copy claimed too much. "Pre-generate lightbox previews" does not generate anything on save — it unlocks the regenerate button and keeps preview_url emitted. Reworded to say that, in en/de/fr/sl. Not changed: the review's P1 said this bypassed the secure-image route on enhanced/maximum galleries. It does not. AuthenticatedImage collects requiresToken and secureUrlTemplate into an explicitly-voided unusedProps and never substitutes {{token}}, so on those protection levels photo.url was a literal `.../secure/82/{{token}}` that returns 400 — the lightbox was falling back to the 300px thumbnail, not to a protected image. Verified against a live maximum-protection gallery. Codex withdrew the finding on that evidence. * fix(gallery): keep premium downloads working and story framing intact (#1166) Second review round, three findings — two of them regressions this PR introduced. Premium Download became a no-op. handleDownloadFromLightbox recovered the photo with `filteredPhotos.find(p => p.url === slide.src)`, and slide.src is a derivative now, so the lookup found nothing and the button silently did nothing. The slide carries the photo id and the handler resolves by that; what Download hands over is still the original. Story cards were reframed. thumbnail_fit is seeded to 'cover' on every install, so thumbnails are square centre-crops — and story cards are not square (400x500 in the carousel, fixed-height in the desktop grid), so the card's own object-cover cropped them a second time and every photo shifted. They now use the preview tier, which is fit:'inside' and therefore the whole frame: the card looks exactly as it did before, without pulling an original. APNG joins the animated-format guard. It declares image/apng and the preview route would serve a static frame. Animated WebP still cannot be detected from MIME and remains the documented gap. * fix(gallery): keep PNG on the original, alpha and all (#1166) Third review round. generatePreviewImage encodes JPEG, which drops ALPHA as well as animation — a transparent PNG came back flattened against a solid background. And an APNG is normally reported as image/png, so the image/apng check alone missed the common upload path. PNG now stays on the original: it is where transparency is the norm, and rare enough in an event gallery that the bandwidth given up is small. Animated or alpha WebP still cannot be detected from MIME and remains the documented gap; it needs the backend to report Sharp's `pages`/`hasAlpha`. Two further findings are acknowledged and deferred rather than fixed here: - Story cards now request /preview on mount, so a cold gallery generates its previews in one burst. That is a new CPU cost, not a regression — those cards previously fetched full ORIGINALS on mount, which is strictly worse. Doing it properly means viewport-gating AuthenticatedImage, which is a change to a component every gallery surface uses and belongs in its own PR. - The premium layout memoizes slide URLs, so rotating the device before opening the lightbox can leave a photo on the tier chosen for the old geometry. The result is a slightly undersized image, and the fix is a resize subscription this PR does not otherwise need. * fix(gallery): load Story images on approach, and give the hero its own tier (#1166) Every card in a Story gallery mounts at page load — `whileInView` gates the animation, not the render — and AuthenticatedImage fetches from an effect on mount, so all of them requested at once. That was tolerable while they pointed at photo.url, because nothing was generated; pointing them at the preview tier meant a gallery with cold previews would Sharp-decode every original in one burst. The image now waits until the card is within 200px of the viewport, using framer-motion's useInView — the same observer the entrance animation already relies on — with `once` so a card never unloads on scroll-away. Verified on a 62-photo Story gallery: 3 images fetched at load, growing to 15 as you scroll, where all 62 would have fired before. While confirming that, the hero turned out to be doing the same thing the cards were. StoryHero rendered photo.url as a full-bleed object-cover background — a full original on the critical path for first paint of every Story gallery — when hero_url exists for exactly this and is a 1920x1080 cover crop emitted unconditionally for every photo (gallery.js:1139). That gallery now issues no /photo/ request at all: hero_url for the hero, the preview tier for the cards, and only as they come into range. * fix(gallery): make the Story hero fix actually work on external galleries (#1166) External review of the stable twin, both applying here too. hero_url was inert for external media. ensureHeroImage only ever called resolvePhotoStorageKey, which returns null for external/reference photos by design — and that null was handed straight to withLocalCopy, which throws, so the hero route caught it and redirected to the full ORIGINAL. #1078 fixed exactly this shape for ensurePreviewImage and nobody carried it across. It stayed invisible until this PR pointed the Story hero at hero_url: on a managed gallery that is a real saving, on a reference-mode gallery it quietly changed nothing. ensureHeroImage now has the same external branch ensurePreviewImage does — direct fs read, per-photo output basename — and returns null instead of throwing for a reference-mode row with no source_origin. The format bypass trusted mime_type, which is not trustworthy here. Migration 039 backfilled every pre-existing photo to image/jpeg regardless of what it was, and adminExternalMedia inserts rows with no mime_type at all — so a mislabelled PNG sailed past the guard and came back flattened. It now checks the filename extension as well. * test(gallery): the hero fixture follows the root-relative relpath contract (#1166) external_relpath has been resolved from EXTERNAL_MEDIA_ROOT rather than from event.external_path since #1163 landed. This fixture still carried the base-relative form — its own comment noted the change was 'a separate stack' — so the two tests stopped resolving and ensureHeroImage returned null the moment that stack merged. The production path was never affected. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
410b8f8f6f |
fix(external-media): record capture dates on import, and backfill existing libraries (#1172) (#1179)
* fix(external-media): record capture dates on import, and backfill existing libraries (#1172) External imports never read EXIF, so photos.captured_at stayed NULL for every row they created. The gallery sorts "Date Taken" with COALESCE(captured_at, uploaded_at), which on a bulk import is the import timestamp — so the sort silently degraded into "order by import batch" with no error and nothing in the UI to say the sort key was missing. The reporter's 12-day trip came back with its first two days at positions 4204-5296 of 5555, because those folders happened to be imported second. - the import reads the capture date next to the sharp().metadata() call that already opens the file, so this costs one more read of the same source rather than a second pass over the mount. Best-effort like the dimensions: a source without EXIF imports with captured_at NULL, as before. - POST /api/admin/photos/repair-capture-dates backfills existing libraries, modelled on the dimension repair beside it — background pass, in-flight guard, status endpoint, and resolvePhotoFilePath, which is what reaches an external row at all. Not a migration: the originals sit on a mount that may be down at upgrade time, reading 8000+ of them would block the boot, and a run that found nothing has to be repeatable. - "no EXIF date" is counted separately from "could not read the file". An operator needs to tell "these files carry no date" from "the mount is broken" before deciding to re-run. - the update is guarded whereNull, so an import finishing mid-run is not overwritten by a slower pass. - every sort branch now carries photos.id as a tiebreaker, not just capture_date. A bulk import writes hundreds of rows inside one second, so uploaded_at and the COALESCE fallback both collapse and the grid reshuffles between loads. id is insertion order, which makes the fallback meaningful. Not addressed: extractCaptureDate reads no OffsetTimeOriginal, and exifr resolves a naive EXIF timestamp against the HOST timezone — so captured_at is not a true instant, and the same file imported on two machines yields two values. That predates this and applies to managed uploads equally; the tests here deliberately assert ordering rather than an absolute instant so they do not encode the bug. Worth its own issue. * fix(capture-dates): read managed originals through storage, skip archived, claim the run flag (#1172) Four holes in the backfill endpoint, all found in review: - Managed photos were resolved with resolvePhotoFilePath, which builds a STORAGE_PATH filesystem path. On an S3 install nothing is there, so every managed row failed. Now split the way the thumbnail regenerator does: external rows read from the mount directly, managed rows go through resolvePhotoStorageKey + withLocalCopy. - Archived events keep their photos rows but their originals are deleted on archive, so those rows failed every run and kept the button lit forever. Excluded from both the job and the status counts. - isRunning was claimed after the candidate query, so two concurrent POSTs could both pass the guard and start a pass. Claimed before the await, with every early exit releasing it. - The noExif comment promised a distinction extractCaptureDate does not make (it returns null for unreadable files too). Reworded to what it is. * chore: drop a stray node_modules symlink committed by mistake The .gitignore pattern is `node_modules/`, which matches a directory and not a symlink of the same name, so a local convenience link slipped past it. It pointed at an absolute path on one machine and would dangle everywhere else, breaking `cd backend && npm install`. * fix(capture-dates): gate the backfill as system maintenance, stop overstating the counters (#1172) The endpoint walks every event in the install and rewrites their metadata, but required only photos.edit — which the built-in team_photographer preset holds (175_granular_permissions_and_presets.js:106). That role exists for a contributing shooter, who should not be able to start a whole-library S3/NAS scan or touch another owner's photos. Now system.manage, with the status endpoint on system.view so the panel simply stays hidden for everyone else. The "without EXIF date" wording also promised a distinction the code does not draw: extractCaptureDate returns null for an unparseable file as well as for one that genuinely carries no date, so both land in that bucket. Reworded to "no date found" / "unreachable" in en, de and fr, which is what the two numbers actually separate. * docs: point the permission note at the follow-up PR (#1172) The dimension repair's matching gate landed in #1182, so the comment no longer needs to describe it as unaddressed. * fix(i18n): align the Slovenian capture-date wording with the other locales (#1172) sl was missed when the counters were reworded from 'without EXIF date' / 'unreadable' to what they actually measure. * fix(capture-dates): gate the status card on the permission the button needs (#1172) system.view and system.manage are independent grants, and StatusTab has no permission gate of its own — a successful status payload is what renders the card and its enabled button (StatusTab.tsx:637). Gating the status endpoint on system.view therefore handed a system.view-only role a live Backfill button whose every click 403s, with no error surfaced by the mutation. The comment above it already claimed this endpoint matched the POST. Now it does. * fix(gallery): make the Date Taken sort correct on SQLite (#1172) photos.captured_at does not hold one type on SQLite. Three writers put three different things in it: integer managed uploads — photoProcessor.js:488 hands knex a Date, which the sqlite3 binding stores as epoch milliseconds text external imports and the backfill, which write ISO-8601 null no capture date, so the sort falls through to uploaded_at, itself text in knex's 'YYYY-MM-DD HH:MM:SS' default shape A plain COALESCE over that is not an ordering. SQLite sorts INTEGER before TEXT unconditionally, so every managed photo carrying EXIF came back ahead of every photo that did not, whatever the dates said — a 2027 capture landing before a 2020 one. Among the text values 'T' (0x54) also outranks the space (0x20), so a same-day ISO 01:15 sorted behind a fallback 23:00. Both failures predate this branch — the first needs only two managed photos — but making that sort correct is what #1172 is about, so it is fixed here rather than left for the issue it belongs to. Normalised in the ORDER BY rather than by rewriting the column: the data fix would have to touch every existing row and every writer, which is a far heavier change than the sort it corrects. The cost is that this sort no longer uses idx_photos_captured_at on SQLite — an acceptable trade on the fallback engine, where the alternative is an index-assisted wrong answer. Postgres is untouched: captured_at is a real timestamp there and COALESCE already compares correctly. The regression tests drive the real gallery route on real SQLite. They write the epoch-millisecond integer directly, because the Date that produces it in production cannot be reproduced inside jest — there the binding's type dispatch misses sandbox Dates and stores "[object Object]" (CLAUDE.md). All four behavioural tests fail on the unfixed ORDER BY; verified by reverting it. * fix(gallery): normalise epoch-integer uploaded_at too, and stop polling a 403 (#1172) Two follow-ups from review. uploaded_at is not always text on SQLite either. A legacy archive restore leaves epoch milliseconds in it — there is a test pinning exactly that (__tests__/integration/sqliteEpochTimestamps.test.js) — and the fallback branch read it with substr(), so '1830297600000' was compared against '2020-01-01 00:00:00' as text and a 2028 upload sorted first. Both columns now get the integer/real branch. The status card also polled every ten seconds regardless of permission. With the endpoint correctly requiring system.manage, anyone who can open the Status tab but cannot run the job would have had a 403 and a logged denial every ten seconds for a panel they were never shown. The query is now gated on the same permission the endpoint requires, so it never starts. * style: quote convention in the capture-sort test (#1172) * fix(capture-dates): skip watcher-imported videos, and make the status counts consistent (#1172) Three follow-ups from review. fileWatcher.processNewPhoto sets type='video' and a video/* mime but never media_type (fileWatcher.js:128-130), so those rows keep the 'image' default from migration 048. Filtering on media_type alone queued every such video on every run — extractCaptureDate returns null for a video, captured_at stays null, and the backlog never cleared. Candidate query and status scope now check all three markers. The status counts were two separate queries, so an import committing a dated photo between them could be counted by the second and not the first: the card then showed withCaptureDate > total and a negative backlog, with the button enabled to "fix" it. One aggregate now. And the card's render checked only the cached payload. TanStack keeps that after `enabled` flips false, so a lower-privileged admin logging in behind a system.manage user inside the cache lifetime would still have seen the card and a button whose POST 403s. The permission is part of the render condition now. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Luca
|
e2844d1909 |
feat(gallery): colour labels for client proofing, and one global default per feedback type (#1044) (#1137)
Colour labels for client proofing, plus the photographer's own stars and colours in the admin grid. - Guest colour labels alongside likes/reactions, opt-in per event (defaults off so live galleries do not change mid-proofing), with 'colors' and 'lightroom' keybind schemes. - One global default per feedback type, replacing the per-type scatter. - Admin marks live in their own table (photo_admin_marks) so they can never reach a guest-facing surface. - XMP export prefers a real label, keeping the rating-derived mapping as a fallback. Review: concurrent-write loss on the mark update path, migration index idempotency and error classification all fixed in 7139bcae; migrations renumbered to 182/183 in 8fecdfae after 180/181 were taken on main. Merged with admin privileges: bypass-size-gate is a required check that fails on size alone for review-bypass authors and never re-evaluates on review, which is its designed behaviour once a maintainer has approved. |
||
|
LucaandPaul Nothaft
|
9431b9f094 |
feat(setup): configure the public address and SMTP in the wizard, not .env (#1104)
* feat(setup): configure the public address and SMTP in the wizard, not .env
A fresh install could not configure its own public address. `general_site_url`
and the `email_configs` row already existed as admin settings, but nothing
could reach them:
- docker-compose.yml injected FRONTEND_URL=${FRONTEND_URL:-http://localhost:3000}
and Dockerfile.aio baked in ENV FRONTEND_URL=http://localhost:3000, so
getFrontendBaseUrl() returned on its first branch every time and the setting
was never read. .env.example shipped the same value as an uncommented
placeholder for FRONTEND_URL / ADMIN_URL / API_URL.
- the wizard never asked for the address at all, and skipped its whole config
step unless a CRM-ish feature was selected — so a gallery-only install was
also never offered SMTP, despite gallery links, guest invites and expiry
warnings all going out through email_configs.
- eleven call sites read process.env.FRONTEND_URL directly rather than the
resolver, three of them defaulting to placeholder hosts that reached real
recipients: https://app.example.com in payment-reminder emails, localhost:3005
in admin invitation emails, https://app.example.com in dev template previews.
Stop injecting a default anywhere, and resolve the origin instead:
FRONTEND_URL -> general_site_url -> the origin the request arrived on ->
whichever exists -> ''. A loopback candidate is treated as unconfigured so the
installs that already have http://localhost:3000 baked into their environment
self-heal; the same guard previously lived inline in routes/gallery.js for the
slideshow QR (#848) and is now shared. The empty return is preserved because
shareLinkService and the SSO redirects in routes/auth rely on it to emit
relative urls — callers needing an absolute url use getAbsoluteFrontendUrl(),
which still ends at http://localhost:3000.
The wizard now persists window.location.origin right after the admin account is
created, so an install that skips the rest still has a usable origin for
background jobs that have no request to derive one from, and offers it as an
editable "Public address" field. Settings -> General shows the field read-only
when FRONTEND_URL pins it, instead of silently ignoring edits.
Also drop the `|| 'mailhog'` fallback when seeding email_configs: that host only
exists in the dev compose profile (which does not even start by default), so a
fresh install came up with a live config pointing nowhere while the wizard
showed empty SMTP fields. With no row, blank fields are the truth and
emailProcessor logs "No email configuration found". Developers set
SMTP_HOST=mailhog explicitly.
backend/src/services/emailService.js is deleted: nothing in backend/ references
it, and it was the only consumer of the SMTP_* variables, which misrepresented
how mail is configured.
Refs #705
* fix(setup): keep FRONTEND_URL ahead of ADMIN_URL/APP_URL when resolving links
The previous commit routed two call sites through the resolver but put the
site-specific variable FIRST, silently reversing precedence:
userManagementService was: FRONTEND_URL || ADMIN_URL || localhost:3005
became: ADMIN_URL || resolver
adminEvents/crud was: FRONTEND_URL || APP_URL || ''
became: APP_URL || resolver
An install with both variables set would have flipped which one won. Call the
resolver first instead — it starts with FRONTEND_URL, so the original relative
order is preserved and only the final fallback changes: localhost:3005 (not
even the frontend's port) and '' (a relative link inside an email) both become
the resolved origin.
Refs #705
* fix(setup): unpin loopback FRONTEND_URL, keep ADMIN_URL/APP_URL reachable
Review feedback on #1104.
isEnvPinned() reported ANY FRONTEND_URL as authoritative, including the
loopback values getFrontendBaseUrl() deliberately demotes. An install
upgrading with the old compose default FRONTEND_URL=http://localhost:3000
therefore resolved its origin from general_site_url correctly, but got the
Site URL field rendered read-only in Settings and skipped by the wizard's
seeding - locking the exact operators this change exists to unblock out of
configuring a public address anywhere. The predicate now mirrors the
resolver, and the derived general_site_url_effective the General tab reads
comes from the same helper instead of re-normalising process.env inline.
APP_URL and ADMIN_URL had become dead code: getFrontendBaseUrl() only
returns falsy when NOTHING is configured, so `|| process.env.ADMIN_URL`
after it never ran once a site URL existed - which after this PR is the
normal case. A split-origin install pointing ADMIN_URL at a separate admin
host got invite links on the public gallery origin instead. They are now
passed as an explicit `override` that resolves directly below FRONTEND_URL,
preserving the historic FRONTEND_URL-before-ADMIN_URL order while beating
the database- and request-derived fallbacks.
general_site_url now feeds the CORS allowlist and the
Access-Control-Allow-Origin header, not just email links, so a schemeless
value is an allowlist entry no browser origin can match. Validate it
server-side in PUT /general (isURL with require_protocol, require_tld off
so LAN/NAS installs on http://nas:3000 still work) and client-side in both
surfaces that write it - type="url" never fires in either, since neither
input sits inside a form.
Two more wizard fixes: the General tab no longer reposts general_site_url
while it is env-pinned, because the field then holds the effective env
value rather than the stored one and the round-trip read as a change to a
protected key, 403ing a settings.edit-without-settings.domains admin on an
unrelated save. And SetupConfigStep validates the From address before
posting - /admin/email/config rejects a blank one, which used to surface as
a generic warning while the wizard advanced from its finally block anyway,
discarding every SMTP value the user had typed, password included. A failed
save now keeps them on the step.
* fix(setup): surface a rejected public address instead of swallowing it
Review round 2 follow-up on #1104, pushed onto the branch.
saveSiteUrl() caught and discarded every error. That was defensible before
round 2 added a server-side URL check, but PUT /general can now answer 400 —
and the two validators disagreed:
http://my_nas.local client: accepted server: rejected
http://foo_bar:3000 client: accepted server: rejected
validate() let those through, the 400 was swallowed, `failed` stayed false and
onDone() ran. The operator finished the wizard believing the public address was
stored when nothing had been. That is the silent misconfiguration this whole
change exists to remove, landing on the LAN and NAS installs it targets.
Three parts:
- saveSiteUrl() throws. finish() resolves it before anything else is posted and
puts the message on the address field rather than the generic "some settings
could not be saved" warning. Skip for now still always leaves, by contract,
but warns instead of dropping the value in silence.
- allow_underscores on the server check, for the same reason require_tld is
off: browsers resolve http://my_nas.local and the client accepts it, so
rejecting it server-side only produced the mismatch above. Both validators
now agree across the LAN/NAS, IDN, bare-IP and scheme-less cases.
- LOOPBACK_BASE_RE anchors its host token. Bare prefix matching also demoted
https://localhost-nas.example.com, and now that this predicate gates the
whole resolver rather than just the slideshow QR, being demoted means a
configured address is silently ignored. 127. stays a bare prefix on purpose:
all of 127.0.0.0/8 is loopback.
Resolver suite 31 passing, up from 26. Mutation-checked: restoring the
unanchored regex fails the three new host-boundary cases.
* fix(settings): don't lock the General tab on a site URL nobody typed
Review follow-up on #1104, pushed onto the branch.
general_site_url was free-text until this PR added a server-side check, so an
upgraded install can hold something schemeless that predates it. The tab
flagged that on load, and `disabled={!!siteUrlError}` then killed Save for
EVERY General setting.
An admin holding settings.edit but not settings.domains could not clear it
either: correcting the address is a change to a protected key and 403s. The
tab has no permission gating, so that role was simply locked out of the tab
with no self-service way back.
That is the same role adminSettings.js:85-95 documents the no-op round-trip
allowance for. The allowance only helps if the request is made, and this
blocked it in the browser first.
Validation now waits until the field is actually edited, and an unchanged
value is dropped from the payload rather than reposted — matching what the
env-pinned case already does one line above, and for the same reason.
stored value invalid, untouched Save works, key not sent
edited to something unusable Save blocked
edited to a usable absolute url saved
Four tests, first coverage for this feature. Mutation-checked: removing the
dirty gate fails the untouched-value case.
---------
Co-authored-by: Paul Nothaft <[email protected]>
|
||
|
Paul NothaftandPaul Nothaft
|
b69dd134d0 |
feat(faces): People in this gallery — face recognition via an optional ML sidecar (#1074) (#1075)
* feat(ml): optional face-detection sidecar, opt-in and inert by default (#1074) First of four PRs for "People in this gallery". This one ships only the sidecar, its wiring and its CI — no schema, no backend code, no UI. Nothing in PicPeak calls it yet. picpeak-ml is a single FastAPI + onnxruntime container: three endpoints (/health, /info, /faces), no database, no volumes, no egress, no model download at runtime. Clustering, person identity and every privacy decision stay in the backend where the data already lives. Models are YuNet (detection) + FaceNet-512 (embedding), both MIT, both pinned by URL and SHA-256 and verified at build time. The licence analysis is in ml/LICENSES.md: the more accurate InsightFace weights are non-commercial-only and PicPeak's users are working photographers, so they are never baked into an image we publish. Two things worth review attention: - Alignment uses a least-squares similarity transform (Umeyama), NOT cv2.estimateAffinePartial2D. RANSAC and LMEDS exist to reject outliers among many correspondences; given five landmarks and no outliers they fit a three-point subset exactly and let the rest drift. Measured on a real off-frontal portrait: eyes and nose pinned to 0.11px, mouth corners 11.8px out on a 160px crop. Umeyama distributes it (max 6.5px, rms 5.1 vs 7.4). The failure mode is silent — a bad warp still yields 512 confident floats — so tests/test_pipeline.py pins it numerically. - FACENET_ONNX_URL has no default and the build fails loudly without it. deepface distributes FaceNet-512 as Keras .h5 only, so the ONNX is produced once by tools/convert_facenet.py and published as a release asset. Converting inside the build would drag TensorFlow through both architecture legs of every build to produce a byte-identical file. The CI jobs are gated on the FACENET_ONNX_URL repository variable and skip cleanly until it is set. Off by default, twice over: the sidecar is behind the `faces` compose profile, and the backend will gate on a `faces` feature flag that defaults to false. FACE_ML_URL defaults to http://picpeak-ml:8000 so the standard deployment needs no configuration — nothing dials that host while the flag is off, which is why a non-resolving default is harmless. Verified: 27 pytest tests green; YuNet loads and detects against a real portrait with its landmark order matching the alignment template index-for-index; both compose files validate and the faces profile is correctly excluded from a default `up`; workflow YAML parses and the job graph resolves. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(ml): pin the converter toolchain, verify parity, drop a false reproducibility claim (#1074) Ran the FaceNet-512 conversion for real and corrected what the previous commit assumed about it. The conversion works: 23,497,424 parameters, 89.6 MB ONNX, and the converted graph matches the Keras original to 2.086e-06 absolute / cosine 1.0000000000. That check is now part of the script rather than something I did once by hand — a subtly wrong graph still returns 512 plausible floats, so it refuses to leave the file on disk if parity fails. Also ran the full pipeline against both real models end to end. The embedding is L2-normalized to 1.000000, and the same face survives being re-rendered: half scale 0.973, double scale 0.984, JPEG q40 0.987, rotated 8 degrees 0.984, brightness +40 0.988. Scale invariance in particular is evidence the alignment warp is doing its job. Corrected claim: the conversion is NOT byte-reproducible. Two runs with the same pinned versions on the same machine gave different SHA-256s. The graphs are functionally identical — same 336 nodes, same 271 initializers, every weight matching to 0.000e+00 — but a few initializer names differ because tf2onnx's traced-op naming is not deterministic (Keras layer naming is deterministic; I checked). The previous commit message and README both claimed byte-identical output. They were wrong, and it matters: anyone re-running the conversion gets a different hash, and without this note that reads like tampering. The build-time SHA-256 pins one published artifact so its URL cannot start serving different bytes; validating a fresh conversion is the parity check's job. requirements-convert.txt now pins the exact set that produced the artifact, including transitive keras/protobuf/numpy, and documents that the converter needs Python 3.11 while the image runs 3.12. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): schema, queue, clustering and API for People in this gallery (#1074) Backend half of the feature. Migration 177, a face-detection queue, the clustering engine, the gallery and admin APIs, and the privacy wiring. No UI yet; nothing is reachable until the `faces` feature flag is on, which defaults to false. The flag is the gate, not FACE_ML_URL. That variable now has a working default (the compose service name), so its presence proves nothing about intent — if it were the gate, every install would poll a hostname that does not resolve. faceQueue re-checks the flag every tick, so turning it off stops the workers without a restart. Visibility scoping is the part worth reviewing closely. Face rows have no concept of photo visibility, but guests are restricted to photos.visibility='visible'. A raw count leaks how many hidden photos someone appears in, and an unscoped cover face renders a crop of a photo the guest may not open — with the best-scoring face being the likeliest pick, so it would happen often rather than rarely. facePeopleService recomputes both per request against the caller's own scope, and event_people.face_count_total is named to be conspicuous in a guest path. Six tests cover it, including the case where a person's photos are ALL hidden and they must vanish entirely. Face data is excluded from backups and .picpeak exports, per the decision in the thread: it is derived, so a restore re-scans rather than carrying biometrics between operators. Three separate mechanisms, because the engines cannot be filtered alike — EXCLUDED_TABLES for export, --exclude-table-data (not --exclude-table; the CREATE TABLE must survive or restore breaks on the first query) for Postgres, and DELETE + VACUUM on the temp copy for SQLite, which has no way to exclude a table from a whole-file .backup. The VACUUM is not cosmetic: without it the pages stay in the file and the claim is false on disk. Archiving now purges face data explicitly. photo_faces cascades off photos, but archive deletes neither the photo rows nor the event, so without this an archived gallery kept its biometrics indefinitely. Other decisions: clustering keeps names across a re-cluster by majority inheritance (without it, one button click silently discards every name the photographer typed); consolidation refuses to merge two people who were named differently; assignment never compares across model_version, since embeddings from two pipelines are not comparable; low-quality faces are stored but left unassigned so they show in "this photo contains" without spawning junk people. Migration is 177, not 174 — 174/175/176 landed on main while this branch was open. 29 tests green: 7 migration (idempotency, down(), cascade, and that installing it enqueues NOTHING), 11 clustering, 11 privacy/visibility. Lint clean; the pre-existing error counts in databaseBackup.js and server.js are unchanged. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): People strip, face filter and admin controls (#1074) Frontend half. Renders nothing anywhere unless the `faces` feature flag is on AND the photographer enabled detection for the gallery — the whole guest surface hangs off one boolean, event.people_enabled, which the server computes from the flag, the per-event toggle and the show-to-guests toggle together. Guest side: a People strip between the filter bar and the grid, circular crops from each person's cover face, an active-filter chip row, and a "Show all" bottom sheet. The face filter composes with category, search, media type and the liked/saved/rated filters in the same useMemo rather than replacing them, so "photos of Anna that I liked" works. Two people selected means AND by default — that is what picking a second face almost always asks for — with a toggle to OR that appears only once a second person is picked. Unnamed people show a photo count and never "Person 7". A number is honest about what the system knows; an invented name is not. There is a test asserting we don't do it. The strip renders nothing below two people, collapses to one line when dismissed (persisted per slug, so dismissing one gallery says nothing about the next), and appears mid-backfill with a progress line rather than blocking the gallery behind a spinner. Avatar crops are computed in ratios of the source dimensions so they survive whatever rendition the browser gets; without width/height they fall back to an uncropped thumbnail, since a wrongly-offset crop is worse than no crop. No new download endpoint: "download these N" rides the existing photoIds path, which already enforces access level and per-category permissions server-side. Adding a person_id selector would have been a second thing to authorize for no gain. Guest-facing copy never says "biometric" or "recognition" — those words describe our implementation, not the guest's experience. The sheet's footnote answers the first question every guest has (where does this go?) inline. The admin card, by contrast, is explicit: it states the controller obligation next to the toggle, and warns that scanning materializes the preview tier on galleries that never generated one, which is real CPU and disk an admin should know about before a 2,000-photo backfill. EN + DE translations. 140 frontend tests green (8 new), tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): measured match threshold, working build defaults, 89MB smaller image (#1074) Ran the Phase 0 spike that had been outstanding, published the model, and fixed what both turned up. THRESHOLD IS NOW MEASURED, NOT GUESSED. LFW's standard 1000-pair protocol run through this exact pipeline (YuNet -> Umeyama alignment -> FaceNet-512 ONNX), 100% detection on 2000 images: same person cosine 0.6958 +/- 0.1415 diff person cosine 0.0849 +/- 0.1674 separation 0.6109 peak accuracy 96.60% @ 0.405 So the pipeline separates people well — the thing I could not previously claim, since every earlier number was the same face re-rendered. Default moves 0.62 -> 0.50. The old value was a placeholder and a bad one: it gave 0% false merges but 22.4% false splits, i.e. roughly one in four same-person pairs failing to join, which fragments a gallery badly. 0.50 gives 1.0% false merge / 8.2% false split. Peak accuracy (0.405) is deliberately NOT chosen: for clustering the two errors do not cost the same. A false split is a duplicate row the photographer can merge away; a false merge puts a stranger into someone's "download my photos" — and until the Phase 2 merge/split UI ships, there is no way to undo one. So this sits on the conservative side of the optimum. The spike is committed as ml/tools/benchmark_threshold.py rather than thrown away, so "why 0.50?" has an answer in six months and a re-tune is one command. BUILD DEFAULTS. FACENET_ONNX_URL/_SHA256 now default to the published ml-models-v1 release asset, so `docker build ml/` and `docker compose --profile faces up` work with no arguments. Blanking either still fails loudly — a URL without a checksum is never acceptable, since the checksum is what makes the URL safe to trust. Found by running compose for real: it failed exactly as designed, which was correct behaviour and a bad out-of-box experience now that a canonical artifact exists. IMAGE SIZE. 389MB -> 300MB single-arch. `chown -R` after COPY rewrote every copied file into a fresh layer, duplicating the 90MB model for nothing; the user is now created before the copies and ownership set via COPY --chown. Also drops pip/setuptools from the runtime image. Measured RSS is 186MiB idle, and the container answers /faces end-to-end in well under the 80-150ms/photo the issue budgeted. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): threshold 0.50 -> 0.60 from real clustering, theme-aware People strip (#1074) Both fixes come from running the feature on an actual gallery — 61 photos, 5 real identities — rather than reasoning about it. THRESHOLD. The LFW pairwise sweep in the previous commit said 0.50, and it was wrong. On a real gallery at 0.50, three of six visible clusters were contaminated: two different people merged into one strip entry, which is the exact failure that puts a stranger into someone's "download my photos". Pairwise error rates do not predict cluster purity. Greedy assignment compounds — one wrong face drags the centroid toward the midpoint between two identities, making the next wrong face likelier. A 1% pairwise false-merge rate is not a 1% chance of a clean gallery, and no amount of staring at an ROC curve would have shown that. Sweep against ground truth (5 identities): 0.50 -> 6 clusters, 3 contaminated 0.56 -> 6 clusters, 0 contaminated 0.60 -> 5 clusters, 0 contaminated <- exactly right 0.64 -> 5 clusters, 0 contaminated, fewer faces assigned 0.60 recovers the right number of people with no contamination; higher only loses coverage. Migration 177 carries the full reasoning so the next person to touch this knows why the obvious pairwise answer is the wrong one. THEME. The People strip hardcoded `text-neutral-800` for named people. On a dark gallery — which the screenshot immediately showed — that renders a named person's label almost invisibly, while UNNAMED people stayed legible. Exactly backwards. Labels, headings, the collapsed summary, the scan line and the filter chip row now read the gallery's own theme tokens (--color-text / --color-muted-text / --color-accent / --color-surface-border) like the rest of the gallery surface. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): keep the mobile filter row inside the viewport (#1074) At 390px the photo count and Clear link were pushed against the right edge by ml-auto and clipped. Only apply it from the sm breakpoint up, where there is room; below that they flow after the chips. Found by screenshotting the real thing on an iPhone-sized viewport. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): complete Phase 1, add People management and auto-categories (#1074) Closes the two Phase 1 gaps, then builds Phase 2 and Phase 3. PHASE 1 GAPS. "Download these N" was specified, described as done in an earlier summary, and never actually built — I had verified the backend needed no new endpoint and let that stand as if the button existed. It now hands the filtered photo ids to the same path as a manual selection, so the server re-applies access level and per-category permissions on the way through. Photos in a downloads-disabled category are excluded client-side too, so the number on the button is the number the guest receives. Hidden entirely when downloads are off for the gallery. Lightbox person chips ("In this photo: Anna") are the second way into the face filter — a guest looking at a photo of themselves can act on it without scrolling back to the strip. Tapping one closes the lightbox and filters the grid behind it. PHASE 2. A People management modal over the endpoints that already existed and were already tested: rename inline, merge (multi-select, first pick is the target so the name a photographer typed survives), split via a face picker, hide, ignore. This matters more than it sounds — clustering deliberately errs toward splitting because a wrong merge puts a stranger into someone's download, and that trade only works if merging is easy. PHASE 3. Rule engine over face_count plus face-area ratio: 0 -> Details, 1 large -> Portraits, 2-5 -> Small groups, >5 -> Groups. The area ratio is what separates "a portrait of someone" from "someone is in this landscape". Three guarantees, all tested: it only ever fills an EMPTY category (enforced in the query AND re-checked in the UPDATE, so a photographer setting one mid-run still wins), everything it touches is marked auto_categorized so undo is exact, and it is a no-op unless separately enabled. Migration 178 adds the column — separate from 177, which has already run wherever this branch is deployed. Verified on the real gallery: 61 photos -> 48 portraits + 13 small groups, undo cleared exactly 61 and left the manual ones alone. Merge moved faces and removed the source. Both confirmed against the database, not just the UI. TWO BUGS THE BROWSER CAUGHT, both invisible to tsc: - The lightbox destructure never landed — my patch targeted a line that has a default value, matched nothing, and failed silently. `people` resolved to something else entirely and the chips would never have rendered. eslint's "outer scope value" warning is what surfaced it. - Admin face thumbnails 403'd because <AuthenticatedImage> attaches whatever gallery token is in session storage; an admin who has also opened one of their own galleries sends a type:"gallery" bearer to an admin route. Admin routes authenticate from the httpOnly cookie, which a plain same-origin <img> sends by itself. Worth noting AdminPhotoGrid has the same latent shape; not touched here. Also: the admin card now reports "N people (M shown to guests)" when those differ, so the settings page and the gallery stop disagreeing without explanation. 45 backend tests (8 new) and 140 frontend tests green; tsc and eslint clean. EN + DE for every new string. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * perf(faces): batch migration DDL and drop the face stack from server.js import (#1074) CI's backend job timed out at 10 minutes on the first run of this branch. Nothing failed — 132 of 182 suites passed and the wall clock ran out. Main does the same 182 in 124s, and where main has 12 suites slow enough for jest to print a duration, this branch had 77. Two changes, both worth making regardless of how much of the gap they close: - Migration 177 added its columns one ALTER TABLE at a time (four on photos, three on events, plus a separate index statement) and seeded settings with a SELECT and an INSERT per key. It now uses one alterTable per table and one SELECT plus one bulk INSERT. 178 folds its index into the same statement as its column. That chain replays in ~90 suites, so statement count there is multiplied by 90. - server.js required faceQueue at module scope, which pulls in axios and — through imageProcessor — sharp. Every supertest suite that imports server.js was paying for a module graph it never uses. Now required inside the startup block, next to the call that needs it. Honest about the evidence: locally the migration delta measures at zero (1.15s vs 1.13s for the same suite, three runs each), so batching alone does not explain an eight-minute regression. A fast local disk and many cores mask per-statement and per-import costs that a two-core runner with a shared disk does not. These are the two real costs this branch added to a path that runs in almost every suite; whether they are sufficient is a question for CI, not for another round of local speculation. 37 face tests still green after the change. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * i18n(faces): complete EN and DE coverage for the face feature (#1074) The admin card and the Features toggle were rendering entirely from inline English `defaultValue` fallbacks — 22 keys existed in no locale file at all, so a German admin saw an English consent notice, English toggles and English buttons. The gallery side was already translated; the admin side was not, and nothing in the toolchain flags this because a `defaultValue` always renders something. Adds the missing `admin.faces.*` (19), `settings.features.faces.*` (2) and shared `common.clear/saved/saveFailed` in both languages. Existing keys are left alone (setdefault, not overwrite), so the shared `common` strings other features rely on are untouched. Committed the audit as frontend/scripts/i18n-faces-audit.py rather than throwing it away: it extracts every t() key the face components actually use and diffs it against each locale, and it also reports German values that are byte-identical to English, which is the usual shape of an untranslated copy-paste. Currently: 69 keys in use, EN complete, DE complete, no identical pairs. Verified in the browser, not just in the JSON — the German card reads "61 / 61 Fotos durchsucht · 16 Personen (5 für Gäste sichtbar)" end to end. Also checked the components for hardcoded user-facing text (JSX nodes, title/aria-label/placeholder attributes) outside t(); there is none. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 13 defects from external review — coordinates, counts, erasure, races (#1074) Codex reviewed the branch against main. Thirteen findings, nine P1. I checked every one against the code and could not dismiss a single one as a false positive, so all thirteen are fixed here. THE WORST ONE: bounding boxes were stored in the wrong coordinate system. The sidecar reports coordinates in the space of the image it was HANDED — which is the ≤1920px preview, not the original — while every consumer compares them against photos.width/height, the original dimensions. A 6000px photo therefore produced boxes ~3x too small and areas ~9x too small: avatar crops landed in the wrong place and the Portraits rule could never fire. It is invisible on any photo already under 1920px, which is exactly why the demo gallery and every screenshot looked correct. Now scaled once in faceProcessor so everything downstream can assume original-image coordinates. ERASURE. The FK cascade on photo_faces is decorative on SQLite: PicPeak never enables `PRAGMA foreign_keys`, so deleting a photo left its embeddings behind. I first enabled the pragma globally and reverted it — six unrelated suites immediately failed on pre-existing dangling references, and switching it on would start rejecting inserts on every existing install. That is a real change worth making, but it is its own PR, not a rider on this one. Instead deletion purges explicitly: purgePhotoFaces in the photo paths (single, bulk, service) and photo_faces/event_people in deleteEventCascade. Tests assert this with the pragma explicitly OFF, so they can only pass if the code does the work. COUNTS. A re-scan deleted the old face rows without undoing their contribution to event_people, so counts inflated on every re-scan and ghost people survived. Now the affected people are recomputed before the replacements are assigned. My own "must not double its faces" test only checked photo_faces rows, which is why it passed throughout. RACES. A worker that finished after an admin purged the event committed its rows anyway — erasure reported success and the data reappeared. The commit is now conditional on the row still being 'processing'. And assignFaces is read-modify-write over an event's people, so two workers lost each other's updates; it is now serialised per event with an in-process mutex plus a Postgres advisory lock for the multi-pod case the queue advertises. METADATA LOSS. Merging discarded the source's name and suppression flags, so a merge could erase a typed name or un-hide someone. Reclustering remembered only people with a label, so an unnamed-but-hidden bystander came back guest-visible after one "Re-group people" — and suppression now propagates to every descendant cluster, not just the majority one. Also: export reset face_status so a restored gallery re-scans instead of claiming to be scanned forever; manual category edits clear auto_categorized so "undo automatic" cannot delete a photographer's own choice; external photos are skipped rather than failed (resolvePhotoStorageKey returns null for them by design); the gallery refetches photo memberships as a scan progresses so filtering is not stale; a failed VACUUM now fails the backup rather than publishing one that may retain biometric pages; and the ML Dockerfile's `|| true` is scoped to the uninstall — as written it was `(install && uninstall) || true`, so a failed dependency install produced a green layer and an image with no onnxruntime. Four new regression tests. Full backend suite failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 12 more defects from review round 2 — cross-event purge, leaks, lifecycle (#1074) Second Codex round on the same diff, now including round 1's fixes. Twelve findings, seven P1. Again none were false positives. SECURITY, AND MINE FROM ROUND 1: the bulk-delete face purge iterated the raw `photoIds` from the request instead of the event-scoped `photos` rows the handler had already validated. purgePhotoFaces has no event scope of its own, so an editor could pass another gallery's photo id and delete its face data — even though the photo deletion right below it was correctly scoped. Fixing one thing and introducing another is exactly why the second round was worth running. ANOTHER VISIBILITY LEAK, same class as the one round 1 fixed: /people returns scan progress, and getScanStatus counted every photo with a face_status — including hidden ones. Guests could read the hidden-photo count off the progress bar while the people list and covers beside it were properly scoped. Now scoped by the same predicate, with the caller passing its audience. RECLUSTER, ROUND 1'S FIX WAS INCOMPLETE. I made suppression follow every descendant but still copied the flags from the majority ANCESTOR. When reclustering merges a visible named person with a hidden one, the majority ancestor is often the visible one — republishing the hidden person's photos. Suppression is now OR-ed across every ancestor contributing faces. The name also now goes to the genuine largest descendant; the previous code took whichever cluster came first in map order, which the comment already claimed it did not. LIFECYCLE. Face data is excluded from backups and exports, but photos. face_status came across intact, so a restored install claimed every photo was scanned while holding no faces — and the worker only claims 'pending', so it stayed that way forever. Now: the SQLite backup requeues in the dump, restore requeues after the pool reinit (the Postgres path cannot rewrite rows inside pg_dump), the portable importer purges LOCAL face tables (they were excluded from the replace list, so another instance's embeddings survived an import with FK checks suspended) and requeues, and archiving disables detection so a restored archive is honestly off rather than enabled-and-empty. WRITE PATHS. Only processPhoto enqueued. The synchronous upload path (chunked-upload completion, watch-folder) left photos unscanned, and replacePhoto kept the OLD image's faces on a row now pointing at a different picture — stale identities shown on the new photo. FRONTEND. PeopleSheet and the admin manager rendered centred thumbnails and ignored the bbox, so on group photos the avatar showed whoever stood in the middle and two people from one photo were indistinguishable — in the manager whose entire job is telling faces apart. The crop maths is now one shared helper (faceCrop.ts) so the three surfaces cannot drift again. Full-page layouts (gallery-premium, gallery-story) render their own lightbox and never received the people props. Backend failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): round 3 — five of round 2's fixes were wrong or no-ops (#1074) Third and final Codex round. Eight findings, four P1 — and the important part is that FIVE of them are defects in round 2's fixes, not in the original code. - The sync-upload enqueue I added was a silent no-op. It queried through `trx` after the transaction had already been committed, which throws "Transaction query already complete" straight into the catch I had wrapped it in. Chunked uploads and watch-folder imports were still never scanned, and the code read as though they were. Uses `db` now. - The post-restore requeue ran BEFORE the files were restored, in both the portable importer and the native restore. The face worker is live during a restore, so it could claim those rows and scan the previous instance's files, or fail them for originals not yet on disk — with nothing to requeue them afterwards. Both now run after file restoration; the native one is extracted into requeueFaceScans() and called from the full and database-only paths. - The admin face crop mixed coordinate spaces: an original-pixel bbox scaled against the THUMBNAIL's natural size. The API now returns the source dimensions alongside the box, so there is one space to reason about. - Forwarding people props through layoutProps did not make them work — the full-page layouts never destructured them. GalleryStoryLayout now threads them to its own lightbox. Genuinely new findings, all in the same class as ones already fixed: - releaseToPending updated unconditionally, so a photo purged while its sidecar request was in flight came back as 'pending' and was rescanned — biometric rows reappearing after the purge reported success. Round 2 fixed exactly this on the COMMIT path and I did not carry it to the retry path. Now guarded on 'processing'. - purgePhotoFaces left face_status alone, so a worker mid-scan still satisfied its commit guard and could write fresh faces into a photo being deleted — orphans, since the FK cascade is inert on SQLite. It now clears the claim as part of the purge. - Phase 3 was unreachable: the migration seeds face_auto_categorize_enabled false and nothing could ever write it, so the rule engine and its undo endpoint returned "disabled" in every real flow. Added GET/PUT and a toggle on the admin card, EN + DE. NOT fixed, deliberately: GalleryPremiumLayout uses yet-another-react-lightbox rather than the shared PhotoLightbox, so person chips there are a real port rather than a prop forward. Recorded as open rather than bodged. Backend failure set identical to origin/main; 41 face tests and 140 frontend tests green; i18n audit reports EN and DE complete at 71 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): block face recognition on the all-in-one image (#1074, #1042) The single-container image cannot run this feature, so it is refused there rather than left to degrade. WHY, since the reason is not obvious from the code: the AIO image runs the backend, the frontend, SQLite and every background worker inside one container aimed at "one photographer plus guests browsing". It has no Redis, SQLite gives it a single writer, and it contains no ML sidecar to talk to. Face detection would add a second image-processing pipeline competing with Sharp for the same CPU and memory. That failure is not loud — the install just becomes slow and looks broken, which is the worst possible shape for a deployment whose whole promise is one container and no decisions. Gated on an explicit PICPEAK_SINGLE_CONTAINER marker, NOT inferred from SERVE_FRONTEND or a SQLite path: plenty of legitimate multi-container setups serve the frontend from the backend or run SQLite, and none of them should lose the feature by accident. Three layers, because the first is the only one that enforces: - faceSettings.isFeatureEnabled() returns false before consulting the flag, so a database restored from a full deployment with `faces` enabled still cannot switch it on here. - The feature-flag API forces `faces: false` in both directions, so the admin UI reflects reality instead of offering a switch that refuses to stay on. - The Features tab renders the card disabled with a plain-language reason, read from a new `single_container` field on /admin/system/version (an endpoint the admin UI already calls). Documented in ml/README.md and .env.example. Three tests pin the behaviour, including that the marker only accepts explicit truthy values. NOTE FOR PR #1068: this expects `Dockerfile.aio` to set `ENV PICPEAK_SINGLE_CONTAINER=true`. That one line lives on that branch and is not in this commit — until it lands, an AIO build would still offer the feature. Worth adding alongside the `Limits` section of docs/single-container.md. 44 face tests green; EN + DE complete at 72 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(faces): pin the bbox coordinate space with a real scale factor (#1074) The coordinate-space bug — boxes stored in preview space while every consumer reads them as original-image pixels — had no test, and could not have been caught by the ones that existed: every photo in the demo gallery is 750px, so the scale factor was always exactly 1.0 and the correction never executed. Verified by hand first, on a real 4000x3000 upload with the face placed off-centre so a wrong crop would be unmistakable. Before the fix the stored box was 1493,204 (preview space, face actually at x≈2850-3618); after, 3110,426 — a factor of 2.083, exactly 4000/1920, landing inside the face. The admin crop then resolved to left=-395px/top=-46px on a 64px window, which is the face centred. That verification is now a test rather than a memory. Three cases: a 4000px photo must scale by 4000/1920, a 1920px photo must NOT change (the case that hid the bug), and a row with no width must fall back to unscaled rather than storing NaN. Note for anyone extending these: jest hoists mock factories above the file, so anything they close over has to be `mock`-prefixed. Getting that wrong fails at transform time with a message that does not name the variable. 47 face tests green. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
8e3573788b |
feat(downloads): per-gallery download resolutions (#858) (#1022)
Clients who need smaller files no longer make the photographer re-export. Two capabilities, both off by default. STANDARD RESOLUTION — the size a gallery hands out for every ordinary download (single, selected, download-all). Global default in Settings, overridable per gallery with the NULL=inherit tri-state. The pre-built download-all zip is built AT the standard resolution, so changing it invalidates those archives, including a fan-out to inheriting galleries. RESOLUTION PICKER — opt-in modal letting guests choose a different size. Custom archives are built as a DB-backed job the client polls, never cached. The picker never offers a size above the standard, and Original reappears only when the admin explicitly allows it. Resize is fit:'inside' + withoutEnlargement — aspect preserved, never upscaled — applied before the watermark, since the mark is sized relative to its input. Three rounds of external review hardened this: job archives are bound to the requester's visibility scope and re-validated at delivery, the streamed download-all path applies the cap, queue admission is bounded, and rejected resolutions no longer inflate download stats. Closes #858. |
||
|
LucaandLuca-Timo
|
2e495d7c48 |
feat(transfers): add PicTransfer — cross-event file transfers (#998)
Closes #997. Send original files from any event as a token-protected download link, with an optional client-upload channel. Strictly opt-in behind a new `transfers` feature flag, default OFF. Migrations 170-172 (transfers, transfer_files, transfer_extra_files, transfer_uploads, transfer_recipients, transfer_downloads, default settings and two email templates) — all hasTable/hasColumn-guarded and idempotent, with destructive statements confined to down(). Backend: transferService (CRUD, 256-bit download token, 6-char upload token, cross-event ZIP streaming of originals), admin CRUD routes, and two public token routes. transferCleanupService runs an hourly retention sweep; source-event photos are never touched. All three routers fail closed via requireFeatureFlag('transfers'). Review closed two ownership blockers, both the same root cause — permissions used where ownership was needed: - photoIds arrived from the request body and were validated only for existence, so a scoped admin could bundle any event's originals and hand them out through the public download token. filterOwnedPhotoIds now resolves ids to their events and gates them through filterOwnedEventIds, on both the create and add-files paths. - The transfer list was unscoped and carried each row's download token, so any admin with events.view could read another's token and fetch their originals. The list is now scoped by created_by, the token/url fields are stripped from the list payload, and a single router.use('/:id', requireTransferOwnership) covers all twelve /:id routes, 404ing foreign and missing alike. The admin photo picker filters its event list to the same rule, so the UI stops offering picks the API would discard. Fork-PR workflows had not been approved since the fix commits, so the PR's green checks were stale against the pre-fix head. Verified by dispatching tests.yml against the actual head: backend and frontend both green. Follow-up: neither ownership guard has a regression test yet. Co-authored-by: Luca-Timo <[email protected]> |
||
|
Luca
|
165cebdb5c |
feat(accounting): re-bill proof attachment, CRM panel & hours↔re-bills cross-add (#979)
Closes #866. Three features, all behind the `incomingInvoices` feature flag: 1. Attach the stored supplier proof PDF to the client-invoice email when a captured invoice is re-billed/passed through, as a SEPARATE attachment so invoice immutability holds. Global default (off), per-customer tri-state override, and per-file selection in a new Send dialog. A missing proof at issue time stamps inbound_documents.proof_attach_error rather than silently dropping, and never blocks the send. Proof filename is a configurable template with {INVOICE} {SUPPLIER} {YEAR} {MONTH} {SEQ}/{SEQ:0Nd} tokens. 2. Re-bills & passthrough panel under CRM → Customer, grouped Open/Sent/Paid with status derived from the linked invoice lifecycle rather than a duplicated column. 3. Cross-add dialog rolling open hours and open re-bills into one invoice, symmetric from both entry points. The two stay distinct, contiguous line groups — never merged into shared line items. Migration 169 is additive, hasColumn-guarded and idempotent. Review (two rounds) closed two concerns: - Storno stranding: nothing cleared inbound_documents.billed_invoice_id when a covering invoice was cancelled, so a Storno'd re-bill showed as Open in the new panel while every billing path filters on that column being NULL — the supplier cost could never be re-billed. releaseRebillsForCancelledInvoice now detaches the linkage on both invoice-cancel paths, with a regression test on the issued-cancel path. - Permission gating: the new controls rendered on data presence alone while their endpoints require accounting.view / accounting.manage / customers.edit. Now gated at both the query and render layers. Known follow-up: two cross-add counter queries are gated on a permission their endpoint does not check (HoursSection.tsx:174, CustomerCrmPanels.tsx:270) — degrades safely, one line each. |