f6b81fabf05ab0bbce48e63bbdf3812b30aa10de
122
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul NothaftandPaul Nothaft
|
a2bf1f644c |
fix(video): try metadata extraction and thumbnail generation independently (#1371)
* fix(video): try metadata extraction and thumbnail generation independently processUploadedVideo() gated everything behind isValidVideo(), which rejects the whole video if ffprobe can't read even one of duration/width/height -- common on some iPhone/Lightroom-exported MP4s (issue 1370). Callers (photoProcessor.js's processPhoto and processUploadedPhotos) already catch that throw and fall back to a static placeholder thumbnail plus a metadata-only retry (codex review of #845), but that fallback never got a REAL thumbnail even when generateVideoThumbnail() would have succeeded on its own -- thumbnailing doesn't need valid duration/width/height, it just seeks and grabs a frame. processUploadedVideo now tries metadata extraction and thumbnail generation independently, keeping whichever succeeds instead of discarding both on a single failed field. The callers' existing throw handling stays as a backstop. Also: extractVideoMetadata stored duration as 0 (not null) whenever ffprobe had no duration field, masking "unknown" as a fake real zero-second clip and defeating downstream `duration != null` checks meant to skip an untrustworthy value. Relates to issue 1370 * fix(video): fall back to the SVG placeholder when thumbnail generation fails processUploadedVideo could return success with thumbnailKey: null when only thumbnail generation failed. The gallery grid (GridGalleryLayout/JustifiedGalleryLayout) falls back to `photo.thumbnail_url || photo.url` when there's no thumbnail, so AuthenticatedImage downloaded the full original video and tried to render it as an <img> -- a broken tile and a potentially huge fetch just from opening the gallery. Falls back to the same ffmpeg-free SVG placeholder the callers already generate for a total processing failure, so a bare thumbnail-generation failure degrades to that placeholder too, never to "no thumbnail at all". Found by codex review. * fix(video): avoid a SQLite connection deadlock in the placeholder fallback generateVideoPlaceholder() unconditionally called getThumbnailSettings(), which queries the database directly (not through any active transaction). videoProcessor.js's new placeholder fallback can run from inside processUploadedPhotos' open per-file SQLite transaction (chunked video upload) -- knex's default SQLite pool has exactly one connection, so that second, un-transacted query deadlocks against the transaction holding it, timing out after acquireConnectionTimeout (60s). Reproduced directly against an isolated SQLite db. generateVideoPlaceholder now skips the settings lookup entirely when the caller supplies explicit width/height, and the video fallback passes the same DEFAULT_THUMBNAIL_WIDTH/HEIGHT the settings lookup would have fallen back to anyway (now exported for reuse). Found by codex review. * fix(video): throw when neither a real thumbnail nor the placeholder can be produced processUploadedVideo returned success with thumbnailKey: null when both the real thumbnail AND the SVG placeholder failed -- a total, systemic failure (storage backend down, disk full), not a quirk of one file. On stable, which doesn't have the #845 call-site fallback, this silently completed the video with no thumbnail at all instead of the retryable 'failed' status a throw here produces. On main, the pre-existing #845 fallback already absorbed this exact case (no behavior change there) -- verified against codex's own git-blame check of the pre-PR stable code before applying this. Now throws in that case, restoring the pre-existing "let the caller mark it failed and retryable" behavior for a genuinely unrecoverable video, while keeping every partial-failure case (the vast majority) resolving with whatever succeeded. Found by codex review. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
59ef2ee9af |
Merge pull request #1361 from PicPeak/feat/usage-reporting-update-prompt
feat(usage): prompt existing admins once for usage reporting after an update |
||
|
Paul Nothaft
|
d20f80112f |
feat(usage): prompt existing admins once for usage reporting after an update
An admin who already had PicPeak installed before the opt-in reporting feature existed never gets asked — the setup wizard only runs once, on a brand-new instance. Adds a one-time modal, shown on the admin's next dashboard visit after updating, offering the same choice the wizard gives a new install. - New `product_usage_state.prompt_shown` column (migration 211) and UsageService.markPromptShown(), set on either outcome (enable or decline) from both this modal and the wizard step, so an installation is never asked twice regardless of which path it took. - New POST /admin/usage/prompt-seen endpoint. - Extracted the wizard's three-point pitch (UsageReportingPitch.tsx) so the modal and the wizard step share identical copy instead of drifting apart. - The modal never shows once participation is already active, and never shows a second time after either the wizard or the modal has been through it once. Depends on #1360 (the setup wizard step this reuses). |
||
|
Paul Nothaft
|
a31a2e25e2 | fix: interrupt idle worker waits during shutdown | ||
|
Paul Nothaft
|
f0e6d2dfb1 |
fix: enforce gallery access and consolidate gallery workflows (#1357)
Harden gallery authentication and authorization, consolidate gallery workflows, and prevent token-bearing URLs from leaking through nginx request error logs. |
||
|
Paul NothaftandPaul Nothaft
|
acb25a9a1c |
fix(images): probe and clean up preview tiers under the extension the encoder actually wrote (#1355)
generatePreviewImage rewrites the output extension to match the encoding it chose, .jpg or .webp for alpha and multi-frame sources. The tier lookup in ensurePreviewImageAtWidth and the cleanup list in previewTierKeys kept the SOURCE extension instead, so for anything but a lowercase .jpg source the stat never matched: every tier request for a .png, .JPG, .jpeg, .heic or RAW photo re-ran Sharp, and cleanup never found the files it left behind, which accumulated for the life of the install. Both now derive every key the tier can live under: the .jpg and .webp candidates, plus the source-extension key last so tiers written before the rewrite are still found by lookup and by cleanup. Follow-up to issue 1020, where the mismatch was identified during review. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
c97341e454 |
fix(images): single-flight lazy rendition generation and keep the old rendition during replacement (#1350)
* fix(images): single-flight lazy rendition generation and keep the old rendition during replacement
The lazy generators in imageProcessor are check-then-generate, and the
check reads the path off the photo row the route already fetched. N
concurrent cold requests for one photo all held a snapshot with the path
still null, all missed, and all ran the same sharp pipeline. Only the
thumbnail tier path had a guard; the canonical thumbnail it falls back
to, heroes, previews and preview tiers had none.
One process-local map now covers every rendition, keyed by photo id and
rendition (`thumbnail:<id>`, `thumbnail:<id>:w<width>`, `hero:<id>`,
`preview:<id>`, `preview:<id>:w<width>`). Concurrent callers share one
promise; the entry is cleared in a finally on success and failure alike
so a rejection cannot poison the key. The tier stat moved inside the
flight so a request arriving as the previous flight clears finds the
written tier instead of missing on a stale probe.
Heroes and previews also deleted the existing object before generating
its replacement, and again in the catch. Both are gone, mirroring what
the thumbnail generator already does: put is the last statement in the
try and replaces atomically on local storage and by key on S3, so the
delete only ever opened a window with no rendition at all, and a source
that failed to read stripped the old rendition with the row still
pointing at it.
No re-read of the photo row inside the flight: the admin regenerate
endpoints force a rebuild by passing a row with the path nulled, and a
re-read would hand back the persisted rendition untouched.
Fixes the single-flight half of issue 1020. The preview cache-key
extension mismatch and any server-wide work queue remain separate.
* fix(images): keep the snapshot validity check outside the single-flight
With the check inside the flight, an admin regeneration (row passed with
the path nulled) could join a viewer's flight for the same photo that was
merely confirming an already-good rendition, and be handed back the very
file it was asked to replace while the endpoint counted a success. Only a
miss enters the flight now; inside it everything is a regeneration.
* fix(images): forced rebuilds run after an in-flight lazy generation instead of adopting it
The admin regenerate endpoints could still join a lazy flight that was
already generating for the same photo. That flight read the thumbnail
settings when it started, so after a settings change it produces exactly
the rendition the regenerate was invoked to replace; adopting it counted
a success while the old size stayed cached.
ensureThumbnail and ensurePreviewImage take `{ force: true }`: skip the
snapshot check and, if a flight is pending, start after it settles. Lazy
misses arriving meanwhile join the forced flight, and an older flight
settling late no longer evicts the newer entry from the map.
* fix(images): key rendition flights by source as well as photo id
replacePhoto keeps the photo id and changes path and filename. Keyed by
id and width alone, a request carrying the replacement row joined a
flight still rendering the file it replaced and was handed the old
image, which the gallery caches for 30 minutes. The tier map this
replaced was keyed by storage key and so already told the two apart.
* test(thumbnails): wait for the regenerate loop's completion line instead of a fixed 150 ms
The loop runs in setImmediate after the response. Under a loaded machine
(fifteen suites in parallel, each booting a migrated SQLite) it took
longer than 150 ms once and the assertions ran against a half-finished
mock call list. Poll the logger spy for the "regeneration complete" line
with a 10 s deadline; the suite also finishes sooner because the wait
ends as soon as the loop does.
---------
Co-authored-by: Paul Nothaft <[email protected]>
|
||
|
Paul NothaftandPaul Nothaft
|
8cc7d7d14a |
feat(external-media): watch reference folders and import new files automatically (#1345)
* feat(external-media): watch reference folders and import new files automatically Managed uploads dropped into storage/events/active are picked up by the chokidar watcher; external media had no equivalent, so a NAS folder that keeps growing needed an admin to open the event and press Import every time. Relates to issue 1187. - The import pass moves out of the route into services/externalImportService.js. The watcher and the Import button now run the identical function; the route only validates and maps errors to status codes. - Mutual exclusion is the per-event claim from maintenanceJobState (`external_import:<id>`, seeded on demand by the new ensure()) instead of the in-process Set. The Set stopped a double-click in one process; the claim also stops the watcher on a second replica, or an admin clicking while the watcher is mid-run elsewhere. The run heartbeats so a claim from a dead process is taken over. - services/externalMediaWatcher.js: per-event opt-in via the new events.external_watch column (migration 208), chokidar with awaitWriteFinish so a copy in flight is not imported half-written, debounced full pass per change, a timer sweep every 15 minutes as the fallback for NFS/SMB mounts that deliver no inotify events, optional stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched events is re-read every minute, so the toggle works from any replica. A watcher that just started runs one pass immediately. - Deletions are ignored on purpose: a file vanishing from a NAS is at least as likely to be a reorganisation or a dropped mount as an intentional removal, and acting on it would delete a guest-visible photo. Rows whose file is gone stay, as they do today. - Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local. - Quiet system passes stay out of the activity log; runs that imported something are logged with actor external-media-watcher. - Frontend: "Watch folder for new files" checkbox under the external folder picker, status line in view mode, EN/DE strings. * fix(external-media): close the review gaps in the folder watcher Codex review of the watcher, round 1. All six findings were real: - Enabling the watcher, or pointing an enabled one at another folder, now requires photos.upload — the permission the manual Import already requires. events.edit alone was a way around it. Only the transition is checked, so a role without photos.upload can still edit an already-watched event. The checkbox is disabled for such roles. - Automatic passes defer files that are still changing: anything modified inside the stability window, or whose size moves across one wait of that window, is left for the next pass. chokidar's awaitWriteFinish only settles the file that fired the event, and the sweep sees no events at all, so a sibling still being copied could be inserted half-written and then skipped forever. - Photos an admin deleted are not brought back by the sweep. The delete routes record the file in external_import_exclusions (migration 209); automatic passes skip the list, the manual Import ignores it and clears it for what it imports. - The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three compose files; they were documented but the backend services use explicit environment lists, so the kill switch did nothing. - A pass re-checks is_active / is_archived at run time, not only in the minutely reconcile. - The lease is renewed on a timer for the whole run, walk included, and ownership is checked before the event row is touched. * fix(external-media): make automatic passes follow the row, not rewrite it Codex review round 2, four findings, all applied: - The event update route drops non-canonical spellings of external_watch and external_path before the permission guard. SQLite resolves column names case-insensitively, so `External_Watch` reached the column while the guard only looked at the lowercase key. - Exclusions are checked per file at insert time, not against a snapshot taken before the settle wait. A photo deleted during the wait was present in the snapshot and got re-inserted by the loop. - An automatic pass no longer writes source_mode / external_path. It re-reads the row after the walk and the settle wait and stops if the folder changed or the event went managed; the manual Import is the only writer. The options are now `automatic` + `settleMs`. - A pass that deferred files re-arms the debounced import, so a file copied just before the watcher started is not stranded when the sweep is disabled. * fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying Codex review round 3, both findings applied: - recordExclusions keys on external_relpath alone. A replaced external photo becomes managed but keeps its relpath on purpose, and deleting that replacement must not republish the NAS original. - An automatic pass checks the full watcher predicate (reference mode, same folder, watch on, active, not archived) before it inserts and on every heartbeat tick during the loop, and stops as soon as the event no longer qualifies. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
5c1e38d921 |
feat(usage): distinguish real edits and template delivery with v5 consent (#1339)
* feat(usage): distinguish real edits and template delivery with v5 consent * fix(usage): exclude queued test messages and count reorders as edits - queueEmail carries usageEligible: false into email_data and the queue processor passes it on, so the dev tools' send-test-email no longer records email_template_delivery once the worker sends it. - event-types/reorder and categories/reorder-global compare the persisted order before and after and record the v5 edit markers only when it changed, matching the display_order edit already counted on PUT. - normalized() builds arrays with Array.from so a row array from the sqlite binding compares equal under Jest's separate realm. * fix(usage): cover per-gallery category order and workflow test runs - categories/reorder records category_editing when an event's override changes; reorder/:eventId records it when an override was actually removed. - send_email and the collections handoff pass usageEligible: false for a workflow test run (engine.testRun sets __test), so a non-dry test send is not counted as template delivery. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
69754f8a2c |
fix(email): scrub gallery passwords from the sent-mail archive (#1340)
* fix(email): scrub gallery passwords from the sent-mail archive
The email queue kept every gallery password and client PIN in clear
text in email_data and rendered_html after the mail was sent, and the
Messages reading pane handed them back to any admin with the messaging
flag. A password hash in the events table bought nothing while the
plaintext sat next to it.
Once a mail is out, or its retries are exhausted, the processor now
masks secret-looking variables (password, passcode, pin) in email_data
and replaces their values in the rendered body, plain and HTML-escaped.
The reading pane applies the same masking to rows archived before this
change. Pending rows keep the real values so a retry still sends them.
Relates to issue 1271
* fix(email): keep a quoted ">" from cutting an attribute value out of redaction
The tag splitter stopped at the first ">", so a template attribute such as
title="{{gallery_password}} > details" left the password unmasked in the
archived HTML while email_data was already masked. The tokenizer is now
quote-aware; a tag with an unbalanced quote falls through as text and is
scrubbed there.
* fix(email): scrub secrets inside HTML comments in the archived body
A comment such as <!-- PIN: {{client_password}} --> was split off as a tag
and its body, which has no attribute, was never scrubbed. Comments are now
one segment and their content is masked whole.
---------
Co-authored-by: Paul Nothaft <[email protected]>
|
||
|
Paul Nothaft
|
aa6e5d613f |
chore(usage): make the capability catalog English-only
features.v2/v3/v4.json carried every name, description and definition twice, once in English and once in German, inside a file that is source code, is vendored byte-identically into the collector and is served as the consented catalog. Source stays English. The German strings already existed a second time in the frontend locale file, which is what the consent dialog actually renders (UsageCatalog reads productUsage.catalog.<key>, never the JSON), so the copy in the catalog was a duplicate that could only ever drift. The `de` fields are gone from all three catalogs, their frontend copies and the inventory definitions; the docs coverage file and FEATURE_COVERAGE.md list English only. Nothing on the wire changes: the report schema is derived from the keys, and the catalog's text is not part of any signature or consent version string. The coverage test now pins the catalog to the en locale verbatim and requires the de locale to cover every key and field, without dictating its wording. The collector holds the same catalog files and needs the same change to stay byte-identical, plus its German catalog strings moved into its own locale file; that is filed there. |
||
|
Paul Nothaft
|
ef8a52f02c | fix(usage): introduce consented v4 without changing historical reports | ||
|
Paul Nothaft
|
02b353e54f |
fix(usage): report restricted gallery downloads in v3 instead of an always-true signal
gallery_downloads.configured was true on every installation with a gallery. allow_downloads ships true — column default in migration 037 and the create route both set it — and the snapshot asked "at least one gallery has it on". The fleet value was ~100% by construction and could not separate a deliberate configuration from an untouched one. v2 consented to that key under that description, so v2 keeps sending it unchanged. v3 replaces it with gallery_downloads_restricted: at least one gallery has downloads switched off, which is the only state of that column anyone actually decides. Same catalog position, so the disclosed capability count stays at 86; the frontend copy, the EN/DE catalog strings, the coverage inventory and FEATURE_COVERAGE.md follow. Done in v3 rather than a v4 because v3 is on main and in no release yet, so nobody has consented to it. The collector carries the same catalog and has to take this change before the release that ships v3. One guard for the window in which :main / :beta images already carried the old v3 catalog. A report queued under it fails local validation on this build, and deliver() left a locally invalid report pending for good, blocking every operation behind it. A report's payload is derived state, so deliver() now rebuilds it from the current snapshot in place and sends that. Packet ID and sequence are kept — a re-signed retry has to reuse them so a lost acknowledgement does not duplicate data — and reports only: a stale registration, deletion or command is a genuine conflict and keeps the existing handling. Tests: the v3 snapshot counts a switched-off gallery and ignores enabled ones, v2 still reports the old key with the old meaning, and a stale queued report goes out rebuilt under the same packet id while a valid one is sent untouched. Relates to issue 1308 |
||
|
Paul Nothaft
|
7ca783f89b | fix(usage): preserve report contracts with compatible receiver validation | ||
|
Paul Nothaft
|
c358bc65f7 | feat(usage): add consented beta capabilities and gallery photo totals | ||
|
Paul Nothaft
|
e40bc474bc |
fix(usage): let an operator clear a participation the collector never accepted
Probing the live collector to settle the delete-sequence question turned up something else: usage.picpeak.app answers a valid usage.v2 registration with INVALID_PACKET while the identical v1 flow is accepted. It does not speak v2 yet — which the deployment notes already require, but the consequence of getting that order wrong was worse than "reports do not send". Opting in to v2 against a v1-only collector left the installation stuck. Registration was refused, so nothing existed at the collector at all; the row sat in activation_pending, disable moved it to deletion_pending, retry was futile forever, and enable refused because the row was not `disabled`. The abandon hatch added earlier did not apply: it was gated on SIGNING_KEY_UNREADABLE. So the most harmless possible failure — nothing registered anywhere — was the one an operator could not clear. The gate is now the property that actually matters: a participation the collector has provably never accepted (sequence 0, no receipt) with a failing delivery can be discarded, from activation_pending as well as deletion_pending. Its receipt records `never-registered` rather than an unconfirmed deletion, because nothing remote exists to be unsure about. A participation the collector *did* accept keeps the old narrow gate and its explicit warning — clearing local state while the collector still holds reports must stay a deliberate, warned-about act. A collector that rejects a registration or a deletion outright now reports SCHEMA_NOT_ACCEPTED instead of DELIVERY_FAILED, and the settings page says the collector does not accept this report version yet. Retrying cannot fix that, and sending the operator to look for a network fault they do not have was wrong. Verified end to end against the live collector: v2 opt-in reports SCHEMA_NOT_ACCEPTED, the exit is offered immediately, the receipt says never-registered, and joining again on v1 registers, reports and withdraws with a collector-confirmed deletion. |
||
|
Paul Nothaft
|
c741dc22c5 |
docs(usage): state in the consent dialog that the connection only runs outwards
The dialog described what is sent and where it goes, but never said which way the connection runs. That is the part an operator is actually being asked to accept: opening an outbound path to someone else's service. PicPeak sends and never pulls. One place in the service reaches the network, it is a POST, and it requests exactly two paths — /api/envelopes, and /api/participant/lookup only when an operator asks for their own export. No scheduled job contacts the collector; the daily rollup is driven solely by an authenticated admin hitting /activity. There is no route the collector could call, and redirect: 'error' means it cannot even point a request somewhere else. From a reply only the acknowledgement for the packet just sent is read, with every field compared against that packet before it is accepted; the stored copy drops the session token and no read path hands it back to the UI. A requested export is streamed to the operator as a file and never interpreted. The consequence is why it belongs in the consent text and not only in the docs: this channel cannot deliver code, configuration or content into an installation, not even from a collector that has been taken over. It is a security property by design rather than by convention. usageOutboundOnly.test.js guards it by source inspection rather than behaviour, because a behavioural test only proves that today's calls behave. It fails the moment someone adds a second fetch, a poll for messages, a scheduled pull, or a public route touching the usage service — verified by injecting each of those. |
||
|
Paul Nothaft
|
1e8b6f1b0f |
fix(usage): close the QA findings on opt-in product usage
A QA exploration of this branch against an isolated rig — own stub collector, SQLite and PostgreSQL — turned up one dead end and a set of signals and controls that did not hold up. This closes all of them. Rotating JWT_SECRET, the documented response to a suspected compromise, made the signing key unreadable. That was already named and documented, but it left no way out: the delete packet can never be signed, so the row stays deletion_pending forever, and enable() refuses because it is not `disabled`. An operator who rotated precisely because the secret was compromised cannot restore it, so the feature was bricked with no control left. POST /usage/abandon is offered only in that state; it drops the local identity and records the receipt as `collector-unconfirmed` rather than claiming a deletion that did not happen. Every failed delivery was retried on the next admin request, and /activity is open to any authenticated admin while the settings ticker fires it every five minutes per open tab — 30 activity calls against a rejecting collector produced 30 outbound requests. Migration 206 adds attempts/next_attempt_at and the unattended sender honours the gate; Retry and opt-out still send immediately, and the tab names the time of the next automatic attempt. Feedback, votes and portal sessions now share an installation-wide budget of 30/hour. They are the only endpoints whose effect is outbound traffic carrying operator-written free text, and the general limiter skips authenticated requests by design. Reading status and withdrawing stay unthrottled. gallery_image_protection was true on a bare install with no galleries: PicPeak ships default_protection_level='standard' and enable_devtools_protection=true, so it reported fleet-wide 100% and could never separate a decision from an untouched default. It now reads only what deviates from the shipped defaults, and the devtools flag is not read at all — being on by default, its only informative state is off, which is the opposite of what the key claims. Also: - the export receipt counted every packet and called the total "usage reports"; reports and participant operations are now counted and named separately - GET /usage/preview no longer persists the custom_css marker, so the transparency view stops changing what will be sent - the feedback route requires every field the packet schema requires, so an API caller gets the missing field named instead of a bare INVALID_PACKET from inside signing - the German strings for this feature use "Sie" throughout, matching the rest of the admin UI; the ignore hint says what ignoring will do rather than stating it as already true - the consent dialog returns focus to the control that opened it - the long buttons wrap instead of running off a 390px viewport - a deletion receipt is labelled as belonging to an earlier participation while a new one is active Regression tests cover each of these, including the delete packet's reuse of the last accepted sequence, which was an unwritten assumption about the collector rather than a defect. |
||
|
Paul Nothaft
|
a7382591bf | feat: expand opt-in capability coverage with versioned consent | ||
|
Paul Nothaft
|
5d31b61c8d | Merge remote-tracking branch 'origin/main' into codex/1110-usage-coverage | ||
|
Paul Nothaft
|
e347f8f40f | fix(usage): minimize session receipts and clarify privacy controls | ||
|
Paul Nothaft
|
32d745b575 |
fix(usage): stop local backups implying S3 use, and make the protocol-error branch reachable
Two findings from the review of the current head. Local backups no longer imply S3. markUsed derived an s3_storage marker from "a backup ran while backup_destination_type is s3" — but the middleware also counts /database-backup/* and /backup/picpeak/export as backups, and those write a local file wherever scheduled backups go. So configuring S3 and downloading a local export reported s3_storage as USED. The middleware now tells markUsed whether the operation writes to the configured destination, and only then is the marker derived. A wrong `true` in this dataset is worse than a missing signal: it is a claim about an install that nobody can check. The ProtocolError branch was dead code. adminUsage matched on `error.name === 'ProtocolError'`, but the class extends Error without setting `name`, so every instance reports 'Error' — verified — and a malformed vote or feedback payload fell through to the global handler, which logs it as an unhandled programming error and answers INTERNAL_ERROR in production, losing the validation code the caller needs. Now matched with instanceof. protocol.cjs is byte-identical with picpeak-usage (diffed against the companion repo), so the fix belongs here rather than in the class. An existing assertion needed updating for the new markUsed argument, and the path split is pinned: /backup/run is destination-driven, /database-backup/backup and /backup/picpeak/export are not. Refs #1110 |
||
|
Paul Nothaft
|
bb76ca5375 |
fix(usage): keep the settings tab usable on a bad collector URL, and report layouts and CSS accurately
Three items, one of which explains an error seen in the app. "The operation could not be completed" could come from a config typo. status() called collectorUrl() bare, and that throws on a bare hostname, a path, a query, or http in production. The settings page renders one generic failure when its status query errors, so a misconfigured USAGE_COLLECTOR_URL replaced the whole tab with that sentence — no cause, and no way to read the status or withdraw, because every control there sits behind that call. The URL is now reported as collector_error: 'INVALID_COLLECTOR_URL' beside the real state, the tab says what is wrong and how to fix it, and the links are only rendered when there is somewhere to point them. gallery_layouts reported grid for every preset-themed install. color_theme holds either a theme object or the NAME of a preset — the theme picker stores names, and eventTypeService seeds them (`theme_preset: 'corporateTimeline'`). Only reading value.galleryLayout made masonry, timeline, mosaic and the two gallery presets invisible. Names now resolve, and an event with no theme of its own resolves through the global one instead of being counted as grid. Only the name -> layout mapping is duplicated, not the presets; frontend/src/types/theme.types.ts stays the source of truth, and an unknown name reports `other` so a preset added later degrades to "something else" rather than quietly inflating the grid count. custom_css missed CSS applied through a template. An enabled css_templates row applied via events.css_template_id is gallery styling by the same definition as the settings fields — the Custom CSS tab is where both are authored — but neither the snapshot nor the middleware saw it, so those installs reported custom_css entirely false. Existence only; template contents are never read. Eleven tests. Reverting each fix in turn fails 3, 1 and 3 of them. Refs #1110 |
||
|
Paul Nothaft
|
9785b636a9 |
fix(usage): take the withdrawal baseline before the lease, not after it
Third and last window in the same race, and again in my own fix. locked() claims the lease and reads the row in two separate statements. Reading the cancellation counter from inside that callback meant a /disable completing in the gap was adopted as this activation's own baseline and silently absorbed — the counter matched, the claim succeeded, and registration went ahead after the operator had withdrawn. The baseline is now read before the lease is taken, which inverts it: every increment from that point on is later than the value the claim tests for, so the claim fails and the withdrawal wins. An increment from before the read is a withdrawal the operator already completed, and a deliberate opt-in afterwards should not be vetoed by it. The test for this passed against the bug on its first two attempts. It stubbed the state read to increment the counter AFTER reading the row, so both the broken and the fixed version saw the old value and behaved identically. The withdrawal has to land before the read returns for the row to carry it — which is the whole point of the window. It now fails without the fix. Refs #1110 |
||
|
Paul Nothaft
|
22da018e1b |
fix(usage): close the remaining withdrawal races, reset per-item name consent
Follow-up review on the previous commit, including a hole in that commit's own fix. The cancellation flag became a counter. Clearing a boolean needed a write of its own, and a /disable landing between the lease and that write was erased — the same race one level down. enable() now records the counter it started with and claims only if it is unchanged, so no clearing write exists to lose. It also fixes the case a boolean could not express at all: a stale cancellation already set, and a fresh one arriving mid-activation, are indistinguishable as flags and obvious as counts. Migration 203, separate from 202 for the reason 202 was separate from 201 — knex will not re-run an applied migration. deliver() re-checks immediately before dispatch. The existing check ran before the binding lookup, which is asynchronous, so a withdrawal that COMPLETED during it still had its registration or report sent afterwards. Not an already-in-flight request — a new one started after the operator had withdrawn. The outbox writes in tick() and command() are conditional on still being active. /disable clears pending_packet without holding the lease, so an unconditional write put a report — or a feedback body and name — back into an outbox the withdrawal had just emptied, where deliver() would then leave it, since it declines to send anything but the delete. Per-item name consent resets with the item. `named` stayed checked after submitting, so the next item carried the previous name automatically, contradicting the anonymous-by-default promise the disclosure makes for each item. The remembered name stays in preferences; attaching it is decided again each time. Two of these tests were worthless when first written and are noted because the pattern keeps recurring: the pre-dispatch case passed without the guard because an empty report payload failed schema validation during signing, so nothing reached the collector for reasons unrelated to the check. With a valid payload it fails without the guard and passes with it. Same for the counter: dropping it from the claim fails two. Refs #1110 |
||
|
Paul Nothaft
|
80e238f0ad |
fix(usage): let a withdrawal win against an activation that is still starting
The last open item from the #1304 review. /disable overlapping an in-flight /enable was silently lost. While activation generates its identity and writes its binding file the row still reads `disabled`, so disable()'s conditional update matched no rows, and the lease conflict raised by its tick() was swallowed as expected noise. The admin was told participation was off; the activation then completed and left it on. An opt-out that does nothing is the one failure this feature cannot have. disable() now records cancel_requested first and unconditionally — before the case-by-case work — and enable() claims its state with a single conditional UPDATE that tests the flag alongside the status. Re-reading the flag and then updating would only have moved the window; making the claim itself carry the condition closes it, so whichever of the two lands first wins outright and the loser writes nothing. Nothing is registered when the claim fails, so there is also nothing to delete remotely — the cancelled activation leaves no identity behind. The flag is cleared at the start of enable(), so a cancellation from an earlier participation cannot veto a later deliberate opt-in. The column is migration 202 rather than an edit to 201. 201 already shipped on this branch and knex records it as applied, so folding the column in would have skipped every database that had already run it and the first /disable would have failed on a missing column. Verified both ways: a fresh install gets the column from 201+202, and a database migrated before 202 existed gains it when 202 arrives. Three tests. With the condition dropped from the claim, the race case fails and the other two pass. Refs #1110 |
||
|
Paul Nothaft
|
c043897b0e |
fix(usage): name the unreadable-key failure, unpin the collector default, align the tab
Review follow-ups on #1304. SIGNING_KEY_UNREADABLE. USAGE_ENCRYPTION_KEY defaults to JWT_SECRET, so rotating JWT_SECRET — the correct response to a suspected compromise — makes the stored Ed25519 key undecryptable. That surfaced as a generic DELIVERY_FAILED which retried forever, and it silently blocks the DELETE packet too: an operator who withdraws has their local state cleared while the collector keeps its copy. decrypt() now tags its own failure and deliver() reports it under its own name, without flagging an identity conflict — an unreadable key is not evidence of a clone. The docs already warned that losing the key breaks deletion signing; they now name the trigger and the error. The collector default is no longer an inline string in the constructor. It is a declared DEFAULT_COLLECTOR_URL, since it is a deployment choice: self-hosters point USAGE_COLLECTOR_URL at their own collector and the UI already derives every link from whatever is configured. schema.cjs is deliberately untouched — it is vendored byte-identical with picpeak-usage, and its $id is a schema identity, not a delivery address. Links in the consent dialog. It named the collector inside prose but never linked it, so an operator deciding whether to opt in could not open the destination or the public schema without retyping a URL. Both are links now, built from the configured collector. UI standards. The tab hand-rolled its surfaces as `<section className="rounded-xl border border-theme …">` and imported Button from a deep path; every other settings tab uses `<Card padding="md">` from the components/common barrel. Converted, with the feedback <form> wrapped rather than replaced so its semantics survive, and headings given the same colour tokens as ImageSecurityTab. The barrel pulls ErrorBoundary -> i18n/config, so the tab's test needed the initReactI18next shim the FaceRecognitionCard test already uses. Not changed: the delete packet reusing the current sequence. The collector handles delete before any sequence check — "possession proof is sufficient for deletion, including when a restored backup has a stale sequence" (picpeak-usage server/collector.js) — so deletion is deliberately sequence-exempt and the client is correct as written. Refs #1110 |
||
|
Paul Nothaft
|
027afb6086 |
fix(security): re-check inline CSS after template substitution
The fourth bypass found in this review, and the one no lexer fix
reaches: sanitizing runs on the stored body, but safeTemplateReplace
rewrites it afterwards, so the string that was validated is not the
string that is sent.
A conditional inside a style attribute can delete the very quoting that
made a url() inert:
style="--x:x{{#if company_name}}'{{/if}};background:url(https://evil…)"
At write time the url() genuinely sits inside a CSS string and is
correctly left alone. Expanding the conditional for a recipient with no
company name removes both quotes and the background goes live —
confirmed end to end against the real functions.
The style-attribute pass now runs again on the substituted output.
Substitution cannot introduce a `"` (values are HTML-escaped), so the
attribute match still holds. body_css is not substituted, so the
<style> block cannot be rewritten after its check and needs nothing.
This is the case the removed newsletter pass had been covering. Rather
than reinstating a second definition of "disallowed", the one definition
now runs at both points where the content changes.
Refs #1264
|
||
|
Paul Nothaft
|
1cf82746b7 |
fix(security): close two CSS url() bypasses the sanitizer dedup exposed
Both found by review against the correct base, and both are cases the
second stripRemoteCssUrls pass had been catching before this PR removed
it. Verified against the real functions before and after.
An escaped quote outside a string. `\'` is an escaped identifier
character, not a string opener, but the scanner stepped onto the
apostrophe, entered string mode and copied the rest of the stylesheet
unexamined — so `.hero{--marker:\';background:url(https://evil/p.gif)}`
kept a live remote URL. Escapes are now consumed as a unit outside
strings.
An unterminated quote. Trusting one meant a single stray apostrophe
disabled scanning for everything after it. An unclosed quote is a parse
error, so the safe reading is to emit it as an ordinary character and
keep scanning; a newline also ends a string, as it does in CSS.
The entity mismatch behind the second case. sanitize-html writes `"`
inside an attribute as `"`, so the scanner and the recipient's
browser disagreed about where strings begin: in
`style="font-family:"don't";background:url(...)"` the browser
decodes first, reads the apostrophe as ordinary text inside a real
string, and fetches the background — a tracking pixel by another name.
Style attributes are now decoded before scanning and re-encoded after,
which also stops the old code silently deleting quotes from the value.
Also detaches the image handlers before releasing the canvas source.
That one did NOT reproduce: measured in both Chromium and WebKit,
neither fires `error` when the attribute is removed after a successful
load. Applied anyway because the ordering is free and the failure it
would cause is silent — canvasFailed set, the canvas swapped for an
<img>, and the image decoded a second time, the exact opposite of what
the release is for.
Refs #1264, #1287
|
||
|
Paul Nothaft
|
fc595409b4 |
feat(crm): newsletter campaigns behind a newsletters flag (#1264)
Part B of #1264. Flag off by default, so an install that never enables it gains no route, no nav entry and no way to mass-mail. A campaign is a body plus a recipient rule. Queueing one writes ordinary email_queue rows (email_type 'newsletter', origin 'campaign', new campaign_id), so retry, rendered_html, sent_at and error_message all come from the existing processor rather than a parallel sender. Throttling staggers scheduled_at; the processor loop is untouched. Two rules the service enforces: no raw HTML is ever stored (sanitized on write and again on render, idempotently), and opt-out is checked at queue time AND again at send time. Migration 199 adds email_campaigns, email_campaign_recipients, email_queue.campaign_id, customer_accounts.marketing_opt_out(_at), and the newsletters.view / newsletters.send permissions. Three rounds of external review are folded in, including several that would otherwise have shipped broken: - Campaign rows never came due on SQLite. queueEmail writes a Date, which the sqlite3 binding stores as epoch ms; ISO text in the same column compares as TEXT against an INTEGER, and SQLite orders every INTEGER below every TEXT. The feature silently sent nothing there. - The flag had no Settings card and no sidebar entry, so it could not be enabled through the UI at all. - Consent is per ADDRESS, not per row: two accounts sharing an inbox meant unsubscribing stopped one and not the other, at both queue and send time. - The unsubscribe GET mutated consent, so a mail-security scanner walking a campaign could have unsubscribed much of the list. GET now confirms, POST acts. - The rate ceiling is clamped to the queue's real throughput (10/min), so the composer's estimate stops being wrong by up to 12x. Closes #1264 |
||
|
Paul Nothaft
|
7c9baff751 |
fix(upload): scope category ids, stop temp-file leaks, split the video cap
Four related fixes on the admin upload/photo path. B5 -- PATCH /photos/:photoId and POST /photos/bulk-update took any parseInt(...) > 0 straight into the update with no existence or scope check, so a photo could be moved into another event's category. The upload route already validated `event_id = X OR is_global` per #500/#525; extracted that query as findScopedCategory() and used it on all three routes so the 400 body is byte-identical. 0/negative/'individual'/'collage'/null still clear without a lookup, so the clear path costs no extra query. B9 -- three distinct temp-file leaks, not one. The validator's size branch never unlinked; the cleanup lived in the final handler, unreachable on any 400; and multer's `destination` callback runs per file and overwrote req.tempUploadPath, so even the success path only ever removed the last file's directory. Now: discardUploadedFiles() runs on every 4xx and the 500 (ENOENT tolerated, and files are only dropped when the whole request is being rejected, so the passing path is untouched); cleanup registered before multer so it also covers multer's own LIMIT_FILE_SIZE return; one directory per request. B8 -- the admin uploader filtered on MIME only, so an oversized file was uploaded in full before the server's 400. Mirrors UserPhotoUpload's existing per-file toast-and-drop. C4 -- general_max_file_size_mb was a single cap for photos and videos, so the 50MB default meant admins could not upload ordinary video without also raising the photo limit. Adds general_max_video_size_mb (default 500MB, clamped by the same 10GB MAX_ALLOWED_FILE_SIZE_MB ceiling, read per request, 60s cache), editable in Settings -> General. Photo uploads are protected from regressing by keeping multer's type-blind limit at max(photoCap, videoCap) and moving the per-kind decision into validateUploadContent, where file.mimetype exists. It 400s with the existing message shape, so an oversized photo is still rejected with the identical body it produced when multer did the rejecting. Known gap: chunked-upload/init still applies the photo cap to video. Making it video-aware would change an existing assertion that pins a 200MB video init being rejected under a 1MB general cap. No component calls that path today and the direction is strict rather than a bypass, so it is left as-is. Guest video uploads still share the single cap in gallery.js. Refs testplan REPORT.md B5, B8, B9, C4. |
||
|
Paul Nothaft
|
103863cbab |
fix(quotes): enforce the status state machine, and correct the table
VALID_QUOTE_TRANSITIONS was a complete-looking quote state machine that nothing consulted, so status changes were unvalidated. Mapping every writer of quotes.status (quoteService.js is the only one -- dealsService, projectService, adminDashboard and customer.js all read) showed the table itself was wrong: six legitimate transitions were missing. sendQuote allows draft/declined/expired -> sent but the table had draft only; adminAcceptQuote allows draft/sent/expired but had sent only; adminDeclineQuote allows draft/sent/expired but had draft/sent; recordResponse had no same-status entry. Enforcing it as written would have broken accept-on-behalf from a draft, resend-after-decline, every expired revival and the 15-minute response-toggle window. So the table is reconciled to reality first, then assertQuoteTransition() (409, QUOTE_INVALID_TRANSITION) is called at all seven sites. Two things worth carrying forward. Nothing in the codebase ever sets 'expired' -- the header comment says "set by the scheduler" and there is no such scheduler; sent -> expired is retained as documented intent only. And the backstop's added value is narrow: every reachable invalid transition is already caught by a call site's own better-worded guard, which fires first. What it newly catches is a status the machine has never heard of -- a legacy or corrupt row like 'cancelled' sails through adminAcceptQuote's guard, which only excludes accepted/declined/converted, and used to be silently overwritten. That is what the new tests pin. Refs testplan REPORT.md B4. |
||
|
Paul Nothaft
|
77b11ab874 |
fix(upload): enforce the chunked-upload cap on bytes received, not declared
The init route checked the client-declared fileSize against general_max_file_size_mb, but nothing checked what then came through the chunk route: a client could declare `fileSize: 1` and stream any amount, and completeUpload only logged the size mismatch before handing the merged file on. The cap the earlier commit added at init was therefore a gate with no fence. The service now carries the cap from init and enforces it on the running byte total per chunk (aborting the upload once crossed, since the chunks on disk are already over the limit), rejects chunk indices outside the announced range, and re-checks the merged file as a backstop. Both routes answer 413/400 for these instead of a blanket 500. |
||
|
Paul Nothaft
|
1d84c738d8 |
test: repair four stale backend suites
All four asserted contracts the product has since moved past. No genuine
product bugs behind any of them; assertions were tightened, not loosened.
adminAuth (3 tests): never mounted errorHandler, so ConflictError/
ValidationError arrived as empty Express defaults. The route also checks
username before email, so the "email conflict" fixture was hitting the
username branch. Mount the handler, fix the fixture, match the real response
shapes.
backupService.enhanced (12 tests): three stacked drifts -- the db mock had no
.returning(), so every runBackup threw at the insert; ensureDatabaseDumpForBackup
now lazily requires ./databaseBackup inside the run, which fails under
mock-fs; and the rsync path moved from exec(shell string) to
spawnAsync('rsync', args) with an isHostAllowed SSRF preflight. Also updates
getBackupStatus to its current shape (frontend aliases, nextScheduledRun null
when no schedule is enabled, #871).
adminSettings.logo: POST /logo gained requirePermission('settings.edit');
the hand-rolled db mock returns a bare Promise from select(), so the
permission lookup threw a TypeError into a 500. Mock the permissions
middleware alongside the already-mocked auth.
crmMintPaths (2 tests): macOS-only. The expected prefix was realpath'd while
the services persist under the raw STORAGE_PATH -- identical on Linux CI
(/var vs /private/var only diverges on macOS), which is why it passed there.
The comment justifying the realpath referenced process.cwd() behaviour the
services no longer have.
Refs testplan REPORT.md #22 (Part 1.2.01).
|
||
|
Paul NothaftandPaul Nothaft
|
44a8c416c7 |
refactor(email): stop reading the webhook response body at all (#1225) (#1239)
The transport carried 41 lines of bounded-read-with-deadline to recover a messageId a receiver MIGHT return. That value is only ever logged — nothing persists it, there is no email_queue.message_id column — and the code to get it produced two of the last four review findings: the size cap made a DELIVERED message retry (axios throws while reading), and the missing deadline let an unclosed stream hang the queue and resend. Not reading the body is how that whole class stops being reachable rather than defended against. responseType 'stream' still keeps axios from buffering; the stream is destroyed immediately and the id is synthesised as before. The status was always the delivery verdict, and it is known before any of this. An 'error' listener goes on before destroy(): destroy can emit on a socket-backed stream, and an unhandled 'error' on a stream throws — which would have turned a receiver's teardown into a failed send. Net: -45 lines of service code, one fewer constant, one fewer test seam, and three of the hardest cases in the suite replaced by two simpler ones. 23 tests, 63 across the email suites, eslint clean. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
6ca8baab23 |
fix(watcher): stop re-importing a photo whose file was replaced (#1226) (#1237)
The existence check matched on filename OR path. replacePhoto regenerates both — a fresh generated filename and a fresh managed path — so a watched-folder photo that had its file replaced stopped matching either arm. The original is still sitting in the watched folder, so the next sweep imported it again and the gallery ended up holding the delivered edit AND the untouched original: the same duplicate shape external_relpath prevents for reference galleries. source_filename is now a third arm. It is the stable key here — written once at ingest by this same path and preserved across a replace by design. Rows predating migration 193 are covered by its backfill: COALESCE(original_filename, filename), and this path never wrote original_filename, so for watcher rows that resolves to the basename being compared. The query is lifted into an exported findExistingPhoto() so the test drives it rather than a copy — the thing under test IS the query, so a query-builder mock would only assert that knex was called the way the test expects. Predates the Lightroom round-trip and applies to the admin replace path too; it became reachable when #1165 brought watcher galleries into round-trip scope. Six tests against a real SQLite database. The load-bearing one fails without the change, verified by removing the arm and re-running; the other five pin what must not move — filename and path matching, the pre-193 backfill shape, a genuinely new file, and event scoping. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
4105c099c9 |
test(external): make the fold-collision guard test the real code (#745 follow-up) (#1234)
The regression test added with #1165 re-implemented the claim ordering and the claim loop inside the test file and asserted against its own copy. It never required externalRelpathFold, so changing the real sort left it green — a guard against silently deleting a client's delivered edit that guarded nothing. The ordering is now a named, exported claimOrderFor() and the test drives it. Verified by sabotage: replacing the comparator with `return 0` fails the test, where before it passed. Three cases added while the seam existed: the managed row wins from BOTH input orders (the original bug was that the survivor was whichever came first, so one order proves nothing), the sort is stable for rows of the same kind, and it does not mutate the caller's array. No behaviour change — the comparator is byte-identical, only lifted out. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
0d41fe5bf1 |
fix(email): keep the webhook payload out of the logs, and bound the response read (#1225) (#1233)
Round 4 of external review, on the merged commit. Both findings are consequences of the round-3 streaming change, which is exactly why the round was worth running. An AxiosError carries the request it failed on: `config.data` is the ENTIRE serialised message, base64 attachments included, and `config.headers` holds the signature. emailProcessor logs the error object and winston serialises it, so a DNS blip or a refused connection wrote password-reset links, guest recovery codes and multi-megabyte invoices into combined.log — verified against axios rather than assumed. Every rejection is now caught and replaced with a message-and-code-only error, so nothing downstream can serialise the request back out of it. readBounded had no deadline. axios' `timeout` covers the response HEADERS, and with responseType 'stream' it has already resolved by the time the body is read — so a receiver that answered 2xx and never closed its body left the await hanging, the queue row stayed pending, and the next processor pass sent the same message again. An unclosed stream was duplicate email. There is now a 10s wall clock that destroys the stream, with the timer unref'd so a hung read cannot hold the process open at exit. 23 transport tests (2 new, both failing without these fixes), 63 across the email suites, eslint clean. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
d62407f431 |
feat(email): webhook transport as an alternative to SMTP (#1225) (#1231)
Setting EMAIL_WEBHOOK_URL makes PicPeak stop sending mail itself and POST each composed message as JSON instead, for something downstream (n8n, Make, a self-hosted relay) to deliver. Unset, every SMTP path is unchanged. Settles the four things #1225 left open: - SSRF: the URL goes through the same DNS-resolving check the outbound webhook worker uses, before every send. Private receivers are opt-in. - Transport security: https is required for anything leaving the machine. The HMAC proves who sent the body, not who can read it, and these bodies carry password-reset links and guest recovery codes. The private-network opt-in doubles as the plaintext opt-in. - Authentication: EMAIL_WEBHOOK_SECRET is required and signs the body as X-PicPeak-Signature, the same scheme as gallery webhooks. A URL without a secret leaves the transport OFF and says so once. - Attachments: carried as base64, not dropped. Oversized ones fail and stay queued rather than arriving without the invoice. Configuration is environment-only on purpose: this redirects every outbound message including password resets, so it must not be changeable from a compromised admin session. Three wiring details decide whether it works at all: docker-compose.yml declares an explicit environment block, so the vars had to be forwarded there; a fresh webhook-only install has no email_configs row (migration 001 seeds it only when SMTP_HOST is set), so the From identity falls back to EMAIL_FROM; and processEmailQueue used to return early when SMTP could not initialise, which would have left the queue permanently unprocessed. guestRecoveryService and the admin test-email endpoint were bypassing the transport — the first dereferenced a null transporter, the second told webhook-only admins to go configure SMTP. emailIntakeService deliberately stays on SMTP: it round-trips a specific mailbox's own credentials. Response handling is streamed and read bounded by hand rather than capped via axios: maxContentLength throws while reading, so a receiver that delivered the mail and then echoed a large body would have been recorded as failed and the message sent again. Note: docker-compose.dev.yml is gitignored and local-only, so the equivalent entries there are not part of this change. docker-compose.production.yml needs none — it passes .env through with env_file. Three rounds of external review; 21 transport tests, 61 across the email suites. |
||
|
Paul NothaftandPaul Nothaft
|
f4c054a661 |
fix(export): name the camera master in photo exports, not the delivered render (#1229) (#1230)
#1165 added photos.source_filename to this service's select, with a comment saying it was there so the Lightroom round-trip could still match after a re-upload — and then nothing read it. Every output path still used original_filename, which is overwritten the first time an edited render is uploaded over a proof (#745). So after a round-trip the exports named the render. Each of these formats exists to help a photographer find the master on disk, and the render's name does not. The XMP case is the sharpest: the sidecar is written next to a RAW master, so a wrongly-named one is never associated with it. Two helpers, because the sites want different things when nothing is known: cameraName() source_filename || original_filename || null cameraFilenameOrStored() the above, else the stored name The dedicated `original_filename` fields (CSV column, JSON key) keep reporting blank/null when unrecorded — echoing the sanitized stored name there would invite a match against a file that does not exist under it. The places that must emit some name (text list, CSV filename cell, XMP sidecar) fall back to the stored one, as they did before. filename_format='stored' is untouched, and rows with no source_filename still resolve to original_filename, so nothing moves for installs that have never run a replacement. Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Luca
|
8db8527f9e |
feat(api): Lightroom round-trip — read proofing marks, put finished edits back (#745) (#1165)
* feat(api): Lightroom round-trip — read marks, put edits back (#745) Gets a client's proofing verdict into a desktop catalogue and a finished edit back over its proof, without anyone re-matching files by hand. Three parts: **Keep the camera filename.** photos.original_filename is the only carrier of `IMG_1234.JPG` — the stored filename is rewritten by generatePhotoFilename. But replacePhoto() overwrites original_filename with whatever name the new file arrives under, so the first re-upload of a renamed render destroys the key the NEXT round-trip needs. Migration 185 adds photos.source_filename, written once at ingest and never touched by a replace, backfilled from original_filename so existing galleries can still match on their first pass. The backfill sits outside the column guard and keys on whereNull, so a run that dies partway self-heals instead of leaving half the rows empty forever. **Read the marks.** GET /api/v1/events/:id/photos returns each photo with its client colour tallies, the caller's own marks, and a merged colour + rating. Guards copied from the sibling upload route (apiTokenAuth + read scope + photos.view + requireEventOwnership). Filters: marked_only, mark_source, color_labels, my_color_labels, min_rating, my_min_rating. The route filters to a page of ids with PhotoFilterBuilder, then enriches just those through photoExportService.getPhotosWithFeedback — the two halves already existed and neither does both, and going id-first keeps the per-colour tally query bounded by page size. services/markMerge.js decides how three possible opinions (guest colours, guest star average, the photographer's own row in photo_admin_marks) collapse into the one colour and one rating Lightroom has room for. Colour goes to the photographer on a tie — one deliberate choice beats an aggregate a tie-break already had to guess at. Rating takes the max, because a rating is a magnitude and losing the higher one quietly demotes a photo somebody rated highly. Its roundRating matches XmpGenerator.mapRating exactly so the API and an XMP sidecar can never disagree about how many stars a photo has. **Put the edit back.** POST /api/v1/events/:id/photos accepts an optional replaces_photo_id and routes to the existing replacePhoto(), preserving the photo's id, feedback, comments and position. The plugin stores the picpeak id on the catalogue photo, so the id survives the editor renaming the render — which makes it the reliable key, not the filename. Scoped to the event in the URL: a token inherits its owner's powers across every event they can see, so an unscoped id would let one gallery overwrite another's photo. For renders whose RAW never went through the plugin, findReplacementCandidate gains an opt-in number_token mode matching on the trailing digit run. Deliberately the LONGEST run and never a fixed last-N slice: multi-camera shoots disambiguate by prefixing the camera index into the number (cam11234.jpg / cam21234.jpg), and a last-4 slice reads 1234 from both bodies and reintroduces exactly the collision the prefix removes. Ambiguity is refused, never guessed. Also drops the multer temp file on the two new early returns — this route only unlinks in its catch block. * refactor(api): one rating-rounding rule, and apply match_mode where it counts Three things the pre-review pass turned up on the round-trip work: - `match_mode` reached the photo-cap pre-count but not the loop that actually picks the replacement target, so asking for `number_token` would have been counted and then quietly ignored. Both call sites now take it. - `number_token` matching read `select('*')` over every photo in the event to compare one digit run. It now reads the three columns the match needs and re-reads the single winner in full, so a 5000-photo event doesn't pull 5000 full rows through memory to answer one question. - `XmpGenerator.mapRating` and `markMerge.roundRating` were the same five thresholds written twice — the second way to do one thing that drifts the moment either is touched. The thresholds now live in markMerge and the generator delegates, which is what keeps a sidecar and the v1 API from ever disagreeing about a photo's star count. * fix(api): keep the new route in the generated OpenAPI spec The `color_labels` description carried an inline JSON example. In an unquoted YAML scalar `{ "green": 2 }` parses as a flow mapping, so swagger-jsdoc threw YAMLSemanticError and dropped the WHOLE route from the spec — visible only as a warning on boot, with the route still serving normally, which is exactly the kind of failure that survives to release. Found by booting a real instance rather than by reading the diff. * fix(api): close the four blockers from review on #1165 1. Replacing an external photo silently kept serving the old file. resolvePhotoStorageKey gives photo.source_origin precedence and returns null for 'reference'/'external', so the edit was uploaded, the row updated and 200 returned while every viewer kept getting the untouched NAS original and the upload sat orphaned. replacePhoto now repoints the row to managed and clears external_relpath. The file on the share is never touched — this moves the pointer, not the data. 2. Every replacement leaked its temp file. putFromFile COPIES on local and uploads on S3; neither consumes the source, and replacePhoto never unlinked it — while the v1 route had disabled its own cleanup on the belief that replacePhoto moved the file. Cleanup now lives in replacePhoto, which closes the admin path too (adminPhotos only unlinks in its new-files branch, so replaced files leaked there as well). The v1 route also unlinks on the FAILURE path, which returned before any cleanup ran. 3. The download-all ZIP is invalidated after a replacement, as adminPhotos.js already does. Without it guests kept downloading the pre-edit photo indefinitely, which defeats the point of the feature. 4. The round-trip could not see reference or watcher galleries at all. fileWatcher and adminExternalMedia never set original_filename — the camera name lives in `filename` for those rows — so the backfill and the GET fallback both produced NULL for exactly the galleries most likely to be driven from Lightroom. The backfill now COALESCEs, both ingest paths set source_filename, and the GET falls back to filename. Concerns: - number_token no longer reads every photo row in the event per file. A LIKE on the digit run narrows the candidate set in SQL first; the exact trailing-run check still decides, so semantics are unchanged. The token is a regex-extracted digit run, so it cannot carry a wildcard. - The replacement's activity entry is scoped to event.id instead of null. The dashboard feed excludes NULL-event rows for scoped callers (GHSA-jhcf), so it was vanishing from the audit trail of the photographer who owns the event. Nit: dropped the unused higherPriorityColor export from markMerge. Three regression tests cover the external repoint, the temp cleanup and the COALESCE backfill. 21/21 pass. * chore(migrations): renumber 185 -> 193 after gallery-folders landed 185_add_category_is_folder.js merged to main while this was in review, so the number the PR reserved is taken and main is now at 192. Knex keys on filename rather than the prefix, so both would have run — but picpeakImportService guards restores with migrationOrder(), which parses that prefix, and two files answering 185 make the forward-only check pass a backup onto a schema missing its columns. Renumbered with every reference: the header comment, the test that requires the path, and the four call-site comments that cite it. The 'migration 182' reference inside it is the colour-labels migration and is unrelated; gallery.js:1134 cites upstream's 185 and is untouched. * fix(api): keep external_relpath when a replacement converts the row The external-photo blocker fix cleared external_relpath along with flipping source_origin, which closed one hole and opened another. adminExternalMedia dedupes a re-scan on (event_id, external_relpath) — routes/adminExternalMedia.js:195 — and migration 186 puts a unique index on exactly that pair. With the column nulled, the next scan of the share would not recognise the NAS original as already imported and would insert it again, so the gallery would end up holding both the edit and a fresh copy of the file it replaced. Only source_origin needs to change: it is what resolvePhotoStorageKey keys on, and every other consumer of external_relpath reads the two together and lets source_origin decide. The stale relpath on a managed row is inert for resolution and still correct as a dedupe key. Test updated to assert the value is kept rather than cleared. * fix(uploads): say when exiftool is missing instead of blaming the RAW A server without exiftool reported `No usable embedded preview in RAW file X.CR3: spawn exiftool ENOENT` for every RAW upload. The headline describes a corrupt photo; the actual cause is a package that was never installed, demoted to a trailing detail. It sends people hunting through their camera files. Hit while testing the Lightroom round-trip (#745): an export of RAW originals failed 11 times with that message, and the file was fine. RAW upload is the only feature that needs exiftool, so an install can be missing it indefinitely and only find out when someone uploads a CR3 — which makes the wording the whole diagnosis. ENOENT now produces a message naming the dependency and the install command for Debian/Alpine/macOS, and breaks out of the tag loop instead of spawning the same missing binary twice more to report the last failure as if it described the photo. A genuinely preview-less RAW still gets the original message. Verified both paths by making exiftool unreachable via PATH rather than mocking: missing tool and unreadable file now report differently. * fix(external): a delivered edit must win a relpath-fold collision Follow-up to keeping external_relpath on a replaced photo. Keeping it is what lets adminExternalMedia still dedupe the folder re-scan, but it also leaves the row inside externalRelpathFold's sweep — and that sweep does not merely rewrite paths, it DELETES collision losers via externalPhotoDedupe. The survivor was whichever row happened to be claimed first, which is iteration order. So a replaced photo — source_origin 'managed', holding the edit the photographer just delivered — could be deleted in favour of the untouched camera original sitting next to it on the share. Managed rows now claim first and therefore survive. The external row that loses is the recoverable one: it is still on the share and a re-scan re-imports it. The edit is not recoverable. Note this is deliberately NOT the "skip managed rows in the fold" shape suggested in review. Skipping would leave those rows holding a base-relative path while every other row moved to root-relative, so the scanner — which computes root-relative — would stop matching them and import the camera original again as a duplicate. That is the exact bug keeping external_relpath exists to prevent, reintroduced through a different door. Rebasing them and protecting them from deletion keeps both properties. |
||
|
Paul NothaftandPaul Nothaft
|
c18f54ede0 |
fix(images): respect EXIF orientation in thumbnails, heroes and previews (#1194)
* fix(images): respect EXIF orientation in thumbnails, heroes and previews (#1185) generateThumbnail, generateHeroImage and generatePreviewImage went straight from sharp(imagePath) to .resize(), so a photo whose Orientation tag is not 1 — routine for portrait shots on bodies that tag rather than rotate the sensor data — was resized from the raw frame and came out sideways. The same pipelines then call .withMetadata(false), stripping the tag from the output, so nothing downstream could correct it either. The download path already had this right: resizeToBox calls probe.rotate() for stills, which is why the same photo looked correct on download and rotated in the gallery. All three generators now do the same, guarded to stills for the reason resizeToBox already documents — .rotate() flattens a multi-frame source. The reporter also spotted the half that compounds it: photos.width/height were stored from sharp's metadata, which reports pixels as STORED, not as displayed. For orientation 5-8 those are swapped, so a portrait photo landed in the database as landscape and masonry/justified sized its tile with the wrong aspect ratio on top of the image being unrotated. A shared orientedDimensions() helper now does that conversion at all four capture sites — managed upload, background processing, external import and the dimension repair — so the stored numbers describe the rotated result the generators now produce. Existing rows keep their pre-rotation dimensions until the photo is reprocessed; the images themselves correct on the next thumbnail/preview regeneration. Tests fail on the unfixed generators — verified by reverting the rotate calls and the swap, which fails 4 of the 7. * fix(images): orient dimensions on every ingest path, and stop guarding rotate where it protects nothing (#1185) Review found the first cut covered four of eight dimension-capture sites. The filesystem watcher, the S3 auto-importer, the v1 upload API and replace-by-name all still persisted raw metadata.width/height, so an orientation 5-8 photo arriving that way got a correctly rotated thumbnail and a database row describing it as landscape — the same aspect-ratio mismatch this PR set out to remove, just on the paths I had not grepped. (I searched for `metadata.width` and the v1 route aliases it to `meta`.) The animated guard was also wrong in two of the three generators. generateThumbnail and generateHeroImage never pass `animated: true`, so they already flatten a multi-frame source to its first frame — skipping .rotate() there protected an animation that was being discarded anyway, while leaving the output in raw orientation against swapped stored dimensions. Both now rotate unconditionally. generatePreviewImage keeps the guard, because it genuinely does open animated sources as animated and .rotate() would flatten them. That leaves one corner unsolved rather than papered over: a multi-frame source that also carries an orientation tag keeps its raw orientation in the preview while the thumbnail and stored dimensions describe the rotated one. GIF has no EXIF and animated WebP effectively never sets it, so it is a real gap but not a common one, and closing it means rotating frame by frame rather than quietly dropping the animation. Documented at the guard. * fix(images): add a recompute mode so existing libraries get corrected too (#1185) The orientation fix only helped new photos. A row affected by the bug has BOTH dimensions stored — just in the raw order — so the repair job's NULL filter could never reach exactly the rows that needed it. Worse, once their thumbnails regenerated rotated, those rows went from consistently-wrong (sideways image in a matching tile) to inconsistent: correct image, wrong-shaped tile. `recompute` widens the candidate set to every image row. Opt-in, because it re-reads every original. It also has to deal with the consequence for faces. Detection runs against the preview and stores boxes in ORIGINAL pixel space, scaled by `photo.width / previewMeta.width` (faceProcessor.js:220-224) — so a photo whose stored dimensions change has face data recorded against a coordinate system that no longer exists, and the overlays crop the wrong region. Photos whose dimensions actually change are requeued for scanning; ones that were already correct are not, or a routine repair would rescan the whole library. Rows with face_status NULL are left alone so installs that never enabled the feature don't start scanning because of a dimension repair. Writing the test for that last rule caught a real bug in it: the candidate query never selected photos.width/height, so `photo.width` was undefined and every row compared as changed. Both columns are selected now. * Revert "fix(images): add a recompute mode so existing libraries get corrected too (#1185)" This reverts commit cb771d08. Review round 3 found five problems, all of them in this addition rather than in the orientation fix itself, and one of them an own-goal: requeueing face scanning makes processPhotoFaces call ensurePreviewImage, which returns the CACHED pre-fix preview when it is still a valid image — so the rescan reads unrotated pixels and scales those boxes by the newly corrected dimensions. That is worse than leaving the data alone. The rest need work this PR should not be carrying: the dimension repair reads originals through resolvePhotoFilePath and plain sharp, so it does nothing on an S3 install and rejects RAW/DNG; recompute pulls archived rows whose originals were deleted on archive; orientation 2, 3 and 4 change the pixels without changing width or height, so a dimension-delta test never notices them; and the dimension write and the face invalidation are not atomic, so a failure between them leaves a row that no retry will ever requeue. Split out so it can be designed and reviewed on its own. The orientation fix — .rotate() in the three generators and orientedDimensions() at all eight ingest sites — is unaffected and stays. * fix(images): the watermarked rendition needs orienting too (#1185) A fourth generator with the same bug, found while reviewing the backfill that builds on this. watermarkService composites and re-encodes through its own sharp pipeline with no .rotate(), and gallery.js serves photos.watermark_path ahead of the original when branding watermarking is on — so on a watermarked gallery the sideways image is precisely what a guest sees. Two details this needed beyond the .rotate() itself: metadata() is read from a separate, unrotated handle. .rotate() does not change what metadata() reports — a 400x200 source tagged orientation 6 still reads 400x200 — and every use of those numbers here is positioning: watermark scale, font size, composite extent. They have to be the DISPLAYED dimensions or the mark is placed against the wrong axis, so they go through orientedDimensions. The composite offsets are floored. getPositionCoordinates derives from the SVG's estimated text extent and returns fractional pixels; sharp rejects a non-integer offset and applyWatermark catches its own error and returns the image unwatermarked. Landing on a whole pixel was luck, and changing the dimensions it is computed from ran out of it — the test surfaced a real "Expected integer for left but received 92.8". --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
849a5807b7 |
fix(admin): make "Storage used" report storage used (#1164) (#1170)
* fix(admin): make "Storage used" report storage used (#1164) The tile summed photos.size_bytes — the catalogued size of the ORIGINALS, which has no relationship to the disk PicPeak runs on. In reference mode those files are never copied and sit on the NAS; duplicate rows counted the same file twice (#1162); and it ignored everything PicPeak genuinely does write locally: thumbnails, previews, hero renditions, watermarks and the per-event download cache. The reporter's tile read ~80 GB against 21 GB of real usage. Worse than the label: the same number drove the storage soft-limit warning bar and, via /storage/info, the recommended soft limit — so a reference-mode install got a disk-capacity recommendation computed from bytes that are not on the disk. - new localStorageUsage service walks the storage root and reports the total plus a breakdown. Walking rather than summing DB columns is the point: thumbnail/preview/hero rows record a key and never a byte count, and orphans from a deleted event or an interrupted import are real bytes. Symlinks are not followed, so a link into the media mount cannot put the NAS back in the total. Cached for 5 minutes, since the dashboard polls. - the dashboard tile and /storage/info now report that, with the catalogued figure kept and labelled as such next to it. A failed measurement reads as "unavailable" rather than substituting a number that means something else. On the local rig: 64.37 MB used against 15.75 MB catalogued, of which 27.9 MB is watermarks and 6.8 MB is download cache — none of which the old figure could see. Not addressed here: `.download-cache/all.zip` still has no TTL or size cap. It is now at least visible in the breakdown, which is what makes the case for capping it. * fix(admin): exclude the media share from local storage usage (#1164) External review found the walk could reintroduce the exact over-count it replaces. EXTERNAL_MEDIA_ROOT's compose default is `<storage>/external-media`, where the NAS is bind-mounted. That is a plain directory, not a symlink, so the symlink guard did not cover it and the walk descended into the share — putting every referenced original back into a figure whose whole purpose is to leave them out, and comparing NAS bytes against statfs() of the local disk. On the reference-mode installs this issue is about, that is the failure mode reappearing inside its own fix. The configured root is now skipped when it lies inside the storage root, and the result reports which path was excluded. A directory that merely shares the name is still counted, because those really are local bytes. Also from the review: - concurrent cold-cache callers now share one walk. /dashboard/stats, /storage/info and the sidebar are routinely requested together, and each was starting its own stat-per-file traversal of the whole library. - storage_partial is surfaced in the StorageInfo type and the sidebar tile, not just the dashboard and analytics cards. An unreadable subtree makes the total a floor, and a floor silently compared against a soft limit reads as "safely under". * fix(admin): do not report a disk walk on an S3 backend (#1164) Second review round. S3 installs were regressed. With STORAGE_BACKEND=s3 the originals, renditions, archives and download caches are objects in the bucket and STORAGE_PATH holds only incidental local files — so the walk reported near-zero and the soft-limit recommendation was derived from it. Those installs now keep the catalogued figure, which is the approximation they had before this PR, and the response says which measurement it is (`storage_measurement: 'disk' | 'catalog'`) so the UI labels it instead of implying a disk measurement that never happened. The Settings → Status storage card ignored storage_partial, formatting a lower bound as exact and deriving the limit percentage from it — so an unreadable subtree could read as safely under the limit. It now carries the same `+` marker as the sidebar and dashboard. * fix(admin): stop rendering an absent measurement as zero usage (#1164) Third review round, two findings. The analytics storage bar coerced a null measurement to 0, drawing an empty bar labelled "0% of limit" and suppressing the over-limit state — reading as plenty of room at exactly the moment nothing is known. It now shows the catalogued figure on S3, where that IS the available answer, and says "no measurement available" rather than inventing a percentage when there is none. /storage/info walked the filesystem before checking the backend and then threw the result away on S3. The sidebar polls that endpoint, so a migrated install still holding a large local tree paid a full stat-per-file traversal on every cold cache for nothing. Gated before the walk, as the dashboard route already was. * fix(admin): tell "no disk to measure" apart from "the measurement failed" (#1164) External review of the stable twin. Both were reported as `storage_measurement: 'catalog'`, so a failed local walk made the dashboard claim the objects live in S3. They are different things — one is a fact about the install, the other is a fault — and there is now an `unavailable` state for the second. The analytics percentage could reach the billions. `safeSoftLimit` fell back to `storageUsed || 1`, and on S3 that is null → 1, while the figure beside it came from `catalogedBytes`. An editor or viewer holds `analytics.view` but not `settings.view`, so `/storage/info` 403s for them and `storageInfo` is undefined — which is exactly when that fallback fires. It now falls back to the measured figure, and suppresses the percentage entirely when there is no real limit rather than dividing usage by itself and always reading 100%. Also lands the AnalyticsPage half of the previous round, which the commit message claimed but the commit did not contain — only its backend counterpart was staged. The stable twin has carried it since it was written, so this is the parity gap in the unusual direction. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
1366d6d14c |
fix(previews): preserve alpha and animation in the preview tier (#1171)
* fix(gallery): stop the lightbox loading originals to display a photo (#1166) The lightbox read `preview_url`, which the server only emits once an admin has flipped lightbox_preview_enabled — off by default. So a stock install fell straight through to `url`, the untouched original: a reporter measured 16.5 MB for a photo whose preview is 345 KB. The lightbox renders its neighbours too, so opening one photo pulled three originals. `slideshow_url` is the same /preview/:id URL, watermark query included, and has been emitted unconditionally for images since #1015 — the slideshow never had a fallback worth taking. Preferring it fixes every existing install with no migration and no admin action, and `url` still backstops videos, where both derivative URLs are null. Verified on the local rig with the toggle off, so the photos API returns preview_url: null exactly as filed. Opening one photo: before GET /photo/82, /photo/81, /photo/21 (3 originals) after GET /preview/82?w=1280, /preview/81, /preview/21 397 KB -> 23 KB per image on that gallery's test photos. The toggle no longer decides whether the lightbox uses previews, so its copy said something untrue; it now describes what it still does, which is pre-generate rather than wait for the first guest to open a photo. Updated in en/de/fr/sl, the locales that carry those keys. * fix(gallery): cover the layouts the lightbox fix missed (#1166) External review found the fix was incomplete, and the review of it found one more. Premium galleries were untouched. PhotoGridWithLayouts returns early for gallery-premium, which builds its own yet-another-react-lightbox slides with `src: photo.url` — so those galleries kept pulling full originals and the reported bandwidth problem remained. They now use lightboxImageUrl for the display source; `download` deliberately stays on photo.url, because what a guest saves must be the original. The Story layout was worse, and neither the issue nor the review caught it: StoryPhotoCard rendered the full original as its GRID TILE, at object-cover in a small card. That is the one place where "hundreds of megabytes for a gallery" was literally true. It now uses the per-device thumbnail tier like PhotoCard, and its PhotoSwipe source uses the preview tier. Animated GIFs keep the original. generatePreviewImage always encodes JPEG, so routing an animated source through the preview tier would have replaced the animation with its first frame — a regression the toggle-off default never had. Animated WebP has the same problem and cannot be distinguished by MIME alone; that needs the backend to report it (Sharp's `metadata.pages > 1`) and is left rather than costing every static-WebP gallery the bandwidth fix. The settings copy claimed too much. "Pre-generate lightbox previews" does not generate anything on save — it unlocks the regenerate button and keeps preview_url emitted. Reworded to say that, in en/de/fr/sl. Not changed: the review's P1 said this bypassed the secure-image route on enhanced/maximum galleries. It does not. AuthenticatedImage collects requiresToken and secureUrlTemplate into an explicitly-voided unusedProps and never substitutes {{token}}, so on those protection levels photo.url was a literal `.../secure/82/{{token}}` that returns 400 — the lightbox was falling back to the 300px thumbnail, not to a protected image. Verified against a live maximum-protection gallery. Codex withdrew the finding on that evidence. * fix(gallery): keep premium downloads working and story framing intact (#1166) Second review round, three findings — two of them regressions this PR introduced. Premium Download became a no-op. handleDownloadFromLightbox recovered the photo with `filteredPhotos.find(p => p.url === slide.src)`, and slide.src is a derivative now, so the lookup found nothing and the button silently did nothing. The slide carries the photo id and the handler resolves by that; what Download hands over is still the original. Story cards were reframed. thumbnail_fit is seeded to 'cover' on every install, so thumbnails are square centre-crops — and story cards are not square (400x500 in the carousel, fixed-height in the desktop grid), so the card's own object-cover cropped them a second time and every photo shifted. They now use the preview tier, which is fit:'inside' and therefore the whole frame: the card looks exactly as it did before, without pulling an original. APNG joins the animated-format guard. It declares image/apng and the preview route would serve a static frame. Animated WebP still cannot be detected from MIME and remains the documented gap. * fix(gallery): keep PNG on the original, alpha and all (#1166) Third review round. generatePreviewImage encodes JPEG, which drops ALPHA as well as animation — a transparent PNG came back flattened against a solid background. And an APNG is normally reported as image/png, so the image/apng check alone missed the common upload path. PNG now stays on the original: it is where transparency is the norm, and rare enough in an event gallery that the bandwidth given up is small. Animated or alpha WebP still cannot be detected from MIME and remains the documented gap; it needs the backend to report Sharp's `pages`/`hasAlpha`. Two further findings are acknowledged and deferred rather than fixed here: - Story cards now request /preview on mount, so a cold gallery generates its previews in one burst. That is a new CPU cost, not a regression — those cards previously fetched full ORIGINALS on mount, which is strictly worse. Doing it properly means viewport-gating AuthenticatedImage, which is a change to a component every gallery surface uses and belongs in its own PR. - The premium layout memoizes slide URLs, so rotating the device before opening the lightbox can leave a photo on the tier chosen for the old geometry. The result is a slightly undersized image, and the fix is a resize subscription this PR does not otherwise need. * fix(gallery): load Story images on approach, and give the hero its own tier (#1166) Every card in a Story gallery mounts at page load — `whileInView` gates the animation, not the render — and AuthenticatedImage fetches from an effect on mount, so all of them requested at once. That was tolerable while they pointed at photo.url, because nothing was generated; pointing them at the preview tier meant a gallery with cold previews would Sharp-decode every original in one burst. The image now waits until the card is within 200px of the viewport, using framer-motion's useInView — the same observer the entrance animation already relies on — with `once` so a card never unloads on scroll-away. Verified on a 62-photo Story gallery: 3 images fetched at load, growing to 15 as you scroll, where all 62 would have fired before. While confirming that, the hero turned out to be doing the same thing the cards were. StoryHero rendered photo.url as a full-bleed object-cover background — a full original on the critical path for first paint of every Story gallery — when hero_url exists for exactly this and is a 1920x1080 cover crop emitted unconditionally for every photo (gallery.js:1139). That gallery now issues no /photo/ request at all: hero_url for the hero, the preview tier for the cards, and only as they come into range. * fix(previews): preserve alpha and animation in the preview tier Follow-up to #1166, which had to bypass the preview tier for GIF, APNG and PNG to avoid a visible regression. This removes the cause. generatePreviewImage encoded JPEG unconditionally. JPEG has no alpha channel and no second frame, so a transparent PNG came back flattened onto a solid background and an animated GIF came back as its first frame — for every consumer of this tier, not just the lightbox: the slideshow (#1015), admin previews, and the face avatars that read it as a whole-frame rendition. It was only invisible by default because the lightbox served originals. Sources with alpha, or more than one page, are now encoded as WebP, which carries both and is still far smaller than the original. Ordinary photos stay JPEG — the common path pays nothing. Two things had to move with it: - The output extension now matches what was written. A PNG source previously produced `preview_foo.png` holding JPEG bytes; harmless while the route hard-coded image/jpeg, wrong once the encoding varies. Existing keys keep working — they are still JPEG and still served as such. - The preview route derives Content-Type from the key. With `nosniff` set, mislabelling would show a broken image rather than being silently corrected. The watermark branch re-encodes to JPEG, so it labels itself explicitly; preserving animation through the watermark compositor is a separate problem. The frontend guess-by-MIME goes away entirely — including the case it could never get right, since a still and an animated WebP declare the same type. Verified on the local rig: a transparent PNG round-trips as `Content-Type: image/webp`, `hasAlpha: true`, 8.3 KB; an ordinary photo still serves `image/jpeg` from a `.jpg` key. * fix(previews): retire the legacy preview keys, and stop mislabelling watermarked ones External review of the stable twin found two defects, both on this branch too. Legacy keys collide with the new naming. The old generator kept the SOURCE basename verbatim while always writing JPEG, so a `.webp` upload produced `previews/preview_shot.webp` holding a JPEG. My PR body claimed "pre-existing keys have no .webp suffix and are JPEG" — that was simply wrong. The route now derives Content-Type from the key and the response carries nosniff, so every photo uploaded as WebP would have rendered as a broken image in the lightbox. Legacy `.png` keys are wrong the other way: flattened JPEGs of what may have been transparent sources, which isPreviewValid would have let stand forever. Migration 188 clears photos.preview_path outright — all of it, not just the suspicious extensions, because a `.jpg` key can equally be a flattened rendition and nothing in the key says so. Previews regenerate lazily on next view under the new encoder, so the cost is one regeneration per photo actually viewed. Storage is untouched, as elsewhere. The watermark branch mislabelled its output. applyWatermark PRESERVES the source format (watermarkService.js:200-211: png stays png, webp stays webp), and its input is the preview — so the output already matches the key the header was derived from. Forcing image/jpeg mislabelled every watermarked WebP preview, and nosniff means the browser would not correct it. The override is gone; the animation loss through the compositor is documented where it happens. * fix(gallery): make the Story hero fix actually work on external galleries (#1166) External review of the stable twin, both applying here too. hero_url was inert for external media. ensureHeroImage only ever called resolvePhotoStorageKey, which returns null for external/reference photos by design — and that null was handed straight to withLocalCopy, which throws, so the hero route caught it and redirected to the full ORIGINAL. #1078 fixed exactly this shape for ensurePreviewImage and nobody carried it across. It stayed invisible until this PR pointed the Story hero at hero_url: on a managed gallery that is a real saving, on a reference-mode gallery it quietly changed nothing. ensureHeroImage now has the same external branch ensurePreviewImage does — direct fs read, per-photo output basename — and returns null instead of throwing for a reference-mode row with no source_origin. The format bypass trusted mime_type, which is not trustworthy here. Migration 039 backfilled every pre-existing photo to image/jpeg regardless of what it was, and adminExternalMedia inserts rows with no mime_type at all — so a mislabelled PNG sailed past the guard and came back flattened. It now checks the filename extension as well. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
77953c15c1 |
fix(gallery): stop the lightbox loading originals to display a photo (#1166) (#1169)
* fix(gallery): stop the lightbox loading originals to display a photo (#1166) The lightbox read `preview_url`, which the server only emits once an admin has flipped lightbox_preview_enabled — off by default. So a stock install fell straight through to `url`, the untouched original: a reporter measured 16.5 MB for a photo whose preview is 345 KB. The lightbox renders its neighbours too, so opening one photo pulled three originals. `slideshow_url` is the same /preview/:id URL, watermark query included, and has been emitted unconditionally for images since #1015 — the slideshow never had a fallback worth taking. Preferring it fixes every existing install with no migration and no admin action, and `url` still backstops videos, where both derivative URLs are null. Verified on the local rig with the toggle off, so the photos API returns preview_url: null exactly as filed. Opening one photo: before GET /photo/82, /photo/81, /photo/21 (3 originals) after GET /preview/82?w=1280, /preview/81, /preview/21 397 KB -> 23 KB per image on that gallery's test photos. The toggle no longer decides whether the lightbox uses previews, so its copy said something untrue; it now describes what it still does, which is pre-generate rather than wait for the first guest to open a photo. Updated in en/de/fr/sl, the locales that carry those keys. * fix(gallery): cover the layouts the lightbox fix missed (#1166) External review found the fix was incomplete, and the review of it found one more. Premium galleries were untouched. PhotoGridWithLayouts returns early for gallery-premium, which builds its own yet-another-react-lightbox slides with `src: photo.url` — so those galleries kept pulling full originals and the reported bandwidth problem remained. They now use lightboxImageUrl for the display source; `download` deliberately stays on photo.url, because what a guest saves must be the original. The Story layout was worse, and neither the issue nor the review caught it: StoryPhotoCard rendered the full original as its GRID TILE, at object-cover in a small card. That is the one place where "hundreds of megabytes for a gallery" was literally true. It now uses the per-device thumbnail tier like PhotoCard, and its PhotoSwipe source uses the preview tier. Animated GIFs keep the original. generatePreviewImage always encodes JPEG, so routing an animated source through the preview tier would have replaced the animation with its first frame — a regression the toggle-off default never had. Animated WebP has the same problem and cannot be distinguished by MIME alone; that needs the backend to report it (Sharp's `metadata.pages > 1`) and is left rather than costing every static-WebP gallery the bandwidth fix. The settings copy claimed too much. "Pre-generate lightbox previews" does not generate anything on save — it unlocks the regenerate button and keeps preview_url emitted. Reworded to say that, in en/de/fr/sl. Not changed: the review's P1 said this bypassed the secure-image route on enhanced/maximum galleries. It does not. AuthenticatedImage collects requiresToken and secureUrlTemplate into an explicitly-voided unusedProps and never substitutes {{token}}, so on those protection levels photo.url was a literal `.../secure/82/{{token}}` that returns 400 — the lightbox was falling back to the 300px thumbnail, not to a protected image. Verified against a live maximum-protection gallery. Codex withdrew the finding on that evidence. * fix(gallery): keep premium downloads working and story framing intact (#1166) Second review round, three findings — two of them regressions this PR introduced. Premium Download became a no-op. handleDownloadFromLightbox recovered the photo with `filteredPhotos.find(p => p.url === slide.src)`, and slide.src is a derivative now, so the lookup found nothing and the button silently did nothing. The slide carries the photo id and the handler resolves by that; what Download hands over is still the original. Story cards were reframed. thumbnail_fit is seeded to 'cover' on every install, so thumbnails are square centre-crops — and story cards are not square (400x500 in the carousel, fixed-height in the desktop grid), so the card's own object-cover cropped them a second time and every photo shifted. They now use the preview tier, which is fit:'inside' and therefore the whole frame: the card looks exactly as it did before, without pulling an original. APNG joins the animated-format guard. It declares image/apng and the preview route would serve a static frame. Animated WebP still cannot be detected from MIME and remains the documented gap. * fix(gallery): keep PNG on the original, alpha and all (#1166) Third review round. generatePreviewImage encodes JPEG, which drops ALPHA as well as animation — a transparent PNG came back flattened against a solid background. And an APNG is normally reported as image/png, so the image/apng check alone missed the common upload path. PNG now stays on the original: it is where transparency is the norm, and rare enough in an event gallery that the bandwidth given up is small. Animated or alpha WebP still cannot be detected from MIME and remains the documented gap; it needs the backend to report Sharp's `pages`/`hasAlpha`. Two further findings are acknowledged and deferred rather than fixed here: - Story cards now request /preview on mount, so a cold gallery generates its previews in one burst. That is a new CPU cost, not a regression — those cards previously fetched full ORIGINALS on mount, which is strictly worse. Doing it properly means viewport-gating AuthenticatedImage, which is a change to a component every gallery surface uses and belongs in its own PR. - The premium layout memoizes slide URLs, so rotating the device before opening the lightbox can leave a photo on the tier chosen for the old geometry. The result is a slightly undersized image, and the fix is a resize subscription this PR does not otherwise need. * fix(gallery): load Story images on approach, and give the hero its own tier (#1166) Every card in a Story gallery mounts at page load — `whileInView` gates the animation, not the render — and AuthenticatedImage fetches from an effect on mount, so all of them requested at once. That was tolerable while they pointed at photo.url, because nothing was generated; pointing them at the preview tier meant a gallery with cold previews would Sharp-decode every original in one burst. The image now waits until the card is within 200px of the viewport, using framer-motion's useInView — the same observer the entrance animation already relies on — with `once` so a card never unloads on scroll-away. Verified on a 62-photo Story gallery: 3 images fetched at load, growing to 15 as you scroll, where all 62 would have fired before. While confirming that, the hero turned out to be doing the same thing the cards were. StoryHero rendered photo.url as a full-bleed object-cover background — a full original on the critical path for first paint of every Story gallery — when hero_url exists for exactly this and is a 1920x1080 cover crop emitted unconditionally for every photo (gallery.js:1139). That gallery now issues no /photo/ request at all: hero_url for the hero, the preview tier for the cards, and only as they come into range. * fix(gallery): make the Story hero fix actually work on external galleries (#1166) External review of the stable twin, both applying here too. hero_url was inert for external media. ensureHeroImage only ever called resolvePhotoStorageKey, which returns null for external/reference photos by design — and that null was handed straight to withLocalCopy, which throws, so the hero route caught it and redirected to the full ORIGINAL. #1078 fixed exactly this shape for ensurePreviewImage and nobody carried it across. It stayed invisible until this PR pointed the Story hero at hero_url: on a managed gallery that is a real saving, on a reference-mode gallery it quietly changed nothing. ensureHeroImage now has the same external branch ensurePreviewImage does — direct fs read, per-photo output basename — and returns null instead of throwing for a reference-mode row with no source_origin. The format bypass trusted mime_type, which is not trustworthy here. Migration 039 backfilled every pre-existing photo to image/jpeg regardless of what it was, and adminExternalMedia inserts rows with no mime_type at all — so a mislabelled PNG sailed past the guard and came back flattened. It now checks the filename extension as well. * test(gallery): the hero fixture follows the root-relative relpath contract (#1166) external_relpath has been resolved from EXTERNAL_MEDIA_ROOT rather than from event.external_path since #1163 landed. This fixture still carried the base-relative form — its own comment noted the change was 'a separate stack' — so the two tests stopped resolving and ensureHeroImage returned null the moment that stack merged. The production path was never affected. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
a7b74bcd87 |
fix(external-media): store external paths from the media root (#1163) (#1168)
* fix(external-media): store external paths from the media root (#1163) Importing a second folder into an event silently invalidated every photo already in it. photos.external_relpath was stored relative to events.external_path, and every import overwrites that column — so the older rows were rebased onto the new folder and their originals resolved to paths that do not exist. Nothing errored, and the grid still looked intact: thumbnails are written to local storage during the import while the base path is still correct. Only what needs the original broke — preview generation, the lightbox, downloads — which presents as a gallery that looks slow rather than one that is broken. The reporter had 7547 of 8004 rows pointing into the void and spent a while chasing it as a CPU problem. - external_relpath is now relative to EXTERNAL_MEDIA_ROOT, so a row is self-describing and nothing an admin does to the event afterwards can move an already-imported photo. - migration 187 folds each event's base path into its rows. Where the current resolution is missing on disk it walks up the base path for an ancestor under which the file IS there — the already-rebased case — and where it finds nothing it leaves the row resolving exactly where it resolves today. Skipped entirely when the media root is unmounted, since every file looks missing then. - the fold also runs after a .picpeak restore: knex_migrations is excluded from the archive, so a pre-#1163 backup would otherwise land base-relative rows on a migrated instance. - drops the duplicate-leaf-segment guess in photoResolver. It papered over this same double-prefixing and actively corrupts a root-relative path whose first segment legitimately repeats (base 'Trip', row 'Trip/x.jpg'). * fix(external-media): verify provenance and fold atomically (#1163) External review found four real defects in the fold. Repair could adopt the wrong file. Existence alone was accepted as proof that an ancestor candidate was the row's original — so a row whose file an admin simply deleted would adopt any same-named file one directory up (base `Trip/Sub`, relpath `photo.jpg`, an unrelated `Trip/photo.jpg`), and downloads would then serve a different photo. Worse than a dead link. An ancestor must now also match photos.size_bytes, which the import recorded from the very file the row describes; rows carrying no size are never repaired from an ancestor. The CURRENT base is still accepted on existence alone, because nothing is being inferred there — that is where the row already resolves. The fold was not atomic. Every UPDATE committed independently and the marker came last, so a process killed mid-fold left converted and unconverted rows with no marker — and the next run folded the converted ones a second time, putting every original one directory deeper with no undo. Probing is now a read-only first phase (so a slow cold NAS does not hold a write transaction open), and every rewrite plus the marker commit together. Failed rewrites certified a partial conversion. The per-row catch counted any error as a collision, carried on, and wrote the marker anyway — leaving that row in the old format for a resolver that now reads it differently. It also could not tell a genuine duplicate from a SQLite lock or I/O fault. Target collisions are now resolved in the planning phase, where they can be identified honestly, and a write that fails rolls the whole fold back. Restore ordering. The fold ran after the face requeue, with the worker live — so a worker could claim an external row while it was still base-relative, resolve it against the wrong path, and burn it to 'failed', a state only an explicit Re-scan clears. The fold now runs first, for the same reason the requeue already sat after restoreFiles. * fix(external-media): close the fold's remaining stranding paths (#1163) Second review round, three findings. A collision loser was left stranded. When an event imported one file through both `Trip` and `Trip/Sub`, two rows folded to the same path and the loser was skipped — keeping a base-relative value that the root-only resolver then reads as `<root>/<relpath>`, permanently wrong, with the marker claiming conversion was complete. It is a duplicate by construction, so it now goes through migration 186's deleteDuplicatePhotos, which reparents its feedback and marks and reconciles the face clusters instead of orphaning them. This branch is rebased onto #1162 for that helper. The other restore path had the same face-ordering bug. restoreService queued face scans in step 6, before step 7c runs pending migrations — so a pre-187 full or database restore handed the live worker rows whose paths were still event-relative, and it burned them to 'failed', a state the later fold does not clear. The requeue now happens after the migrations, where the files already are. A failed conversion was reported as a clean restore. The fold is transactional, so a failure leaves every external path in the old format under a resolver that reads from the media root — every original unreachable. It was logged as a warning and the restore returned success. It now returns externalPathsConverted/externalPathError, and suppresses the face requeue, which would otherwise mark those photos failed on top. * fix(external-media): make the fold safe against its own intermediate states (#1163) Third review round, four findings. A one-pass rewrite could collide with itself. Every FINAL path is distinct, but a final value can equal another row's CURRENT one — `photo.jpg` repairing to `Trip/photo.jpg` while the row already holding `Trip/photo.jpg` folds deeper — so the update violated migration 186's unique index halfway through. On Postgres that surfaces as 23505, which run-migrations-safe.js mistakes for "schema already exists" and records 187 as applied after the rollback, leaving every path unconverted with nothing to retry. Rows now park on a per-row staging value first, and migration 187 re-throws without the driver's code so the runner cannot misread it. The bulk update targeted rows the plan never saw. Phase 1 probes outside the transaction and can run for minutes; an import finishing in that window inserts an already root-relative row, and `where event_id` prefixed it again with the stale base. It now updates by the ids phase 1 captured. The restore UI never showed a conversion failure. The API carried externalPathsConverted, but PicpeakBackupCard neither declared nor read it and showed a green success either way — so an admin whose external originals were all unreachable was told the restore worked. restoreService requeued faces even when the migrations failed. The step 7c catch is deliberately non-fatal, so a pre-187 backup whose fold never ran still handed the live worker event-relative paths to burn to 'failed'. * fix(external-media): the fold's staging value must be storable on Postgres (#1163) External review of the stable twin caught this, and it was on both branches. The two-pass rewrite parks each row on a temporary value, and that value was written with a leading NUL. SQLite stores NUL in TEXT without complaint; Postgres rejects it outright — "invalid byte sequence for encoding UTF8: 0x00" — so migration 187 rolled back on exactly the installs that need the two-pass repair, and only on the engine most of them run. Restores hit the same wall and reported the conversion as failed. The prefix is ordinary text now. It still cannot collide with a real relative path and is still obviously wrong if a crash leaves one behind. Adds a gated Postgres test alongside the existing picpeakRestorePg one, because a SQLite-only suite structurally cannot catch this class: restoring the NUL makes exactly the two-pass repair case fail with that error, and nothing else. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul NothaftandPaul Nothaft
|
06da1b9f7e |
fix(external-media): one row per external file per event (#1162) (#1167)
* fix(external-media): one row per external file per event (#1162) Two overlapping import-external runs against the same event inserted every file twice. The route checked for an existing external_relpath and then inserted, with an fs.stat and a sharp().metadata() read sitting in between — a window wide enough for both runs to see "not there". A reporter's event held 8004 rows for 6012 distinct paths. Nothing at the storage layer stopped it: migration 041 created only a NON-unique (event_id, source_origin) index. - migration 186 removes the duplicates that already exist and adds a partial unique index on (event_id, external_relpath). The survivor is the lowest id that has a thumbnail, so a half-finished import does not cost a grid tile, and hero references are repointed first because the FK is SET NULL. - the route treats a unique violation as a skip and carries on, so a second writer this process cannot see (another replica) converges instead of duplicating or 500ing. - a second import while one is already running now gets a 409 rather than walking the whole tree to have every insert bounce. The duplicates' thumbnail files are left behind as unreferenced bytes — a migration is the wrong place to reach into storage, which may be S3. * fix(external-media): keep dependent rows and legacy restores intact (#1162) External review found two real defects in the dedupe half of this fix. Dangling rows on SQLite. Every FK into photos declares ON DELETE CASCADE, but PicPeak never sets `PRAGMA foreign_keys = ON` — the codebase says so where it deletes an event (adminEvents/helpers.js:245) — so on every SQLite install the cascade is inert and deleting a duplicate photo left its face embeddings, guest feedback and admin marks behind, pointing at an id that no longer exists. Biometric data outliving its photo is exactly the invariant the event delete goes out of its way to hold. Dependents are now handled explicitly, and moved rather than discarded where they can be: the duplicates were separate tiles in the grid, so a guest's comment or an admin's rating could legitimately be on either, and dropping it inside a fix for silent data loss would be its own bug. Where the target already holds an equivalent row — the same guest's like, the same admin's mark, the same transfer's entry — the loser is dropped, because those tables mean one row per (photo, actor). photo_faces is the deliberate exception: both rows were scanned, so moving would duplicate every embedding and split the person clusters built from them. Legacy restores. Suspending FK enforcement does not suspend a UNIQUE index on either engine, so a .picpeak backup taken before migration 186 — carrying exactly the duplicates it removes — would hit the new index mid-batchInsert and roll the whole restore back, after every table had already been emptied. The restore now drops the index for the load and rebuilds it after running the same dedupe. Also: a failed CREATE INDEX is no longer swallowed. Recording the migration as applied without it leaves the install permanently racy, with nothing to trigger a retry. The shared work moves to services/externalPhotoDedupe.js, which the migration and the restore both call. * fix(external-media): reconcile derived state around the dedupe (#1162) Second review round, four more real findings. The index throw did not actually stop anything. run-migrations-safe.js treats 23505 as "schema already exists" and marks the migration applied (run-migrations-safe.js:138) — and a CREATE UNIQUE INDEX that finds duplicate rows raises exactly 23505 on Postgres. A replica inserting one between the dedupe and the index lock is a real rolling-deploy shape, and the outcome was the thing the throw was added to prevent. The index is now verified against the catalog afterwards, and failure raises a code-less error the runner cannot mistake for idempotence. Two people sharing a device were treated as one. photo_feedback carries both guest_identifier (per device) and guest_id (per person, migration 078), and feedbackService scopes by guest_id when present. Keying equivalence on the identifier alone deleted one of two different people's ratings. It now uses the same COALESCE rule the service does. Deleting faces raw left ghost people. event_people counts and centroids are derived from the photo_faces rows being removed, and #1132's separation snapshots hold a copy of each side's centroid — which is why faceProcessor exposes purgePhotoFaces and says it is "called from every photo-deletion path". The dedupe now goes through it. Reparenting feedback left the survivor's totals stale. photos carries denormalized feedback_count / like_count / average_rating / favorite_count and the later reaction and colour counts, so a survivor that now owns feedback kept rendering zero. updatePhotoFeedbackStats takes a trx so the dedupe can recompute on its own connection. Also: the equivalence-key delimiter was a literal NUL byte, which made git classify the whole file as binary and hide its diff. Escaped. * fix(external-media): stop the dedupe discarding half-states (#1162) Third review round. Five findings, four applied. - is_hidden joins the feedback equivalence key. feedbackService lets a moderator-hidden row coexist with the guest's visible replacement and counts only the visible one, so ignoring it deleted the visible row as redundant. - admin marks merge instead of dropping. rating and color_label are written independently, so the same admin can have rated one tile and coloured the other; the loser now hands over any field the winner has no value for. - a survivor that loses the only completed scan is requeued. Otherwise the purge takes the sole embeddings and nothing re-queues it — the photo just silently stops having a face. - view_count and download_count are carried over. Those are real interactions recorded per row, and dropping them quietly lowered the engagement the admin grid shows. Not applied: repointing a category hero can in principle land on a survivor in another category. It needs the two duplicate rows to have been re-categorised apart after the racing import, and the result is a cosmetic hero mismatch that the admin category routes already guard on write. Not worth the extra branch in a data migration. * fix(external-media): invalidate the download zip when duplicates are removed (#1162) External review of the stable twin. Applies to both branches. The pre-built "download everything" archive still contained the duplicate rows the dedupe had just deleted, so guests kept receiving them until something else happened to invalidate it. Every ordinary photo-deletion path calls downloadZipService.invalidate for exactly this reason. The columns are cleared rather than the service being called: that service carries debounce timers and a regeneration queue, which is not something a migration should start. getZipInfo already treats a cleared record as a cache miss and rebuilds on the next request, so this is the durable half of what invalidate does. The stale object is left in storage for the same reason the duplicates' thumbnails are — a migration is the wrong place to reach into a backend that may be S3. --------- Co-authored-by: Paul Nothaft <[email protected]> |
||
|
Paul Nothaft
|
00b20b2d72 |
fix(gallery): guest filters respect show_feedback_to_guests, and marks survive a mid-write clear (#1147)
Two follow-ups from the review of #1137. Filters were a second way to read hidden feedback. Every token on /photos is an OR of two halves: what THIS viewer marked, and what ANYONE marked. The response fields built from the second half — like_count, comment_count, color_label_count — are all gated on show_feedback_to_guests. The filter was not, so with the setting off a guest could still send ?filter=liked and get back exactly the photos other people liked, across all five tokens. Reachable by a direct API caller holding a gallery token; the frontend never sends filter to this endpoint. The half it left standing was also the wrong half. It read guest_identifier from the guest_id QUERY PARAMETER, which never matched anything — the frontend invents that string in localStorage and never sends it when submitting feedback, while submissions store generateGuestIdentifier(req). So gating the aggregate would have emptied these filters rather than narrowing them to 'mine', and accepting a caller-supplied identifier was a way back through the gate. Resolved from the request now, hidden rows excluded to match what the viewer can see. A mark whose row is cleared mid-write lost its value. #1137 fixed two calls both writing; this is one clearing while another sets. The clear empties the row, the row is deleted for being empty, and the setter's update matches nothing — the caller told 'no mark'. A zero-row update now reports itself and the caller re-reads, bounded at three passes, throwing rather than reporting a success that did not happen. Merged with admin privileges: the author cannot self-approve. |
||
|
Luca
|
e2844d1909 |
feat(gallery): colour labels for client proofing, and one global default per feedback type (#1044) (#1137)
Colour labels for client proofing, plus the photographer's own stars and colours in the admin grid. - Guest colour labels alongside likes/reactions, opt-in per event (defaults off so live galleries do not change mid-proofing), with 'colors' and 'lightroom' keybind schemes. - One global default per feedback type, replacing the per-type scatter. - Admin marks live in their own table (photo_admin_marks) so they can never reach a guest-facing surface. - XMP export prefers a real label, keeping the rating-derived mapping as a fallback. Review: concurrent-write loss on the mark update path, migration index idempotency and error classification all fixed in 7139bcae; migrations renumbered to 182/183 in 8fecdfae after 180/181 were taken on main. Merged with admin privileges: bypass-size-gate is a required check that fails on size alone for review-bypass authors and never re-evaluates on review, which is its designed behaviour once a maintainer has approved. |
||
|
Paul Nothaft
|
97d92f8428 |
fix(thumbnails): regenerate external photos instead of dropping their tiers (#1129)
POST /admin/thumbnails/regenerate resolved every source as storage/events/active/<photo.path> and fs.access'd it. External and reference rows are not there, so every one failed and was counted as an error — and because the tier deletion runs first, the endpoint dropped every ?w= tier and rebuilt nothing, leaving the library worse than before it ran. The UI reported success either way. Now routed through ensureThumbnail, which resolves both source kinds, uses the per-photo ext<id>_ output name, and writes thumbnail_path back itself. Review rounds also removed both destructive deletes in generateThumbnail: the pre-delete ran before sharp opened the source, so an unreadable source left the previous rendition gone and the database pointing at it — across a bulk run, the whole gallery. Neither delete was needed, since put stages to a temp file and renames atomically and is the last statement in the try. Videos are filtered out, and the superseded rendition is removed only when the storage key actually moved, compared through the same canonicalisation the backends apply so a legacy backslash path is not mistaken for a different object. Reported by @BraynArts, who also identified the fix. |
