5db0a76cce94de03f86295ba2bd6ba526661d16d
131 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Paul Nothaft
|
98d25601b4 |
fix(gallery): cap how many cached zips rebuild at once in the background (#1418)
invalidateAll() invalidates every event holding a cached zip, and each invalidate() arms its own debounce timer in the same tick — so they all fired together and every one of them started building at once. Each build opens its own storage reads, so flipping a global setting across 25 events was enough to exhaust the S3 agent pool and stall uploads, thumbnails and gallery reads until the burst drained. Nothing capped it. Background rebuilds now run two at a time. The cap is deliberately only on that path: a foreground generateZip — a guest actually waiting on a download — is never queued behind a settings-change burst, which would trade one stall for another. stop() drains anything parked for a slot, so shutdown cannot hang on a queue that will never move. This is the second half of the problem. The first half — an individual build opening one unbounded read per photo — is the storage-read guard that already landed for the guest download routes and the job builder; the cached-zip builder's own copy is still in an open PR. Relates to issue 1399 Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Trung-Tin Pham
|
f094cc06a7 |
fix(gallery): stop the pre-zip build leaking storage reads (#1402)
Building the download-all archive opened one storage read per photo and handed each stream to archiver, which uses them one at a time. Every read past the one being written parked an S3 socket with a full receive buffer, and both early exits walked away from all of them: archiver's abort() does not touch its source streams, and the error path only removed the temp directory. Nothing else reclaims those sockets either, because the SDK arms its socket timeout on a 3s delay and clears it as soon as the response headers land. On a live server 43 of the 50 pooled sockets ended up stuck for days and photo uploads stopped completing, with nothing logged. The common way in is an ordinary upload. invalidate() runs on every photo upload, delete and bulk edit, and it aborted an in-flight build. Track every open read and destroy them on every exit path, cancel the in-flight build from invalidate() rather than waiting for the loop to reach its next version check, and cap reads in flight at 2. A build of 120 photos peaked at 50 concurrent GETs before, the whole agent pool, which starved uploads and thumbnails on its own. Only S3 deployments are affected. Local filesystem installs take the archive.file() branch and open no sockets. |
||
|
Paul Nothaft
|
4622478e44 |
fix(upload): stop buffering a chunk body before anything checks its size (#1406)
* fix(upload): stop buffering a chunk body before anything checks its size The chunk route drained the whole request into an array and concatenated it before calling uploadChunk — which is where every check lives. So a 300MB body sent against an unknown upload id was read in full, cost ~300MB of heap, and was only then answered with an error. The per-file cap was real but applied after the damage, and nothing looked at Content-Length at all. uploadChunk now takes the request stream itself and consumes nothing until the upload id, the chunk index and the declared Content-Length have all been checked against the remaining allowance. A body that clears those is streamed straight to the chunk file under a hard byte cap, so a sender that lies about its length — or sends none, the Transfer-Encoding: chunked case — is cut off at the allowance instead of being read to the end. A Buffer is still accepted, so the existing callers and chunkedUploadSizeCap tests are untouched. Worth noting the old code silently did nothing when handed a stream: fs.promises.writeFile accepts an async iterable, so the body was written while `chunkData.length` was undefined and the cap comparison was NaN > max, i.e. always false. Reachable only for an authenticated admin holding photos.upload on an event they own, and only once the CSRF gate lets application/octet-stream through. Relates to issue 1403 * fix(upload): harden the chunk stream against abort, retry and cap failures Three failure paths the streaming rewrite introduced, all found by external review. None existed in the buffered version: the async iterator it replaced rejected a dead request on its own, and never opened the chunk file until it already held the whole body. - An already-destroyed request hung the call forever. If the client hangs up while auth and ownership are awaiting the database, pipe() emits neither `end` nor `error`, so the promise never settled and the write descriptor stayed open. Checked up front now, alongside `aborted` and a `close` without `readableEnded` for a body cut short mid-flight. - A failed re-send destroyed the chunk it was replacing. createWriteStream truncates on open, so re-sending an index and then failing left receivedChunks and chunkSizes still claiming the old copy: status reported 100% and completeUpload died on ENOENT. Chunks are staged through a sibling .part file and renamed only on success. - Tripping the cap stopped the 413 from reaching the client. `source` is the IncomingMessage, so destroying it destroyed the socket under the response and the client saw a connection reset instead of the size-limit JSON. The read is paused instead, which is all the cap needs. Relates to issue 1403 * fix(upload): isolate chunk staging per attempt and close before cleanup Two races found by a second review round, both in the staging logic added by the previous commit. - Two in-flight sends of the same chunk index shared one `.part` path, so whichever renamed first published bytes the other had already truncated. An acknowledged 10-byte chunk could end up 2 bytes. The staging suffix is now per-attempt rather than per-index. - Unlinking the partial file raced the write stream's pending open(). destroy() does not await it, so the unlink failed with ENOENT and the open then recreated the `.part` file after cleanup had supposedly finished — reported reproducible in 121 of 300 immediately-failing streams. Cleanup now waits for the stream to close. Relates to issue 1403 * fix(upload): revalidate the per-file cap before publishing a chunk `allowance` is computed before the body arrives, so a chunk that completed while this one was still streaming was not counted in it. Two overlapping 0.75MB chunks under a 1MB cap were therefore both accepted, leaving 1.5MB on disk; enough concurrent streams could go well past the cap before anyone asked to complete the upload. The buffered version got this right for free, because it only ever checked after reading the whole body. The streaming version keeps its pre-read check — that is what makes a too-large request cheap — and asks again with the current aggregate before renaming the staging file into place. Relates to issue 1403 * fix(upload): answer client-caused chunk states with their own status code An unknown, finished or expired upload id, and a complete call with chunks missing, were plain Errors with no statusCode, so both routes fell through to the blanket 500 and logged at error level. All four are the client's mistake: they read as a backend fault in monitoring and invite a retry that can never succeed. They now carry 404, 409, 410 and 400 respectively, and both routes pass a tagged status through instead of matching on the two they happened to know about. Only genuinely unexpected errors reach the 500 and the error log. No client is affected: uploadLargeFile, the only caller of this endpoint family, still has no callers of its own. Relates to issue 1403 * fix(upload): retire the connection after an early refusal, clean up a failed publish Two more from review. Refusing a body before reading it is the point of the streaming cap, but the unread bytes are still in flight on a connection the response advertises as keep-alive. Node does not drain them, so the NEXT request on that socket hangs until it times out — reproducible with an 8MB body against a 1MB cap. The error response now sets Connection: close whenever the request was not read to the end. A failed rename — ENOSPC, a vanished directory — left the fully written staging file behind. Staging names are per-attempt, so a client that retries instead of aborting accumulates one per try until the upload expires. The partial is removed on that path too. Relates to issue 1403 --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
316bcbd679 |
fix(backend): contain and sanitize the SQLite restore source path (#1384)
The restore flow accepted an unvalidated database.backup_file from the manifest (absolute paths and traversal both worked, and no containment check enforced the configured backup root), then interpolated it unescaped into a `sqlite3 .restore '<path>'` command, letting an attacker-chosen source file replace the live database. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
ec03089d57 |
fix(backend): validate the S3 endpoint host before the restore download (#1383)
* fix(backend): validate the S3 endpoint host before the restore download downloadFileFromS3() built an S3StorageAdapter and called .download() directly, skipping the isHostAllowed() private-IP/DNS-rebinding guard that testConnection() applies elsewhere — an admin with backup.restore could point the configured S3 endpoint at an internal/metadata address for unauthenticated egress via the server. * fix(backend): pin the restore S3 download to its validated DNS resolution isHostAllowed() was check-then-connect: the AWS SDK re-resolves the endpoint hostname independently when it actually connects, so a DNS rebinding condition between the preflight check and the real connection could still reach a private/internal address. Reuses the same pinnedRequestOptions() primitive webhookDeliveryWorker.js already uses, wired into the S3Client's requestHandler. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
b798d8e4c1 |
fix(backend): use the strong password generator for resets and enforce must_change_password (#1387)
Admin password reset generated a ~2^21-entropy password from a small wordlist instead of the already-available generateSecurePassword(16), and must_change_password was written on reset but never checked by any route-blocking logic — a reset user could keep using the old session/password indefinitely. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
59ea83c84e |
fix(backend): require actor to hold every permission of a role they grant (#1378)
* fix(backend): require actor to hold every permission of a role they grant Any admin with `users.edit` could grant an arbitrary non-super_admin role — including one carrying far more permissions than they themselves hold — via PUT /api/admin/users/:id. The role-change path never called the existing assertActorMayGrant() guard that already protects role create/edit. * fix(backend): apply the same role-grant guard to admin invitations createInvitation() only blocked granting super_admin — the same users.create-holder-can-invite-into-any-role escalation that updateAdminUser() was fixed for (GHSA-rv8w-m6mx-7j4q) was still open via POST /admin/users/invite. Reuses assertActorMayGrant(). --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
cdde937d7f |
fix(backend): reject a replayed TOTP code within its validity window (#1389)
* fix(backend): reject a replayed TOTP code within its validity window verifyTotp() was stateless — otplib's window:1 tolerance meant the same 6-digit code could complete two independent logins inside its ~90s validity window. Track each admin's last-consumed step and reject a code that doesn't advance past it. * fix(backend): make the TOTP replay-tracking persist atomic verifyTotpEncryptedStep() read two_factor_last_used_step, then a plain UPDATE wrote the new step with no conditional guard — two concurrent requests carrying the same captured code could both pass the check before either UPDATE landed. The persist is now a conditional UPDATE (only advances the step, checked via affected-row count), so a losing concurrent request is correctly treated as a replay. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
e3247911a0 |
fix(backend): shorten payment-check token TTL and notify admin on use (#1385)
The unauthenticated payment-check magic link (intentional, matches publicQuotes.js) had a 30-day token lifetime and wrote to the invoice ledger silently. Shortened the TTL and added a best-effort admin notification on every write via this route, so the no-login convenience stays but an admin always sees the action happen. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
a2bf1f644c |
fix(video): try metadata extraction and thumbnail generation independently (#1371)
* fix(video): try metadata extraction and thumbnail generation independently processUploadedVideo() gated everything behind isValidVideo(), which rejects the whole video if ffprobe can't read even one of duration/width/height -- common on some iPhone/Lightroom-exported MP4s (issue 1370). Callers (photoProcessor.js's processPhoto and processUploadedPhotos) already catch that throw and fall back to a static placeholder thumbnail plus a metadata-only retry (codex review of #845), but that fallback never got a REAL thumbnail even when generateVideoThumbnail() would have succeeded on its own -- thumbnailing doesn't need valid duration/width/height, it just seeks and grabs a frame. processUploadedVideo now tries metadata extraction and thumbnail generation independently, keeping whichever succeeds instead of discarding both on a single failed field. The callers' existing throw handling stays as a backstop. Also: extractVideoMetadata stored duration as 0 (not null) whenever ffprobe had no duration field, masking "unknown" as a fake real zero-second clip and defeating downstream `duration != null` checks meant to skip an untrustworthy value. Relates to issue 1370 * fix(video): fall back to the SVG placeholder when thumbnail generation fails processUploadedVideo could return success with thumbnailKey: null when only thumbnail generation failed. The gallery grid (GridGalleryLayout/JustifiedGalleryLayout) falls back to `photo.thumbnail_url || photo.url` when there's no thumbnail, so AuthenticatedImage downloaded the full original video and tried to render it as an <img> -- a broken tile and a potentially huge fetch just from opening the gallery. Falls back to the same ffmpeg-free SVG placeholder the callers already generate for a total processing failure, so a bare thumbnail-generation failure degrades to that placeholder too, never to "no thumbnail at all". Found by codex review. * fix(video): avoid a SQLite connection deadlock in the placeholder fallback generateVideoPlaceholder() unconditionally called getThumbnailSettings(), which queries the database directly (not through any active transaction). videoProcessor.js's new placeholder fallback can run from inside processUploadedPhotos' open per-file SQLite transaction (chunked video upload) -- knex's default SQLite pool has exactly one connection, so that second, un-transacted query deadlocks against the transaction holding it, timing out after acquireConnectionTimeout (60s). Reproduced directly against an isolated SQLite db. generateVideoPlaceholder now skips the settings lookup entirely when the caller supplies explicit width/height, and the video fallback passes the same DEFAULT_THUMBNAIL_WIDTH/HEIGHT the settings lookup would have fallen back to anyway (now exported for reuse). Found by codex review. * fix(video): throw when neither a real thumbnail nor the placeholder can be produced processUploadedVideo returned success with thumbnailKey: null when both the real thumbnail AND the SVG placeholder failed -- a total, systemic failure (storage backend down, disk full), not a quirk of one file. On stable, which doesn't have the #845 call-site fallback, this silently completed the video with no thumbnail at all instead of the retryable 'failed' status a throw here produces. On main, the pre-existing #845 fallback already absorbed this exact case (no behavior change there) -- verified against codex's own git-blame check of the pre-PR stable code before applying this. Now throws in that case, restoring the pre-existing "let the caller mark it failed and retryable" behavior for a genuinely unrecoverable video, while keeping every partial-failure case (the vast majority) resolving with whatever succeeded. Found by codex review. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
59ef2ee9af |
Merge pull request #1361 from PicPeak/feat/usage-reporting-update-prompt
feat(usage): prompt existing admins once for usage reporting after an update |
||
|
Paul Nothaft
|
d20f80112f |
feat(usage): prompt existing admins once for usage reporting after an update
An admin who already had PicPeak installed before the opt-in reporting feature existed never gets asked — the setup wizard only runs once, on a brand-new instance. Adds a one-time modal, shown on the admin's next dashboard visit after updating, offering the same choice the wizard gives a new install. - New `product_usage_state.prompt_shown` column (migration 211) and UsageService.markPromptShown(), set on either outcome (enable or decline) from both this modal and the wizard step, so an installation is never asked twice regardless of which path it took. - New POST /admin/usage/prompt-seen endpoint. - Extracted the wizard's three-point pitch (UsageReportingPitch.tsx) so the modal and the wizard step share identical copy instead of drifting apart. - The modal never shows once participation is already active, and never shows a second time after either the wizard or the modal has been through it once. Depends on #1360 (the setup wizard step this reuses). |
||
|
Paul Nothaft
|
a31a2e25e2 | fix: interrupt idle worker waits during shutdown | ||
|
Paul Nothaft
|
f0e6d2dfb1 |
fix: enforce gallery access and consolidate gallery workflows (#1357)
Harden gallery authentication and authorization, consolidate gallery workflows, and prevent token-bearing URLs from leaking through nginx request error logs. |
||
|
Paul Nothaft
|
acb25a9a1c |
fix(images): probe and clean up preview tiers under the extension the encoder actually wrote (#1355)
generatePreviewImage rewrites the output extension to match the encoding it chose, .jpg or .webp for alpha and multi-frame sources. The tier lookup in ensurePreviewImageAtWidth and the cleanup list in previewTierKeys kept the SOURCE extension instead, so for anything but a lowercase .jpg source the stat never matched: every tier request for a .png, .JPG, .jpeg, .heic or RAW photo re-ran Sharp, and cleanup never found the files it left behind, which accumulated for the life of the install. Both now derive every key the tier can live under: the .jpg and .webp candidates, plus the source-extension key last so tiers written before the rewrite are still found by lookup and by cleanup. Follow-up to issue 1020, where the mismatch was identified during review. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
c97341e454 |
fix(images): single-flight lazy rendition generation and keep the old rendition during replacement (#1350)
* fix(images): single-flight lazy rendition generation and keep the old rendition during replacement
The lazy generators in imageProcessor are check-then-generate, and the
check reads the path off the photo row the route already fetched. N
concurrent cold requests for one photo all held a snapshot with the path
still null, all missed, and all ran the same sharp pipeline. Only the
thumbnail tier path had a guard; the canonical thumbnail it falls back
to, heroes, previews and preview tiers had none.
One process-local map now covers every rendition, keyed by photo id and
rendition (`thumbnail:<id>`, `thumbnail:<id>:w<width>`, `hero:<id>`,
`preview:<id>`, `preview:<id>:w<width>`). Concurrent callers share one
promise; the entry is cleared in a finally on success and failure alike
so a rejection cannot poison the key. The tier stat moved inside the
flight so a request arriving as the previous flight clears finds the
written tier instead of missing on a stale probe.
Heroes and previews also deleted the existing object before generating
its replacement, and again in the catch. Both are gone, mirroring what
the thumbnail generator already does: put is the last statement in the
try and replaces atomically on local storage and by key on S3, so the
delete only ever opened a window with no rendition at all, and a source
that failed to read stripped the old rendition with the row still
pointing at it.
No re-read of the photo row inside the flight: the admin regenerate
endpoints force a rebuild by passing a row with the path nulled, and a
re-read would hand back the persisted rendition untouched.
Fixes the single-flight half of issue 1020. The preview cache-key
extension mismatch and any server-wide work queue remain separate.
* fix(images): keep the snapshot validity check outside the single-flight
With the check inside the flight, an admin regeneration (row passed with
the path nulled) could join a viewer's flight for the same photo that was
merely confirming an already-good rendition, and be handed back the very
file it was asked to replace while the endpoint counted a success. Only a
miss enters the flight now; inside it everything is a regeneration.
* fix(images): forced rebuilds run after an in-flight lazy generation instead of adopting it
The admin regenerate endpoints could still join a lazy flight that was
already generating for the same photo. That flight read the thumbnail
settings when it started, so after a settings change it produces exactly
the rendition the regenerate was invoked to replace; adopting it counted
a success while the old size stayed cached.
ensureThumbnail and ensurePreviewImage take `{ force: true }`: skip the
snapshot check and, if a flight is pending, start after it settles. Lazy
misses arriving meanwhile join the forced flight, and an older flight
settling late no longer evicts the newer entry from the map.
* fix(images): key rendition flights by source as well as photo id
replacePhoto keeps the photo id and changes path and filename. Keyed by
id and width alone, a request carrying the replacement row joined a
flight still rendering the file it replaced and was handed the old
image, which the gallery caches for 30 minutes. The tier map this
replaced was keyed by storage key and so already told the two apart.
* test(thumbnails): wait for the regenerate loop's completion line instead of a fixed 150 ms
The loop runs in setImmediate after the response. Under a loaded machine
(fifteen suites in parallel, each booting a migrated SQLite) it took
longer than 150 ms once and the assertions ran against a half-finished
mock call list. Poll the logger spy for the "regeneration complete" line
with a 10 s deadline; the suite also finishes sooner because the wait
ends as soon as the loop does.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
|
||
|
Paul Nothaft
|
8cc7d7d14a |
feat(external-media): watch reference folders and import new files automatically (#1345)
* feat(external-media): watch reference folders and import new files automatically Managed uploads dropped into storage/events/active are picked up by the chokidar watcher; external media had no equivalent, so a NAS folder that keeps growing needed an admin to open the event and press Import every time. Relates to issue 1187. - The import pass moves out of the route into services/externalImportService.js. The watcher and the Import button now run the identical function; the route only validates and maps errors to status codes. - Mutual exclusion is the per-event claim from maintenanceJobState (`external_import:<id>`, seeded on demand by the new ensure()) instead of the in-process Set. The Set stopped a double-click in one process; the claim also stops the watcher on a second replica, or an admin clicking while the watcher is mid-run elsewhere. The run heartbeats so a claim from a dead process is taken over. - services/externalMediaWatcher.js: per-event opt-in via the new events.external_watch column (migration 208), chokidar with awaitWriteFinish so a copy in flight is not imported half-written, debounced full pass per change, a timer sweep every 15 minutes as the fallback for NFS/SMB mounts that deliver no inotify events, optional stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched events is re-read every minute, so the toggle works from any replica. A watcher that just started runs one pass immediately. - Deletions are ignored on purpose: a file vanishing from a NAS is at least as likely to be a reorganisation or a dropped mount as an intentional removal, and acting on it would delete a guest-visible photo. Rows whose file is gone stay, as they do today. - Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local. - Quiet system passes stay out of the activity log; runs that imported something are logged with actor external-media-watcher. - Frontend: "Watch folder for new files" checkbox under the external folder picker, status line in view mode, EN/DE strings. * fix(external-media): close the review gaps in the folder watcher Codex review of the watcher, round 1. All six findings were real: - Enabling the watcher, or pointing an enabled one at another folder, now requires photos.upload — the permission the manual Import already requires. events.edit alone was a way around it. Only the transition is checked, so a role without photos.upload can still edit an already-watched event. The checkbox is disabled for such roles. - Automatic passes defer files that are still changing: anything modified inside the stability window, or whose size moves across one wait of that window, is left for the next pass. chokidar's awaitWriteFinish only settles the file that fired the event, and the sweep sees no events at all, so a sibling still being copied could be inserted half-written and then skipped forever. - Photos an admin deleted are not brought back by the sweep. The delete routes record the file in external_import_exclusions (migration 209); automatic passes skip the list, the manual Import ignores it and clears it for what it imports. - The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three compose files; they were documented but the backend services use explicit environment lists, so the kill switch did nothing. - A pass re-checks is_active / is_archived at run time, not only in the minutely reconcile. - The lease is renewed on a timer for the whole run, walk included, and ownership is checked before the event row is touched. * fix(external-media): make automatic passes follow the row, not rewrite it Codex review round 2, four findings, all applied: - The event update route drops non-canonical spellings of external_watch and external_path before the permission guard. SQLite resolves column names case-insensitively, so `External_Watch` reached the column while the guard only looked at the lowercase key. - Exclusions are checked per file at insert time, not against a snapshot taken before the settle wait. A photo deleted during the wait was present in the snapshot and got re-inserted by the loop. - An automatic pass no longer writes source_mode / external_path. It re-reads the row after the walk and the settle wait and stops if the folder changed or the event went managed; the manual Import is the only writer. The options are now `automatic` + `settleMs`. - A pass that deferred files re-arms the debounced import, so a file copied just before the watcher started is not stranded when the sweep is disabled. * fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying Codex review round 3, both findings applied: - recordExclusions keys on external_relpath alone. A replaced external photo becomes managed but keeps its relpath on purpose, and deleting that replacement must not republish the NAS original. - An automatic pass checks the full watcher predicate (reference mode, same folder, watch on, active, not archived) before it inserts and on every heartbeat tick during the loop, and stops as soon as the event no longer qualifies. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
5c1e38d921 |
feat(usage): distinguish real edits and template delivery with v5 consent (#1339)
* feat(usage): distinguish real edits and template delivery with v5 consent * fix(usage): exclude queued test messages and count reorders as edits - queueEmail carries usageEligible: false into email_data and the queue processor passes it on, so the dev tools' send-test-email no longer records email_template_delivery once the worker sends it. - event-types/reorder and categories/reorder-global compare the persisted order before and after and record the v5 edit markers only when it changed, matching the display_order edit already counted on PUT. - normalized() builds arrays with Array.from so a row array from the sqlite binding compares equal under Jest's separate realm. * fix(usage): cover per-gallery category order and workflow test runs - categories/reorder records category_editing when an event's override changes; reorder/:eventId records it when an override was actually removed. - send_email and the collections handoff pass usageEligible: false for a workflow test run (engine.testRun sets __test), so a non-dry test send is not counted as template delivery. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
69754f8a2c |
fix(email): scrub gallery passwords from the sent-mail archive (#1340)
* fix(email): scrub gallery passwords from the sent-mail archive
The email queue kept every gallery password and client PIN in clear
text in email_data and rendered_html after the mail was sent, and the
Messages reading pane handed them back to any admin with the messaging
flag. A password hash in the events table bought nothing while the
plaintext sat next to it.
Once a mail is out, or its retries are exhausted, the processor now
masks secret-looking variables (password, passcode, pin) in email_data
and replaces their values in the rendered body, plain and HTML-escaped.
The reading pane applies the same masking to rows archived before this
change. Pending rows keep the real values so a retry still sends them.
Relates to issue 1271
* fix(email): keep a quoted ">" from cutting an attribute value out of redaction
The tag splitter stopped at the first ">", so a template attribute such as
title="{{gallery_password}} > details" left the password unmasked in the
archived HTML while email_data was already masked. The tokenizer is now
quote-aware; a tag with an unbalanced quote falls through as text and is
scrubbed there.
* fix(email): scrub secrets inside HTML comments in the archived body
A comment such as <!-- PIN: {{client_password}} --> was split off as a tag
and its body, which has no attribute, was never scrubbed. Comments are now
one segment and their content is masked whole.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
|
||
|
Paul Nothaft
|
aa6e5d613f |
chore(usage): make the capability catalog English-only
features.v2/v3/v4.json carried every name, description and definition twice, once in English and once in German, inside a file that is source code, is vendored byte-identically into the collector and is served as the consented catalog. Source stays English. The German strings already existed a second time in the frontend locale file, which is what the consent dialog actually renders (UsageCatalog reads productUsage.catalog.<key>, never the JSON), so the copy in the catalog was a duplicate that could only ever drift. The `de` fields are gone from all three catalogs, their frontend copies and the inventory definitions; the docs coverage file and FEATURE_COVERAGE.md list English only. Nothing on the wire changes: the report schema is derived from the keys, and the catalog's text is not part of any signature or consent version string. The coverage test now pins the catalog to the en locale verbatim and requires the de locale to cover every key and field, without dictating its wording. The collector holds the same catalog files and needs the same change to stay byte-identical, plus its German catalog strings moved into its own locale file; that is filed there. |
||
|
Paul Nothaft
|
ef8a52f02c | fix(usage): introduce consented v4 without changing historical reports | ||
|
Paul Nothaft
|
02b353e54f |
fix(usage): report restricted gallery downloads in v3 instead of an always-true signal
gallery_downloads.configured was true on every installation with a gallery. allow_downloads ships true — column default in migration 037 and the create route both set it — and the snapshot asked "at least one gallery has it on". The fleet value was ~100% by construction and could not separate a deliberate configuration from an untouched one. v2 consented to that key under that description, so v2 keeps sending it unchanged. v3 replaces it with gallery_downloads_restricted: at least one gallery has downloads switched off, which is the only state of that column anyone actually decides. Same catalog position, so the disclosed capability count stays at 86; the frontend copy, the EN/DE catalog strings, the coverage inventory and FEATURE_COVERAGE.md follow. Done in v3 rather than a v4 because v3 is on main and in no release yet, so nobody has consented to it. The collector carries the same catalog and has to take this change before the release that ships v3. One guard for the window in which :main / :beta images already carried the old v3 catalog. A report queued under it fails local validation on this build, and deliver() left a locally invalid report pending for good, blocking every operation behind it. A report's payload is derived state, so deliver() now rebuilds it from the current snapshot in place and sends that. Packet ID and sequence are kept — a re-signed retry has to reuse them so a lost acknowledgement does not duplicate data — and reports only: a stale registration, deletion or command is a genuine conflict and keeps the existing handling. Tests: the v3 snapshot counts a switched-off gallery and ignores enabled ones, v2 still reports the old key with the old meaning, and a stale queued report goes out rebuilt under the same packet id while a valid one is sent untouched. Relates to issue 1308 |
||
|
Paul Nothaft
|
7ca783f89b | fix(usage): preserve report contracts with compatible receiver validation | ||
|
Paul Nothaft
|
c358bc65f7 | feat(usage): add consented beta capabilities and gallery photo totals | ||
|
Paul Nothaft
|
e40bc474bc |
fix(usage): let an operator clear a participation the collector never accepted
Probing the live collector to settle the delete-sequence question turned up something else: usage.picpeak.app answers a valid usage.v2 registration with INVALID_PACKET while the identical v1 flow is accepted. It does not speak v2 yet — which the deployment notes already require, but the consequence of getting that order wrong was worse than "reports do not send". Opting in to v2 against a v1-only collector left the installation stuck. Registration was refused, so nothing existed at the collector at all; the row sat in activation_pending, disable moved it to deletion_pending, retry was futile forever, and enable refused because the row was not `disabled`. The abandon hatch added earlier did not apply: it was gated on SIGNING_KEY_UNREADABLE. So the most harmless possible failure — nothing registered anywhere — was the one an operator could not clear. The gate is now the property that actually matters: a participation the collector has provably never accepted (sequence 0, no receipt) with a failing delivery can be discarded, from activation_pending as well as deletion_pending. Its receipt records `never-registered` rather than an unconfirmed deletion, because nothing remote exists to be unsure about. A participation the collector *did* accept keeps the old narrow gate and its explicit warning — clearing local state while the collector still holds reports must stay a deliberate, warned-about act. A collector that rejects a registration or a deletion outright now reports SCHEMA_NOT_ACCEPTED instead of DELIVERY_FAILED, and the settings page says the collector does not accept this report version yet. Retrying cannot fix that, and sending the operator to look for a network fault they do not have was wrong. Verified end to end against the live collector: v2 opt-in reports SCHEMA_NOT_ACCEPTED, the exit is offered immediately, the receipt says never-registered, and joining again on v1 registers, reports and withdraws with a collector-confirmed deletion. |
||
|
Paul Nothaft
|
c741dc22c5 |
docs(usage): state in the consent dialog that the connection only runs outwards
The dialog described what is sent and where it goes, but never said which way the connection runs. That is the part an operator is actually being asked to accept: opening an outbound path to someone else's service. PicPeak sends and never pulls. One place in the service reaches the network, it is a POST, and it requests exactly two paths — /api/envelopes, and /api/participant/lookup only when an operator asks for their own export. No scheduled job contacts the collector; the daily rollup is driven solely by an authenticated admin hitting /activity. There is no route the collector could call, and redirect: 'error' means it cannot even point a request somewhere else. From a reply only the acknowledgement for the packet just sent is read, with every field compared against that packet before it is accepted; the stored copy drops the session token and no read path hands it back to the UI. A requested export is streamed to the operator as a file and never interpreted. The consequence is why it belongs in the consent text and not only in the docs: this channel cannot deliver code, configuration or content into an installation, not even from a collector that has been taken over. It is a security property by design rather than by convention. usageOutboundOnly.test.js guards it by source inspection rather than behaviour, because a behavioural test only proves that today's calls behave. It fails the moment someone adds a second fetch, a poll for messages, a scheduled pull, or a public route touching the usage service — verified by injecting each of those. |
||
|
Paul Nothaft
|
1e8b6f1b0f |
fix(usage): close the QA findings on opt-in product usage
A QA exploration of this branch against an isolated rig — own stub collector, SQLite and PostgreSQL — turned up one dead end and a set of signals and controls that did not hold up. This closes all of them. Rotating JWT_SECRET, the documented response to a suspected compromise, made the signing key unreadable. That was already named and documented, but it left no way out: the delete packet can never be signed, so the row stays deletion_pending forever, and enable() refuses because it is not `disabled`. An operator who rotated precisely because the secret was compromised cannot restore it, so the feature was bricked with no control left. POST /usage/abandon is offered only in that state; it drops the local identity and records the receipt as `collector-unconfirmed` rather than claiming a deletion that did not happen. Every failed delivery was retried on the next admin request, and /activity is open to any authenticated admin while the settings ticker fires it every five minutes per open tab — 30 activity calls against a rejecting collector produced 30 outbound requests. Migration 206 adds attempts/next_attempt_at and the unattended sender honours the gate; Retry and opt-out still send immediately, and the tab names the time of the next automatic attempt. Feedback, votes and portal sessions now share an installation-wide budget of 30/hour. They are the only endpoints whose effect is outbound traffic carrying operator-written free text, and the general limiter skips authenticated requests by design. Reading status and withdrawing stay unthrottled. gallery_image_protection was true on a bare install with no galleries: PicPeak ships default_protection_level='standard' and enable_devtools_protection=true, so it reported fleet-wide 100% and could never separate a decision from an untouched default. It now reads only what deviates from the shipped defaults, and the devtools flag is not read at all — being on by default, its only informative state is off, which is the opposite of what the key claims. Also: - the export receipt counted every packet and called the total "usage reports"; reports and participant operations are now counted and named separately - GET /usage/preview no longer persists the custom_css marker, so the transparency view stops changing what will be sent - the feedback route requires every field the packet schema requires, so an API caller gets the missing field named instead of a bare INVALID_PACKET from inside signing - the German strings for this feature use "Sie" throughout, matching the rest of the admin UI; the ignore hint says what ignoring will do rather than stating it as already true - the consent dialog returns focus to the control that opened it - the long buttons wrap instead of running off a 390px viewport - a deletion receipt is labelled as belonging to an earlier participation while a new one is active Regression tests cover each of these, including the delete packet's reuse of the last accepted sequence, which was an unwritten assumption about the collector rather than a defect. |
||
|
Paul Nothaft
|
a7382591bf | feat: expand opt-in capability coverage with versioned consent | ||
|
Paul Nothaft
|
5d31b61c8d | Merge remote-tracking branch 'origin/main' into codex/1110-usage-coverage | ||
|
Paul Nothaft
|
e347f8f40f | fix(usage): minimize session receipts and clarify privacy controls | ||
|
Paul Nothaft
|
32d745b575 |
fix(usage): stop local backups implying S3 use, and make the protocol-error branch reachable
Two findings from the review of the current head. Local backups no longer imply S3. markUsed derived an s3_storage marker from "a backup ran while backup_destination_type is s3" — but the middleware also counts /database-backup/* and /backup/picpeak/export as backups, and those write a local file wherever scheduled backups go. So configuring S3 and downloading a local export reported s3_storage as USED. The middleware now tells markUsed whether the operation writes to the configured destination, and only then is the marker derived. A wrong `true` in this dataset is worse than a missing signal: it is a claim about an install that nobody can check. The ProtocolError branch was dead code. adminUsage matched on `error.name === 'ProtocolError'`, but the class extends Error without setting `name`, so every instance reports 'Error' — verified — and a malformed vote or feedback payload fell through to the global handler, which logs it as an unhandled programming error and answers INTERNAL_ERROR in production, losing the validation code the caller needs. Now matched with instanceof. protocol.cjs is byte-identical with picpeak-usage (diffed against the companion repo), so the fix belongs here rather than in the class. An existing assertion needed updating for the new markUsed argument, and the path split is pinned: /backup/run is destination-driven, /database-backup/backup and /backup/picpeak/export are not. Refs #1110 |
||
|
Paul Nothaft
|
bb76ca5375 |
fix(usage): keep the settings tab usable on a bad collector URL, and report layouts and CSS accurately
Three items, one of which explains an error seen in the app. "The operation could not be completed" could come from a config typo. status() called collectorUrl() bare, and that throws on a bare hostname, a path, a query, or http in production. The settings page renders one generic failure when its status query errors, so a misconfigured USAGE_COLLECTOR_URL replaced the whole tab with that sentence — no cause, and no way to read the status or withdraw, because every control there sits behind that call. The URL is now reported as collector_error: 'INVALID_COLLECTOR_URL' beside the real state, the tab says what is wrong and how to fix it, and the links are only rendered when there is somewhere to point them. gallery_layouts reported grid for every preset-themed install. color_theme holds either a theme object or the NAME of a preset — the theme picker stores names, and eventTypeService seeds them (`theme_preset: 'corporateTimeline'`). Only reading value.galleryLayout made masonry, timeline, mosaic and the two gallery presets invisible. Names now resolve, and an event with no theme of its own resolves through the global one instead of being counted as grid. Only the name -> layout mapping is duplicated, not the presets; frontend/src/types/theme.types.ts stays the source of truth, and an unknown name reports `other` so a preset added later degrades to "something else" rather than quietly inflating the grid count. custom_css missed CSS applied through a template. An enabled css_templates row applied via events.css_template_id is gallery styling by the same definition as the settings fields — the Custom CSS tab is where both are authored — but neither the snapshot nor the middleware saw it, so those installs reported custom_css entirely false. Existence only; template contents are never read. Eleven tests. Reverting each fix in turn fails 3, 1 and 3 of them. Refs #1110 |
||
|
Paul Nothaft
|
9785b636a9 |
fix(usage): take the withdrawal baseline before the lease, not after it
Third and last window in the same race, and again in my own fix. locked() claims the lease and reads the row in two separate statements. Reading the cancellation counter from inside that callback meant a /disable completing in the gap was adopted as this activation's own baseline and silently absorbed — the counter matched, the claim succeeded, and registration went ahead after the operator had withdrawn. The baseline is now read before the lease is taken, which inverts it: every increment from that point on is later than the value the claim tests for, so the claim fails and the withdrawal wins. An increment from before the read is a withdrawal the operator already completed, and a deliberate opt-in afterwards should not be vetoed by it. The test for this passed against the bug on its first two attempts. It stubbed the state read to increment the counter AFTER reading the row, so both the broken and the fixed version saw the old value and behaved identically. The withdrawal has to land before the read returns for the row to carry it — which is the whole point of the window. It now fails without the fix. Refs #1110 |
||
|
Paul Nothaft
|
22da018e1b |
fix(usage): close the remaining withdrawal races, reset per-item name consent
Follow-up review on the previous commit, including a hole in that commit's own fix. The cancellation flag became a counter. Clearing a boolean needed a write of its own, and a /disable landing between the lease and that write was erased — the same race one level down. enable() now records the counter it started with and claims only if it is unchanged, so no clearing write exists to lose. It also fixes the case a boolean could not express at all: a stale cancellation already set, and a fresh one arriving mid-activation, are indistinguishable as flags and obvious as counts. Migration 203, separate from 202 for the reason 202 was separate from 201 — knex will not re-run an applied migration. deliver() re-checks immediately before dispatch. The existing check ran before the binding lookup, which is asynchronous, so a withdrawal that COMPLETED during it still had its registration or report sent afterwards. Not an already-in-flight request — a new one started after the operator had withdrawn. The outbox writes in tick() and command() are conditional on still being active. /disable clears pending_packet without holding the lease, so an unconditional write put a report — or a feedback body and name — back into an outbox the withdrawal had just emptied, where deliver() would then leave it, since it declines to send anything but the delete. Per-item name consent resets with the item. `named` stayed checked after submitting, so the next item carried the previous name automatically, contradicting the anonymous-by-default promise the disclosure makes for each item. The remembered name stays in preferences; attaching it is decided again each time. Two of these tests were worthless when first written and are noted because the pattern keeps recurring: the pre-dispatch case passed without the guard because an empty report payload failed schema validation during signing, so nothing reached the collector for reasons unrelated to the check. With a valid payload it fails without the guard and passes with it. Same for the counter: dropping it from the claim fails two. Refs #1110 |
||
|
Paul Nothaft
|
80e238f0ad |
fix(usage): let a withdrawal win against an activation that is still starting
The last open item from the #1304 review. /disable overlapping an in-flight /enable was silently lost. While activation generates its identity and writes its binding file the row still reads `disabled`, so disable()'s conditional update matched no rows, and the lease conflict raised by its tick() was swallowed as expected noise. The admin was told participation was off; the activation then completed and left it on. An opt-out that does nothing is the one failure this feature cannot have. disable() now records cancel_requested first and unconditionally — before the case-by-case work — and enable() claims its state with a single conditional UPDATE that tests the flag alongside the status. Re-reading the flag and then updating would only have moved the window; making the claim itself carry the condition closes it, so whichever of the two lands first wins outright and the loser writes nothing. Nothing is registered when the claim fails, so there is also nothing to delete remotely — the cancelled activation leaves no identity behind. The flag is cleared at the start of enable(), so a cancellation from an earlier participation cannot veto a later deliberate opt-in. The column is migration 202 rather than an edit to 201. 201 already shipped on this branch and knex records it as applied, so folding the column in would have skipped every database that had already run it and the first /disable would have failed on a missing column. Verified both ways: a fresh install gets the column from 201+202, and a database migrated before 202 existed gains it when 202 arrives. Three tests. With the condition dropped from the claim, the race case fails and the other two pass. Refs #1110 |
||
|
Paul Nothaft
|
c043897b0e |
fix(usage): name the unreadable-key failure, unpin the collector default, align the tab
Review follow-ups on #1304. SIGNING_KEY_UNREADABLE. USAGE_ENCRYPTION_KEY defaults to JWT_SECRET, so rotating JWT_SECRET — the correct response to a suspected compromise — makes the stored Ed25519 key undecryptable. That surfaced as a generic DELIVERY_FAILED which retried forever, and it silently blocks the DELETE packet too: an operator who withdraws has their local state cleared while the collector keeps its copy. decrypt() now tags its own failure and deliver() reports it under its own name, without flagging an identity conflict — an unreadable key is not evidence of a clone. The docs already warned that losing the key breaks deletion signing; they now name the trigger and the error. The collector default is no longer an inline string in the constructor. It is a declared DEFAULT_COLLECTOR_URL, since it is a deployment choice: self-hosters point USAGE_COLLECTOR_URL at their own collector and the UI already derives every link from whatever is configured. schema.cjs is deliberately untouched — it is vendored byte-identical with picpeak-usage, and its $id is a schema identity, not a delivery address. Links in the consent dialog. It named the collector inside prose but never linked it, so an operator deciding whether to opt in could not open the destination or the public schema without retyping a URL. Both are links now, built from the configured collector. UI standards. The tab hand-rolled its surfaces as `<section className="rounded-xl border border-theme …">` and imported Button from a deep path; every other settings tab uses `<Card padding="md">` from the components/common barrel. Converted, with the feedback <form> wrapped rather than replaced so its semantics survive, and headings given the same colour tokens as ImageSecurityTab. The barrel pulls ErrorBoundary -> i18n/config, so the tab's test needed the initReactI18next shim the FaceRecognitionCard test already uses. Not changed: the delete packet reusing the current sequence. The collector handles delete before any sequence check — "possession proof is sufficient for deletion, including when a restored backup has a stale sequence" (picpeak-usage server/collector.js) — so deletion is deliberately sequence-exempt and the client is correct as written. Refs #1110 |
||
|
Paul Nothaft
|
027afb6086 |
fix(security): re-check inline CSS after template substitution
The fourth bypass found in this review, and the one no lexer fix
reaches: sanitizing runs on the stored body, but safeTemplateReplace
rewrites it afterwards, so the string that was validated is not the
string that is sent.
A conditional inside a style attribute can delete the very quoting that
made a url() inert:
style="--x:x{{#if company_name}}'{{/if}};background:url(https://evil…)"
At write time the url() genuinely sits inside a CSS string and is
correctly left alone. Expanding the conditional for a recipient with no
company name removes both quotes and the background goes live —
confirmed end to end against the real functions.
The style-attribute pass now runs again on the substituted output.
Substitution cannot introduce a `"` (values are HTML-escaped), so the
attribute match still holds. body_css is not substituted, so the
<style> block cannot be rewritten after its check and needs nothing.
This is the case the removed newsletter pass had been covering. Rather
than reinstating a second definition of "disallowed", the one definition
now runs at both points where the content changes.
Refs #1264
|
||
|
Paul Nothaft
|
1cf82746b7 |
fix(security): close two CSS url() bypasses the sanitizer dedup exposed
Both found by review against the correct base, and both are cases the
second stripRemoteCssUrls pass had been catching before this PR removed
it. Verified against the real functions before and after.
An escaped quote outside a string. `\'` is an escaped identifier
character, not a string opener, but the scanner stepped onto the
apostrophe, entered string mode and copied the rest of the stylesheet
unexamined — so `.hero{--marker:\';background:url(https://evil/p.gif)}`
kept a live remote URL. Escapes are now consumed as a unit outside
strings.
An unterminated quote. Trusting one meant a single stray apostrophe
disabled scanning for everything after it. An unclosed quote is a parse
error, so the safe reading is to emit it as an ordinary character and
keep scanning; a newline also ends a string, as it does in CSS.
The entity mismatch behind the second case. sanitize-html writes `"`
inside an attribute as `"`, so the scanner and the recipient's
browser disagreed about where strings begin: in
`style="font-family:"don't";background:url(...)"` the browser
decodes first, reads the apostrophe as ordinary text inside a real
string, and fetches the background — a tracking pixel by another name.
Style attributes are now decoded before scanning and re-encoded after,
which also stops the old code silently deleting quotes from the value.
Also detaches the image handlers before releasing the canvas source.
That one did NOT reproduce: measured in both Chromium and WebKit,
neither fires `error` when the attribute is removed after a successful
load. Applied anyway because the ordering is free and the failure it
would cause is silent — canvasFailed set, the canvas swapped for an
<img>, and the image decoded a second time, the exact opposite of what
the release is for.
Refs #1264, #1287
|
||
|
Paul Nothaft
|
fc595409b4 |
feat(crm): newsletter campaigns behind a newsletters flag (#1264)
Part B of #1264. Flag off by default, so an install that never enables it gains no route, no nav entry and no way to mass-mail. A campaign is a body plus a recipient rule. Queueing one writes ordinary email_queue rows (email_type 'newsletter', origin 'campaign', new campaign_id), so retry, rendered_html, sent_at and error_message all come from the existing processor rather than a parallel sender. Throttling staggers scheduled_at; the processor loop is untouched. Two rules the service enforces: no raw HTML is ever stored (sanitized on write and again on render, idempotently), and opt-out is checked at queue time AND again at send time. Migration 199 adds email_campaigns, email_campaign_recipients, email_queue.campaign_id, customer_accounts.marketing_opt_out(_at), and the newsletters.view / newsletters.send permissions. Three rounds of external review are folded in, including several that would otherwise have shipped broken: - Campaign rows never came due on SQLite. queueEmail writes a Date, which the sqlite3 binding stores as epoch ms; ISO text in the same column compares as TEXT against an INTEGER, and SQLite orders every INTEGER below every TEXT. The feature silently sent nothing there. - The flag had no Settings card and no sidebar entry, so it could not be enabled through the UI at all. - Consent is per ADDRESS, not per row: two accounts sharing an inbox meant unsubscribing stopped one and not the other, at both queue and send time. - The unsubscribe GET mutated consent, so a mail-security scanner walking a campaign could have unsubscribed much of the list. GET now confirms, POST acts. - The rate ceiling is clamped to the queue's real throughput (10/min), so the composer's estimate stops being wrong by up to 12x. Closes #1264 |
||
|
Paul Nothaft
|
7c9baff751 |
fix(upload): scope category ids, stop temp-file leaks, split the video cap
Four related fixes on the admin upload/photo path. B5 -- PATCH /photos/:photoId and POST /photos/bulk-update took any parseInt(...) > 0 straight into the update with no existence or scope check, so a photo could be moved into another event's category. The upload route already validated `event_id = X OR is_global` per #500/#525; extracted that query as findScopedCategory() and used it on all three routes so the 400 body is byte-identical. 0/negative/'individual'/'collage'/null still clear without a lookup, so the clear path costs no extra query. B9 -- three distinct temp-file leaks, not one. The validator's size branch never unlinked; the cleanup lived in the final handler, unreachable on any 400; and multer's `destination` callback runs per file and overwrote req.tempUploadPath, so even the success path only ever removed the last file's directory. Now: discardUploadedFiles() runs on every 4xx and the 500 (ENOENT tolerated, and files are only dropped when the whole request is being rejected, so the passing path is untouched); cleanup registered before multer so it also covers multer's own LIMIT_FILE_SIZE return; one directory per request. B8 -- the admin uploader filtered on MIME only, so an oversized file was uploaded in full before the server's 400. Mirrors UserPhotoUpload's existing per-file toast-and-drop. C4 -- general_max_file_size_mb was a single cap for photos and videos, so the 50MB default meant admins could not upload ordinary video without also raising the photo limit. Adds general_max_video_size_mb (default 500MB, clamped by the same 10GB MAX_ALLOWED_FILE_SIZE_MB ceiling, read per request, 60s cache), editable in Settings -> General. Photo uploads are protected from regressing by keeping multer's type-blind limit at max(photoCap, videoCap) and moving the per-kind decision into validateUploadContent, where file.mimetype exists. It 400s with the existing message shape, so an oversized photo is still rejected with the identical body it produced when multer did the rejecting. Known gap: chunked-upload/init still applies the photo cap to video. Making it video-aware would change an existing assertion that pins a 200MB video init being rejected under a 1MB general cap. No component calls that path today and the direction is strict rather than a bypass, so it is left as-is. Guest video uploads still share the single cap in gallery.js. Refs testplan REPORT.md B5, B8, B9, C4. |
||
|
Paul Nothaft
|
103863cbab |
fix(quotes): enforce the status state machine, and correct the table
VALID_QUOTE_TRANSITIONS was a complete-looking quote state machine that nothing consulted, so status changes were unvalidated. Mapping every writer of quotes.status (quoteService.js is the only one -- dealsService, projectService, adminDashboard and customer.js all read) showed the table itself was wrong: six legitimate transitions were missing. sendQuote allows draft/declined/expired -> sent but the table had draft only; adminAcceptQuote allows draft/sent/expired but had sent only; adminDeclineQuote allows draft/sent/expired but had draft/sent; recordResponse had no same-status entry. Enforcing it as written would have broken accept-on-behalf from a draft, resend-after-decline, every expired revival and the 15-minute response-toggle window. So the table is reconciled to reality first, then assertQuoteTransition() (409, QUOTE_INVALID_TRANSITION) is called at all seven sites. Two things worth carrying forward. Nothing in the codebase ever sets 'expired' -- the header comment says "set by the scheduler" and there is no such scheduler; sent -> expired is retained as documented intent only. And the backstop's added value is narrow: every reachable invalid transition is already caught by a call site's own better-worded guard, which fires first. What it newly catches is a status the machine has never heard of -- a legacy or corrupt row like 'cancelled' sails through adminAcceptQuote's guard, which only excludes accepted/declined/converted, and used to be silently overwritten. That is what the new tests pin. Refs testplan REPORT.md B4. |
||
|
Paul Nothaft
|
77b11ab874 |
fix(upload): enforce the chunked-upload cap on bytes received, not declared
The init route checked the client-declared fileSize against general_max_file_size_mb, but nothing checked what then came through the chunk route: a client could declare `fileSize: 1` and stream any amount, and completeUpload only logged the size mismatch before handing the merged file on. The cap the earlier commit added at init was therefore a gate with no fence. The service now carries the cap from init and enforces it on the running byte total per chunk (aborting the upload once crossed, since the chunks on disk are already over the limit), rejects chunk indices outside the announced range, and re-checks the merged file as a backstop. Both routes answer 413/400 for these instead of a blanket 500. |
||
|
Paul Nothaft
|
1d84c738d8 |
test: repair four stale backend suites
All four asserted contracts the product has since moved past. No genuine
product bugs behind any of them; assertions were tightened, not loosened.
adminAuth (3 tests): never mounted errorHandler, so ConflictError/
ValidationError arrived as empty Express defaults. The route also checks
username before email, so the "email conflict" fixture was hitting the
username branch. Mount the handler, fix the fixture, match the real response
shapes.
backupService.enhanced (12 tests): three stacked drifts -- the db mock had no
.returning(), so every runBackup threw at the insert; ensureDatabaseDumpForBackup
now lazily requires ./databaseBackup inside the run, which fails under
mock-fs; and the rsync path moved from exec(shell string) to
spawnAsync('rsync', args) with an isHostAllowed SSRF preflight. Also updates
getBackupStatus to its current shape (frontend aliases, nextScheduledRun null
when no schedule is enabled, #871).
adminSettings.logo: POST /logo gained requirePermission('settings.edit');
the hand-rolled db mock returns a bare Promise from select(), so the
permission lookup threw a TypeError into a 500. Mock the permissions
middleware alongside the already-mocked auth.
crmMintPaths (2 tests): macOS-only. The expected prefix was realpath'd while
the services persist under the raw STORAGE_PATH -- identical on Linux CI
(/var vs /private/var only diverges on macOS), which is why it passed there.
The comment justifying the realpath referenced process.cwd() behaviour the
services no longer have.
Refs testplan REPORT.md #22 (Part 1.2.01).
|
||
|
Paul Nothaft
|
44a8c416c7 |
refactor(email): stop reading the webhook response body at all (#1225) (#1239)
The transport carried 41 lines of bounded-read-with-deadline to recover a messageId a receiver MIGHT return. That value is only ever logged — nothing persists it, there is no email_queue.message_id column — and the code to get it produced two of the last four review findings: the size cap made a DELIVERED message retry (axios throws while reading), and the missing deadline let an unclosed stream hang the queue and resend. Not reading the body is how that whole class stops being reachable rather than defended against. responseType 'stream' still keeps axios from buffering; the stream is destroyed immediately and the id is synthesised as before. The status was always the delivery verdict, and it is known before any of this. An 'error' listener goes on before destroy(): destroy can emit on a socket-backed stream, and an unhandled 'error' on a stream throws — which would have turned a receiver's teardown into a failed send. Net: -45 lines of service code, one fewer constant, one fewer test seam, and three of the hardest cases in the suite replaced by two simpler ones. 23 tests, 63 across the email suites, eslint clean. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
6ca8baab23 |
fix(watcher): stop re-importing a photo whose file was replaced (#1226) (#1237)
The existence check matched on filename OR path. replacePhoto regenerates both — a fresh generated filename and a fresh managed path — so a watched-folder photo that had its file replaced stopped matching either arm. The original is still sitting in the watched folder, so the next sweep imported it again and the gallery ended up holding the delivered edit AND the untouched original: the same duplicate shape external_relpath prevents for reference galleries. source_filename is now a third arm. It is the stable key here — written once at ingest by this same path and preserved across a replace by design. Rows predating migration 193 are covered by its backfill: COALESCE(original_filename, filename), and this path never wrote original_filename, so for watcher rows that resolves to the basename being compared. The query is lifted into an exported findExistingPhoto() so the test drives it rather than a copy — the thing under test IS the query, so a query-builder mock would only assert that knex was called the way the test expects. Predates the Lightroom round-trip and applies to the admin replace path too; it became reachable when #1165 brought watcher galleries into round-trip scope. Six tests against a real SQLite database. The load-bearing one fails without the change, verified by removing the arm and re-running; the other five pin what must not move — filename and path matching, the pre-193 backfill shape, a genuinely new file, and event scoping. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
4105c099c9 |
test(external): make the fold-collision guard test the real code (#745 follow-up) (#1234)
The regression test added with #1165 re-implemented the claim ordering and the claim loop inside the test file and asserted against its own copy. It never required externalRelpathFold, so changing the real sort left it green — a guard against silently deleting a client's delivered edit that guarded nothing. The ordering is now a named, exported claimOrderFor() and the test drives it. Verified by sabotage: replacing the comparator with `return 0` fails the test, where before it passed. Three cases added while the seam existed: the managed row wins from BOTH input orders (the original bug was that the survivor was whichever came first, so one order proves nothing), the sort is stable for rows of the same kind, and it does not mutate the caller's array. No behaviour change — the comparator is byte-identical, only lifted out. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
0d41fe5bf1 |
fix(email): keep the webhook payload out of the logs, and bound the response read (#1225) (#1233)
Round 4 of external review, on the merged commit. Both findings are consequences of the round-3 streaming change, which is exactly why the round was worth running. An AxiosError carries the request it failed on: `config.data` is the ENTIRE serialised message, base64 attachments included, and `config.headers` holds the signature. emailProcessor logs the error object and winston serialises it, so a DNS blip or a refused connection wrote password-reset links, guest recovery codes and multi-megabyte invoices into combined.log — verified against axios rather than assumed. Every rejection is now caught and replaced with a message-and-code-only error, so nothing downstream can serialise the request back out of it. readBounded had no deadline. axios' `timeout` covers the response HEADERS, and with responseType 'stream' it has already resolved by the time the body is read — so a receiver that answered 2xx and never closed its body left the await hanging, the queue row stayed pending, and the next processor pass sent the same message again. An unclosed stream was duplicate email. There is now a 10s wall clock that destroys the stream, with the timer unref'd so a hung read cannot hold the process open at exit. 23 transport tests (2 new, both failing without these fixes), 63 across the email suites, eslint clean. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
d62407f431 |
feat(email): webhook transport as an alternative to SMTP (#1225) (#1231)
Setting EMAIL_WEBHOOK_URL makes PicPeak stop sending mail itself and POST each composed message as JSON instead, for something downstream (n8n, Make, a self-hosted relay) to deliver. Unset, every SMTP path is unchanged. Settles the four things #1225 left open: - SSRF: the URL goes through the same DNS-resolving check the outbound webhook worker uses, before every send. Private receivers are opt-in. - Transport security: https is required for anything leaving the machine. The HMAC proves who sent the body, not who can read it, and these bodies carry password-reset links and guest recovery codes. The private-network opt-in doubles as the plaintext opt-in. - Authentication: EMAIL_WEBHOOK_SECRET is required and signs the body as X-PicPeak-Signature, the same scheme as gallery webhooks. A URL without a secret leaves the transport OFF and says so once. - Attachments: carried as base64, not dropped. Oversized ones fail and stay queued rather than arriving without the invoice. Configuration is environment-only on purpose: this redirects every outbound message including password resets, so it must not be changeable from a compromised admin session. Three wiring details decide whether it works at all: docker-compose.yml declares an explicit environment block, so the vars had to be forwarded there; a fresh webhook-only install has no email_configs row (migration 001 seeds it only when SMTP_HOST is set), so the From identity falls back to EMAIL_FROM; and processEmailQueue used to return early when SMTP could not initialise, which would have left the queue permanently unprocessed. guestRecoveryService and the admin test-email endpoint were bypassing the transport — the first dereferenced a null transporter, the second told webhook-only admins to go configure SMTP. emailIntakeService deliberately stays on SMTP: it round-trips a specific mailbox's own credentials. Response handling is streamed and read bounded by hand rather than capped via axios: maxContentLength throws while reading, so a receiver that delivered the mail and then echoed a large body would have been recorded as failed and the message sent again. Note: docker-compose.dev.yml is gitignored and local-only, so the equivalent entries there are not part of this change. docker-compose.production.yml needs none — it passes .env through with env_file. Three rounds of external review; 21 transport tests, 61 across the email suites. |
||
|
Paul Nothaft
|
f4c054a661 |
fix(export): name the camera master in photo exports, not the delivered render (#1229) (#1230)
#1165 added photos.source_filename to this service's select, with a comment saying it was there so the Lightroom round-trip could still match after a re-upload — and then nothing read it. Every output path still used original_filename, which is overwritten the first time an edited render is uploaded over a proof (#745). So after a round-trip the exports named the render. Each of these formats exists to help a photographer find the master on disk, and the render's name does not. The XMP case is the sharpest: the sidecar is written next to a RAW master, so a wrongly-named one is never associated with it. Two helpers, because the sites want different things when nothing is known: cameraName() source_filename || original_filename || null cameraFilenameOrStored() the above, else the stored name The dedicated `original_filename` fields (CSV column, JSON key) keep reporting blank/null when unrecorded — echoing the sanitized stored name there would invite a match against a file that does not exist under it. The places that must emit some name (text list, CSV filename cell, XMP sidecar) fall back to the stored one, as they did before. filename_format='stored' is untouched, and rows with no source_filename still resolve to original_filename, so nothing moves for installs that have never run a replacement. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Luca
|
8db8527f9e |
feat(api): Lightroom round-trip — read proofing marks, put finished edits back (#745) (#1165)
* feat(api): Lightroom round-trip — read marks, put edits back (#745) Gets a client's proofing verdict into a desktop catalogue and a finished edit back over its proof, without anyone re-matching files by hand. Three parts: **Keep the camera filename.** photos.original_filename is the only carrier of `IMG_1234.JPG` — the stored filename is rewritten by generatePhotoFilename. But replacePhoto() overwrites original_filename with whatever name the new file arrives under, so the first re-upload of a renamed render destroys the key the NEXT round-trip needs. Migration 185 adds photos.source_filename, written once at ingest and never touched by a replace, backfilled from original_filename so existing galleries can still match on their first pass. The backfill sits outside the column guard and keys on whereNull, so a run that dies partway self-heals instead of leaving half the rows empty forever. **Read the marks.** GET /api/v1/events/:id/photos returns each photo with its client colour tallies, the caller's own marks, and a merged colour + rating. Guards copied from the sibling upload route (apiTokenAuth + read scope + photos.view + requireEventOwnership). Filters: marked_only, mark_source, color_labels, my_color_labels, min_rating, my_min_rating. The route filters to a page of ids with PhotoFilterBuilder, then enriches just those through photoExportService.getPhotosWithFeedback — the two halves already existed and neither does both, and going id-first keeps the per-colour tally query bounded by page size. services/markMerge.js decides how three possible opinions (guest colours, guest star average, the photographer's own row in photo_admin_marks) collapse into the one colour and one rating Lightroom has room for. Colour goes to the photographer on a tie — one deliberate choice beats an aggregate a tie-break already had to guess at. Rating takes the max, because a rating is a magnitude and losing the higher one quietly demotes a photo somebody rated highly. Its roundRating matches XmpGenerator.mapRating exactly so the API and an XMP sidecar can never disagree about how many stars a photo has. **Put the edit back.** POST /api/v1/events/:id/photos accepts an optional replaces_photo_id and routes to the existing replacePhoto(), preserving the photo's id, feedback, comments and position. The plugin stores the picpeak id on the catalogue photo, so the id survives the editor renaming the render — which makes it the reliable key, not the filename. Scoped to the event in the URL: a token inherits its owner's powers across every event they can see, so an unscoped id would let one gallery overwrite another's photo. For renders whose RAW never went through the plugin, findReplacementCandidate gains an opt-in number_token mode matching on the trailing digit run. Deliberately the LONGEST run and never a fixed last-N slice: multi-camera shoots disambiguate by prefixing the camera index into the number (cam11234.jpg / cam21234.jpg), and a last-4 slice reads 1234 from both bodies and reintroduces exactly the collision the prefix removes. Ambiguity is refused, never guessed. Also drops the multer temp file on the two new early returns — this route only unlinks in its catch block. * refactor(api): one rating-rounding rule, and apply match_mode where it counts Three things the pre-review pass turned up on the round-trip work: - `match_mode` reached the photo-cap pre-count but not the loop that actually picks the replacement target, so asking for `number_token` would have been counted and then quietly ignored. Both call sites now take it. - `number_token` matching read `select('*')` over every photo in the event to compare one digit run. It now reads the three columns the match needs and re-reads the single winner in full, so a 5000-photo event doesn't pull 5000 full rows through memory to answer one question. - `XmpGenerator.mapRating` and `markMerge.roundRating` were the same five thresholds written twice — the second way to do one thing that drifts the moment either is touched. The thresholds now live in markMerge and the generator delegates, which is what keeps a sidecar and the v1 API from ever disagreeing about a photo's star count. * fix(api): keep the new route in the generated OpenAPI spec The `color_labels` description carried an inline JSON example. In an unquoted YAML scalar `{ "green": 2 }` parses as a flow mapping, so swagger-jsdoc threw YAMLSemanticError and dropped the WHOLE route from the spec — visible only as a warning on boot, with the route still serving normally, which is exactly the kind of failure that survives to release. Found by booting a real instance rather than by reading the diff. * fix(api): close the four blockers from review on #1165 1. Replacing an external photo silently kept serving the old file. resolvePhotoStorageKey gives photo.source_origin precedence and returns null for 'reference'/'external', so the edit was uploaded, the row updated and 200 returned while every viewer kept getting the untouched NAS original and the upload sat orphaned. replacePhoto now repoints the row to managed and clears external_relpath. The file on the share is never touched — this moves the pointer, not the data. 2. Every replacement leaked its temp file. putFromFile COPIES on local and uploads on S3; neither consumes the source, and replacePhoto never unlinked it — while the v1 route had disabled its own cleanup on the belief that replacePhoto moved the file. Cleanup now lives in replacePhoto, which closes the admin path too (adminPhotos only unlinks in its new-files branch, so replaced files leaked there as well). The v1 route also unlinks on the FAILURE path, which returned before any cleanup ran. 3. The download-all ZIP is invalidated after a replacement, as adminPhotos.js already does. Without it guests kept downloading the pre-edit photo indefinitely, which defeats the point of the feature. 4. The round-trip could not see reference or watcher galleries at all. fileWatcher and adminExternalMedia never set original_filename — the camera name lives in `filename` for those rows — so the backfill and the GET fallback both produced NULL for exactly the galleries most likely to be driven from Lightroom. The backfill now COALESCEs, both ingest paths set source_filename, and the GET falls back to filename. Concerns: - number_token no longer reads every photo row in the event per file. A LIKE on the digit run narrows the candidate set in SQL first; the exact trailing-run check still decides, so semantics are unchanged. The token is a regex-extracted digit run, so it cannot carry a wildcard. - The replacement's activity entry is scoped to event.id instead of null. The dashboard feed excludes NULL-event rows for scoped callers (GHSA-jhcf), so it was vanishing from the audit trail of the photographer who owns the event. Nit: dropped the unused higherPriorityColor export from markMerge. Three regression tests cover the external repoint, the temp cleanup and the COALESCE backfill. 21/21 pass. * chore(migrations): renumber 185 -> 193 after gallery-folders landed 185_add_category_is_folder.js merged to main while this was in review, so the number the PR reserved is taken and main is now at 192. Knex keys on filename rather than the prefix, so both would have run — but picpeakImportService guards restores with migrationOrder(), which parses that prefix, and two files answering 185 make the forward-only check pass a backup onto a schema missing its columns. Renumbered with every reference: the header comment, the test that requires the path, and the four call-site comments that cite it. The 'migration 182' reference inside it is the colour-labels migration and is unrelated; gallery.js:1134 cites upstream's 185 and is untouched. * fix(api): keep external_relpath when a replacement converts the row The external-photo blocker fix cleared external_relpath along with flipping source_origin, which closed one hole and opened another. adminExternalMedia dedupes a re-scan on (event_id, external_relpath) — routes/adminExternalMedia.js:195 — and migration 186 puts a unique index on exactly that pair. With the column nulled, the next scan of the share would not recognise the NAS original as already imported and would insert it again, so the gallery would end up holding both the edit and a fresh copy of the file it replaced. Only source_origin needs to change: it is what resolvePhotoStorageKey keys on, and every other consumer of external_relpath reads the two together and lets source_origin decide. The stale relpath on a managed row is inert for resolution and still correct as a dedupe key. Test updated to assert the value is kept rather than cleared. * fix(uploads): say when exiftool is missing instead of blaming the RAW A server without exiftool reported `No usable embedded preview in RAW file X.CR3: spawn exiftool ENOENT` for every RAW upload. The headline describes a corrupt photo; the actual cause is a package that was never installed, demoted to a trailing detail. It sends people hunting through their camera files. Hit while testing the Lightroom round-trip (#745): an export of RAW originals failed 11 times with that message, and the file was fine. RAW upload is the only feature that needs exiftool, so an install can be missing it indefinitely and only find out when someone uploads a CR3 — which makes the wording the whole diagnosis. ENOENT now produces a message naming the dependency and the install command for Debian/Alpine/macOS, and breaks out of the tag loop instead of spawning the same missing binary twice more to report the last failure as if it described the photo. A genuinely preview-less RAW still gets the original message. Verified both paths by making exiftool unreachable via PATH rather than mocking: missing tool and unreadable file now report differently. * fix(external): a delivered edit must win a relpath-fold collision Follow-up to keeping external_relpath on a replaced photo. Keeping it is what lets adminExternalMedia still dedupe the folder re-scan, but it also leaves the row inside externalRelpathFold's sweep — and that sweep does not merely rewrite paths, it DELETES collision losers via externalPhotoDedupe. The survivor was whichever row happened to be claimed first, which is iteration order. So a replaced photo — source_origin 'managed', holding the edit the photographer just delivered — could be deleted in favour of the untouched camera original sitting next to it on the share. Managed rows now claim first and therefore survive. The external row that loses is the recoverable one: it is still on the share and a re-scan re-imports it. The edit is not recoverable. Note this is deliberately NOT the "skip managed rows in the fold" shape suggested in review. Skipping would leave those rows holding a base-relative path while every other row moved to root-relative, so the scanner — which computes root-relative — would stop matching them and import the camera original again as a duplicate. That is the exact bug keeping external_relpath exists to prevent, reintroduced through a different door. Rebasing them and protecting them from deletion keeps both properties. |
