1372 Commits

Author	SHA1	Message	Date
Paul Nothaft	18bc1b77cb	fix(upload): retire the connection after an early refusal, clean up a failed publish ... Two more from review. Refusing a body before reading it is the point of the streaming cap, but the unread bytes are still in flight on a connection the response advertises as keep-alive. Node does not drain them, so the NEXT request on that socket hangs until it times out — reproducible with an 8MB body against a 1MB cap. The error response now sets Connection: close whenever the request was not read to the end. A failed rename — ENOSPC, a vanished directory — left the fully written staging file behind. Staging names are per-attempt, so a client that retries instead of aborting accumulates one per try until the upload expires. The partial is removed on that path too. Relates to issue 1403	2026-09-11 10:22:22 +02:00
Paul Nothaft	48423c95e6	fix(upload): answer client-caused chunk states with their own status code ... An unknown, finished or expired upload id, and a complete call with chunks missing, were plain Errors with no statusCode, so both routes fell through to the blanket 500 and logged at error level. All four are the client's mistake: they read as a backend fault in monitoring and invite a retry that can never succeed. They now carry 404, 409, 410 and 400 respectively, and both routes pass a tagged status through instead of matching on the two they happened to know about. Only genuinely unexpected errors reach the 500 and the error log. No client is affected: uploadLargeFile, the only caller of this endpoint family, still has no callers of its own. Relates to issue 1403	2026-09-11 09:30:09 +02:00
Paul Nothaft	0435840a0f	fix(upload): revalidate the per-file cap before publishing a chunk ... `allowance` is computed before the body arrives, so a chunk that completed while this one was still streaming was not counted in it. Two overlapping 0.75MB chunks under a 1MB cap were therefore both accepted, leaving 1.5MB on disk; enough concurrent streams could go well past the cap before anyone asked to complete the upload. The buffered version got this right for free, because it only ever checked after reading the whole body. The streaming version keeps its pre-read check — that is what makes a too-large request cheap — and asks again with the current aggregate before renaming the staging file into place. Relates to issue 1403	2026-09-11 09:17:56 +02:00
Paul Nothaft	b87ca5fba1	fix(upload): isolate chunk staging per attempt and close before cleanup ... Two races found by a second review round, both in the staging logic added by the previous commit. - Two in-flight sends of the same chunk index shared one `.part` path, so whichever renamed first published bytes the other had already truncated. An acknowledged 10-byte chunk could end up 2 bytes. The staging suffix is now per-attempt rather than per-index. - Unlinking the partial file raced the write stream's pending open(). destroy() does not await it, so the unlink failed with ENOENT and the open then recreated the `.part` file after cleanup had supposedly finished — reported reproducible in 121 of 300 immediately-failing streams. Cleanup now waits for the stream to close. Relates to issue 1403	2026-09-11 09:06:17 +02:00
Paul Nothaft	9986d6708d	fix(upload): harden the chunk stream against abort, retry and cap failures ... Three failure paths the streaming rewrite introduced, all found by external review. None existed in the buffered version: the async iterator it replaced rejected a dead request on its own, and never opened the chunk file until it already held the whole body. - An already-destroyed request hung the call forever. If the client hangs up while auth and ownership are awaiting the database, pipe() emits neither `end` nor `error`, so the promise never settled and the write descriptor stayed open. Checked up front now, alongside `aborted` and a `close` without `readableEnded` for a body cut short mid-flight. - A failed re-send destroyed the chunk it was replacing. createWriteStream truncates on open, so re-sending an index and then failing left receivedChunks and chunkSizes still claiming the old copy: status reported 100% and completeUpload died on ENOENT. Chunks are staged through a sibling .part file and renamed only on success. - Tripping the cap stopped the 413 from reaching the client. `source` is the IncomingMessage, so destroying it destroyed the socket under the response and the client saw a connection reset instead of the size-limit JSON. The read is paused instead, which is all the cap needs. Relates to issue 1403	2026-09-11 08:55:12 +02:00
Paul Nothaft	d89401810f	fix(upload): stop buffering a chunk body before anything checks its size ... The chunk route drained the whole request into an array and concatenated it before calling uploadChunk — which is where every check lives. So a 300MB body sent against an unknown upload id was read in full, cost ~300MB of heap, and was only then answered with an error. The per-file cap was real but applied after the damage, and nothing looked at Content-Length at all. uploadChunk now takes the request stream itself and consumes nothing until the upload id, the chunk index and the declared Content-Length have all been checked against the remaining allowance. A body that clears those is streamed straight to the chunk file under a hard byte cap, so a sender that lies about its length — or sends none, the Transfer-Encoding: chunked case — is cut off at the allowance instead of being read to the end. A Buffer is still accepted, so the existing callers and chunkedUploadSizeCap tests are untouched. Worth noting the old code silently did nothing when handed a stream: fs.promises.writeFile accepts an async iterable, so the body was written while `chunkData.length` was undefined and the cap comparison was NaN > max, i.e. always false. Reachable only for an authenticated admin holding photos.upload on an event they own, and only once the CSRF gate lets application/octet-stream through. Relates to issue 1403	2026-09-11 08:30:19 +02:00
Paul Nothaft	c4d89c9d64	chore(main): release 3.131.3-beta.0 (#1373)	2026-09-10 11:54:50 +00:00
Paul Nothaft	a2bf1f644c	fix(video): try metadata extraction and thumbnail generation independently (#1371) ... * fix(video): try metadata extraction and thumbnail generation independently processUploadedVideo() gated everything behind isValidVideo(), which rejects the whole video if ffprobe can't read even one of duration/width/height -- common on some iPhone/Lightroom-exported MP4s (issue 1370). Callers (photoProcessor.js's processPhoto and processUploadedPhotos) already catch that throw and fall back to a static placeholder thumbnail plus a metadata-only retry (codex review of #845), but that fallback never got a REAL thumbnail even when generateVideoThumbnail() would have succeeded on its own -- thumbnailing doesn't need valid duration/width/height, it just seeks and grabs a frame. processUploadedVideo now tries metadata extraction and thumbnail generation independently, keeping whichever succeeds instead of discarding both on a single failed field. The callers' existing throw handling stays as a backstop. Also: extractVideoMetadata stored duration as 0 (not null) whenever ffprobe had no duration field, masking "unknown" as a fake real zero-second clip and defeating downstream `duration != null` checks meant to skip an untrustworthy value. Relates to issue 1370 * fix(video): fall back to the SVG placeholder when thumbnail generation fails processUploadedVideo could return success with thumbnailKey: null when only thumbnail generation failed. The gallery grid (GridGalleryLayout/JustifiedGalleryLayout) falls back to `photo.thumbnail_url || photo.url` when there's no thumbnail, so AuthenticatedImage downloaded the full original video and tried to render it as an <img> -- a broken tile and a potentially huge fetch just from opening the gallery. Falls back to the same ffmpeg-free SVG placeholder the callers already generate for a total processing failure, so a bare thumbnail-generation failure degrades to that placeholder too, never to "no thumbnail at all". Found by codex review. * fix(video): avoid a SQLite connection deadlock in the placeholder fallback generateVideoPlaceholder() unconditionally called getThumbnailSettings(), which queries the database directly (not through any active transaction). videoProcessor.js's new placeholder fallback can run from inside processUploadedPhotos' open per-file SQLite transaction (chunked video upload) -- knex's default SQLite pool has exactly one connection, so that second, un-transacted query deadlocks against the transaction holding it, timing out after acquireConnectionTimeout (60s). Reproduced directly against an isolated SQLite db. generateVideoPlaceholder now skips the settings lookup entirely when the caller supplies explicit width/height, and the video fallback passes the same DEFAULT_THUMBNAIL_WIDTH/HEIGHT the settings lookup would have fallen back to anyway (now exported for reuse). Found by codex review. * fix(video): throw when neither a real thumbnail nor the placeholder can be produced processUploadedVideo returned success with thumbnailKey: null when both the real thumbnail AND the SVG placeholder failed -- a total, systemic failure (storage backend down, disk full), not a quirk of one file. On stable, which doesn't have the #845 call-site fallback, this silently completed the video with no thumbnail at all instead of the retryable 'failed' status a throw here produces. On main, the pre-existing #845 fallback already absorbed this exact case (no behavior change there) -- verified against codex's own git-blame check of the pre-PR stable code before applying this. Now throws in that case, restoring the pre-existing "let the caller mark it failed and retryable" behavior for a genuinely unrecoverable video, while keeping every partial-failure case (the vast majority) resolving with whatever succeeded. Found by codex review. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-10 13:48:48 +02:00
Paul Nothaft	443ec91de9	chore(main): release 3.131.2-beta.0 (#1368)	2026-09-09 20:35:16 +00:00
Paul Nothaft	15cd5ede82	fix(backup): honor the configured database-backup destination path (#1366) ... * fix(backup): stop ignoring the configured database-backup destination path databaseBackupService.getBackupConfig() returns the raw database_backup_*-prefixed setting keys, but backup() and startScheduledBackups() destructured unprefixed names off that object (destinationPath, compress, enabled, schedule, retentionDays, emailOnSuccess/Failure). None of those keys ever existed on the config object, so every read silently fell through to its hardcoded default. The visible symptom (reported in issue 1365): the inline database dump that runs before every file backup (default ON) always tried to create /backup/database, regardless of what an admin configured, and died with EACCES on the read-only default path — before the file backup's own (correctly wired) backup_destination_path was ever reached. The standalone scheduled database-backup runner had the same bug: config.enabled was always undefined, so it silently never started regardless of database_backup_enabled. Also fixes saveManifestToLocal's manifest-directory fallback, which hardcoded /backup instead of matching the sane getStoragePath()/backups default used everywhere else for a missing backup_destination_path. Relates to issue 1365 * fix(backup): reject a database-backup destination inside a public static mount Making database_backup_destination_path actually take effect reopens a GHSA-jw8m-43r2-jqrm-class exfiltration path: that setting is writable via PUT /api/admin/database-backup/config under backup.create alone (the built-in admin role has it without settings.edit or backup.restore), with no path validation. Before this fix the setting was silently ignored (the destructuring bug), so pointing it at the public uploads/logos or fonts mount was harmless; now that it is honored, it needed the same defense GHSA-jw8m already applies to the per-request override. Rejects the setting at both the config write (immediate 400) and, defensively, at backup() time before mkdir. Found by codex review. * fix(backup): close two gaps codex round 2 found in the destination guard - The public-roots list missed the bundled fallback fonts dir (backend/assets/fonts, also mounted at /fonts, and nodejs-owned per the Dockerfile's COPY --chown so it's writable at runtime). - The comparison was case-sensitive; on a case-insensitive-but- preserving filesystem (APFS, NTFS, Docker Desktop bind mounts of either) STORAGE_PATH/UPLOADS/Logos names the same directory as uploads/logos on disk. Now compares lowercased. - database_backup_retention_days reached cleanupOldBackups unvalidated. A value <= 0 pushes the cutoff to today or the future, deleting every completed backup on the next scheduled run -- a backup.create holder achieving what backup.delete gates on the manual /cleanup route. Rejected at config-write time (400) and defensively inside cleanupOldBackups itself. - The scheduled-backup cron callback closed over retention_days from schedule-start time; a retention-only /config update (which doesn't restart the schedule) ran stale until restart. Re-reads it on every tick instead. Found by codex review, round 2. * fix(backup): resolve symlinks and add the all-in-one frontend dir to the destination guard Codex round 3 found two more bypasses of the public-root guard, both specific to the all-in-one image (Dockerfile.aio): - /app/frontend/dist (FRONTEND_DIR) ships nodejs-owned and is served unauthenticated as the built SPA -- missing from the protected-roots list. - /app/storage is a symlink to /data/storage (the actual STORAGE_PATH). A destination given as /app/storage/uploads/logos passed the guard's lexical path.resolve() comparison while resolving, on disk, to the exact same directory as the protected STORAGE_PATH/uploads/logos. isUnderPubliclyServableRoot now resolves symlinks in whatever prefix of each path already exists (resolveRealish) before comparing, rather than relying on path.resolve() alone. Also restores three fs.mkdir spies in the test file that were never un-spied, which silently leaked a rejected mock into any later test doing a real fs.mkdir -- exactly what the new symlink test needed to set up its fixture. Found by codex review, round 3. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-09 22:27:18 +02:00
Paul Nothaft	6d717544ec	chore(main): release 3.131.1-beta.0 (#1364)	2026-09-08 20:08:38 +00:00
Paul Nothaft	de3ae1f176	chore(main): release 3.131.0-beta.0 (#1362)	2026-09-08 17:32:31 +00:00
Paul Nothaft	59ef2ee9af	Merge pull request #1361 from PicPeak/feat/usage-reporting-update-prompt ... feat(usage): prompt existing admins once for usage reporting after an update	2026-09-08 19:23:54 +02:00
Paul Nothaft	ff23efec81	chore(usage): renumber prompt_shown migration to 212 ... #1359 independently added 211_revocations_without_expiry.js against the same main baseline. Renumbering this one to 212 keeps the migrations directory sequentially numbered once both land, regardless of merge order. No functional change — same up()/down(), same column.	2026-09-08 19:22:01 +02:00
Paul Nothaft	9a437ee9e1	fix(usage): preserve consent choices and make the prompt accessible	2026-09-08 17:22:47 +02:00
Paul Nothaft	d20f80112f	feat(usage): prompt existing admins once for usage reporting after an update ... An admin who already had PicPeak installed before the opt-in reporting feature existed never gets asked — the setup wizard only runs once, on a brand-new instance. Adds a one-time modal, shown on the admin's next dashboard visit after updating, offering the same choice the wizard gives a new install. - New `product_usage_state.prompt_shown` column (migration 211) and UsageService.markPromptShown(), set on either outcome (enable or decline) from both this modal and the wizard step, so an installation is never asked twice regardless of which path it took. - New POST /admin/usage/prompt-seen endpoint. - Extracted the wizard's three-point pitch (UsageReportingPitch.tsx) so the modal and the wizard step share identical copy instead of drifting apart. - The modal never shows once participation is already active, and never shows a second time after either the wizard or the modal has been through it once. Depends on #1360 (the setup wizard step this reuses).	2026-09-08 16:25:51 +02:00
Paul Nothaft	662516a5ad	fix: retain revocations for tokens without expiry	2026-09-08 15:54:01 +02:00
Paul Nothaft	a31a2e25e2	fix: interrupt idle worker waits during shutdown	2026-09-08 15:54:01 +02:00
Paul Nothaft	20a291eb34	chore(main): release 3.130.2-beta.0 (#1358)	2026-09-08 13:40:19 +00:00
Paul Nothaft	f0e6d2dfb1	fix: enforce gallery access and consolidate gallery workflows (#1357) ... Harden gallery authentication and authorization, consolidate gallery workflows, and prevent token-bearing URLs from leaking through nginx request error logs.	2026-09-08 15:34:09 +02:00
Paul Nothaft	895e5ab3cc	chore(main): release 3.130.1-beta.0 (#1354)	2026-09-08 06:53:51 +00:00
Paul Nothaft	acb25a9a1c	fix(images): probe and clean up preview tiers under the extension the encoder actually wrote (#1355) ... generatePreviewImage rewrites the output extension to match the encoding it chose, .jpg or .webp for alpha and multi-frame sources. The tier lookup in ensurePreviewImageAtWidth and the cleanup list in previewTierKeys kept the SOURCE extension instead, so for anything but a lowercase .jpg source the stat never matched: every tier request for a .png, .JPG, .jpeg, .heic or RAW photo re-ran Sharp, and cleanup never found the files it left behind, which accumulated for the life of the install. Both now derive every key the tier can live under: the .jpg and .webp candidates, plus the source-extension key last so tiers written before the rewrite are still found by lookup and by cleanup. Follow-up to issue 1020, where the mismatch was identified during review. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-08 08:41:53 +02:00
Paul Nothaft	c97341e454	fix(images): single-flight lazy rendition generation and keep the old rendition during replacement (#1350) ... * fix(images): single-flight lazy rendition generation and keep the old rendition during replacement The lazy generators in imageProcessor are check-then-generate, and the check reads the path off the photo row the route already fetched. N concurrent cold requests for one photo all held a snapshot with the path still null, all missed, and all ran the same sharp pipeline. Only the thumbnail tier path had a guard; the canonical thumbnail it falls back to, heroes, previews and preview tiers had none. One process-local map now covers every rendition, keyed by photo id and rendition (`thumbnail:<id>`, `thumbnail:<id>:w<width>`, `hero:<id>`, `preview:<id>`, `preview:<id>:w<width>`). Concurrent callers share one promise; the entry is cleared in a finally on success and failure alike so a rejection cannot poison the key. The tier stat moved inside the flight so a request arriving as the previous flight clears finds the written tier instead of missing on a stale probe. Heroes and previews also deleted the existing object before generating its replacement, and again in the catch. Both are gone, mirroring what the thumbnail generator already does: put is the last statement in the try and replaces atomically on local storage and by key on S3, so the delete only ever opened a window with no rendition at all, and a source that failed to read stripped the old rendition with the row still pointing at it. No re-read of the photo row inside the flight: the admin regenerate endpoints force a rebuild by passing a row with the path nulled, and a re-read would hand back the persisted rendition untouched. Fixes the single-flight half of issue 1020. The preview cache-key extension mismatch and any server-wide work queue remain separate. * fix(images): keep the snapshot validity check outside the single-flight With the check inside the flight, an admin regeneration (row passed with the path nulled) could join a viewer's flight for the same photo that was merely confirming an already-good rendition, and be handed back the very file it was asked to replace while the endpoint counted a success. Only a miss enters the flight now; inside it everything is a regeneration. * fix(images): forced rebuilds run after an in-flight lazy generation instead of adopting it The admin regenerate endpoints could still join a lazy flight that was already generating for the same photo. That flight read the thumbnail settings when it started, so after a settings change it produces exactly the rendition the regenerate was invoked to replace; adopting it counted a success while the old size stayed cached. ensureThumbnail and ensurePreviewImage take `{ force: true }`: skip the snapshot check and, if a flight is pending, start after it settles. Lazy misses arriving meanwhile join the forced flight, and an older flight settling late no longer evicts the newer entry from the map. * fix(images): key rendition flights by source as well as photo id replacePhoto keeps the photo id and changes path and filename. Keyed by id and width alone, a request carrying the replacement row joined a flight still rendering the file it replaced and was handed the old image, which the gallery caches for 30 minutes. The tier map this replaced was keyed by storage key and so already told the two apart. * test(thumbnails): wait for the regenerate loop's completion line instead of a fixed 150 ms The loop runs in setImmediate after the response. Under a loaded machine (fifteen suites in parallel, each booting a migrated SQLite) it took longer than 150 ms once and the assertions ran against a half-finished mock call list. Poll the logger spy for the "regeneration complete" line with a 10 s deadline; the suite also finishes sooner because the wait ends as soon as the loop does. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-08 08:33:04 +02:00
Paul Nothaft	7e9a61c245	chore(main): release 3.130.0-beta.0 (#1347)	2026-09-07 20:36:18 +00:00
Paul Nothaft	810801a9ab	fix(events): drop non-canonical keys from the event update before any check runs (#1346) ... PUT /admin/events/:id spreads the body into the UPDATE. SQLite resolves quoted identifiers case-insensitively, so `{ "Event_Name": ... }` lands on event_name there — while every check in the handler (validators, the field-level permission guards, the deny-set) keys on the exact lowercase name. The deny-set already case-folded for its own columns; every other column was reachable through a spelling variant. Every events column and every input-only key the handler accepts is lowercase snake_case, so a key with any uppercase in it is not something a legitimate client sends. Such keys are now removed before anything looks at the body. Postgres was unaffected (quoted identifiers are case-sensitive there; a variant produced a 500 instead). Surfaced by the Codex review of the folder-watcher change, where a photos.upload guard on external_watch could be walked around this way. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 22:30:13 +02:00
Paul Nothaft	8cc7d7d14a	feat(external-media): watch reference folders and import new files automatically (#1345) ... * feat(external-media): watch reference folders and import new files automatically Managed uploads dropped into storage/events/active are picked up by the chokidar watcher; external media had no equivalent, so a NAS folder that keeps growing needed an admin to open the event and press Import every time. Relates to issue 1187. - The import pass moves out of the route into services/externalImportService.js. The watcher and the Import button now run the identical function; the route only validates and maps errors to status codes. - Mutual exclusion is the per-event claim from maintenanceJobState (`external_import:<id>`, seeded on demand by the new ensure()) instead of the in-process Set. The Set stopped a double-click in one process; the claim also stops the watcher on a second replica, or an admin clicking while the watcher is mid-run elsewhere. The run heartbeats so a claim from a dead process is taken over. - services/externalMediaWatcher.js: per-event opt-in via the new events.external_watch column (migration 208), chokidar with awaitWriteFinish so a copy in flight is not imported half-written, debounced full pass per change, a timer sweep every 15 minutes as the fallback for NFS/SMB mounts that deliver no inotify events, optional stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched events is re-read every minute, so the toggle works from any replica. A watcher that just started runs one pass immediately. - Deletions are ignored on purpose: a file vanishing from a NAS is at least as likely to be a reorganisation or a dropped mount as an intentional removal, and acting on it would delete a guest-visible photo. Rows whose file is gone stay, as they do today. - Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local. - Quiet system passes stay out of the activity log; runs that imported something are logged with actor external-media-watcher. - Frontend: "Watch folder for new files" checkbox under the external folder picker, status line in view mode, EN/DE strings. * fix(external-media): close the review gaps in the folder watcher Codex review of the watcher, round 1. All six findings were real: - Enabling the watcher, or pointing an enabled one at another folder, now requires photos.upload — the permission the manual Import already requires. events.edit alone was a way around it. Only the transition is checked, so a role without photos.upload can still edit an already-watched event. The checkbox is disabled for such roles. - Automatic passes defer files that are still changing: anything modified inside the stability window, or whose size moves across one wait of that window, is left for the next pass. chokidar's awaitWriteFinish only settles the file that fired the event, and the sweep sees no events at all, so a sibling still being copied could be inserted half-written and then skipped forever. - Photos an admin deleted are not brought back by the sweep. The delete routes record the file in external_import_exclusions (migration 209); automatic passes skip the list, the manual Import ignores it and clears it for what it imports. - The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three compose files; they were documented but the backend services use explicit environment lists, so the kill switch did nothing. - A pass re-checks is_active / is_archived at run time, not only in the minutely reconcile. - The lease is renewed on a timer for the whole run, walk included, and ownership is checked before the event row is touched. * fix(external-media): make automatic passes follow the row, not rewrite it Codex review round 2, four findings, all applied: - The event update route drops non-canonical spellings of external_watch and external_path before the permission guard. SQLite resolves column names case-insensitively, so `External_Watch` reached the column while the guard only looked at the lowercase key. - Exclusions are checked per file at insert time, not against a snapshot taken before the settle wait. A photo deleted during the wait was present in the snapshot and got re-inserted by the loop. - An automatic pass no longer writes source_mode / external_path. It re-reads the row after the walk and the settle wait and stops if the folder changed or the event went managed; the manual Import is the only writer. The options are now `automatic` + `settleMs`. - A pass that deferred files re-arms the debounced import, so a file copied just before the watcher started is not stranded when the sweep is disabled. * fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying Codex review round 3, both findings applied: - recordExclusions keys on external_relpath alone. A replaced external photo becomes managed but keeps its relpath on purpose, and deleting that replacement must not republish the NAS original. - An automatic pass checks the full watcher predicate (reference mode, same folder, watch on, active, not archived) before it inserts and on every heartbeat tick during the loop, and stops as soon as the event no longer qualifies. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 22:29:55 +02:00
Paul Nothaft	1b3f721d10	chore(main): release 3.129.0-beta.0 (#1344)	2026-09-07 18:47:26 +00:00
Paul Nothaft	fb9da72f14	feat(security): opt-in recoverable gallery passwords (#1341) ... * feat(security): opt-in recoverable gallery passwords Gallery passwords are bcrypt hashes, so an admin who needs to hand a password to a client a second time has to reset it, which invalidates what the client already has. This adds a security setting, security_gallery_password_recoverable, off by default, that keeps an AES-256-GCM encrypted copy of each gallery password and client PIN next to the hash. The key is derived from GALLERY_PASSWORD_ENCRYPTION_KEY or JWT_SECRET. While the setting is on: - create, publish, send-later, edit, reset and the v1 API write the copy alongside the hash; turning a gallery's password requirement off clears it - GET /api/admin/events/:id/password returns the copy to admins with events.edit and ownership, and writes a gallery_password_viewed activity entry on every real reveal - resend-email uses the stored password instead of the "set at creation" sentinel, so the client receives what already works Switching the setting off purges every stored copy. Login and hash verification are untouched; the copy is never read on the gallery side. The Security tab carries the toggle with a warning that stays visible, and the event page shows "Show password" with copy buttons only while the setting is on and the gallery has a secret. Relates to issue 1271 * fix(security): close the write-versus-switch-off race in the password vault The recoverable setting is read while an event insert is assembled and the client-PIN hash awaits after that, so a settings request that switched the feature off and purged in that gap was overtaken by the insert. Every write site now re-reads the setting right after its statement and clears its own row when the setting is off; the settings writer flips the value before it purges, so either the purge or the re-check catches the row. * fix(security): resend carries the stored client PIN and link; deterministic tamper test The creation mail includes the client-access link and PIN; a resend only sent the gallery password even when a stored PIN was available. The ciphertext tamper assertion replaced the last two characters with a constant, which was a no-op roughly once in 4096 runs. * fix(security): drop the revealed password after Send gallery email The send-later route can replace the password; the share card keys its revealed copy on the event query's refetch time, so invalidate the event after the send like the other password-changing mutations do. * fix(security): purge leftovers before the setting write when turning recovery on Switching on wrote the setting first and purged after, so a password write that read the new "on" in between stored a copy the purge then deleted. Turning on now purges before the write; turning off keeps purging after it, which together with the write-site re-check leaves the vault holding exactly what was written while the setting was on. * chore(security): drop the duplicate rateLimitService import left by the rebase * chore(usage): register the password recovery routes in the v5 coverage inventory The inventory moved from v4 to v5 on main; the entry added by this branch followed it. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 20:36:48 +02:00
Paul Nothaft	16b79ee119	chore(main): release 3.128.0-beta.0 (#1343)	2026-09-07 18:10:48 +00:00
Paul Nothaft	5c1e38d921	feat(usage): distinguish real edits and template delivery with v5 consent (#1339) ... * feat(usage): distinguish real edits and template delivery with v5 consent * fix(usage): exclude queued test messages and count reorders as edits - queueEmail carries usageEligible: false into email_data and the queue processor passes it on, so the dev tools' send-test-email no longer records email_template_delivery once the worker sends it. - event-types/reorder and categories/reorder-global compare the persisted order before and after and record the v5 edit markers only when it changed, matching the display_order edit already counted on PUT. - normalized() builds arrays with Array.from so a row array from the sqlite binding compares equal under Jest's separate realm. * fix(usage): cover per-gallery category order and workflow test runs - categories/reorder records category_editing when an event's override changes; reorder/:eventId records it when an override was actually removed. - send_email and the collections handoff pass usageEligible: false for a workflow test run (engine.testRun sets __test), so a non-dry test send is not counted as template delivery. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 20:03:15 +02:00
Paul Nothaft	69754f8a2c	fix(email): scrub gallery passwords from the sent-mail archive (#1340) ... * fix(email): scrub gallery passwords from the sent-mail archive The email queue kept every gallery password and client PIN in clear text in email_data and rendered_html after the mail was sent, and the Messages reading pane handed them back to any admin with the messaging flag. A password hash in the events table bought nothing while the plaintext sat next to it. Once a mail is out, or its retries are exhausted, the processor now masks secret-looking variables (password, passcode, pin) in email_data and replaces their values in the rendered body, plain and HTML-escaped. The reading pane applies the same masking to rows archived before this change. Pending rows keep the real values so a retry still sends them. Relates to issue 1271 * fix(email): keep a quoted ">" from cutting an attribute value out of redaction The tag splitter stopped at the first ">", so a template attribute such as title="{{gallery_password}} > details" left the password unmasked in the archived HTML while email_data was already masked. The tokenizer is now quote-aware; a tag with an unbalanced quote falls through as text and is scrubbed there. * fix(email): scrub secrets inside HTML comments in the archived body A comment such as <!-- PIN: {{client_password}} --> was split off as a tag and its body, which has no attribute, was never scrubbed. Comments are now one segment and their content is masked whole. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 20:02:57 +02:00
Paul Nothaft	8017370271	feat(settings): expose the API rate limiter in the Security tab (#1338) ... * feat(settings): expose the API rate limiter in the Security tab The general per-IP limiter had six settings in app_settings and a backend route to write them, and no screen. Installs ran on the code fallback — 300 requests per 15 minutes per IP — with no way to see it, which is how issue 1287 played out: a 546-photo gallery exhausted the budget for one viewer and the operator learned about the setting from a grep of the backend log. Security tab: a card with the six settings, the validation ranges the route enforces, a one-line explanation per field, and a note that the unit is the client IP — an office or household behind one NAT shares a budget, and behind a proxy TRUST_PROXY has to cover the proxy or every visitor shares its address. The tab's Save button saves the limiter through its own route. The limiter values are checked against the route's ranges before anything is written and the limiter is written first, so a rejected value cannot leave the password/session settings half-saved behind a failure toast. Backend, three things the screen needed: - The settings read fills the six keys with the code defaults when they have no row, so the form shows the budget in force rather than an empty field; the defaults live in one exported constant the limiter itself reads. - The write route upserts instead of updating: on a fresh install, which has no rows, the old UPDATE matched nothing and the route answered 200 while changing nothing. - The live limiter instances move into rateLimitService and the write route rebuilds them. express-rate-limit fixes windowMs when an instance is built — max and skip re-read the settings per request, the window does not — so a saved window used to apply only after a restart. The gates in server.js resolve the instance per request through the service's getters. A rebuild starts fresh counters, which on a settings change is acceptable. The limiters get explicit MemoryStores and a rebuild shuts the superseded ones down, because a store keeps a cleanup interval alive for as long as it exists and dropping the reference alone would leak one timer per save. Tests: the read surfaces defaults and honours the key filter; the write creates rows on a fresh database, the limiter sees the values immediately and hands the gates a fresh instance; existing rows are updated not duplicated; out-of-range values are rejected. The tab renders the values, edits through the hook state, carries the ranges, saves with the tab's button, and the pre-write validation accepts the bounds and rejects outside them and cleared fields. Relates to issue 1337 * docs(security): point the rate limiter doc at the Security tab and the upsert --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 20:02:40 +02:00
Paul Nothaft	834111fc66	chore(main): release 3.127.2-beta.0 (#1342)	2026-09-07 14:40:56 +00:00
Paul Nothaft	9bbdca9fc5	docs(security): correct the rate limiter defaults and how they are set (#1336) ... * docs(security): correct the rate limiter defaults and how they are set SECURITY_LOGGING.md said rate_limit_max_requests defaults to 1000 and that the settings are configurable via the admin panel. The code fallback when no app_settings row exists is 300 (19e125d8), a fresh install has no row, and there is no admin screen: the settings are written by PUT /api/admin/settings/security/rate-limit, which nothing in the frontend calls. The reporter of issue 1287 ran on the 300 default without any way to see it. The table now carries the real defaults, what the auth budget counts, the exemptions including the gallery-image one from v3.127.0-beta.0, and the TRUST_PROXY and shared-NAT caveats. Relates to issue 1287 * docs(security): note the rate-limit route only updates existing rows --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>	2026-09-07 16:35:13 +02:00
Paul Nothaft	e43216311b	chore(main): release 3.127.1-beta.0 (#1334)	2026-09-07 07:48:14 +00:00
Paul Nothaft	1f09ac2d0d	chore(main): release 3.127.0-beta.0 (#1331)	2026-09-07 07:29:24 +00:00
Paul Nothaft	f63347d79a	Merge pull request #1330 from PicPeak/chore/usage-catalog-english-only ... chore(usage): make the capability catalog English-only	2026-09-07 09:19:09 +02:00
Paul Nothaft	ac24319135	Merge pull request #1324 from PicPeak/fix/1287-gallery-images-rate-limit ... fix(security): stop a gallery viewer's own image fetches spending the anonymous budget	2026-09-07 09:18:46 +02:00
Paul Nothaft	e06d0b4513	fix(setup): require Node 22.12 for sanitize-html	2026-09-07 09:00:01 +02:00
Paul Nothaft	aa6e5d613f	chore(usage): make the capability catalog English-only ... features.v2/v3/v4.json carried every name, description and definition twice, once in English and once in German, inside a file that is source code, is vendored byte-identically into the collector and is served as the consented catalog. Source stays English. The German strings already existed a second time in the frontend locale file, which is what the consent dialog actually renders (UsageCatalog reads productUsage.catalog.<key>, never the JSON), so the copy in the catalog was a duplicate that could only ever drift. The `de` fields are gone from all three catalogs, their frontend copies and the inventory definitions; the docs coverage file and FEATURE_COVERAGE.md list English only. Nothing on the wire changes: the report schema is derived from the keys, and the catalog's text is not part of any signature or consent version string. The coverage test now pins the catalog to the en locale verbatim and requires the de locale to cover every key and field, without dictating its wording. The collector holds the same catalog files and needs the same change to stay byte-identical, plus its German catalog strings moved into its own locale file; that is filed there.	2026-09-07 08:26:10 +02:00
Paul Nothaft	65831785a2	fix(security): bump sanitize-html to 2.17.7 ... Trivy flags the backend image on two sanitize-html advisories, both fixed upstream: - CVE-2026-63670 (fixed 2.17.6): a literal solidus after a raw-text end tag (`</textarea/>`) is treated as text by htmlparser2 and re-emitted unescaped, so disallowed markup passes when textarea or xmp is in allowedTags. - CVE-2026-84371 (fixed 2.17.7): an SVG SMIL animation whose attributeName selects href lets the sibling values/from/to/by attributes carry URLs past the scheme policy. 2.17.5 -> 2.17.7, exact pin as before. The new version brings its own htmlparser2 12 / domhandler 6 / domutils 4 / dom-serializer 3 / entities 8 tree under node_modules/sanitize-html; nothing else in the lock moves. That tree is ESM-only, so the backend now needs unflagged require(esm): Node 20.19+ or 22.12+. The image is node:22-alpine and CI runs 22, but engines.node still admitted 22.0-22.11, where require('sanitize-html') throws ERR_REQUIRE_ESM at startup (publicSiteService loads it during initialisation). engines is now ^20.19.0 || >=22.12.0 and the native setup script's Node check enforces the same range instead of accepting any 22.x. On the supported versions the sanitiser behaves identically to 2.17.5 on the tracker and newsletter fixtures. Jest 29's CommonJS registry cannot evaluate ESM either, so every suite importing a route or service that uses the sanitiser would fail at import. jest.config.js now maps `sanitize-html` to jest.sanitizeHtml.js, which hands that one module to Node's real loader via process.getBuiltinModule('module') — a plain require('module') inside Jest is Jest's wrapper and returns an empty object for this package. Verified against a real 2.17.7 install: the sanitiser suites and a settings route suite pass; without the mapper they fail with "Cannot use import statement outside a module".	2026-09-06 23:06:43 +02:00
Paul Nothaft	7b2dd3fab1	fix(security): stop a gallery viewer's own image fetches spending the anonymous budget ... The general per-IP limiter was inert until b0f33c17 registered it ahead of the routers (budget 100, raised to 300 by 19e125d8), and 839bf4e4 then stopped gallery tokens from earning the authenticated skip. Since that release every guest has been paying for the gallery's thumbnails out of 300 requests per 15 minutes per IP. A 546-photo grid runs out mid-scroll: the remaining tiles come back 429, which the frontend turned into blank tiles with no error and no retry, and the next refresh finds the photo list limited too. Reproduced in iOS Safari against a seeded 546-photo Grid gallery: loading in bursts, then nothing, no console output, no recovery — the exact shape of the large-gallery report, whose plateaus were 125, 235, 300 and 308 tiles. A token that verifies and names the gallery in the path is now exempt on the image routes only: thumbnail, preview, hero and photo, GET only. The photo list, downloads, feedback and every write stay on the budget, a token for one gallery buys nothing on another, and the exemption rides on skip_authenticated so an operator who turns the skip off counts guests too. The 839bf4e4 concern — a free token as an unlimited budget on every /api route — stays closed; what this hands back is the bandwidth of routes a 300-request budget never bounded anyway. Relates to issue 1287	2026-09-06 22:47:37 +02:00
Paul Nothaft	83b74e2256	chore(main): release 3.126.3-beta.0 (#1323)	2026-09-06 19:45:17 +00:00
Paul Nothaft	ef8a52f02c	fix(usage): introduce consented v4 without changing historical reports	2026-09-06 21:34:04 +02:00
Paul Nothaft	3cc893126d	Merge remote-tracking branch 'origin/main' into codex/usage-v4-client	2026-09-06 21:18:58 +02:00
Paul Nothaft	6459bff50e	chore(main): release 3.126.2-beta.0 (#1319)	2026-09-06 19:13:42 +00:00
Paul Nothaft	c4b03a831f	Merge pull request #1315 from PicPeak/fix/1287-retry-failed-tiles ... fix(gallery): retry a failed image fetch once the tile is back on screen	2026-09-06 21:07:22 +02:00
Paul Nothaft	02b353e54f	fix(usage): report restricted gallery downloads in v3 instead of an always-true signal ... gallery_downloads.configured was true on every installation with a gallery. allow_downloads ships true — column default in migration 037 and the create route both set it — and the snapshot asked "at least one gallery has it on". The fleet value was ~100% by construction and could not separate a deliberate configuration from an untouched one. v2 consented to that key under that description, so v2 keeps sending it unchanged. v3 replaces it with gallery_downloads_restricted: at least one gallery has downloads switched off, which is the only state of that column anyone actually decides. Same catalog position, so the disclosed capability count stays at 86; the frontend copy, the EN/DE catalog strings, the coverage inventory and FEATURE_COVERAGE.md follow. Done in v3 rather than a v4 because v3 is on main and in no release yet, so nobody has consented to it. The collector carries the same catalog and has to take this change before the release that ships v3. One guard for the window in which :main / :beta images already carried the old v3 catalog. A report queued under it fails local validation on this build, and deliver() left a locally invalid report pending for good, blocking every operation behind it. A report's payload is derived state, so deliver() now rebuilds it from the current snapshot in place and sends that. Packet ID and sequence are kept — a re-signed retry has to reuse them so a lost acknowledgement does not duplicate data — and reports only: a stale registration, deletion or command is a genuine conflict and keeps the existing handling. Tests: the v3 snapshot counts a switched-off gallery and ignores enabled ones, v2 still reports the old key with the old meaning, and a stale queued report goes out rebuilt under the same packet id while a valid one is sent untouched. Relates to issue 1308	2026-09-06 20:42:28 +02:00
Paul Nothaft	77ae94e649	fix(gallery): retry a failed image fetch once the tile is back on screen ... A rejected fetch in AuthenticatedImage set the error state, rendered nothing, and never asked again. The fetch effect only re-runs when its inputs change, and for a grid tile they never do — so a transient failure (a hiccup on cellular, or Safari cancelling loads when the tab goes to the background) was a permanently blank tile with no request in flight and nothing in any log. Grid passes no fallbackSrc, so there was not even a broken-image icon to point at. The retry is bounded and gated. Three attempts with a doubling delay (2 s, 4 s, 8 s), and an attempt fires only once the placeholder intersects the viewport and the document is visible, so a tile that failed while the user was away retries when they come back rather than while they are still gone. A new src gets a fresh budget. The fallbackSrc path is untouched: it already renders a plain <img> and should not loop. Two refinements from review. A final 4xx (anything but 408 and 429) exhausts the budget at once: an expired gallery token or a missing photo cannot be retried into existence, and on a 68-tile viewport three retries each would be ~200 requests that cannot succeed. And a 429's Retry-After is honoured as the minimum delay, because the backoff alone would spend every retry inside a 15-minute rate-limit window and leave the tile blank after the limit had lifted. Retry-After is not CORS-safelisted, so server.js now exposes it for split-origin deployments alongside Content-Disposition. The error branch now renders the same grey box as the loading state instead of null. That is what the retry effect observes, and it is also something the user can see. The empty-src branch now clears the error flag too, so a tile whose src is removed after a failure does not keep showing the failure box. Nine tests in AuthenticatedImage.retry.test.tsx; the retry cases fail against the previous version. Not presented as the fix for the iOS report. It closes the one gap that turns a transient failure into a permanent one, which the reporter asked for in the original issue, and it is worth having on any device. Relates to issue 1287	2026-09-06 20:13:57 +02:00
Paul Nothaft	25c3e3d7b8	chore(main): release 3.126.1-beta.0 (#1314)	2026-09-06 17:59:34 +00:00