Two more from review.
Refusing a body before reading it is the point of the streaming cap, but the
unread bytes are still in flight on a connection the response advertises as
keep-alive. Node does not drain them, so the NEXT request on that socket hangs
until it times out — reproducible with an 8MB body against a 1MB cap. The
error response now sets Connection: close whenever the request was not read to
the end.
A failed rename — ENOSPC, a vanished directory — left the fully written
staging file behind. Staging names are per-attempt, so a client that retries
instead of aborting accumulates one per try until the upload expires. The
partial is removed on that path too.
Relates to issue 1403
An unknown, finished or expired upload id, and a complete call with chunks
missing, were plain Errors with no statusCode, so both routes fell through to
the blanket 500 and logged at error level. All four are the client's mistake:
they read as a backend fault in monitoring and invite a retry that can never
succeed.
They now carry 404, 409, 410 and 400 respectively, and both routes pass a
tagged status through instead of matching on the two they happened to know
about. Only genuinely unexpected errors reach the 500 and the error log.
No client is affected: uploadLargeFile, the only caller of this endpoint
family, still has no callers of its own.
Relates to issue 1403
`allowance` is computed before the body arrives, so a chunk that completed
while this one was still streaming was not counted in it. Two overlapping
0.75MB chunks under a 1MB cap were therefore both accepted, leaving 1.5MB on
disk; enough concurrent streams could go well past the cap before anyone asked
to complete the upload.
The buffered version got this right for free, because it only ever checked
after reading the whole body. The streaming version keeps its pre-read check —
that is what makes a too-large request cheap — and asks again with the current
aggregate before renaming the staging file into place.
Relates to issue 1403
Two races found by a second review round, both in the staging logic added by
the previous commit.
- Two in-flight sends of the same chunk index shared one `.part` path, so
whichever renamed first published bytes the other had already truncated. An
acknowledged 10-byte chunk could end up 2 bytes. The staging suffix is now
per-attempt rather than per-index.
- Unlinking the partial file raced the write stream's pending open(). destroy()
does not await it, so the unlink failed with ENOENT and the open then
recreated the `.part` file after cleanup had supposedly finished — reported
reproducible in 121 of 300 immediately-failing streams. Cleanup now waits for
the stream to close.
Relates to issue 1403
Three failure paths the streaming rewrite introduced, all found by external
review. None existed in the buffered version: the async iterator it replaced
rejected a dead request on its own, and never opened the chunk file until it
already held the whole body.
- An already-destroyed request hung the call forever. If the client hangs up
while auth and ownership are awaiting the database, pipe() emits neither
`end` nor `error`, so the promise never settled and the write descriptor
stayed open. Checked up front now, alongside `aborted` and a `close` without
`readableEnded` for a body cut short mid-flight.
- A failed re-send destroyed the chunk it was replacing. createWriteStream
truncates on open, so re-sending an index and then failing left
receivedChunks and chunkSizes still claiming the old copy: status reported
100% and completeUpload died on ENOENT. Chunks are staged through a sibling
.part file and renamed only on success.
- Tripping the cap stopped the 413 from reaching the client. `source` is the
IncomingMessage, so destroying it destroyed the socket under the response
and the client saw a connection reset instead of the size-limit JSON. The
read is paused instead, which is all the cap needs.
Relates to issue 1403
The chunk route drained the whole request into an array and concatenated it
before calling uploadChunk — which is where every check lives. So a 300MB body
sent against an unknown upload id was read in full, cost ~300MB of heap, and
was only then answered with an error. The per-file cap was real but applied
after the damage, and nothing looked at Content-Length at all.
uploadChunk now takes the request stream itself and consumes nothing until the
upload id, the chunk index and the declared Content-Length have all been
checked against the remaining allowance. A body that clears those is streamed
straight to the chunk file under a hard byte cap, so a sender that lies about
its length — or sends none, the Transfer-Encoding: chunked case — is cut off
at the allowance instead of being read to the end.
A Buffer is still accepted, so the existing callers and chunkedUploadSizeCap
tests are untouched. Worth noting the old code silently did nothing when
handed a stream: fs.promises.writeFile accepts an async iterable, so the body
was written while `chunkData.length` was undefined and the cap comparison was
NaN > max, i.e. always false.
Reachable only for an authenticated admin holding photos.upload on an event
they own, and only once the CSRF gate lets application/octet-stream through.
Relates to issue 1403
* fix(video): try metadata extraction and thumbnail generation independently
processUploadedVideo() gated everything behind isValidVideo(), which
rejects the whole video if ffprobe can't read even one of
duration/width/height -- common on some iPhone/Lightroom-exported
MP4s (issue 1370). Callers (photoProcessor.js's processPhoto and
processUploadedPhotos) already catch that throw and fall back to a
static placeholder thumbnail plus a metadata-only retry (codex
review of #845), but that fallback never got a REAL thumbnail even
when generateVideoThumbnail() would have succeeded on its own --
thumbnailing doesn't need valid duration/width/height, it just seeks
and grabs a frame.
processUploadedVideo now tries metadata extraction and thumbnail
generation independently, keeping whichever succeeds instead of
discarding both on a single failed field. The callers' existing
throw handling stays as a backstop.
Also: extractVideoMetadata stored duration as 0 (not null) whenever
ffprobe had no duration field, masking "unknown" as a fake real
zero-second clip and defeating downstream `duration != null` checks
meant to skip an untrustworthy value.
Relates to issue 1370
* fix(video): fall back to the SVG placeholder when thumbnail generation fails
processUploadedVideo could return success with thumbnailKey: null
when only thumbnail generation failed. The gallery grid
(GridGalleryLayout/JustifiedGalleryLayout) falls back to
`photo.thumbnail_url || photo.url` when there's no thumbnail, so
AuthenticatedImage downloaded the full original video and tried to
render it as an <img> -- a broken tile and a potentially huge
fetch just from opening the gallery.
Falls back to the same ffmpeg-free SVG placeholder the callers
already generate for a total processing failure, so a bare
thumbnail-generation failure degrades to that placeholder too,
never to "no thumbnail at all".
Found by codex review.
* fix(video): avoid a SQLite connection deadlock in the placeholder fallback
generateVideoPlaceholder() unconditionally called getThumbnailSettings(),
which queries the database directly (not through any active transaction).
videoProcessor.js's new placeholder fallback can run from inside
processUploadedPhotos' open per-file SQLite transaction (chunked video
upload) -- knex's default SQLite pool has exactly one connection, so
that second, un-transacted query deadlocks against the transaction
holding it, timing out after acquireConnectionTimeout (60s). Reproduced
directly against an isolated SQLite db.
generateVideoPlaceholder now skips the settings lookup entirely when
the caller supplies explicit width/height, and the video fallback
passes the same DEFAULT_THUMBNAIL_WIDTH/HEIGHT the settings lookup
would have fallen back to anyway (now exported for reuse).
Found by codex review.
* fix(video): throw when neither a real thumbnail nor the placeholder can be produced
processUploadedVideo returned success with thumbnailKey: null when
both the real thumbnail AND the SVG placeholder failed -- a total,
systemic failure (storage backend down, disk full), not a quirk of
one file. On stable, which doesn't have the #845 call-site fallback,
this silently completed the video with no thumbnail at all instead
of the retryable 'failed' status a throw here produces. On main,
the pre-existing #845 fallback already absorbed this exact case
(no behavior change there) -- verified against codex's own
git-blame check of the pre-PR stable code before applying this.
Now throws in that case, restoring the pre-existing "let the caller
mark it failed and retryable" behavior for a genuinely unrecoverable
video, while keeping every partial-failure case (the vast majority)
resolving with whatever succeeded.
Found by codex review.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* fix(backup): stop ignoring the configured database-backup destination path
databaseBackupService.getBackupConfig() returns the raw
database_backup_*-prefixed setting keys, but backup() and
startScheduledBackups() destructured unprefixed names off that
object (destinationPath, compress, enabled, schedule,
retentionDays, emailOnSuccess/Failure). None of those keys ever
existed on the config object, so every read silently fell through
to its hardcoded default.
The visible symptom (reported in issue 1365): the inline database
dump that runs before every file backup (default ON) always tried
to create /backup/database, regardless of what an admin configured,
and died with EACCES on the read-only default path — before the
file backup's own (correctly wired) backup_destination_path was
ever reached. The standalone scheduled database-backup runner had
the same bug: config.enabled was always undefined, so it silently
never started regardless of database_backup_enabled.
Also fixes saveManifestToLocal's manifest-directory fallback,
which hardcoded /backup instead of matching the sane
getStoragePath()/backups default used everywhere else for a
missing backup_destination_path.
Relates to issue 1365
* fix(backup): reject a database-backup destination inside a public static mount
Making database_backup_destination_path actually take effect
reopens a GHSA-jw8m-43r2-jqrm-class exfiltration path: that
setting is writable via PUT /api/admin/database-backup/config
under backup.create alone (the built-in admin role has it
without settings.edit or backup.restore), with no path
validation. Before this fix the setting was silently ignored
(the destructuring bug), so pointing it at the public
uploads/logos or fonts mount was harmless; now that it is
honored, it needed the same defense GHSA-jw8m already applies
to the per-request override.
Rejects the setting at both the config write (immediate 400)
and, defensively, at backup() time before mkdir.
Found by codex review.
* fix(backup): close two gaps codex round 2 found in the destination guard
- The public-roots list missed the bundled fallback fonts dir
(backend/assets/fonts, also mounted at /fonts, and nodejs-owned
per the Dockerfile's COPY --chown so it's writable at runtime).
- The comparison was case-sensitive; on a case-insensitive-but-
preserving filesystem (APFS, NTFS, Docker Desktop bind mounts of
either) STORAGE_PATH/UPLOADS/Logos names the same directory as
uploads/logos on disk. Now compares lowercased.
- database_backup_retention_days reached cleanupOldBackups
unvalidated. A value <= 0 pushes the cutoff to today or the
future, deleting every completed backup on the next scheduled
run -- a backup.create holder achieving what backup.delete
gates on the manual /cleanup route. Rejected at config-write
time (400) and defensively inside cleanupOldBackups itself.
- The scheduled-backup cron callback closed over retention_days
from schedule-start time; a retention-only /config update
(which doesn't restart the schedule) ran stale until restart.
Re-reads it on every tick instead.
Found by codex review, round 2.
* fix(backup): resolve symlinks and add the all-in-one frontend dir to the destination guard
Codex round 3 found two more bypasses of the public-root guard,
both specific to the all-in-one image (Dockerfile.aio):
- /app/frontend/dist (FRONTEND_DIR) ships nodejs-owned and is
served unauthenticated as the built SPA -- missing from the
protected-roots list.
- /app/storage is a symlink to /data/storage (the actual
STORAGE_PATH). A destination given as /app/storage/uploads/logos
passed the guard's lexical path.resolve() comparison while
resolving, on disk, to the exact same directory as the protected
STORAGE_PATH/uploads/logos. isUnderPubliclyServableRoot now
resolves symlinks in whatever prefix of each path already
exists (resolveRealish) before comparing, rather than relying on
path.resolve() alone.
Also restores three fs.mkdir spies in the test file that were
never un-spied, which silently leaked a rejected mock into any
later test doing a real fs.mkdir -- exactly what the new symlink
test needed to set up its fixture.
Found by codex review, round 3.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
#1359 independently added 211_revocations_without_expiry.js against
the same main baseline. Renumbering this one to 212 keeps the
migrations directory sequentially numbered once both land, regardless
of merge order. No functional change — same up()/down(), same column.
An admin who already had PicPeak installed before the opt-in reporting
feature existed never gets asked — the setup wizard only runs once, on
a brand-new instance. Adds a one-time modal, shown on the admin's next
dashboard visit after updating, offering the same choice the wizard
gives a new install.
- New `product_usage_state.prompt_shown` column (migration 211) and
UsageService.markPromptShown(), set on either outcome (enable or
decline) from both this modal and the wizard step, so an
installation is never asked twice regardless of which path it took.
- New POST /admin/usage/prompt-seen endpoint.
- Extracted the wizard's three-point pitch (UsageReportingPitch.tsx)
so the modal and the wizard step share identical copy instead of
drifting apart.
- The modal never shows once participation is already active, and
never shows a second time after either the wizard or the modal has
been through it once.
Depends on #1360 (the setup wizard step this reuses).
Harden gallery authentication and authorization, consolidate gallery workflows, and prevent token-bearing URLs from leaking through nginx request error logs.
generatePreviewImage rewrites the output extension to match the encoding
it chose, .jpg or .webp for alpha and multi-frame sources. The tier
lookup in ensurePreviewImageAtWidth and the cleanup list in
previewTierKeys kept the SOURCE extension instead, so for anything but a
lowercase .jpg source the stat never matched: every tier request for a
.png, .JPG, .jpeg, .heic or RAW photo re-ran Sharp, and cleanup never
found the files it left behind, which accumulated for the life of the
install.
Both now derive every key the tier can live under: the .jpg and .webp
candidates, plus the source-extension key last so tiers written before
the rewrite are still found by lookup and by cleanup.
Follow-up to issue 1020, where the mismatch was identified during review.
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* fix(images): single-flight lazy rendition generation and keep the old rendition during replacement
The lazy generators in imageProcessor are check-then-generate, and the
check reads the path off the photo row the route already fetched. N
concurrent cold requests for one photo all held a snapshot with the path
still null, all missed, and all ran the same sharp pipeline. Only the
thumbnail tier path had a guard; the canonical thumbnail it falls back
to, heroes, previews and preview tiers had none.
One process-local map now covers every rendition, keyed by photo id and
rendition (`thumbnail:<id>`, `thumbnail:<id>:w<width>`, `hero:<id>`,
`preview:<id>`, `preview:<id>:w<width>`). Concurrent callers share one
promise; the entry is cleared in a finally on success and failure alike
so a rejection cannot poison the key. The tier stat moved inside the
flight so a request arriving as the previous flight clears finds the
written tier instead of missing on a stale probe.
Heroes and previews also deleted the existing object before generating
its replacement, and again in the catch. Both are gone, mirroring what
the thumbnail generator already does: put is the last statement in the
try and replaces atomically on local storage and by key on S3, so the
delete only ever opened a window with no rendition at all, and a source
that failed to read stripped the old rendition with the row still
pointing at it.
No re-read of the photo row inside the flight: the admin regenerate
endpoints force a rebuild by passing a row with the path nulled, and a
re-read would hand back the persisted rendition untouched.
Fixes the single-flight half of issue 1020. The preview cache-key
extension mismatch and any server-wide work queue remain separate.
* fix(images): keep the snapshot validity check outside the single-flight
With the check inside the flight, an admin regeneration (row passed with
the path nulled) could join a viewer's flight for the same photo that was
merely confirming an already-good rendition, and be handed back the very
file it was asked to replace while the endpoint counted a success. Only a
miss enters the flight now; inside it everything is a regeneration.
* fix(images): forced rebuilds run after an in-flight lazy generation instead of adopting it
The admin regenerate endpoints could still join a lazy flight that was
already generating for the same photo. That flight read the thumbnail
settings when it started, so after a settings change it produces exactly
the rendition the regenerate was invoked to replace; adopting it counted
a success while the old size stayed cached.
ensureThumbnail and ensurePreviewImage take `{ force: true }`: skip the
snapshot check and, if a flight is pending, start after it settles. Lazy
misses arriving meanwhile join the forced flight, and an older flight
settling late no longer evicts the newer entry from the map.
* fix(images): key rendition flights by source as well as photo id
replacePhoto keeps the photo id and changes path and filename. Keyed by
id and width alone, a request carrying the replacement row joined a
flight still rendering the file it replaced and was handed the old
image, which the gallery caches for 30 minutes. The tier map this
replaced was keyed by storage key and so already told the two apart.
* test(thumbnails): wait for the regenerate loop's completion line instead of a fixed 150 ms
The loop runs in setImmediate after the response. Under a loaded machine
(fifteen suites in parallel, each booting a migrated SQLite) it took
longer than 150 ms once and the assertions ran against a half-finished
mock call list. Poll the logger spy for the "regeneration complete" line
with a 10 s deadline; the suite also finishes sooner because the wait
ends as soon as the loop does.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
PUT /admin/events/:id spreads the body into the UPDATE. SQLite resolves
quoted identifiers case-insensitively, so `{ "Event_Name": ... }` lands
on event_name there — while every check in the handler (validators, the
field-level permission guards, the deny-set) keys on the exact lowercase
name. The deny-set already case-folded for its own columns; every other
column was reachable through a spelling variant.
Every events column and every input-only key the handler accepts is
lowercase snake_case, so a key with any uppercase in it is not something
a legitimate client sends. Such keys are now removed before anything
looks at the body. Postgres was unaffected (quoted identifiers are
case-sensitive there; a variant produced a 500 instead).
Surfaced by the Codex review of the folder-watcher change, where a
photos.upload guard on external_watch could be walked around this way.
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* feat(external-media): watch reference folders and import new files automatically
Managed uploads dropped into storage/events/active are picked up by the
chokidar watcher; external media had no equivalent, so a NAS folder that
keeps growing needed an admin to open the event and press Import every
time. Relates to issue 1187.
- The import pass moves out of the route into
services/externalImportService.js. The watcher and the Import button
now run the identical function; the route only validates and maps
errors to status codes.
- Mutual exclusion is the per-event claim from maintenanceJobState
(`external_import:<id>`, seeded on demand by the new ensure()) instead
of the in-process Set. The Set stopped a double-click in one process;
the claim also stops the watcher on a second replica, or an admin
clicking while the watcher is mid-run elsewhere. The run heartbeats so
a claim from a dead process is taken over.
- services/externalMediaWatcher.js: per-event opt-in via the new
events.external_watch column (migration 208), chokidar with
awaitWriteFinish so a copy in flight is not imported half-written,
debounced full pass per change, a timer sweep every 15 minutes as the
fallback for NFS/SMB mounts that deliver no inotify events, optional
stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched
events is re-read every minute, so the toggle works from any replica.
A watcher that just started runs one pass immediately.
- Deletions are ignored on purpose: a file vanishing from a NAS is at
least as likely to be a reorganisation or a dropped mount as an
intentional removal, and acting on it would delete a guest-visible
photo. Rows whose file is gone stay, as they do today.
- Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local.
- Quiet system passes stay out of the activity log; runs that imported
something are logged with actor external-media-watcher.
- Frontend: "Watch folder for new files" checkbox under the external
folder picker, status line in view mode, EN/DE strings.
* fix(external-media): close the review gaps in the folder watcher
Codex review of the watcher, round 1. All six findings were real:
- Enabling the watcher, or pointing an enabled one at another folder,
now requires photos.upload — the permission the manual Import already
requires. events.edit alone was a way around it. Only the transition
is checked, so a role without photos.upload can still edit an
already-watched event. The checkbox is disabled for such roles.
- Automatic passes defer files that are still changing: anything
modified inside the stability window, or whose size moves across one
wait of that window, is left for the next pass. chokidar's
awaitWriteFinish only settles the file that fired the event, and the
sweep sees no events at all, so a sibling still being copied could be
inserted half-written and then skipped forever.
- Photos an admin deleted are not brought back by the sweep. The delete
routes record the file in external_import_exclusions (migration 209);
automatic passes skip the list, the manual Import ignores it and
clears it for what it imports.
- The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three
compose files; they were documented but the backend services use
explicit environment lists, so the kill switch did nothing.
- A pass re-checks is_active / is_archived at run time, not only in the
minutely reconcile.
- The lease is renewed on a timer for the whole run, walk included, and
ownership is checked before the event row is touched.
* fix(external-media): make automatic passes follow the row, not rewrite it
Codex review round 2, four findings, all applied:
- The event update route drops non-canonical spellings of external_watch
and external_path before the permission guard. SQLite resolves column
names case-insensitively, so `External_Watch` reached the column while
the guard only looked at the lowercase key.
- Exclusions are checked per file at insert time, not against a
snapshot taken before the settle wait. A photo deleted during the wait
was present in the snapshot and got re-inserted by the loop.
- An automatic pass no longer writes source_mode / external_path. It
re-reads the row after the walk and the settle wait and stops if the
folder changed or the event went managed; the manual Import is the
only writer. The options are now `automatic` + `settleMs`.
- A pass that deferred files re-arms the debounced import, so a file
copied just before the watcher started is not stranded when the sweep
is disabled.
* fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying
Codex review round 3, both findings applied:
- recordExclusions keys on external_relpath alone. A replaced external
photo becomes managed but keeps its relpath on purpose, and deleting
that replacement must not republish the NAS original.
- An automatic pass checks the full watcher predicate (reference mode,
same folder, watch on, active, not archived) before it inserts and on
every heartbeat tick during the loop, and stops as soon as the event
no longer qualifies.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* feat(security): opt-in recoverable gallery passwords
Gallery passwords are bcrypt hashes, so an admin who needs to hand a
password to a client a second time has to reset it, which invalidates
what the client already has. This adds a security setting,
security_gallery_password_recoverable, off by default, that keeps an
AES-256-GCM encrypted copy of each gallery password and client PIN next
to the hash. The key is derived from GALLERY_PASSWORD_ENCRYPTION_KEY or
JWT_SECRET.
While the setting is on:
- create, publish, send-later, edit, reset and the v1 API write the copy
alongside the hash; turning a gallery's password requirement off
clears it
- GET /api/admin/events/:id/password returns the copy to admins with
events.edit and ownership, and writes a gallery_password_viewed
activity entry on every real reveal
- resend-email uses the stored password instead of the "set at creation"
sentinel, so the client receives what already works
Switching the setting off purges every stored copy. Login and hash
verification are untouched; the copy is never read on the gallery side.
The Security tab carries the toggle with a warning that stays visible,
and the event page shows "Show password" with copy buttons only while
the setting is on and the gallery has a secret.
Relates to issue 1271
* fix(security): close the write-versus-switch-off race in the password vault
The recoverable setting is read while an event insert is assembled and the
client-PIN hash awaits after that, so a settings request that switched the
feature off and purged in that gap was overtaken by the insert. Every write
site now re-reads the setting right after its statement and clears its own
row when the setting is off; the settings writer flips the value before it
purges, so either the purge or the re-check catches the row.
* fix(security): resend carries the stored client PIN and link; deterministic tamper test
The creation mail includes the client-access link and PIN; a resend only
sent the gallery password even when a stored PIN was available. The
ciphertext tamper assertion replaced the last two characters with a
constant, which was a no-op roughly once in 4096 runs.
* fix(security): drop the revealed password after Send gallery email
The send-later route can replace the password; the share card keys its
revealed copy on the event query's refetch time, so invalidate the event
after the send like the other password-changing mutations do.
* fix(security): purge leftovers before the setting write when turning recovery on
Switching on wrote the setting first and purged after, so a password write
that read the new "on" in between stored a copy the purge then deleted.
Turning on now purges before the write; turning off keeps purging after it,
which together with the write-site re-check leaves the vault holding
exactly what was written while the setting was on.
* chore(security): drop the duplicate rateLimitService import left by the rebase
* chore(usage): register the password recovery routes in the v5 coverage inventory
The inventory moved from v4 to v5 on main; the entry added by this branch
followed it.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* feat(usage): distinguish real edits and template delivery with v5 consent
* fix(usage): exclude queued test messages and count reorders as edits
- queueEmail carries usageEligible: false into email_data and the queue
processor passes it on, so the dev tools' send-test-email no longer
records email_template_delivery once the worker sends it.
- event-types/reorder and categories/reorder-global compare the persisted
order before and after and record the v5 edit markers only when it
changed, matching the display_order edit already counted on PUT.
- normalized() builds arrays with Array.from so a row array from the sqlite
binding compares equal under Jest's separate realm.
* fix(usage): cover per-gallery category order and workflow test runs
- categories/reorder records category_editing when an event's override
changes; reorder/:eventId records it when an override was actually
removed.
- send_email and the collections handoff pass usageEligible: false for a
workflow test run (engine.testRun sets __test), so a non-dry test send is
not counted as template delivery.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* fix(email): scrub gallery passwords from the sent-mail archive
The email queue kept every gallery password and client PIN in clear
text in email_data and rendered_html after the mail was sent, and the
Messages reading pane handed them back to any admin with the messaging
flag. A password hash in the events table bought nothing while the
plaintext sat next to it.
Once a mail is out, or its retries are exhausted, the processor now
masks secret-looking variables (password, passcode, pin) in email_data
and replaces their values in the rendered body, plain and HTML-escaped.
The reading pane applies the same masking to rows archived before this
change. Pending rows keep the real values so a retry still sends them.
Relates to issue 1271
* fix(email): keep a quoted ">" from cutting an attribute value out of redaction
The tag splitter stopped at the first ">", so a template attribute such as
title="{{gallery_password}} > details" left the password unmasked in the
archived HTML while email_data was already masked. The tokenizer is now
quote-aware; a tag with an unbalanced quote falls through as text and is
scrubbed there.
* fix(email): scrub secrets inside HTML comments in the archived body
A comment such as <!-- PIN: {{client_password}} --> was split off as a tag
and its body, which has no attribute, was never scrubbed. Comments are now
one segment and their content is masked whole.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
* feat(settings): expose the API rate limiter in the Security tab
The general per-IP limiter had six settings in app_settings and a
backend route to write them, and no screen. Installs ran on the code
fallback — 300 requests per 15 minutes per IP — with no way to see it,
which is how issue 1287 played out: a 546-photo gallery exhausted the
budget for one viewer and the operator learned about the setting from
a grep of the backend log.
Security tab: a card with the six settings, the validation ranges the
route enforces, a one-line explanation per field, and a note that the
unit is the client IP — an office or household behind one NAT shares
a budget, and behind a proxy TRUST_PROXY has to cover the proxy or
every visitor shares its address. The tab's Save button saves the
limiter through its own route. The limiter values are checked against
the route's ranges before anything is written and the limiter is
written first, so a rejected value cannot leave the password/session
settings half-saved behind a failure toast.
Backend, three things the screen needed:
- The settings read fills the six keys with the code defaults when
they have no row, so the form shows the budget in force rather than
an empty field; the defaults live in one exported constant the
limiter itself reads.
- The write route upserts instead of updating: on a fresh install,
which has no rows, the old UPDATE matched nothing and the route
answered 200 while changing nothing.
- The live limiter instances move into rateLimitService and the
write route rebuilds them. express-rate-limit fixes windowMs when an
instance is built — max and skip re-read the settings per request,
the window does not — so a saved window used to apply only after a
restart. The gates in server.js resolve the instance per request
through the service's getters. A rebuild starts fresh counters,
which on a settings change is acceptable. The limiters get explicit
MemoryStores and a rebuild shuts the superseded ones down, because a
store keeps a cleanup interval alive for as long as it exists and
dropping the reference alone would leak one timer per save.
Tests: the read surfaces defaults and honours the key filter; the
write creates rows on a fresh database, the limiter sees the values
immediately and hands the gates a fresh instance; existing rows are
updated not duplicated; out-of-range values are rejected. The tab
renders the values, edits through the hook state, carries the ranges,
saves with the tab's button, and the pre-write validation accepts the
bounds and rejects outside them and cleared fields.
Relates to issue 1337
* docs(security): point the rate limiter doc at the Security tab and the upsert
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
features.v2/v3/v4.json carried every name, description and definition
twice, once in English and once in German, inside a file that is source
code, is vendored byte-identically into the collector and is served as
the consented catalog. Source stays English. The German strings already
existed a second time in the frontend locale file, which is what the
consent dialog actually renders (UsageCatalog reads
productUsage.catalog.<key>, never the JSON), so the copy in the catalog
was a duplicate that could only ever drift.
The `de` fields are gone from all three catalogs, their frontend copies
and the inventory definitions; the docs coverage file and
FEATURE_COVERAGE.md list English only. Nothing on the wire changes: the
report schema is derived from the keys, and the catalog's text is not
part of any signature or consent version string.
The coverage test now pins the catalog to the en locale verbatim and
requires the de locale to cover every key and field, without dictating
its wording. The collector holds the same catalog files and needs the
same change to stay byte-identical, plus its German catalog strings
moved into its own locale file; that is filed there.
The general per-IP limiter was inert until b0f33c17 registered it ahead
of the routers (budget 100, raised to 300 by 19e125d8), and 839bf4e4
then stopped gallery tokens from earning the authenticated skip. Since
that release every guest has been paying for the gallery's thumbnails
out of 300 requests per 15 minutes per IP. A 546-photo grid runs out
mid-scroll: the remaining tiles come back 429, which the frontend
turned into blank tiles with no error and no retry, and the next
refresh finds the photo list limited too.
Reproduced in iOS Safari against a seeded 546-photo Grid gallery:
loading in bursts, then nothing, no console output, no recovery — the
exact shape of the large-gallery report, whose plateaus were 125, 235,
300 and 308 tiles.
A token that verifies and names the gallery in the path is now exempt
on the image routes only: thumbnail, preview, hero and photo, GET only.
The photo list, downloads, feedback and every write stay on the budget,
a token for one gallery buys nothing on another, and the exemption
rides on skip_authenticated so an operator who turns the skip off
counts guests too. The 839bf4e4 concern — a free token as an unlimited
budget on every /api route — stays closed; what this hands back is the
bandwidth of routes a 300-request budget never bounded anyway.
Relates to issue 1287
gallery_downloads.configured was true on every installation with a
gallery. allow_downloads ships true — column default in migration 037
and the create route both set it — and the snapshot asked "at least one
gallery has it on". The fleet value was ~100% by construction and could
not separate a deliberate configuration from an untouched one.
v2 consented to that key under that description, so v2 keeps sending it
unchanged. v3 replaces it with gallery_downloads_restricted: at least
one gallery has downloads switched off, which is the only state of that
column anyone actually decides. Same catalog position, so the disclosed
capability count stays at 86; the frontend copy, the EN/DE catalog
strings, the coverage inventory and FEATURE_COVERAGE.md follow.
Done in v3 rather than a v4 because v3 is on main and in no release
yet, so nobody has consented to it. The collector carries the same
catalog and has to take this change before the release that ships v3.
One guard for the window in which :main / :beta images already carried
the old v3 catalog. A report queued under it fails local validation on
this build, and deliver() left a locally invalid report pending for
good, blocking every operation behind it. A report's payload is derived
state, so deliver() now rebuilds it from the current snapshot in place
and sends that. Packet ID and sequence are kept — a re-signed retry has
to reuse them so a lost acknowledgement does not duplicate data — and
reports only: a stale registration, deletion or command is a genuine
conflict and keeps the existing handling.
Tests: the v3 snapshot counts a switched-off gallery and ignores
enabled ones, v2 still reports the old key with the old meaning, and a
stale queued report goes out rebuilt under the same packet id while a
valid one is sent untouched.
Relates to issue 1308
Probing the live collector to settle the delete-sequence question turned up
something else: usage.picpeak.app answers a valid usage.v2 registration with
INVALID_PACKET while the identical v1 flow is accepted. It does not speak v2
yet — which the deployment notes already require, but the consequence of
getting that order wrong was worse than "reports do not send".
Opting in to v2 against a v1-only collector left the installation stuck.
Registration was refused, so nothing existed at the collector at all; the row
sat in activation_pending, disable moved it to deletion_pending, retry was
futile forever, and enable refused because the row was not `disabled`. The
abandon hatch added earlier did not apply: it was gated on
SIGNING_KEY_UNREADABLE. So the most harmless possible failure — nothing
registered anywhere — was the one an operator could not clear.
The gate is now the property that actually matters: a participation the
collector has provably never accepted (sequence 0, no receipt) with a failing
delivery can be discarded, from activation_pending as well as
deletion_pending. Its receipt records `never-registered` rather than an
unconfirmed deletion, because nothing remote exists to be unsure about. A
participation the collector *did* accept keeps the old narrow gate and its
explicit warning — clearing local state while the collector still holds
reports must stay a deliberate, warned-about act.
A collector that rejects a registration or a deletion outright now reports
SCHEMA_NOT_ACCEPTED instead of DELIVERY_FAILED, and the settings page says the
collector does not accept this report version yet. Retrying cannot fix that,
and sending the operator to look for a network fault they do not have was
wrong.
Verified end to end against the live collector: v2 opt-in reports
SCHEMA_NOT_ACCEPTED, the exit is offered immediately, the receipt says
never-registered, and joining again on v1 registers, reports and withdraws
with a collector-confirmed deletion.
The dialog described what is sent and where it goes, but never said which way
the connection runs. That is the part an operator is actually being asked to
accept: opening an outbound path to someone else's service.
PicPeak sends and never pulls. One place in the service reaches the network,
it is a POST, and it requests exactly two paths — /api/envelopes, and
/api/participant/lookup only when an operator asks for their own export. No
scheduled job contacts the collector; the daily rollup is driven solely by an
authenticated admin hitting /activity. There is no route the collector could
call, and redirect: 'error' means it cannot even point a request somewhere
else. From a reply only the acknowledgement for the packet just sent is read,
with every field compared against that packet before it is accepted; the
stored copy drops the session token and no read path hands it back to the UI.
A requested export is streamed to the operator as a file and never
interpreted.
The consequence is why it belongs in the consent text and not only in the
docs: this channel cannot deliver code, configuration or content into an
installation, not even from a collector that has been taken over. It is a
security property by design rather than by convention.
usageOutboundOnly.test.js guards it by source inspection rather than
behaviour, because a behavioural test only proves that today's calls behave.
It fails the moment someone adds a second fetch, a poll for messages, a
scheduled pull, or a public route touching the usage service — verified by
injecting each of those.
A QA exploration of this branch against an isolated rig — own stub
collector, SQLite and PostgreSQL — turned up one dead end and a set of
signals and controls that did not hold up. This closes all of them.
Rotating JWT_SECRET, the documented response to a suspected compromise,
made the signing key unreadable. That was already named and documented,
but it left no way out: the delete packet can never be signed, so the
row stays deletion_pending forever, and enable() refuses because it is
not `disabled`. An operator who rotated precisely because the secret was
compromised cannot restore it, so the feature was bricked with no
control left. POST /usage/abandon is offered only in that state; it
drops the local identity and records the receipt as
`collector-unconfirmed` rather than claiming a deletion that did not
happen.
Every failed delivery was retried on the next admin request, and
/activity is open to any authenticated admin while the settings ticker
fires it every five minutes per open tab — 30 activity calls against a
rejecting collector produced 30 outbound requests. Migration 206 adds
attempts/next_attempt_at and the unattended sender honours the gate;
Retry and opt-out still send immediately, and the tab names the time of
the next automatic attempt.
Feedback, votes and portal sessions now share an installation-wide
budget of 30/hour. They are the only endpoints whose effect is outbound
traffic carrying operator-written free text, and the general limiter
skips authenticated requests by design. Reading status and withdrawing
stay unthrottled.
gallery_image_protection was true on a bare install with no galleries:
PicPeak ships default_protection_level='standard' and
enable_devtools_protection=true, so it reported fleet-wide 100% and
could never separate a decision from an untouched default. It now reads
only what deviates from the shipped defaults, and the devtools flag is
not read at all — being on by default, its only informative state is
off, which is the opposite of what the key claims.
Also:
- the export receipt counted every packet and called the total "usage
reports"; reports and participant operations are now counted and named
separately
- GET /usage/preview no longer persists the custom_css marker, so the
transparency view stops changing what will be sent
- the feedback route requires every field the packet schema requires,
so an API caller gets the missing field named instead of a bare
INVALID_PACKET from inside signing
- the German strings for this feature use "Sie" throughout, matching the
rest of the admin UI; the ignore hint says what ignoring will do
rather than stating it as already true
- the consent dialog returns focus to the control that opened it
- the long buttons wrap instead of running off a 390px viewport
- a deletion receipt is labelled as belonging to an earlier
participation while a new one is active
Regression tests cover each of these, including the delete packet's
reuse of the last accepted sequence, which was an unwritten assumption
about the collector rather than a defect.
#1298 and #1303 merged together. #1298 taught the creation paths to
resolve a fragmentation_level default; #1303 removed everything that
consumed it. Neither conflicted textually, so main ended up validating
the field on create and update, copying it on duplicate, resolving
default_fragmentation_level for it, and advertising it in the v1 API
docs — for a value nothing reads and a setting the Image Security tab no
longer exposes.
Inert rather than broken, which is exactly why it needed removing on
purpose: dead code that contradicts the PR that just deleted the feature
is how the next reader concludes fragmentation still works.
The events.fragmentation_level column and the app_settings row stay, as
#1303 decided — dropping a column is irreversible and the stored values
are harmless once nothing reads them.
Refs #1300
Three findings, one of them mine and CI-affecting.
The Postgres suite gets its own schema. CI hands every gated suite the
same PICPEAK_PG_TEST_URL and runs jest with parallel workers, and both
picpeakRestorePg and externalRelpathFoldPg drop and recreate `events`
and `app_settings` in it — so the suite I added would have destroyed
their fixtures and vice versa, intermittently. It now creates and drops
its own `usage_pg_test` schema and reaches the tables through
searchPath, which works because the service queries unqualified names.
Verified on a clean database: after the run `public` still holds zero
tables. My first attempt at this silently did not apply — the
replacement anchor had been reformatted by eslint and I printed success
without asserting the match, which is why the first "isolated" claim was
wrong.
Webhook-only installs are no longer counted as SMTP users. With
EMAIL_WEBHOOK_URL and EMAIL_WEBHOOK_SECRET set, adminEmail sends
/email/test through the webhook transport and never touches SMTP (#1225
added that path), but the rule recorded the permanent `smtp` marker
anyway. Gated on the transport that is actually configured.
Activation is written atomically with its acknowledgement. Split across
two updates, a failure or a stop between them left the row
activation_pending with pending_packet already cleared — registered with
the collector, and permanently stuck locally, because tick() has nothing
to retry from there. The register case now sets status in the same write
and is guarded precisely on activation_pending rather than merely "not
withdrawing".
Refs #1110
Two findings from the review of the current head.
Local backups no longer imply S3. markUsed derived an s3_storage marker
from "a backup ran while backup_destination_type is s3" — but the
middleware also counts /database-backup/* and /backup/picpeak/export as
backups, and those write a local file wherever scheduled backups go. So
configuring S3 and downloading a local export reported s3_storage as
USED. The middleware now tells markUsed whether the operation writes to
the configured destination, and only then is the marker derived. A wrong
`true` in this dataset is worse than a missing signal: it is a claim
about an install that nobody can check.
The ProtocolError branch was dead code. adminUsage matched on
`error.name === 'ProtocolError'`, but the class extends Error without
setting `name`, so every instance reports 'Error' — verified — and a
malformed vote or feedback payload fell through to the global handler,
which logs it as an unhandled programming error and answers
INTERNAL_ERROR in production, losing the validation code the caller
needs. Now matched with instanceof. protocol.cjs is byte-identical with
picpeak-usage (diffed against the companion repo), so the fix belongs
here rather than in the class.
An existing assertion needed updating for the new markUsed argument, and
the path split is pinned: /backup/run is destination-driven,
/database-backup/backup and /backup/picpeak/export are not.
Refs #1110
Everything about this feature had been exercised on SQLite only, which is
the engine least likely to show its problems.
Adds __tests__/integration/productUsagePg.test.js, following the gated
pattern the .picpeak restore suites use: it runs the real migrations
201-203 against a real PostgreSQL and covers what SQLite cannot answer.
node-postgres returns bigint as a STRING, and the withdrawal guard
compares `cancel_seq` — a `'1' !== 1` slip there would let an activation
complete after an opt-out, and SQLite, which hands back a number, would
never show it. Booleans are real booleans rather than 0/1, which is what
every `configured` signal in a report is built from. And markUsed takes
SELECT ... FOR UPDATE on this engine only.
Seven cases, all passing against PostgreSQL 15. Removing the
compare-and-swap condition fails the withdrawal case there too, so the
suite has teeth on that engine and not only on SQLite. CI already
provides PICPEAK_PG_TEST_URL, so these run there rather than skipping.
The collector default is harder to lose. An unset, empty or
whitespace-only USAGE_COLLECTOR_URL now falls back to
https://usage.picpeak.app — deployments that template the variable in
(docker-compose writes ${USAGE_COLLECTOR_URL:-...}) can hand over an
empty string, and that has to mean "use the default" rather than "no
collector". A value that is present but malformed is still reported as a
configuration error instead of being silently replaced: quietly
retargeting a self-hoster's collector at ours would send their reports
somewhere they did not choose.
Refs #1110
Three items, one of which explains an error seen in the app.
"The operation could not be completed" could come from a config typo.
status() called collectorUrl() bare, and that throws on a bare hostname,
a path, a query, or http in production. The settings page renders one
generic failure when its status query errors, so a misconfigured
USAGE_COLLECTOR_URL replaced the whole tab with that sentence — no
cause, and no way to read the status or withdraw, because every control
there sits behind that call. The URL is now reported as
collector_error: 'INVALID_COLLECTOR_URL' beside the real state, the tab
says what is wrong and how to fix it, and the links are only rendered
when there is somewhere to point them.
gallery_layouts reported grid for every preset-themed install.
color_theme holds either a theme object or the NAME of a preset — the
theme picker stores names, and eventTypeService seeds them
(`theme_preset: 'corporateTimeline'`). Only reading value.galleryLayout
made masonry, timeline, mosaic and the two gallery presets invisible.
Names now resolve, and an event with no theme of its own resolves
through the global one instead of being counted as grid. Only the
name -> layout mapping is duplicated, not the presets;
frontend/src/types/theme.types.ts stays the source of truth, and an
unknown name reports `other` so a preset added later degrades to
"something else" rather than quietly inflating the grid count.
custom_css missed CSS applied through a template. An enabled
css_templates row applied via events.css_template_id is gallery styling
by the same definition as the settings fields — the Custom CSS tab is
where both are authored — but neither the snapshot nor the middleware
saw it, so those installs reported custom_css entirely false. Existence
only; template contents are never read.
Eleven tests. Reverting each fix in turn fails 3, 1 and 3 of them.
Refs #1110
Third and last window in the same race, and again in my own fix.
locked() claims the lease and reads the row in two separate statements.
Reading the cancellation counter from inside that callback meant a
/disable completing in the gap was adopted as this activation's own
baseline and silently absorbed — the counter matched, the claim
succeeded, and registration went ahead after the operator had withdrawn.
The baseline is now read before the lease is taken, which inverts it:
every increment from that point on is later than the value the claim
tests for, so the claim fails and the withdrawal wins. An increment from
before the read is a withdrawal the operator already completed, and a
deliberate opt-in afterwards should not be vetoed by it.
The test for this passed against the bug on its first two attempts. It
stubbed the state read to increment the counter AFTER reading the row,
so both the broken and the fixed version saw the old value and behaved
identically. The withdrawal has to land before the read returns for the
row to carry it — which is the whole point of the window. It now fails
without the fix.
Refs #1110
Follow-up review on the previous commit, including a hole in that
commit's own fix.
The cancellation flag became a counter. Clearing a boolean needed a
write of its own, and a /disable landing between the lease and that
write was erased — the same race one level down. enable() now records
the counter it started with and claims only if it is unchanged, so no
clearing write exists to lose. It also fixes the case a boolean could
not express at all: a stale cancellation already set, and a fresh one
arriving mid-activation, are indistinguishable as flags and obvious as
counts. Migration 203, separate from 202 for the reason 202 was separate
from 201 — knex will not re-run an applied migration.
deliver() re-checks immediately before dispatch. The existing check ran
before the binding lookup, which is asynchronous, so a withdrawal that
COMPLETED during it still had its registration or report sent
afterwards. Not an already-in-flight request — a new one started after
the operator had withdrawn.
The outbox writes in tick() and command() are conditional on still being
active. /disable clears pending_packet without holding the lease, so an
unconditional write put a report — or a feedback body and name — back
into an outbox the withdrawal had just emptied, where deliver() would
then leave it, since it declines to send anything but the delete.
Per-item name consent resets with the item. `named` stayed checked after
submitting, so the next item carried the previous name automatically,
contradicting the anonymous-by-default promise the disclosure makes for
each item. The remembered name stays in preferences; attaching it is
decided again each time.
Two of these tests were worthless when first written and are noted
because the pattern keeps recurring: the pre-dispatch case passed
without the guard because an empty report payload failed schema
validation during signing, so nothing reached the collector for reasons
unrelated to the check. With a valid payload it fails without the guard
and passes with it. Same for the counter: dropping it from the claim
fails two.
Refs #1110
The last open item from the #1304 review.
/disable overlapping an in-flight /enable was silently lost. While
activation generates its identity and writes its binding file the row
still reads `disabled`, so disable()'s conditional update matched no
rows, and the lease conflict raised by its tick() was swallowed as
expected noise. The admin was told participation was off; the activation
then completed and left it on. An opt-out that does nothing is the one
failure this feature cannot have.
disable() now records cancel_requested first and unconditionally —
before the case-by-case work — and enable() claims its state with a
single conditional UPDATE that tests the flag alongside the status.
Re-reading the flag and then updating would only have moved the window;
making the claim itself carry the condition closes it, so whichever of
the two lands first wins outright and the loser writes nothing.
Nothing is registered when the claim fails, so there is also nothing to
delete remotely — the cancelled activation leaves no identity behind.
The flag is cleared at the start of enable(), so a cancellation from an
earlier participation cannot veto a later deliberate opt-in.
The column is migration 202 rather than an edit to 201. 201 already
shipped on this branch and knex records it as applied, so folding the
column in would have skipped every database that had already run it and
the first /disable would have failed on a missing column. Verified both
ways: a fresh install gets the column from 201+202, and a database
migrated before 202 existed gains it when 202 arrives.
Three tests. With the condition dropped from the claim, the race case
fails and the other two pass.
Refs #1110
Review follow-ups on #1304.
SIGNING_KEY_UNREADABLE. USAGE_ENCRYPTION_KEY defaults to JWT_SECRET, so
rotating JWT_SECRET — the correct response to a suspected compromise —
makes the stored Ed25519 key undecryptable. That surfaced as a generic
DELIVERY_FAILED which retried forever, and it silently blocks the DELETE
packet too: an operator who withdraws has their local state cleared
while the collector keeps its copy. decrypt() now tags its own failure
and deliver() reports it under its own name, without flagging an
identity conflict — an unreadable key is not evidence of a clone. The
docs already warned that losing the key breaks deletion signing; they
now name the trigger and the error.
The collector default is no longer an inline string in the constructor.
It is a declared DEFAULT_COLLECTOR_URL, since it is a deployment choice:
self-hosters point USAGE_COLLECTOR_URL at their own collector and the UI
already derives every link from whatever is configured. schema.cjs is
deliberately untouched — it is vendored byte-identical with
picpeak-usage, and its $id is a schema identity, not a delivery address.
Links in the consent dialog. It named the collector inside prose but
never linked it, so an operator deciding whether to opt in could not
open the destination or the public schema without retyping a URL. Both
are links now, built from the configured collector.
UI standards. The tab hand-rolled its surfaces as
`<section className="rounded-xl border border-theme …">` and imported
Button from a deep path; every other settings tab uses `<Card
padding="md">` from the components/common barrel. Converted, with the
feedback <form> wrapped rather than replaced so its semantics survive,
and headings given the same colour tokens as ImageSecurityTab. The
barrel pulls ErrorBoundary -> i18n/config, so the tab's test needed the
initReactI18next shim the FaceRecognitionCard test already uses.
Not changed: the delete packet reusing the current sequence. The
collector handles delete before any sequence check — "possession proof
is sufficient for deletion, including when a restored backup has a
stale sequence" (picpeak-usage server/collector.js) — so deletion is
deliberately sequence-exempt and the client is correct as written.
Refs #1110
Fifth bypass, and the same root cause as the first: sanitizeCSS
validated, then kept rewriting.
`<[^>]*>` deletes the span it matches, and `<">` takes a quote with it.
So `--x:x<">;background:url(https://evil.example/p.gif);--y:x<">` was
scanned with the url() safely inside a string, and the tag strip below
then removed the quotes that made it so — shipping a live remote
background with no warning.
The file already carried the rule: "any pass that can join tokens has to
happen before validation, not after." It has now been broken three
separate times — by the HTML-comment strip (#1290), the control-
character strip, and the tag strip. Rather than fix a third instance in
place, the URL scan is now the LAST step, so what is validated is always
the bytes that get served.
All eight known bypass classes are pinned, together with the legitimate
data: URI, quoted font stack and escaped selector that must survive
untouched.
Refs #1264
The fourth bypass found in this review, and the one no lexer fix
reaches: sanitizing runs on the stored body, but safeTemplateReplace
rewrites it afterwards, so the string that was validated is not the
string that is sent.
A conditional inside a style attribute can delete the very quoting that
made a url() inert:
style="--x:x{{#if company_name}}'{{/if}};background:url(https://evil…)"
At write time the url() genuinely sits inside a CSS string and is
correctly left alone. Expanding the conditional for a recipient with no
company name removes both quotes and the background goes live —
confirmed end to end against the real functions.
The style-attribute pass now runs again on the substituted output.
Substitution cannot introduce a `"` (values are HTML-escaped), so the
attribute match still holds. body_css is not substituted, so the
<style> block cannot be rewritten after its check and needs nothing.
This is the case the removed newsletter pass had been covering. Rather
than reinstating a second definition of "disallowed", the one definition
now runs at both points where the content changes.
Refs #1264
Third bypass of this scanner found in one review pass, and the same
shape as the others: the lexer and a browser disagreeing about where a
token begins.
JavaScript's `\s` matches U+00A0; CSS whitespace is exactly space, tab,
LF, CR and FF. Skipping an NBSP as whitespace let the scanner read the
quote after it as a legitimate quoted data: URI and swallow a remote
url() inside that "string" —
.a{background:url(<NBSP>"data:image/png);background:url(https://evil…);--x:");}
came through untouched, with no warning, and survived re-sanitising. A
browser treats NBSP as an ordinary character, so that is an UNQUOTED
url-token ending at the first `)`, leaving the remote background live.
All three token readers now use an explicit CSS whitespace class.
Ordinary spacing around a data: URI still works, and is pinned.
Refs #1264
My previous commit introduced this. Handling `\` outside strings before
readIdentifier meant a LEADING escape was eaten before the url check
saw it: `\75` is the CSS escape for `u`, so `.a{background:\75rl(...)}`
is url() to a browser and passed through untouched, with no warning —
a bypass the base version did not have. An escape mid-identifier
(`u\72l`) was unaffected, which is why the first tests missed it.
The escape branch now runs AFTER readIdentifier, which already decodes
leading escapes itself. What is left for it is the case it was added
for: `\'`, which must not be read as opening a string.
Both spellings are pinned, along with the legitimate escaped selector
and data: URI that must survive untouched.
Refs #1264
Paul Nothaft
