Commit Graph

2312 Commits

Author SHA1 Message Date
Paul Nothaft 65831785a2 fix(security): bump sanitize-html to 2.17.7
Trivy flags the backend image on two sanitize-html advisories, both
fixed upstream:

- CVE-2026-63670 (fixed 2.17.6): a literal solidus after a raw-text end
  tag (`</textarea/>`) is treated as text by htmlparser2 and re-emitted
  unescaped, so disallowed markup passes when textarea or xmp is in
  allowedTags.
- CVE-2026-84371 (fixed 2.17.7): an SVG SMIL animation whose
  attributeName selects href lets the sibling values/from/to/by
  attributes carry URLs past the scheme policy.

2.17.5 -> 2.17.7, exact pin as before. The new version brings its own
htmlparser2 12 / domhandler 6 / domutils 4 / dom-serializer 3 /
entities 8 tree under node_modules/sanitize-html; nothing else in the
lock moves.

That tree is ESM-only, so the backend now needs unflagged require(esm):
Node 20.19+ or 22.12+. The image is node:22-alpine and CI runs 22, but
engines.node still admitted 22.0-22.11, where require('sanitize-html')
throws ERR_REQUIRE_ESM at startup (publicSiteService loads it during
initialisation). engines is now ^20.19.0 || >=22.12.0 and the native
setup script's Node check enforces the same range instead of accepting
any 22.x. On the supported versions the sanitiser behaves identically
to 2.17.5 on the tracker and newsletter fixtures.

Jest 29's CommonJS registry cannot evaluate ESM either, so every suite
importing a route or service that uses the sanitiser would fail at
import. jest.config.js now maps `sanitize-html` to jest.sanitizeHtml.js,
which hands that one module to Node's real loader via
process.getBuiltinModule('module') — a plain require('module') inside
Jest is Jest's wrapper and returns an empty object for this package.
Verified against a real 2.17.7 install: the sanitiser suites and a
settings route suite pass; without the mapper they fail with "Cannot
use import statement outside a module".
2026-09-06 23:06:43 +02:00
Paul Nothaft 83b74e2256 chore(main): release 3.126.3-beta.0 (#1323)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 9m30s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 10m21s
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 13m48s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / smoke-aio (push) Failing after 12m33s
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / summary (push) Has been cancelled
v3.126.3-beta.0
2026-09-06 19:45:17 +00:00
Paul Nothaft 5306fe378c Merge pull request #1318 from PicPeak/fix/1308-gallery-downloads-signal
fix(usage): introduce consented v4 download restriction reporting
2026-09-06 21:40:32 +02:00
Paul Nothaft ef8a52f02c fix(usage): introduce consented v4 without changing historical reports 2026-09-06 21:34:04 +02:00
Paul Nothaft 3cc893126d Merge remote-tracking branch 'origin/main' into codex/usage-v4-client 2026-09-06 21:18:58 +02:00
Paul Nothaft 6459bff50e chore(main): release 3.126.2-beta.0 (#1319)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 10m16s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 10m38s
Build and Push Docker Images / smoke-aio (push) Failing after 13m25s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 14m46s
Build and Push Docker Images / summary (push) Has been cancelled
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
v3.126.2-beta.0
2026-09-06 19:13:42 +00:00
Paul Nothaft c4b03a831f Merge pull request #1315 from PicPeak/fix/1287-retry-failed-tiles
fix(gallery): retry a failed image fetch once the tile is back on screen
2026-09-06 21:07:22 +02:00
Paul Nothaft b3937d0b8c Merge pull request #1312 from PicPeak/fix/1287-release-offscreen-tiles
fix(gallery): release grid tiles once they are far enough out of view
2026-09-06 21:07:19 +02:00
Paul Nothaft 02b353e54f fix(usage): report restricted gallery downloads in v3 instead of an always-true signal
gallery_downloads.configured was true on every installation with a
gallery. allow_downloads ships true — column default in migration 037
and the create route both set it — and the snapshot asked "at least one
gallery has it on". The fleet value was ~100% by construction and could
not separate a deliberate configuration from an untouched one.

v2 consented to that key under that description, so v2 keeps sending it
unchanged. v3 replaces it with gallery_downloads_restricted: at least
one gallery has downloads switched off, which is the only state of that
column anyone actually decides. Same catalog position, so the disclosed
capability count stays at 86; the frontend copy, the EN/DE catalog
strings, the coverage inventory and FEATURE_COVERAGE.md follow.

Done in v3 rather than a v4 because v3 is on main and in no release
yet, so nobody has consented to it. The collector carries the same
catalog and has to take this change before the release that ships v3.

One guard for the window in which :main / :beta images already carried
the old v3 catalog. A report queued under it fails local validation on
this build, and deliver() left a locally invalid report pending for
good, blocking every operation behind it. A report's payload is derived
state, so deliver() now rebuilds it from the current snapshot in place
and sends that. Packet ID and sequence are kept — a re-signed retry has
to reuse them so a lost acknowledgement does not duplicate data — and
reports only: a stale registration, deletion or command is a genuine
conflict and keeps the existing handling.

Tests: the v3 snapshot counts a switched-off gallery and ignores
enabled ones, v2 still reports the old key with the old meaning, and a
stale queued report goes out rebuilt under the same packet id while a
valid one is sent untouched.

Relates to issue 1308
2026-09-06 20:42:28 +02:00
Paul Nothaft 77ae94e649 fix(gallery): retry a failed image fetch once the tile is back on screen
A rejected fetch in AuthenticatedImage set the error state, rendered
nothing, and never asked again. The fetch effect only re-runs when its
inputs change, and for a grid tile they never do — so a transient
failure (a hiccup on cellular, or Safari cancelling loads when the tab
goes to the background) was a permanently blank tile with no request in
flight and nothing in any log. Grid passes no fallbackSrc, so there was
not even a broken-image icon to point at.

The retry is bounded and gated. Three attempts with a doubling delay
(2 s, 4 s, 8 s), and an attempt fires only once the placeholder
intersects the viewport and the document is visible, so a tile that
failed while the user was away retries when they come back rather than
while they are still gone. A new src gets a fresh budget. The
fallbackSrc path is untouched: it already renders a plain <img> and
should not loop.

Two refinements from review. A final 4xx (anything but 408 and 429)
exhausts the budget at once: an expired gallery token or a missing
photo cannot be retried into existence, and on a 68-tile viewport three
retries each would be ~200 requests that cannot succeed. And a 429's
Retry-After is honoured as the minimum delay, because the backoff alone
would spend every retry inside a 15-minute rate-limit window and leave
the tile blank after the limit had lifted. Retry-After is not
CORS-safelisted, so server.js now exposes it for split-origin
deployments alongside Content-Disposition.

The error branch now renders the same grey box as the loading state
instead of null. That is what the retry effect observes, and it is
also something the user can see. The empty-src branch now clears the
error flag too, so a tile whose src is removed after a failure does not
keep showing the failure box.

Nine tests in AuthenticatedImage.retry.test.tsx; the retry cases fail
against the previous version.

Not presented as the fix for the iOS report. It closes the one gap that
turns a transient failure into a permanent one, which the reporter asked
for in the original issue, and it is worth having on any device.

Relates to issue 1287
2026-09-06 20:13:57 +02:00
Paul Nothaft 25c3e3d7b8 chore(main): release 3.126.1-beta.0 (#1314)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 9m31s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 10m13s
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 13m47s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / smoke-aio (push) Failing after 12m34s
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
Build and Push Docker Images / summary (push) Has been cancelled
v3.126.1-beta.0
2026-09-06 17:59:34 +00:00
Paul Nothaft b801f3a6b8 Merge pull request #1313 from PicPeak/codex/usage-report-compatibility
fix(usage): preserve compatibility with old and partial reports
2026-09-06 19:54:33 +02:00
Paul Nothaft 7ca783f89b fix(usage): preserve report contracts with compatible receiver validation 2026-09-06 19:45:41 +02:00
Paul Nothaft 63ab731cee chore(main): release 3.126.0-beta.0 (#1311)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 10m57s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 11m15s
Build and Push Docker Images / smoke-aio (push) Failing after 12m15s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 13m40s
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
Build and Push Docker Images / summary (push) Has been cancelled
v3.126.0-beta.0
2026-09-06 17:43:14 +00:00
Paul Nothaft a6a1db5254 fix(gallery): release grid tiles once they are far enough out of view
The pre-load band made tiles arrive in time. It did nothing about them never
leaving. PhotoCard latched its observer with triggerOnce, so a tile that had
been scrolled past stayed mounted for the life of the page — holding its
object URL, and where image protection is on a canvas sized to the image that
the browser is not permitted to evict.

Measured in Chrome on a seeded 546-photo grid, scrolling top to bottom:
mounted tiles climb 24 → 100 → 212 → 364 → 546 and never fall. That is a
monotonically growing retained set, which is the profile a memory-constrained
browser discards a tab over — the reported symptom on iOS Safari 18.1 being
tiles that stop appearing and a blank page after refresh. With this change the
same scroll peaks at 68.

PhotoCard now takes an optional outer band. The inner band, unchanged, decides
when a tile starts loading; the outer one decides when it is far enough away
to unmount, and unmounting is what actually frees anything, because
AuthenticatedImage revokes its object URL and drops the canvas in its cleanup.
The gap between the bands is the hysteresis: at three viewport heights against
a one-viewport load band, a tile travels two further viewport heights after it
stops loading before it is released, so ordinary scrolling never crosses both
edges. Thumbnails are served private, max-age=1800, so returning costs a cache
hit rather than a round trip.

Opt-in per layout, and only Grid opts in. Its skeleton is aspect-square and
holds the tile's box exactly, so releasing shifts nothing; the measured
layouts have no such guarantee. Without the prop the observer keeps its
original latch, so every other layout behaves exactly as before — pinned by a
test, since that is the half most easily broken by accident.

This is not presented as the fix for the iOS report. It removes the mechanism
that best explains it, and it is worth having on any device; whether it is the
mechanism still needs a measurement from the phone that failed.

Relates to issue 1287
2026-09-06 19:36:36 +02:00
Paul Nothaft b0bb65d0d2 Merge pull request #1310 from PicPeak/codex/usage-v3-features
feat(usage): add beta capabilities and gallery/photo totals with explicit consent
2026-09-06 19:33:47 +02:00
Paul Nothaft d3622de81a chore(main): release 3.125.0-beta.0 (#1309)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 9m14s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 10m12s
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 14m13s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / smoke-aio (push) Failing after 13m0s
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
Build and Push Docker Images / summary (push) Has been cancelled
v3.125.0-beta.0
2026-09-06 17:27:47 +00:00
Paul Nothaft c358bc65f7 feat(usage): add consented beta capabilities and gallery photo totals 2026-09-06 19:23:14 +02:00
Paul Nothaft 35b42bba9d Merge pull request #1304 from PicPeak/codex/1110-product-usage
feat: add opt-in product usage and feedback (#1110)
2026-09-06 19:22:07 +02:00
Paul Nothaft e40bc474bc fix(usage): let an operator clear a participation the collector never accepted
Probing the live collector to settle the delete-sequence question turned up
something else: usage.picpeak.app answers a valid usage.v2 registration with
INVALID_PACKET while the identical v1 flow is accepted. It does not speak v2
yet — which the deployment notes already require, but the consequence of
getting that order wrong was worse than "reports do not send".

Opting in to v2 against a v1-only collector left the installation stuck.
Registration was refused, so nothing existed at the collector at all; the row
sat in activation_pending, disable moved it to deletion_pending, retry was
futile forever, and enable refused because the row was not `disabled`. The
abandon hatch added earlier did not apply: it was gated on
SIGNING_KEY_UNREADABLE. So the most harmless possible failure — nothing
registered anywhere — was the one an operator could not clear.

The gate is now the property that actually matters: a participation the
collector has provably never accepted (sequence 0, no receipt) with a failing
delivery can be discarded, from activation_pending as well as
deletion_pending. Its receipt records `never-registered` rather than an
unconfirmed deletion, because nothing remote exists to be unsure about. A
participation the collector *did* accept keeps the old narrow gate and its
explicit warning — clearing local state while the collector still holds
reports must stay a deliberate, warned-about act.

A collector that rejects a registration or a deletion outright now reports
SCHEMA_NOT_ACCEPTED instead of DELIVERY_FAILED, and the settings page says the
collector does not accept this report version yet. Retrying cannot fix that,
and sending the operator to look for a network fault they do not have was
wrong.

Verified end to end against the live collector: v2 opt-in reports
SCHEMA_NOT_ACCEPTED, the exit is offered immediately, the receipt says
never-registered, and joining again on v1 registers, reports and withdraws
with a collector-confirmed deletion.
2026-09-06 18:54:58 +02:00
Paul Nothaft c741dc22c5 docs(usage): state in the consent dialog that the connection only runs outwards
The dialog described what is sent and where it goes, but never said which way
the connection runs. That is the part an operator is actually being asked to
accept: opening an outbound path to someone else's service.

PicPeak sends and never pulls. One place in the service reaches the network,
it is a POST, and it requests exactly two paths — /api/envelopes, and
/api/participant/lookup only when an operator asks for their own export. No
scheduled job contacts the collector; the daily rollup is driven solely by an
authenticated admin hitting /activity. There is no route the collector could
call, and redirect: 'error' means it cannot even point a request somewhere
else. From a reply only the acknowledgement for the packet just sent is read,
with every field compared against that packet before it is accepted; the
stored copy drops the session token and no read path hands it back to the UI.
A requested export is streamed to the operator as a file and never
interpreted.

The consequence is why it belongs in the consent text and not only in the
docs: this channel cannot deliver code, configuration or content into an
installation, not even from a collector that has been taken over. It is a
security property by design rather than by convention.

usageOutboundOnly.test.js guards it by source inspection rather than
behaviour, because a behavioural test only proves that today's calls behave.
It fails the moment someone adds a second fetch, a poll for messages, a
scheduled pull, or a public route touching the usage service — verified by
injecting each of those.
2026-09-06 17:58:19 +02:00
Paul Nothaft 1e8b6f1b0f fix(usage): close the QA findings on opt-in product usage
A QA exploration of this branch against an isolated rig — own stub
collector, SQLite and PostgreSQL — turned up one dead end and a set of
signals and controls that did not hold up. This closes all of them.

Rotating JWT_SECRET, the documented response to a suspected compromise,
made the signing key unreadable. That was already named and documented,
but it left no way out: the delete packet can never be signed, so the
row stays deletion_pending forever, and enable() refuses because it is
not `disabled`. An operator who rotated precisely because the secret was
compromised cannot restore it, so the feature was bricked with no
control left. POST /usage/abandon is offered only in that state; it
drops the local identity and records the receipt as
`collector-unconfirmed` rather than claiming a deletion that did not
happen.

Every failed delivery was retried on the next admin request, and
/activity is open to any authenticated admin while the settings ticker
fires it every five minutes per open tab — 30 activity calls against a
rejecting collector produced 30 outbound requests. Migration 206 adds
attempts/next_attempt_at and the unattended sender honours the gate;
Retry and opt-out still send immediately, and the tab names the time of
the next automatic attempt.

Feedback, votes and portal sessions now share an installation-wide
budget of 30/hour. They are the only endpoints whose effect is outbound
traffic carrying operator-written free text, and the general limiter
skips authenticated requests by design. Reading status and withdrawing
stay unthrottled.

gallery_image_protection was true on a bare install with no galleries:
PicPeak ships default_protection_level='standard' and
enable_devtools_protection=true, so it reported fleet-wide 100% and
could never separate a decision from an untouched default. It now reads
only what deviates from the shipped defaults, and the devtools flag is
not read at all — being on by default, its only informative state is
off, which is the opposite of what the key claims.

Also:
- the export receipt counted every packet and called the total "usage
  reports"; reports and participant operations are now counted and named
  separately
- GET /usage/preview no longer persists the custom_css marker, so the
  transparency view stops changing what will be sent
- the feedback route requires every field the packet schema requires,
  so an API caller gets the missing field named instead of a bare
  INVALID_PACKET from inside signing
- the German strings for this feature use "Sie" throughout, matching the
  rest of the admin UI; the ignore hint says what ignoring will do
  rather than stating it as already true
- the consent dialog returns focus to the control that opened it
- the long buttons wrap instead of running off a 390px viewport
- a deletion receipt is labelled as belonging to an earlier
  participation while a new one is active

Regression tests cover each of these, including the delete packet's
reuse of the last accepted sequence, which was an unwritten assumption
about the collector rather than a defect.
2026-09-06 17:40:43 +02:00
Paul Nothaft a7382591bf feat: expand opt-in capability coverage with versioned consent 2026-09-06 00:56:58 +02:00
Paul Nothaft 5d31b61c8d Merge remote-tracking branch 'origin/main' into codex/1110-usage-coverage 2026-09-05 23:59:54 +02:00
Paul Nothaft a5ff926409 chore(main): release 3.124.1-beta.0 (#1307)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 9m30s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 10m2s
Build and Push Docker Images / smoke-aio (push) Failing after 11m43s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 13m9s
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
Build and Push Docker Images / summary (push) Has been cancelled
v3.124.1-beta.0
2026-09-05 21:55:53 +00:00
Paul Nothaft 5dda14f727 Merge pull request #1306 from PicPeak/fix/strand-fragmentation-defaults
fix: remove the fragmentation handling stranded by #1303
2026-09-05 23:49:33 +02:00
Paul Nothaft b4c2c40550 chore(main): release 3.124.0-beta.0 (#1305)
Build and Push Docker Images / build-backend (linux/amd64, ubuntu-latest) (push) Successful in 9m28s
Build and Push Docker Images / build-frontend (linux/amd64, ubuntu-latest) (push) Successful in 10m6s
Build and Push Docker Images / build-aio (linux/amd64, ubuntu-latest) (push) Successful in 14m27s
Build and Push Docker Images / build-ml (linux/amd64, ubuntu-latest) (push) Has been skipped
Build and Push Docker Images / smoke-aio (push) Failing after 13m18s
Build and Push Docker Images / merge-backend (push) Has been cancelled
Build and Push Docker Images / build-frontend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-frontend (push) Has been cancelled
Build and Push Docker Images / build-aio (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / build-backend (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-aio (push) Has been cancelled
Build and Push Docker Images / build-ml (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Build and Push Docker Images / merge-ml (push) Has been cancelled
Build and Push Docker Images / summary (push) Has been cancelled
Build and Push Docker Images / dockerhub-descriptions (push) Has been cancelled
v3.124.0-beta.0
2026-09-05 21:45:17 +00:00
Paul Nothaft e347f8f40f fix(usage): minimize session receipts and clarify privacy controls 2026-09-05 23:44:15 +02:00
Paul Nothaft 7ff8caf9d7 fix: remove the fragmentation handling stranded by #1303
#1298 and #1303 merged together. #1298 taught the creation paths to
resolve a fragmentation_level default; #1303 removed everything that
consumed it. Neither conflicted textually, so main ended up validating
the field on create and update, copying it on duplicate, resolving
default_fragmentation_level for it, and advertising it in the v1 API
docs — for a value nothing reads and a setting the Image Security tab no
longer exposes.

Inert rather than broken, which is exactly why it needed removing on
purpose: dead code that contradicts the PR that just deleted the feature
is how the next reader concludes fragmentation still works.

The events.fragmentation_level column and the app_settings row stay, as
#1303 decided — dropping a column is irreversible and the stored values
are harmless once nothing reads them.

Refs #1300
2026-09-05 23:41:17 +02:00
Paul Nothaft ae23b1adea Merge pull request #1303 from PicPeak/fix/1300-remove-fragmentation
fix: remove the image-fragmentation surface
2026-09-05 23:37:14 +02:00
Paul Nothaft 0536c86ec9 Merge pull request #1302 from PicPeak/feat/newsletter-large-send-warning
feat(newsletters): warn about deliverability before a large send
2026-09-05 23:37:00 +02:00
Paul Nothaft 2e9bd540c9 Merge pull request #1298 from PicPeak/fix/1296-dead-canvas-setting
fix(security): apply the Image-security defaults instead of storing them (#1296)
2026-09-05 23:36:53 +02:00
Paul Nothaft 905fc595e3 Merge pull request #1295 from PicPeak/fix/post-merge-followups
fix(gallery): image-loading follow-ups — pre-load band, decode release, sanitizer dedup
2026-09-05 23:36:45 +02:00
Paul Nothaft 1f316ef91c Merge pull request #1299 from PicPeak/fix/1297-inert-protection-props
fix(gallery): remove the inert image-protection prop surface from AuthenticatedImage
2026-09-05 23:36:27 +02:00
Paul Nothaft cc263f2e87 fix(usage): isolate the Postgres fixture, and stop two more wrong signals
Three findings, one of them mine and CI-affecting.

The Postgres suite gets its own schema. CI hands every gated suite the
same PICPEAK_PG_TEST_URL and runs jest with parallel workers, and both
picpeakRestorePg and externalRelpathFoldPg drop and recreate `events`
and `app_settings` in it — so the suite I added would have destroyed
their fixtures and vice versa, intermittently. It now creates and drops
its own `usage_pg_test` schema and reaches the tables through
searchPath, which works because the service queries unqualified names.
Verified on a clean database: after the run `public` still holds zero
tables. My first attempt at this silently did not apply — the
replacement anchor had been reformatted by eslint and I printed success
without asserting the match, which is why the first "isolated" claim was
wrong.

Webhook-only installs are no longer counted as SMTP users. With
EMAIL_WEBHOOK_URL and EMAIL_WEBHOOK_SECRET set, adminEmail sends
/email/test through the webhook transport and never touches SMTP (#1225
added that path), but the rule recorded the permanent `smtp` marker
anyway. Gated on the transport that is actually configured.

Activation is written atomically with its acknowledgement. Split across
two updates, a failure or a stop between them left the row
activation_pending with pending_packet already cleared — registered with
the collector, and permanently stuck locally, because tick() has nothing
to retry from there. The register case now sets status in the same write
and is guarded precisely on activation_pending rather than merely "not
withdrawing".

Refs #1110
2026-09-05 23:26:41 +02:00
Paul Nothaft 32d745b575 fix(usage): stop local backups implying S3 use, and make the protocol-error branch reachable
Two findings from the review of the current head.

Local backups no longer imply S3. markUsed derived an s3_storage marker
from "a backup ran while backup_destination_type is s3" — but the
middleware also counts /database-backup/* and /backup/picpeak/export as
backups, and those write a local file wherever scheduled backups go. So
configuring S3 and downloading a local export reported s3_storage as
USED. The middleware now tells markUsed whether the operation writes to
the configured destination, and only then is the marker derived. A wrong
`true` in this dataset is worse than a missing signal: it is a claim
about an install that nobody can check.

The ProtocolError branch was dead code. adminUsage matched on
`error.name === 'ProtocolError'`, but the class extends Error without
setting `name`, so every instance reports 'Error' — verified — and a
malformed vote or feedback payload fell through to the global handler,
which logs it as an unhandled programming error and answers
INTERNAL_ERROR in production, losing the validation code the caller
needs. Now matched with instanceof. protocol.cjs is byte-identical with
picpeak-usage (diffed against the companion repo), so the fix belongs
here rather than in the class.

An existing assertion needed updating for the new markUsed argument, and
the path split is pinned: /backup/run is destination-driven,
/database-backup/backup and /backup/picpeak/export are not.

Refs #1110
2026-09-05 23:16:48 +02:00
Paul Nothaft c7cedb00d6 test(usage): prove product usage works on PostgreSQL, and harden the collector default
Everything about this feature had been exercised on SQLite only, which is
the engine least likely to show its problems.

Adds __tests__/integration/productUsagePg.test.js, following the gated
pattern the .picpeak restore suites use: it runs the real migrations
201-203 against a real PostgreSQL and covers what SQLite cannot answer.
node-postgres returns bigint as a STRING, and the withdrawal guard
compares `cancel_seq` — a `'1' !== 1` slip there would let an activation
complete after an opt-out, and SQLite, which hands back a number, would
never show it. Booleans are real booleans rather than 0/1, which is what
every `configured` signal in a report is built from. And markUsed takes
SELECT ... FOR UPDATE on this engine only.

Seven cases, all passing against PostgreSQL 15. Removing the
compare-and-swap condition fails the withdrawal case there too, so the
suite has teeth on that engine and not only on SQLite. CI already
provides PICPEAK_PG_TEST_URL, so these run there rather than skipping.

The collector default is harder to lose. An unset, empty or
whitespace-only USAGE_COLLECTOR_URL now falls back to
https://usage.picpeak.app — deployments that template the variable in
(docker-compose writes ${USAGE_COLLECTOR_URL:-...}) can hand over an
empty string, and that has to mean "use the default" rather than "no
collector". A value that is present but malformed is still reported as a
configuration error instead of being silently replaced: quietly
retargeting a self-hoster's collector at ours would send their reports
somewhere they did not choose.

Refs #1110
2026-09-05 22:53:57 +02:00
Paul Nothaft 75ef137b7d fix(usage): drop the tinted block and stop the modal opening with a focus ring
Two things reported on the reformatted consent modal.

The green box is gone. Setting "what is never included" apart as a
tinted panel broke the rhythm of the sections and read as an arbitrary
highlight rather than emphasis. All six sections are uniform now; the
icon and heading are enough to tell them apart.

The green bars across the disclosure were a focus ring, not a border.
showModal() focuses the first focusable descendant, which since the
reformat was the scrollable region I had given tabIndex={0} — so its
inset ring was drawn for every user the moment the dialog opened, and
because the dialog clips its sides a full-width inset ring appears as
two coloured bars. Focus now goes to the dialog itself, which is also
the better screen-reader behaviour: the title is announced on open, and
the region's ring appears only when someone deliberately tabs to it. It
is a thinner, softer ring for that case. The dialog suppresses its own
ring, since that focus is programmatic rather than keyboard navigation.

The collector shown in the transport sentence was never wrong: it
interpolates the configured collector, and the screenshots showing
http://127.0.0.1:9 were taken on a rig deliberately pointed at a dead
loopback port so they could not reach production. Re-checked with
USAGE_COLLECTOR_URL unset: the sentence reads
https://usage.picpeak.app and both links resolve there.

Refs #1110
2026-09-05 22:49:21 +02:00
Paul Nothaft a9e51d8fd7 fix(usage): reformat the consent modal so the disclosure can be read
It was seven anonymous paragraphs in one scrolling block, with the title
and the buttons scrolling away with them. The scroll container is
keyboard-focusable, and unstyled it drew a default focus ring, so the
disclosure also looked like a giant textarea.

Now: a fixed header carrying the icon, title and purpose; a scroll
region with six labelled sections, each with a small heading and icon so
the thing can be scanned rather than only read; and a fixed footer with
the consent checkbox and the actions, which no longer scroll out of
reach on a short screen. "What is never included" is set apart as a
tinted block, since it is the part that answers the question an operator
actually has. The focus ring is now a deliberate inset ring on a
labelled region, which is correct for keyboard use instead of an
accident that looked like a form field.

Dark mode is fixed as part of this, and it was my own doing: the dialog
used `bg-theme-surface`, which does not follow dark mode, and the
section text I added carries dark: variants. Light surface plus
near-white text is unreadable. The surface is class-driven now —
neutral-800, which is what `.card` resolves to in dark and what the rest
of the admin UI uses. Verified in both themes through the app's own
theme toggle rather than by forcing the class, which is what produced
the misleading half-state the first time I looked.

Six section headings added in EN and DE.

Refs #1110
2026-09-05 22:43:20 +02:00
Paul Nothaft bb76ca5375 fix(usage): keep the settings tab usable on a bad collector URL, and report layouts and CSS accurately
Three items, one of which explains an error seen in the app.

"The operation could not be completed" could come from a config typo.
status() called collectorUrl() bare, and that throws on a bare hostname,
a path, a query, or http in production. The settings page renders one
generic failure when its status query errors, so a misconfigured
USAGE_COLLECTOR_URL replaced the whole tab with that sentence — no
cause, and no way to read the status or withdraw, because every control
there sits behind that call. The URL is now reported as
collector_error: 'INVALID_COLLECTOR_URL' beside the real state, the tab
says what is wrong and how to fix it, and the links are only rendered
when there is somewhere to point them.

gallery_layouts reported grid for every preset-themed install.
color_theme holds either a theme object or the NAME of a preset — the
theme picker stores names, and eventTypeService seeds them
(`theme_preset: 'corporateTimeline'`). Only reading value.galleryLayout
made masonry, timeline, mosaic and the two gallery presets invisible.
Names now resolve, and an event with no theme of its own resolves
through the global one instead of being counted as grid. Only the
name -> layout mapping is duplicated, not the presets;
frontend/src/types/theme.types.ts stays the source of truth, and an
unknown name reports `other` so a preset added later degrades to
"something else" rather than quietly inflating the grid count.

custom_css missed CSS applied through a template. An enabled
css_templates row applied via events.css_template_id is gallery styling
by the same definition as the settings fields — the Custom CSS tab is
where both are authored — but neither the snapshot nor the middleware
saw it, so those installs reported custom_css entirely false. Existence
only; template contents are never read.

Eleven tests. Reverting each fix in turn fails 3, 1 and 3 of them.

Refs #1110
2026-09-05 22:33:25 +02:00
Paul Nothaft 9785b636a9 fix(usage): take the withdrawal baseline before the lease, not after it
Third and last window in the same race, and again in my own fix.

locked() claims the lease and reads the row in two separate statements.
Reading the cancellation counter from inside that callback meant a
/disable completing in the gap was adopted as this activation's own
baseline and silently absorbed — the counter matched, the claim
succeeded, and registration went ahead after the operator had withdrawn.

The baseline is now read before the lease is taken, which inverts it:
every increment from that point on is later than the value the claim
tests for, so the claim fails and the withdrawal wins. An increment from
before the read is a withdrawal the operator already completed, and a
deliberate opt-in afterwards should not be vetoed by it.

The test for this passed against the bug on its first two attempts. It
stubbed the state read to increment the counter AFTER reading the row,
so both the broken and the fixed version saw the old value and behaved
identically. The withdrawal has to land before the read returns for the
row to carry it — which is the whole point of the window. It now fails
without the fix.

Refs #1110
2026-09-05 22:17:44 +02:00
Paul Nothaft 22da018e1b fix(usage): close the remaining withdrawal races, reset per-item name consent
Follow-up review on the previous commit, including a hole in that
commit's own fix.

The cancellation flag became a counter. Clearing a boolean needed a
write of its own, and a /disable landing between the lease and that
write was erased — the same race one level down. enable() now records
the counter it started with and claims only if it is unchanged, so no
clearing write exists to lose. It also fixes the case a boolean could
not express at all: a stale cancellation already set, and a fresh one
arriving mid-activation, are indistinguishable as flags and obvious as
counts. Migration 203, separate from 202 for the reason 202 was separate
from 201 — knex will not re-run an applied migration.

deliver() re-checks immediately before dispatch. The existing check ran
before the binding lookup, which is asynchronous, so a withdrawal that
COMPLETED during it still had its registration or report sent
afterwards. Not an already-in-flight request — a new one started after
the operator had withdrawn.

The outbox writes in tick() and command() are conditional on still being
active. /disable clears pending_packet without holding the lease, so an
unconditional write put a report — or a feedback body and name — back
into an outbox the withdrawal had just emptied, where deliver() would
then leave it, since it declines to send anything but the delete.

Per-item name consent resets with the item. `named` stayed checked after
submitting, so the next item carried the previous name automatically,
contradicting the anonymous-by-default promise the disclosure makes for
each item. The remembered name stays in preferences; attaching it is
decided again each time.

Two of these tests were worthless when first written and are noted
because the pattern keeps recurring: the pre-dispatch case passed
without the guard because an empty report payload failed schema
validation during signing, so nothing reached the collector for reasons
unrelated to the check. With a valid payload it fails without the guard
and passes with it. Same for the counter: dropping it from the claim
fails two.

Refs #1110
2026-09-05 22:09:39 +02:00
Paul Nothaft 80e238f0ad fix(usage): let a withdrawal win against an activation that is still starting
The last open item from the #1304 review.

/disable overlapping an in-flight /enable was silently lost. While
activation generates its identity and writes its binding file the row
still reads `disabled`, so disable()'s conditional update matched no
rows, and the lease conflict raised by its tick() was swallowed as
expected noise. The admin was told participation was off; the activation
then completed and left it on. An opt-out that does nothing is the one
failure this feature cannot have.

disable() now records cancel_requested first and unconditionally —
before the case-by-case work — and enable() claims its state with a
single conditional UPDATE that tests the flag alongside the status.
Re-reading the flag and then updating would only have moved the window;
making the claim itself carry the condition closes it, so whichever of
the two lands first wins outright and the loser writes nothing.

Nothing is registered when the claim fails, so there is also nothing to
delete remotely — the cancelled activation leaves no identity behind.
The flag is cleared at the start of enable(), so a cancellation from an
earlier participation cannot veto a later deliberate opt-in.

The column is migration 202 rather than an edit to 201. 201 already
shipped on this branch and knex records it as applied, so folding the
column in would have skipped every database that had already run it and
the first /disable would have failed on a missing column. Verified both
ways: a fresh install gets the column from 201+202, and a database
migrated before 202 existed gains it when 202 arrives.

Three tests. With the condition dropped from the claim, the race case
fails and the other two pass.

Refs #1110
2026-09-05 21:58:04 +02:00
Paul Nothaft 4944b9b3b6 fix(usage): scope the participation notice, highlight it, and call ignoring what it is
It appears on the dashboard and settings only. It is an invitation, not
an alert, so it belongs on pages an admin opens deliberately rather than
on top of whatever task they are in the middle of.

The activity ticker deliberately did NOT move with it. That ticker is
what triggers the daily rollup — the backend has no scheduler — so
tying it to the banner would have stopped reporting for an admin who
works on Events and never opens the dashboard, and stopped it entirely
for a participating install, where the banner never renders at all. The
effect stays mounted on every admin page and only the visible aside is
scoped. Two tests pin exactly that, because it is the kind of thing a
later refactor would helpfully "clean up".

Highlighted like the migration banner it sits under: tinted surface,
border, icon, a title line above the body. It was previously the same
neutral surface as the page behind it and read as filler.

"Not now" is now "Ignore". The button calls dismiss(), which persists
notice_dismissed on the server — the invitation never comes back. "Not
now" promised otherwise. The label says what happens and a hint says
where to join later.

Only shown while participation is off. activation_pending,
deletion_pending and identity_conflict are in-flight states the settings
page explains properly; inviting someone to join in the middle of their
own withdrawal would be worse than saying nothing.

The first version of these tests was worthless: the negative cases
asserted absence after waiting only for the status call, so the
component was still rendering null for want of data and every one passed
with the gates removed. They now wait for the query cache to fill.
Removing the route gate fails 4; removing the status gate fails 6.

Refs #1110
2026-09-05 21:42:21 +02:00
Paul Nothaft 83fbb63e13 fix(usage): protect a pending withdrawal, widen the backup signal, explain an unreadable key
Three of four findings from the follow-up review.

A withdrawal is no longer clobbered by the instance-copy check. That
update was unconditional, so an opt-out arriving while the binding
lookup was in flight was replaced by identity_conflict — and tick()
stops there, so the deletion the operator asked for was never sent. It
now carries the same whereNot('deletion_pending') guard the
collector-conflict handler beside it already had.

Scheduled database backups count as a configured backup. The middleware
records /backup/* and /database-backup/* under one capability, but
`configured` read only backup_enabled, so an install whose only backup
is the scheduled database one reported used: true, configured: false —
a contradiction in the dataset this feature exists to produce.

SIGNING_KEY_UNREADABLE gets its own message. Naming the error in the
previous commit was half the job: the settings page still showed the
generic retry/disable advice, and neither action can succeed without the
original encryption material. It now says what happened and what is
actually required, in EN and DE.

NOT fixed, and reported instead: /disable overlapping an in-flight
/enable. While activation is still doing its slow work the row still
reads `disabled`, so disable's conditional update matches nothing and
the lease conflict from its tick() is swallowed — the operator is told
participation is off while activation completes and leaves it on.
Closing it properly needs a persisted cancellation flag that enable
checks before finalising: taking the lease cannot help, since it either
conflicts immediately or would block the request for the 60s lease. That
is a schema and state-machine decision for the author, not something to
restructure underneath them.

Refs #1110
2026-09-05 21:34:25 +02:00
Paul Nothaft c043897b0e fix(usage): name the unreadable-key failure, unpin the collector default, align the tab
Review follow-ups on #1304.

SIGNING_KEY_UNREADABLE. USAGE_ENCRYPTION_KEY defaults to JWT_SECRET, so
rotating JWT_SECRET — the correct response to a suspected compromise —
makes the stored Ed25519 key undecryptable. That surfaced as a generic
DELIVERY_FAILED which retried forever, and it silently blocks the DELETE
packet too: an operator who withdraws has their local state cleared
while the collector keeps its copy. decrypt() now tags its own failure
and deliver() reports it under its own name, without flagging an
identity conflict — an unreadable key is not evidence of a clone. The
docs already warned that losing the key breaks deletion signing; they
now name the trigger and the error.

The collector default is no longer an inline string in the constructor.
It is a declared DEFAULT_COLLECTOR_URL, since it is a deployment choice:
self-hosters point USAGE_COLLECTOR_URL at their own collector and the UI
already derives every link from whatever is configured. schema.cjs is
deliberately untouched — it is vendored byte-identical with
picpeak-usage, and its $id is a schema identity, not a delivery address.

Links in the consent dialog. It named the collector inside prose but
never linked it, so an operator deciding whether to opt in could not
open the destination or the public schema without retyping a URL. Both
are links now, built from the configured collector.

UI standards. The tab hand-rolled its surfaces as
`<section className="rounded-xl border border-theme …">` and imported
Button from a deep path; every other settings tab uses `<Card
padding="md">` from the components/common barrel. Converted, with the
feedback <form> wrapped rather than replaced so its semantics survive,
and headings given the same colour tokens as ImageSecurityTab. The
barrel pulls ErrorBoundary -> i18n/config, so the tab's test needed the
initReactI18next shim the FaceRecognitionCard test already uses.

Not changed: the delete packet reusing the current sequence. The
collector handles delete before any sequence check — "possession proof
is sufficient for deletion, including when a restored backup has a
stale sequence" (picpeak-usage server/collector.js) — so deletion is
deliberately sequence-exempt and the client is correct as written.

Refs #1110
2026-09-05 21:23:22 +02:00
Paul Nothaft 1151e96144 fix(security): validate CSS urls last, after every pass that moves text
Fifth bypass, and the same root cause as the first: sanitizeCSS
validated, then kept rewriting.

`<[^>]*>` deletes the span it matches, and `<">` takes a quote with it.
So `--x:x<">;background:url(https://evil.example/p.gif);--y:x<">` was
scanned with the url() safely inside a string, and the tag strip below
then removed the quotes that made it so — shipping a live remote
background with no warning.

The file already carried the rule: "any pass that can join tokens has to
happen before validation, not after." It has now been broken three
separate times — by the HTML-comment strip (#1290), the control-
character strip, and the tag strip. Rather than fix a third instance in
place, the URL scan is now the LAST step, so what is validated is always
the bytes that get served.

All eight known bypass classes are pinned, together with the legitimate
data: URI, quoted font stack and escaped selector that must survive
untouched.

Refs #1264
2026-09-05 15:30:04 +02:00
Paul Nothaft 027afb6086 fix(security): re-check inline CSS after template substitution
The fourth bypass found in this review, and the one no lexer fix
reaches: sanitizing runs on the stored body, but safeTemplateReplace
rewrites it afterwards, so the string that was validated is not the
string that is sent.

A conditional inside a style attribute can delete the very quoting that
made a url() inert:

  style="--x:x{{#if company_name}}'{{/if}};background:url(https://evil…)"

At write time the url() genuinely sits inside a CSS string and is
correctly left alone. Expanding the conditional for a recipient with no
company name removes both quotes and the background goes live —
confirmed end to end against the real functions.

The style-attribute pass now runs again on the substituted output.
Substitution cannot introduce a `"` (values are HTML-escaped), so the
attribute match still holds. body_css is not substituted, so the
<style> block cannot be rewritten after its check and needs nothing.

This is the case the removed newsletter pass had been covering. Rather
than reinstating a second definition of "disallowed", the one definition
now runs at both points where the content changes.

Refs #1264
2026-09-05 15:21:40 +02:00
Paul Nothaft 4196e83a5f fix(security): use CSS whitespace, not JavaScript's, in the url() reader
Third bypass of this scanner found in one review pass, and the same
shape as the others: the lexer and a browser disagreeing about where a
token begins.

JavaScript's `\s` matches U+00A0; CSS whitespace is exactly space, tab,
LF, CR and FF. Skipping an NBSP as whitespace let the scanner read the
quote after it as a legitimate quoted data: URI and swallow a remote
url() inside that "string" —

  .a{background:url(<NBSP>"data:image/png);background:url(https://evil…);--x:");}

came through untouched, with no warning, and survived re-sanitising. A
browser treats NBSP as an ordinary character, so that is an UNQUOTED
url-token ending at the first `)`, leaving the remote background live.

All three token readers now use an explicit CSS whitespace class.
Ordinary spacing around a data: URI still works, and is pinned.

Refs #1264
2026-09-05 14:34:55 +02:00
Paul Nothaft b6dc0991ce fix(security): check for an escaped identifier before consuming the escape
My previous commit introduced this. Handling `\` outside strings before
readIdentifier meant a LEADING escape was eaten before the url check
saw it: `\75` is the CSS escape for `u`, so `.a{background:\75rl(...)}`
is url() to a browser and passed through untouched, with no warning —
a bypass the base version did not have. An escape mid-identifier
(`u\72l`) was unaffected, which is why the first tests missed it.

The escape branch now runs AFTER readIdentifier, which already decodes
leading escapes itself. What is left for it is the case it was added
for: `\'`, which must not be read as opening a string.

Both spellings are pinned, along with the legitimate escaped selector
and data: URI that must survive untouched.

Refs #1264
2026-09-05 14:26:51 +02:00