paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(usage): close the QA findings on opt-in product usage

Browse Source 
A QA exploration of this branch against an isolated rig — own stub
collector, SQLite and PostgreSQL — turned up one dead end and a set of
signals and controls that did not hold up. This closes all of them.

Rotating JWT_SECRET, the documented response to a suspected compromise,
made the signing key unreadable. That was already named and documented,
but it left no way out: the delete packet can never be signed, so the
row stays deletion_pending forever, and enable() refuses because it is
not `disabled`. An operator who rotated precisely because the secret was
compromised cannot restore it, so the feature was bricked with no
control left. POST /usage/abandon is offered only in that state; it
drops the local identity and records the receipt as
`collector-unconfirmed` rather than claiming a deletion that did not
happen.

Every failed delivery was retried on the next admin request, and
/activity is open to any authenticated admin while the settings ticker
fires it every five minutes per open tab — 30 activity calls against a
rejecting collector produced 30 outbound requests. Migration 206 adds
attempts/next_attempt_at and the unattended sender honours the gate;
Retry and opt-out still send immediately, and the tab names the time of
the next automatic attempt.

Feedback, votes and portal sessions now share an installation-wide
budget of 30/hour. They are the only endpoints whose effect is outbound
traffic carrying operator-written free text, and the general limiter
skips authenticated requests by design. Reading status and withdrawing
stay unthrottled.

gallery_image_protection was true on a bare install with no galleries:
PicPeak ships default_protection_level='standard' and
enable_devtools_protection=true, so it reported fleet-wide 100% and
could never separate a decision from an untouched default. It now reads
only what deviates from the shipped defaults, and the devtools flag is
not read at all — being on by default, its only informative state is
off, which is the opposite of what the key claims.

Also:
- the export receipt counted every packet and called the total "usage
  reports"; reports and participant operations are now counted and named
  separately
- GET /usage/preview no longer persists the custom_css marker, so the
  transparency view stops changing what will be sent
- the feedback route requires every field the packet schema requires,
  so an API caller gets the missing field named instead of a bare
  INVALID_PACKET from inside signing
- the German strings for this feature use "Sie" throughout, matching the
  rest of the admin UI; the ignore hint says what ignoring will do
  rather than stating it as already true
- the consent dialog returns focus to the control that opened it
- the long buttons wrap instead of running off a 390px viewport
- a deletion receipt is labelled as belonging to an earlier
  participation while a new one is active

Regression tests cover each of these, including the delete packet's
reuse of the last accepted sequence, which was an unwritten assumption
about the collector rather than a defect.
This commit is contained in:
Paul Nothaft
2026-09-06 17:40:43 +02:00
parent a7382591bf
commit 1e8b6f1b0f
20 changed files with 1076 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/adminUsage.js
+66 -19
View File
@@ -1,5 +1,6 @@
const express = require('express');
const crypto = require('crypto');
const rateLimit = require('express-rate-limit');
const { adminAuth } = require('../middleware/auth');
const { requirePermission } = require('../middleware/permissions');
const { ValidationError } = require('../utils/errors');
@@ -21,6 +22,30 @@ const wrap = (fn) => (req, res, next) =>
.json({ error: 'Invalid usage request', code: error.code });
next(error);
});
// The three routes below are the only ones whose effect is an outbound
// request to someone else's service, carrying operator-written free text
// (title 120, body 4000, name 80). The platform's general limiter skips
// authenticated requests by design, which is right for endpoints that only
// touch this installation and wrong for a relay: without this an admin
// session can push unbounded traffic at the collector.
//
// Keyed to the installation, not the caller's IP, because the budget being
// protected is "how much this install relays", and per-process because that
// is the same store the rest of the app uses — a multi-replica deployment
// gets one budget per replica, which still bounds the shape that matters.
const outboundLimiter = rateLimit({
windowMs: 60 * 60 * 1000,
max: 30,
keyGenerator: () => 'usage-outbound',
standardHeaders: true,
legacyHeaders: false,
handler: (_req, res) =>
res.status(429).json({
error: 'Too many usage submissions. Try again later.',
code: 'USAGE_RATE_LIMITED'
})
});
router.use(adminAuth);
router.use((_req, res, next) => {
res.set('Cache-Control', 'no-store');
@@ -66,9 +91,17 @@ router.post(
'/disable',
wrap(async (_req, res) => res.json(await service.disable()))
);
// Reachable only from a withdrawal whose delete packet can never be signed;
// the service refuses in every other state. See UsageService.abandon().
router.post(
'/abandon',
wrap(async (_req, res) => res.json(await service.abandon()))
);
// An operator asking for a retry skips the delivery backoff — that button
// exists precisely to not wait for the next window.
router.post(
'/retry',
wrap(async (_req, res) => res.json(await service.tick()))
wrap(async (_req, res) => res.json(await service.tick({ force: true })))
);
router.get(
'/preview',
@@ -84,29 +117,41 @@ router.put(
'/feedback-preferences',
wrap(async (req, res) => res.json(await service.preferences(req.body)))
);
// Every field the packet schema requires. The allowlist used to let `name`,
// `allow_public` and `allow_marketing` be omitted, and the packet schema —
// which requires all of them — then failed with a bare INVALID_PACKET instead
// of naming the missing field. The UI always sends them; anything driving the
// API directly did not, and got an error it could not act on.
const FEEDBACK_FIELDS = [
'kind',
'title',
'body',
'name',
'allow_public',
'allow_marketing'
];
router.post(
'/feedback',
outboundLimiter,
wrap(async (req, res) => {
const body = req.body;
if (
!body ||
Object.keys(body).some(
(k) =>
![
'kind',
'title',
'body',
'name',
'allow_public',
'allow_marketing'
].includes(k)
) ||
typeof body.title !== 'string' ||
!body.title.trim() ||
typeof body.body !== 'string' ||
!body.body.trim()
)
if (!body || typeof body !== 'object')
throw new ValidationError('Invalid feedback');
const unknown = Object.keys(body).filter(
(key) => !FEEDBACK_FIELDS.includes(key)
);
if (unknown.length)
throw new ValidationError(
`Unknown feedback fields: ${unknown.join(', ')}`
);
for (const key of ['kind', 'title', 'body', 'name'])
if (typeof body[key] !== 'string')
throw new ValidationError(`Feedback field "${key}" must be a string`);
for (const key of ['allow_public', 'allow_marketing'])
if (typeof body[key] !== 'boolean')
throw new ValidationError(`Feedback field "${key}" must be a boolean`);
if (!body.title.trim() || !body.body.trim())
throw new ValidationError('Feedback title and body are required');
res.json(
await service.command('feedback', {
...body,
@@ -117,10 +162,12 @@ router.post(
);
router.post(
'/vote',
outboundLimiter,
wrap(async (req, res) => res.json(await service.command('vote', req.body)))
);
router.post(
'/portal-session',
outboundLimiter,
wrap(async (_req, res) => {
const result = await service.command('session', {});
res.json({
backend/src/usage/UsageService.js
+154 -9
View File
@@ -248,6 +248,20 @@ class UsageService {
consent_update_available: state.status === 'active' && this.schemaVersion(state) !== CURRENT_SCHEMA_VERSION,
last_report_date: state.last_report_date,
last_error: state.last_error,
// Epoch ms, or null when nothing is being paced. The settings page shows
// it so a waiting install reads as "waiting" rather than as broken.
retry_after:
Number(state.next_attempt_at || 0) > this.now()
? Number(state.next_attempt_at)
: null,
// The one failure the operator cannot retry their way out of: the
// signing key is unreadable, so the delete packet can never be signed.
// Without this flag the settings page has no way to offer the only
// remaining exit (abandon), and the install sits in deletion_pending
// forever.
can_abandon:
state.status === 'deletion_pending' &&
state.last_error === 'SIGNING_KEY_UNREADABLE',
pending_action: state.pending_packet
? JSON.parse(state.pending_packet).action
: null,
@@ -278,6 +292,29 @@ class UsageService {
}
}
// Consecutive failures pace the unattended sender: 2, 4, 8, 16, 32 minutes,
// then hourly. Capped rather than unbounded because a collector that comes
// back after a long outage should be noticed within the hour, and a report
// is only due once per UTC day anyway.
backoffMs(attempts) {
return Math.min(2 ** Math.max(1, attempts), 60) * 60000;
}
async noteDeliveryFailure() {
const state = await this.state();
const attempts = Number(state?.attempts || 0) + 1;
await this.db('product_usage_state')
.where({ id: 1 })
.update({
attempts,
next_attempt_at: this.now() + this.backoffMs(attempts)
});
}
async clearDeliveryBackoff() {
await this.db('product_usage_state')
.where({ id: 1 })
.update({ attempts: 0, next_attempt_at: 0 });
}
async dismiss() {
await this.db('product_usage_state')
.where({ id: 1 })
@@ -329,7 +366,9 @@ class UsageService {
instance_binding: instanceBinding,
sequence: 0,
pending_packet: JSON.stringify(pending),
last_error: null
last_error: null,
attempts: 0,
next_attempt_at: 0
});
// Withdrawn while activating. Participation stays off and nothing was
// registered, so there is nothing to delete remotely either.
@@ -363,11 +402,13 @@ class UsageService {
pending_packet: null,
last_packet: null,
last_receipt: null,
last_report_date: null
last_report_date: null,
attempts: 0,
next_attempt_at: 0
});
await this.db('product_usage_markers').delete();
try {
await this.tick();
await this.tick({ force: true });
} catch (error) {
// A sender may still own the lease. Collection is already stopped and
// the next admin activity retries deletion after that sender finishes.
@@ -376,6 +417,71 @@ class UsageService {
return this.status();
}
// The escape hatch for a withdrawal that can never be signed. When
// USAGE_ENCRYPTION_KEY — or the JWT_SECRET it falls back to — has been
// rotated, the private key is unreadable, so the delete packet cannot be
// produced at all. Retrying and disabling both no-op forever, and enable()
// refuses because the row is not `disabled`: the feature is bricked with no
// control left. Restoring the old key material is the correct fix and stays
// the documented one, but an operator who rotated because of a suspected
// compromise no longer has it.
//
// This drops the local identity and says so honestly: collection is already
// stopped, but the collector was never told, so the receipt records
// `collector-unconfirmed` rather than claiming a deletion that did not
// happen. Deliberately not folded into enable() — abandoning an
// unconfirmed deletion is its own decision, not a side effect of opting in.
async abandon() {
await this.locked(async (state) => {
if (
state.status !== 'deletion_pending' ||
state.last_error !== 'SIGNING_KEY_UNREADABLE'
)
throw new ConflictError(
'Only an unsignable withdrawal can be abandoned'
);
await fs.unlink(this.bindingPath).catch((error) => {
if (error.code !== 'ENOENT') throw error;
});
await this.db('product_usage_markers').delete();
const receipts = state.privacy_receipts
? JSON.parse(state.privacy_receipts)
: {};
await this.db('product_usage_state')
.where({ id: 1, status: 'deletion_pending' })
.update({
status: 'disabled',
installation_id: null,
public_key: null,
private_key_encrypted: null,
instance_binding: null,
pending_packet: null,
last_packet: null,
last_receipt: null,
last_report_date: null,
last_error: null,
sequence: 0,
attempts: 0,
next_attempt_at: 0,
feedback_preferences: null,
privacy_receipts: JSON.stringify({
...receipts,
last_abandonment: {
receipt_version: 'local-audit.v1',
kind: 'abandonment',
receipt_id: crypto.randomUUID(),
confirmed_at: new Date(this.now()).toISOString(),
status: 'collector-unconfirmed',
reason: 'SIGNING_KEY_UNREADABLE',
installation_id: state.installation_id,
scope: ['local identity', 'local markers', 'local key material']
}
})
});
});
return this.status();
}
async post(pathname, body, maxResponseBytes = 65536) {
const response = await this.fetch(`${this.collectorUrl()}${pathname}`, {
method: 'POST',
@@ -496,6 +602,8 @@ class UsageService {
last_report_date: null,
last_error: null,
sequence: 0,
attempts: 0,
next_attempt_at: 0,
feedback_preferences: null
});
} else {
@@ -505,6 +613,8 @@ class UsageService {
sequence: packet.sequence,
pending_packet: null,
last_error: null,
attempts: 0,
next_attempt_at: 0,
last_receipt: JSON.stringify(storedReceipt)
};
if (packet.action === 'report') {
@@ -556,7 +666,12 @@ class UsageService {
await this.db('product_usage_state')
.where({ id: 1 })
.whereNot({ status: 'deletion_pending' })
.update({ pending_packet: null, last_error: 'REQUEST_REJECTED' });
.update({
pending_packet: null,
last_error: 'REQUEST_REJECTED',
attempts: 0,
next_attempt_at: 0
});
return null;
}
const conflict = [
@@ -574,6 +689,11 @@ class UsageService {
await this.db('product_usage_state')
.where({ id: 1 })
.update({ last_error: code });
// Paced, not abandoned: the packet stays pending and the operator can
// still force a retry from the settings page. Only the automatic sender
// waits, which is what stops one permanently rejected packet from
// producing one collector request per admin click.
await this.noteDeliveryFailure();
if (conflict && packet.action !== 'delete') {
await this.db('product_usage_state')
.where({ id: 1 })
@@ -584,9 +704,15 @@ class UsageService {
}
}
async tick() {
// `force` is what the Retry button and /disable pass: an operator asking for
// an attempt now must not be held behind a backoff they can see and want to
// skip. The unattended callers — /activity and the settings ticker — leave
// it off, so a failing packet costs one request per backoff window instead
// of one per admin action.
async tick({ force = false } = {}) {
await this.locked(async (state) => {
if (state.status === 'disabled') return;
if (!force && Number(state.next_attempt_at || 0) > this.now()) return;
if (state.status === 'deletion_pending') {
const packet = makePacket(state, 'delete', Number(state.sequence), {}, this.schemaVersion(state));
state.pending_packet = JSON.stringify(packet);
@@ -666,7 +792,13 @@ class UsageService {
});
}
async snapshot(version) {
// `persist` is false for the settings preview. snapshot() records applied
// custom CSS as a lifetime marker, which meant the "see exactly what would
// be sent" view changed what gets sent — a read with a write behind it, in
// the one place whose whole job is transparency. The reported value is
// unaffected: the marker is derived here either way, and the next real
// report persists it.
async snapshot(version, { persist = true } = {}) {
version = version || this.schemaVersion(await this.state());
const rows = await this.db('app_settings')
.whereIn('setting_key', SETTING_KEYS)
@@ -777,7 +909,7 @@ class UsageService {
// Applied CSS is already a capability in use; no visitor observation is
// needed. Remember its presence as a coarse lifetime marker after consent.
if (features.custom_css.configured) {
await this.markUsed(['custom_css']);
if (persist) await this.markUsed(['custom_css']);
features.custom_css.used = true;
}
const now = new Date(this.now()).toISOString();
@@ -797,7 +929,7 @@ class UsageService {
const state = await this.state();
if (state.status !== 'active')
throw new ConflictError('Usage participation is not active');
return this.snapshot();
return this.snapshot(null, { persist: false });
}
async command(action, payload) {
let receipt;
@@ -894,10 +1026,23 @@ class UsageService {
kind: 'export',
receipt_id: crypto.randomUUID(),
confirmed_at: new Date(this.now()).toISOString(),
// Reports only. Counting every packet — feedback, votes, portal
// sessions, the registration — and labelling the total "usage
// reports" made a privacy receipt state something untrue about
// its own contents, which is exactly the document that has to be
// exact. `packet_count` keeps the total available alongside it.
report_count: Array.isArray(result.packets)
? result.packets.filter(
(envelope) => envelope?.packet?.action === 'report'
).length
: 0,
packet_count: Array.isArray(result.packets)
? result.packets.length
: 0,
scope: ['unique accepted usage reports']
scope: [
'accepted usage reports',
'accepted participant operations'
]
}
})
});
backend/src/usage/expandedSnapshot.js
+16 -4
View File
@@ -91,12 +91,24 @@ async function expandSnapshot(db, { features, flags, used, now }) {
result.gallery_expiration.configured = await exists('events', ['expires_at'], (query) => query.whereNotNull('expires_at'));
result.download_resolution_picker.configured = truth(settings.download_resolution_picker_enabled) ||
await enabled('events', 'download_resolution_picker_enabled');
result.gallery_image_protection.configured = ['standard', 'enhanced', 'maximum'].includes(settings.default_protection_level) ||
truth(settings.enable_devtools_protection) || truth(settings.enable_canvas_rendering);
for (const column of ['disable_right_click', 'enable_devtools_protection', 'use_canvas_rendering'])
// Only what an operator actually changed. PicPeak ships
// default_protection_level='standard' and enable_devtools_protection=true —
// globally and on every event row — so accepting either as evidence made
// this signal `true` on a bare install with no galleries at all. It reported
// fleet-wide 100% and could never separate a deliberate configuration from
// an untouched one, which is a field that costs consent budget and explains
// nothing. `enable_devtools_protection` is therefore not read at all: being
// on by default, its only informative state is off, which is the opposite
// of what this key claims. The remaining inputs each ship off ('standard'
// protection, no canvas rendering, right-click allowed), so a true here is
// always a decision someone made.
result.gallery_image_protection.configured =
['enhanced', 'maximum'].includes(settings.default_protection_level) ||
truth(settings.enable_canvas_rendering);
for (const column of ['disable_right_click', 'use_canvas_rendering'])
result.gallery_image_protection.configured ||= await enabled('events', column);
result.gallery_image_protection.configured ||= await exists('events', ['protection_level'], (query) =>
query.whereIn('protection_level', ['standard', 'enhanced', 'maximum']));
query.whereIn('protection_level', ['enhanced', 'maximum']));
for (const [suffix, column] of Object.entries({
likes: 'allow_likes', ratings: 'allow_ratings', comments: 'allow_comments',
favorites: 'allow_favorites', reactions: 'allow_reactions', color_labels: 'allow_color_labels'
backend/src/usage/features.v2.json
+2 -2
View File
@@ -1252,8 +1252,8 @@
"de": "Bildschutz aktiviert"
},
"configured": {
"en": "Enabled in applicable gallery/global configuration; only existence across the installation, never gallery IDs or counts.",
"de": "In der betreffenden Galerie-/Globalkonfiguration aktiviert; nur installationsweite Existenz, niemals Galeriekennungen oder Anzahlen."
"en": "Enabled beyond the shipped defaults — a stronger protection level, canvas rendering, or right-click disabled — globally or on at least one gallery; only existence across the installation, never gallery IDs or counts.",
"de": "Über die Auslieferungsvorgaben hinaus aktiviert — höhere Schutzstufe, Canvas-Rendering oder deaktivierter Rechtsklick — global oder in mindestens einer Galerie; nur installationsweite Existenz, niemals Galeriekennungen oder Anzahlen."
},
"used": null
},