fix(usage): close the QA findings on opt-in product usage

A QA exploration of this branch against an isolated rig — own stub
collector, SQLite and PostgreSQL — turned up one dead end and a set of
signals and controls that did not hold up. This closes all of them.

Rotating JWT_SECRET, the documented response to a suspected compromise,
made the signing key unreadable. That was already named and documented,
but it left no way out: the delete packet can never be signed, so the
row stays deletion_pending forever, and enable() refuses because it is
not `disabled`. An operator who rotated precisely because the secret was
compromised cannot restore it, so the feature was bricked with no
control left. POST /usage/abandon is offered only in that state; it
drops the local identity and records the receipt as
`collector-unconfirmed` rather than claiming a deletion that did not
happen.

Every failed delivery was retried on the next admin request, and
/activity is open to any authenticated admin while the settings ticker
fires it every five minutes per open tab — 30 activity calls against a
rejecting collector produced 30 outbound requests. Migration 206 adds
attempts/next_attempt_at and the unattended sender honours the gate;
Retry and opt-out still send immediately, and the tab names the time of
the next automatic attempt.

Feedback, votes and portal sessions now share an installation-wide
budget of 30/hour. They are the only endpoints whose effect is outbound
traffic carrying operator-written free text, and the general limiter
skips authenticated requests by design. Reading status and withdrawing
stay unthrottled.

gallery_image_protection was true on a bare install with no galleries:
PicPeak ships default_protection_level='standard' and
enable_devtools_protection=true, so it reported fleet-wide 100% and
could never separate a decision from an untouched default. It now reads
only what deviates from the shipped defaults, and the devtools flag is
not read at all — being on by default, its only informative state is
off, which is the opposite of what the key claims.

Also:
- the export receipt counted every packet and called the total "usage
  reports"; reports and participant operations are now counted and named
  separately
- GET /usage/preview no longer persists the custom_css marker, so the
  transparency view stops changing what will be sent
- the feedback route requires every field the packet schema requires,
  so an API caller gets the missing field named instead of a bare
  INVALID_PACKET from inside signing
- the German strings for this feature use "Sie" throughout, matching the
  rest of the admin UI; the ignore hint says what ignoring will do
  rather than stating it as already true
- the consent dialog returns focus to the control that opened it
- the long buttons wrap instead of running off a 390px viewport
- a deletion receipt is labelled as belonging to an earlier
  participation while a new one is active

Regression tests cover each of these, including the delete packet's
reuse of the last accepted sequence, which was an unwritten assumption
about the collector rather than a defect.
This commit is contained in:
Paul Nothaft
2026-09-06 17:40:43 +02:00
parent a7382591bf
commit 1e8b6f1b0f
20 changed files with 1076 additions and 62 deletions
@@ -18,6 +18,7 @@ const knex = require('knex');
const fs = require('fs');
const os = require('os');
const path = require('path');
const { generateIdentity, makePacket } = require('../../src/usage/protocol.cjs');
const PG_URL = process.env.PICPEAK_PG_TEST_URL;
const maybe = PG_URL ? describe : describe.skip;
@@ -52,6 +53,7 @@ maybe('product usage on Postgres', () => {
await require('../../migrations/core/203_product_usage_cancel_seq').up(db);
await require('../../migrations/core/204_product_usage_privacy_receipts').up(db);
await require('../../migrations/core/205_product_usage_consent_version').up(db);
await require('../../migrations/core/206_product_usage_delivery_backoff').up(db);
await db.schema.createTable('app_settings', (t) => {
t.string('setting_key').primary(); t.text('setting_value'); t.string('setting_type');
@@ -114,6 +116,56 @@ maybe('product usage on Postgres', () => {
expect(cols.sequence).toBeDefined();
expect(cols.privacy_receipts).toBeDefined();
expect(cols.consent_version).toBeDefined();
// next_attempt_at is a bigint like sequence and cancel_seq, so pg hands it
// back as a STRING — the tick() gate compares it against a number.
expect(cols.attempts).toBeDefined();
expect(cols.next_attempt_at).toBeDefined();
});
it('reruns the backoff migration safely', async () => {
const migration = require('../../migrations/core/206_product_usage_delivery_backoff');
await migration.up(db);
await migration.up(db);
const row = await db('product_usage_state').where({ id: 1 }).first();
expect(Number(row.attempts)).toBe(0);
expect(Number(row.next_attempt_at)).toBe(0);
});
it('honours the retry gate even though pg returns next_attempt_at as a string', async () => {
let clock = 5_000_000;
let calls = 0;
const identity = generateIdentity();
const client = service({
now: () => clock,
fetch: async () => { calls += 1; throw new Error('collector unreachable'); },
});
await db('product_usage_state').where({ id: 1 }).update({
status: 'active',
consent_version: 'usage-consent.v2',
installation_id: identity.installation_id,
public_key: identity.public_key,
private_key_encrypted: client.encrypt(identity.private_key),
sequence: 1,
attempts: 0,
next_attempt_at: 0,
pending_packet: JSON.stringify(makePacket(identity, 'session', 2, {}, 'usage.v2')),
});
await client.tick();
expect(calls).toBe(1);
const paced = await db('product_usage_state').where({ id: 1 }).first();
// A '5000120000' > 5000000 string comparison would be a different answer.
expect(typeof paced.next_attempt_at).toBe('string');
await client.tick();
expect(calls).toBe(1);
clock = Number(paced.next_attempt_at) + 1;
await client.tick();
expect(calls).toBe(2);
await db('product_usage_state').where({ id: 1 }).update({
status: 'disabled', pending_packet: null, attempts: 0, next_attempt_at: 0,
});
});
it('reruns the receipt migration safely and scrubs legacy plaintext sessions', async () => {
@@ -31,6 +31,7 @@ jest.mock('../../src/services/productUsageService', () =>
'dismiss',
'enable',
'disable',
'abandon',
'preview',
'export',
'preferences',
@@ -125,6 +126,7 @@ const ROUTES = [
['post', '/enable'],
['post', '/consent'],
['post', '/disable'],
['post', '/abandon'],
['post', '/retry'],
['post', '/dismiss'],
['get', '/preview'],
@@ -230,3 +232,89 @@ test('only a backup that writes to the configured destination flags S3', () => {
['/backup/picpeak/export', false],
]);
});
// The route allowlist and the packet schema have to agree. The allowlist used
// to let `name`, `allow_public` and `allow_marketing` be omitted while the
// schema requires all three, so an API caller got a bare INVALID_PACKET from
// deep inside signing instead of being told which field was missing.
const VALID_FEEDBACK = {
kind: 'feedback',
title: 'Title',
body: 'Body',
name: '',
allow_public: false,
allow_marketing: false
};
test.each([
['no body at all', {}],
['missing name', { ...VALID_FEEDBACK, name: undefined }],
['missing allow_public', { ...VALID_FEEDBACK, allow_public: undefined }],
['missing allow_marketing', { ...VALID_FEEDBACK, allow_marketing: undefined }],
['a boolean sent as a string', { ...VALID_FEEDBACK, allow_public: 'true' }],
['a title of only whitespace', { ...VALID_FEEDBACK, title: ' ' }],
['an unknown field', { ...VALID_FEEDBACK, ownerId: 7 }]
])('feedback rejects %s before anything is signed', async (_label, data) => {
const response = await request(app)
.post('/api/admin/usage/feedback')
.set('Authorization', `Bearer ${token('admin')}`)
.send(JSON.parse(JSON.stringify(data)))
.expect(400);
// Named, not a bare protocol failure the caller cannot act on.
expect(response.body.code).toBe('VALIDATION_ERROR');
expect(service.command).not.toHaveBeenCalled();
});
test('feedback accepts the complete payload and mints the id server-side', async () => {
await request(app)
.post('/api/admin/usage/feedback')
.set('Authorization', `Bearer ${token('admin')}`)
.send({ ...VALID_FEEDBACK, name: 'QA' })
.expect(200);
expect(service.command).toHaveBeenCalledWith(
'feedback',
expect.objectContaining({ name: 'QA', feedback_id: expect.any(String) })
);
});
// Runs last on purpose: the limiter's budget is per-process and shared with
// every test above that reaches an outbound route, so consuming it here cannot
// starve them. The assertion is deliberately about the property — some request
// is refused and the service stops being called — rather than an exact count,
// which would depend on how much budget earlier tests used.
test('the outbound routes are throttled so an admin session cannot flood the collector', async () => {
const codes = [];
for (let i = 0; i < 45; i += 1) {
const response = await request(app)
.post('/api/admin/usage/feedback')
.set('Authorization', `Bearer ${token('admin')}`)
.send({ ...VALID_FEEDBACK, title: `flood ${i}` });
codes.push(response.status);
if (response.status === 429) {
expect(response.body.code).toBe('USAGE_RATE_LIMITED');
break;
}
}
expect(codes).toContain(429);
expect(service.command.mock.calls.length).toBeLessThan(codes.length);
// The same budget covers the other two routes that relay to the collector.
await request(app)
.post('/api/admin/usage/vote')
.set('Authorization', `Bearer ${token('admin')}`)
.send({ feedback_id: '11111111-1111-4111-8111-111111111111', voted: true })
.expect(429);
await request(app)
.post('/api/admin/usage/portal-session')
.set('Authorization', `Bearer ${token('admin')}`)
.expect(429);
// Reading status and withdrawing must never be throttled: those are how an
// operator sees what is happening and how they get out.
await request(app)
.get('/api/admin/usage')
.set('Authorization', `Bearer ${token('admin')}`)
.expect(200);
await request(app)
.post('/api/admin/usage/disable')
.set('Authorization', `Bearer ${token('admin')}`)
.expect(200);
});
@@ -57,6 +57,8 @@ async function bootDb() {
t.string('lease_token', 36);
t.bigInteger('lease_until').notNullable().defaultTo(0);
t.bigInteger('cancel_seq').notNullable().defaultTo(0);
t.integer('attempts').notNullable().defaultTo(0);
t.bigInteger('next_attempt_at').notNullable().defaultTo(0);
});
await db.schema.createTable('product_usage_markers', (t) => {
t.string('feature', 60).primary();
@@ -0,0 +1,147 @@
/**
* Two things the participant is entitled to have stated exactly.
*
* The export receipt is a privacy document — the artefact an operator shows a
* third party — so a count in it has to mean what its label says. It counted
* every packet in the participation (feedback, votes, portal sessions, the
* registration) and called the total "usage reports": an install that had sent
* one report and twenty feedback items reported twenty-one reports.
*
* The delete packet's sequence is the other: it reuses the last ACCEPTED
* sequence rather than taking the next one, unlike every other action. That is
* a contract with the collector, not an implementation detail — if the
* collector ever enforced strictly increasing sequences per installation, the
* withdrawal would be rejected forever and the operator could never leave. It
* is pinned here so the assumption is written down and a change to it has to
* be deliberate.
*/
const knex = require('knex');
const { UsageService } = require('../../src/usage/UsageService');
const { generateIdentity, verifyEnvelope } = require('../../src/usage/protocol.cjs');
const SECRET = 's'.repeat(48);
async function bootDb() {
const db = knex({
client: 'sqlite3',
connection: { filename: ':memory:' },
useNullAsDefault: true,
});
await db.schema.createTable('product_usage_state', (t) => {
t.integer('id').primary();
t.string('status', 30).notNullable().defaultTo('disabled');
t.string('consent_version', 40).notNullable().defaultTo('usage-consent.v2');
t.boolean('notice_dismissed').notNullable().defaultTo(false);
t.string('installation_id', 64);
t.string('public_key', 59);
t.text('private_key_encrypted');
t.string('instance_binding', 64);
t.bigInteger('sequence').notNullable().defaultTo(0);
t.text('pending_packet');
t.text('last_packet');
t.text('last_receipt');
t.text('privacy_receipts');
t.string('last_report_date', 10);
t.string('last_error', 80);
t.text('feedback_preferences');
t.string('lease_token', 36);
t.bigInteger('lease_until').notNullable().defaultTo(0);
t.bigInteger('cancel_seq').notNullable().defaultTo(0);
t.integer('attempts').notNullable().defaultTo(0);
t.bigInteger('next_attempt_at').notNullable().defaultTo(0);
});
await db('product_usage_state').insert({ id: 1 });
await db.schema.createTable('product_usage_markers', (t) => t.string('feature', 60).primary());
return db;
}
const envelope = (action) => ({ packet: { action, installation_id: 'a'.repeat(64) } });
describe('the export receipt states what it actually counted', () => {
let db;
afterEach(async () => { if (db) await db.destroy(); db = null; });
const exportWith = async (packets) => {
db = await bootDb();
await db('product_usage_state').where({ id: 1 }).update({
status: 'active',
installation_id: 'a'.repeat(64),
});
const service = new UsageService(db, {
secret: SECRET,
endpoint: 'https://usage.example.test',
now: () => Date.parse('2026-09-06T12:00:00.000Z'),
fetch: async () => ({
ok: true,
headers: { get: () => null },
body: (async function* () {
yield Buffer.from(JSON.stringify({ installation_id: 'a'.repeat(64), packets }));
})(),
}),
});
await service.export();
return JSON.parse((await db('product_usage_state').where({ id: 1 }).first()).privacy_receipts)
.last_export;
};
it('counts reports as reports and everything else separately', async () => {
const receipt = await exportWith([
envelope('register'),
envelope('report'),
envelope('consent'),
envelope('feedback'),
envelope('feedback'),
envelope('vote'),
envelope('session'),
]);
expect(receipt.report_count).toBe(1);
expect(receipt.packet_count).toBe(7);
expect(receipt.scope).toEqual([
'accepted usage reports',
'accepted participant operations',
]);
});
it('reports zero rather than a total when no report was ever accepted', async () => {
const receipt = await exportWith([envelope('register'), envelope('feedback')]);
expect(receipt.report_count).toBe(0);
expect(receipt.packet_count).toBe(2);
});
});
describe('the delete packet reuses the last accepted sequence', () => {
let db;
afterEach(async () => { if (db) await db.destroy(); db = null; });
it('sends the accepted sequence, not the next one', async () => {
db = await bootDb();
const identity = generateIdentity();
const sent = [];
const service = new UsageService(db, {
secret: SECRET,
endpoint: 'https://usage.example.test',
now: () => Date.parse('2026-09-06T12:00:00.000Z'),
bindingPath: `${require('os').tmpdir()}/usage-delete-seq-${Date.now()}.key`,
fetch: async (_url, options) => {
const body = JSON.parse(options.body);
sent.push(verifyEnvelope(body, Date.parse('2026-09-06T12:00:00.000Z')));
// Deliberately not a sequence-enforcing collector: this test pins what
// PicPeak sends, and the collector contract is what must match it.
throw new Error('stop after capturing the packet');
},
});
await db('product_usage_state').where({ id: 1 }).update({
status: 'deletion_pending',
installation_id: identity.installation_id,
public_key: identity.public_key,
private_key_encrypted: service.encrypt(identity.private_key),
sequence: 7,
});
await service.tick({ force: true });
expect(sent).toHaveLength(1);
expect(sent[0].action).toBe('delete');
expect(sent[0].sequence).toBe(7);
});
});
@@ -10,6 +10,7 @@
*/
const knex = require('knex');
const { UsageService } = require('../../src/usage/UsageService');
const { generateIdentity, makePacket } = require('../../src/usage/protocol.cjs');
const SECRET_A = 'a'.repeat(48);
const SECRET_B = 'b'.repeat(48);
@@ -39,6 +40,8 @@ async function bootDb() {
t.text('feedback_preferences');
t.string('lease_token', 36);
t.bigInteger('lease_until').notNullable().defaultTo(0);
t.integer('attempts').notNullable().defaultTo(0);
t.bigInteger('next_attempt_at').notNullable().defaultTo(0);
});
await db('product_usage_state').insert({ id: 1 });
return db;
@@ -94,3 +97,166 @@ describe('usage signing key becomes unreadable after secret rotation', () => {
expect(row.status).toBe('active');
});
});
/**
* Naming the failure told the operator what happened but left them nowhere to
* go: the delete packet can never be signed, so the row stays in
* deletion_pending forever, and enable() refuses because it is not `disabled`.
* An operator who rotated the secret precisely because it was compromised
* cannot restore it, so without an exit the feature is bricked.
*/
describe('abandoning a withdrawal that can never be signed', () => {
let db;
afterEach(async () => { if (db) await db.destroy(); db = null; });
const stuck = async () => {
db = await bootDb();
await db.schema.createTable('product_usage_markers', (t) => {
t.string('feature', 60).primary();
});
await db('product_usage_markers').insert({ feature: 'crm' });
await db('product_usage_state').where({ id: 1 }).update({
status: 'deletion_pending',
installation_id: 'a'.repeat(64),
public_key: 'p'.repeat(59),
private_key_encrypted: new UsageService(db, { secret: SECRET_A }).encrypt('key'),
sequence: 4,
last_error: 'SIGNING_KEY_UNREADABLE',
});
return new UsageService(db, {
secret: SECRET_B,
endpoint: 'https://usage.example.test',
bindingPath: `${require('os').tmpdir()}/usage-abandon-${Date.now()}.key`,
fetch: () => { throw new Error('network must not be reached'); },
});
};
it('clears the local identity and records the deletion as unconfirmed', async () => {
const service = await stuck();
const status = await service.abandon();
expect(status.status).toBe('disabled');
expect(status.installation_id).toBeNull();
const row = await db('product_usage_state').where({ id: 1 }).first();
expect(row.private_key_encrypted).toBeNull();
expect(row.public_key).toBeNull();
expect(row.last_error).toBeNull();
expect(await db('product_usage_markers').count('* as c').first()).toEqual({ c: 0 });
// The receipt must not claim a deletion the collector never confirmed.
const receipt = JSON.parse(row.privacy_receipts).last_abandonment;
expect(receipt.status).toBe('collector-unconfirmed');
expect(receipt.reason).toBe('SIGNING_KEY_UNREADABLE');
expect(receipt.installation_id).toBe('a'.repeat(64));
});
it('lets the operator rejoin afterwards', async () => {
const service = await stuck();
await service.abandon();
expect((await service.state()).status).toBe('disabled');
});
it('refuses on a withdrawal that is merely undelivered', async () => {
const service = await stuck();
await db('product_usage_state').where({ id: 1 }).update({ last_error: 'DELIVERY_FAILED' });
await expect(service.abandon()).rejects.toThrow(/abandoned/);
expect((await service.state()).installation_id).toBe('a'.repeat(64));
});
it('refuses while participation is active', async () => {
const service = await stuck();
await db('product_usage_state').where({ id: 1 }).update({ status: 'active' });
await expect(service.abandon()).rejects.toThrow(/abandoned/);
expect((await service.state()).installation_id).toBe('a'.repeat(64));
});
});
/**
* Every failed delivery used to be retried on the next admin request, and
* /activity is open to any authenticated admin while the settings ticker fires
* it every five minutes per open tab. A permanently rejected packet therefore
* produced one collector request per admin action, indefinitely.
*/
describe('delivery backoff', () => {
let db;
afterEach(async () => { if (db) await db.destroy(); db = null; });
// A real identity and a schema-valid packet, so the failure happens where
// this test claims it does — at the network — rather than in signPacket.
const activeWithPendingPacket = async (fetchImpl, now) => {
db = await bootDb();
await db.schema.createTable('product_usage_markers', (t) => {
t.string('feature', 60).primary();
});
const service = new UsageService(db, {
secret: SECRET_A,
endpoint: 'https://usage.example.test',
now: () => now(),
fetch: fetchImpl,
});
const identity = generateIdentity();
await db('product_usage_state').where({ id: 1 }).update({
status: 'active',
consent_version: 'usage-consent.v2',
installation_id: identity.installation_id,
public_key: identity.public_key,
private_key_encrypted: service.encrypt(identity.private_key),
sequence: 1,
pending_packet: JSON.stringify(
makePacket(identity, 'session', 2, {}, 'usage.v2')
),
});
return service;
};
it('paces the next unattended attempt after a failure, and lets Retry skip it', async () => {
let clock = 1_000_000;
let calls = 0;
const service = await activeWithPendingPacket(() => {
calls += 1;
throw new Error('collector unreachable');
}, () => clock);
await service.tick();
expect(calls).toBe(1);
const paced = await service.state();
expect(Number(paced.attempts)).toBe(1);
expect(Number(paced.next_attempt_at)).toBeGreaterThan(clock);
// The unattended callers — /activity and the settings ticker — wait.
await service.tick();
await service.tick();
expect(calls).toBe(1);
// The operator pressing Retry does not.
await service.tick({ force: true });
expect(calls).toBe(2);
expect(Number((await service.state()).attempts)).toBe(2);
// Once the window passes, the automatic sender tries again on its own.
clock = Number((await service.state()).next_attempt_at) + 1;
await service.tick();
expect(calls).toBe(3);
});
it('grows the wait with consecutive failures and caps it at an hour', () => {
const service = new UsageService(null, { secret: SECRET_A, endpoint: 'https://usage.example.test' });
expect(service.backoffMs(1)).toBe(2 * 60000);
expect(service.backoffMs(3)).toBe(8 * 60000);
expect(service.backoffMs(20)).toBe(60 * 60000);
});
it('clears the pacing once a packet is accepted', async () => {
const clock = 1_000_000;
const service = await activeWithPendingPacket(async () => {
throw new Error('collector unreachable');
}, () => clock);
await service.tick();
expect(Number((await service.state()).attempts)).toBe(1);
await service.clearDeliveryBackoff();
const cleared = await service.state();
expect(Number(cleared.attempts)).toBe(0);
expect(Number(cleared.next_attempt_at)).toBe(0);
});
});
@@ -34,6 +34,8 @@ async function bootDb() {
t.text('feedback_preferences'); t.string('lease_token', 36);
t.bigInteger('lease_until').notNullable().defaultTo(0);
t.bigInteger('cancel_seq').notNullable().defaultTo(0);
t.integer('attempts').notNullable().defaultTo(0);
t.bigInteger('next_attempt_at').notNullable().defaultTo(0);
});
await db('product_usage_state').insert({ id: 1 });
await db.schema.createTable('product_usage_markers', (t) => t.string('feature', 60).primary());
@@ -196,6 +198,116 @@ describe('S3 use is only implied by backups that write to the destination', () =
});
});
/**
* A signal whose answer is fixed by the shipped defaults is not a signal.
* PicPeak ships default_protection_level='standard' and
* enable_devtools_protection=true, so accepting either as evidence made
* gallery_image_protection true on a bare install with no galleries — a
* fleet-wide 100% that cannot separate a decision from an untouched default.
*/
describe('gallery_image_protection reports decisions, not shipped defaults', () => {
let db;
afterEach(async () => { if (db) await db.destroy(); db = null; });
const v2 = async () => {
db = await bootDb();
await db.schema.alterTable('events', (t) => {
for (const column of ['disable_right_click', 'enable_devtools_protection', 'use_canvas_rendering']) t.boolean(column);
t.string('protection_level');
});
await db('product_usage_state').where({ id: 1 })
.update({ status: 'active', consent_version: 'usage-consent.v2' });
return service(db);
};
const shipped = async () => {
// Exactly what migration 038 seeds, plus an event carrying the column
// defaults from the same migration.
await db('app_settings').insert([
{ setting_key: 'default_protection_level', setting_value: '"standard"' },
{ setting_key: 'enable_devtools_protection', setting_value: 'true' },
{ setting_key: 'enable_canvas_rendering', setting_value: 'false' },
]);
await db('events').insert({
protection_level: 'standard',
enable_devtools_protection: true,
use_canvas_rendering: false,
disable_right_click: false,
});
};
it('is false on a bare install with no galleries at all', async () => {
const client = await v2();
expect((await client.snapshot()).features.gallery_image_protection)
.toEqual({ configured: false });
});
it('is false when every value is still the shipped default', async () => {
const client = await v2();
await shipped();
expect((await client.snapshot()).features.gallery_image_protection)
.toEqual({ configured: false });
});
it('ignores the devtools flag entirely, since it ships on', async () => {
const client = await v2();
await shipped();
// Turning it OFF is the only informative state it has, and that is the
// opposite of what this key claims — so neither state may set it.
await db('app_settings').where({ setting_key: 'enable_devtools_protection' })
.update({ setting_value: 'false' });
await db('events').update({ enable_devtools_protection: false });
expect((await client.snapshot()).features.gallery_image_protection)
.toEqual({ configured: false });
});
it.each([
['a stronger global level', async (db) => db('app_settings').where({ setting_key: 'default_protection_level' }).update({ setting_value: '"maximum"' })],
['global canvas rendering', async (db) => db('app_settings').where({ setting_key: 'enable_canvas_rendering' }).update({ setting_value: 'true' })],
['a stronger level on one gallery', async (db) => db('events').update({ protection_level: 'enhanced' })],
['canvas rendering on one gallery', async (db) => db('events').update({ use_canvas_rendering: true })],
['right-click disabled on one gallery', async (db) => db('events').update({ disable_right_click: true })],
])('is true for %s', async (_label, change) => {
const client = await v2();
await shipped();
await change(db);
expect((await client.snapshot()).features.gallery_image_protection)
.toEqual({ configured: true });
});
});
/**
* The settings preview is the "see exactly what would be sent" view. It shared
* snapshot() with the real sender, and snapshot() records applied custom CSS
* as a lifetime marker — so reading the transparency view wrote a marker.
*/
describe('preview does not change what will be sent', () => {
let db;
afterEach(async () => { if (db) await db.destroy(); db = null; });
const withAppliedCss = async () => {
db = await bootDb();
await db('product_usage_state').where({ id: 1 })
.update({ status: 'active', consent_version: 'usage-consent.v2' });
await db('app_settings').insert({
setting_key: 'general_custom_css', setting_value: '".x{}"'
});
return service(db);
};
it('reports custom_css as used without persisting the marker', async () => {
const client = await withAppliedCss();
const preview = await client.preview();
expect(preview.features.custom_css).toEqual({ configured: true, used: true });
expect(await db('product_usage_markers').pluck('feature')).toEqual([]);
});
it('still persists it when the sender builds the real report', async () => {
const client = await withAppliedCss();
await client.snapshot();
expect(await db('product_usage_markers').pluck('feature')).toEqual(['custom_css']);
});
});
describe('v2 technical configuration and privacy boundaries', () => {
let db;
let savedEnv;
@@ -0,0 +1,31 @@
// Retry pacing for the collector. Without it every failed packet was retried
// on the next admin request: /activity is open to any authenticated admin and
// the settings ticker fires it every five minutes per open tab, so an
// installation whose packet the collector rejects permanently hammered it
// once per admin action, forever, with a failing request sitting on the
// critical path of that action.
//
// `attempts` counts consecutive failures and `next_attempt_at` is the epoch-ms
// gate the automatic sender honours. Explicit operator actions — Retry and
// Disable — pass through regardless; the point is to pace the unattended loop,
// not to make the admin wait out a backoff they asked to skip.
exports.up = async function (knex) {
if (!(await knex.schema.hasTable('product_usage_state'))) return;
if (!(await knex.schema.hasColumn('product_usage_state', 'attempts')))
await knex.schema.alterTable('product_usage_state', (t) => {
t.integer('attempts').notNullable().defaultTo(0);
});
if (!(await knex.schema.hasColumn('product_usage_state', 'next_attempt_at')))
await knex.schema.alterTable('product_usage_state', (t) => {
t.bigInteger('next_attempt_at').notNullable().defaultTo(0);
});
};
exports.down = async function (knex) {
if (!(await knex.schema.hasTable('product_usage_state'))) return;
for (const column of ['attempts', 'next_attempt_at'])
if (await knex.schema.hasColumn('product_usage_state', column))
await knex.schema.alterTable('product_usage_state', (t) => {
t.dropColumn(column);
});
};
+66 -19
View File
@@ -1,5 +1,6 @@
const express = require('express');
const crypto = require('crypto');
const rateLimit = require('express-rate-limit');
const { adminAuth } = require('../middleware/auth');
const { requirePermission } = require('../middleware/permissions');
const { ValidationError } = require('../utils/errors');
@@ -21,6 +22,30 @@ const wrap = (fn) => (req, res, next) =>
.json({ error: 'Invalid usage request', code: error.code });
next(error);
});
// The three routes below are the only ones whose effect is an outbound
// request to someone else's service, carrying operator-written free text
// (title 120, body 4000, name 80). The platform's general limiter skips
// authenticated requests by design, which is right for endpoints that only
// touch this installation and wrong for a relay: without this an admin
// session can push unbounded traffic at the collector.
//
// Keyed to the installation, not the caller's IP, because the budget being
// protected is "how much this install relays", and per-process because that
// is the same store the rest of the app uses — a multi-replica deployment
// gets one budget per replica, which still bounds the shape that matters.
const outboundLimiter = rateLimit({
windowMs: 60 * 60 * 1000,
max: 30,
keyGenerator: () => 'usage-outbound',
standardHeaders: true,
legacyHeaders: false,
handler: (_req, res) =>
res.status(429).json({
error: 'Too many usage submissions. Try again later.',
code: 'USAGE_RATE_LIMITED'
})
});
router.use(adminAuth);
router.use((_req, res, next) => {
res.set('Cache-Control', 'no-store');
@@ -66,9 +91,17 @@ router.post(
'/disable',
wrap(async (_req, res) => res.json(await service.disable()))
);
// Reachable only from a withdrawal whose delete packet can never be signed;
// the service refuses in every other state. See UsageService.abandon().
router.post(
'/abandon',
wrap(async (_req, res) => res.json(await service.abandon()))
);
// An operator asking for a retry skips the delivery backoff — that button
// exists precisely to not wait for the next window.
router.post(
'/retry',
wrap(async (_req, res) => res.json(await service.tick()))
wrap(async (_req, res) => res.json(await service.tick({ force: true })))
);
router.get(
'/preview',
@@ -84,29 +117,41 @@ router.put(
'/feedback-preferences',
wrap(async (req, res) => res.json(await service.preferences(req.body)))
);
// Every field the packet schema requires. The allowlist used to let `name`,
// `allow_public` and `allow_marketing` be omitted, and the packet schema —
// which requires all of them — then failed with a bare INVALID_PACKET instead
// of naming the missing field. The UI always sends them; anything driving the
// API directly did not, and got an error it could not act on.
const FEEDBACK_FIELDS = [
'kind',
'title',
'body',
'name',
'allow_public',
'allow_marketing'
];
router.post(
'/feedback',
outboundLimiter,
wrap(async (req, res) => {
const body = req.body;
if (
!body ||
Object.keys(body).some(
(k) =>
![
'kind',
'title',
'body',
'name',
'allow_public',
'allow_marketing'
].includes(k)
) ||
typeof body.title !== 'string' ||
!body.title.trim() ||
typeof body.body !== 'string' ||
!body.body.trim()
)
if (!body || typeof body !== 'object')
throw new ValidationError('Invalid feedback');
const unknown = Object.keys(body).filter(
(key) => !FEEDBACK_FIELDS.includes(key)
);
if (unknown.length)
throw new ValidationError(
`Unknown feedback fields: ${unknown.join(', ')}`
);
for (const key of ['kind', 'title', 'body', 'name'])
if (typeof body[key] !== 'string')
throw new ValidationError(`Feedback field "${key}" must be a string`);
for (const key of ['allow_public', 'allow_marketing'])
if (typeof body[key] !== 'boolean')
throw new ValidationError(`Feedback field "${key}" must be a boolean`);
if (!body.title.trim() || !body.body.trim())
throw new ValidationError('Feedback title and body are required');
res.json(
await service.command('feedback', {
...body,
@@ -117,10 +162,12 @@ router.post(
);
router.post(
'/vote',
outboundLimiter,
wrap(async (req, res) => res.json(await service.command('vote', req.body)))
);
router.post(
'/portal-session',
outboundLimiter,
wrap(async (_req, res) => {
const result = await service.command('session', {});
res.json({
+154 -9
View File
@@ -248,6 +248,20 @@ class UsageService {
consent_update_available: state.status === 'active' && this.schemaVersion(state) !== CURRENT_SCHEMA_VERSION,
last_report_date: state.last_report_date,
last_error: state.last_error,
// Epoch ms, or null when nothing is being paced. The settings page shows
// it so a waiting install reads as "waiting" rather than as broken.
retry_after:
Number(state.next_attempt_at || 0) > this.now()
? Number(state.next_attempt_at)
: null,
// The one failure the operator cannot retry their way out of: the
// signing key is unreadable, so the delete packet can never be signed.
// Without this flag the settings page has no way to offer the only
// remaining exit (abandon), and the install sits in deletion_pending
// forever.
can_abandon:
state.status === 'deletion_pending' &&
state.last_error === 'SIGNING_KEY_UNREADABLE',
pending_action: state.pending_packet
? JSON.parse(state.pending_packet).action
: null,
@@ -278,6 +292,29 @@ class UsageService {
}
}
// Consecutive failures pace the unattended sender: 2, 4, 8, 16, 32 minutes,
// then hourly. Capped rather than unbounded because a collector that comes
// back after a long outage should be noticed within the hour, and a report
// is only due once per UTC day anyway.
backoffMs(attempts) {
return Math.min(2 ** Math.max(1, attempts), 60) * 60000;
}
async noteDeliveryFailure() {
const state = await this.state();
const attempts = Number(state?.attempts || 0) + 1;
await this.db('product_usage_state')
.where({ id: 1 })
.update({
attempts,
next_attempt_at: this.now() + this.backoffMs(attempts)
});
}
async clearDeliveryBackoff() {
await this.db('product_usage_state')
.where({ id: 1 })
.update({ attempts: 0, next_attempt_at: 0 });
}
async dismiss() {
await this.db('product_usage_state')
.where({ id: 1 })
@@ -329,7 +366,9 @@ class UsageService {
instance_binding: instanceBinding,
sequence: 0,
pending_packet: JSON.stringify(pending),
last_error: null
last_error: null,
attempts: 0,
next_attempt_at: 0
});
// Withdrawn while activating. Participation stays off and nothing was
// registered, so there is nothing to delete remotely either.
@@ -363,11 +402,13 @@ class UsageService {
pending_packet: null,
last_packet: null,
last_receipt: null,
last_report_date: null
last_report_date: null,
attempts: 0,
next_attempt_at: 0
});
await this.db('product_usage_markers').delete();
try {
await this.tick();
await this.tick({ force: true });
} catch (error) {
// A sender may still own the lease. Collection is already stopped and
// the next admin activity retries deletion after that sender finishes.
@@ -376,6 +417,71 @@ class UsageService {
return this.status();
}
// The escape hatch for a withdrawal that can never be signed. When
// USAGE_ENCRYPTION_KEY — or the JWT_SECRET it falls back to — has been
// rotated, the private key is unreadable, so the delete packet cannot be
// produced at all. Retrying and disabling both no-op forever, and enable()
// refuses because the row is not `disabled`: the feature is bricked with no
// control left. Restoring the old key material is the correct fix and stays
// the documented one, but an operator who rotated because of a suspected
// compromise no longer has it.
//
// This drops the local identity and says so honestly: collection is already
// stopped, but the collector was never told, so the receipt records
// `collector-unconfirmed` rather than claiming a deletion that did not
// happen. Deliberately not folded into enable() — abandoning an
// unconfirmed deletion is its own decision, not a side effect of opting in.
async abandon() {
await this.locked(async (state) => {
if (
state.status !== 'deletion_pending' ||
state.last_error !== 'SIGNING_KEY_UNREADABLE'
)
throw new ConflictError(
'Only an unsignable withdrawal can be abandoned'
);
await fs.unlink(this.bindingPath).catch((error) => {
if (error.code !== 'ENOENT') throw error;
});
await this.db('product_usage_markers').delete();
const receipts = state.privacy_receipts
? JSON.parse(state.privacy_receipts)
: {};
await this.db('product_usage_state')
.where({ id: 1, status: 'deletion_pending' })
.update({
status: 'disabled',
installation_id: null,
public_key: null,
private_key_encrypted: null,
instance_binding: null,
pending_packet: null,
last_packet: null,
last_receipt: null,
last_report_date: null,
last_error: null,
sequence: 0,
attempts: 0,
next_attempt_at: 0,
feedback_preferences: null,
privacy_receipts: JSON.stringify({
...receipts,
last_abandonment: {
receipt_version: 'local-audit.v1',
kind: 'abandonment',
receipt_id: crypto.randomUUID(),
confirmed_at: new Date(this.now()).toISOString(),
status: 'collector-unconfirmed',
reason: 'SIGNING_KEY_UNREADABLE',
installation_id: state.installation_id,
scope: ['local identity', 'local markers', 'local key material']
}
})
});
});
return this.status();
}
async post(pathname, body, maxResponseBytes = 65536) {
const response = await this.fetch(`${this.collectorUrl()}${pathname}`, {
method: 'POST',
@@ -496,6 +602,8 @@ class UsageService {
last_report_date: null,
last_error: null,
sequence: 0,
attempts: 0,
next_attempt_at: 0,
feedback_preferences: null
});
} else {
@@ -505,6 +613,8 @@ class UsageService {
sequence: packet.sequence,
pending_packet: null,
last_error: null,
attempts: 0,
next_attempt_at: 0,
last_receipt: JSON.stringify(storedReceipt)
};
if (packet.action === 'report') {
@@ -556,7 +666,12 @@ class UsageService {
await this.db('product_usage_state')
.where({ id: 1 })
.whereNot({ status: 'deletion_pending' })
.update({ pending_packet: null, last_error: 'REQUEST_REJECTED' });
.update({
pending_packet: null,
last_error: 'REQUEST_REJECTED',
attempts: 0,
next_attempt_at: 0
});
return null;
}
const conflict = [
@@ -574,6 +689,11 @@ class UsageService {
await this.db('product_usage_state')
.where({ id: 1 })
.update({ last_error: code });
// Paced, not abandoned: the packet stays pending and the operator can
// still force a retry from the settings page. Only the automatic sender
// waits, which is what stops one permanently rejected packet from
// producing one collector request per admin click.
await this.noteDeliveryFailure();
if (conflict && packet.action !== 'delete') {
await this.db('product_usage_state')
.where({ id: 1 })
@@ -584,9 +704,15 @@ class UsageService {
}
}
async tick() {
// `force` is what the Retry button and /disable pass: an operator asking for
// an attempt now must not be held behind a backoff they can see and want to
// skip. The unattended callers — /activity and the settings ticker — leave
// it off, so a failing packet costs one request per backoff window instead
// of one per admin action.
async tick({ force = false } = {}) {
await this.locked(async (state) => {
if (state.status === 'disabled') return;
if (!force && Number(state.next_attempt_at || 0) > this.now()) return;
if (state.status === 'deletion_pending') {
const packet = makePacket(state, 'delete', Number(state.sequence), {}, this.schemaVersion(state));
state.pending_packet = JSON.stringify(packet);
@@ -666,7 +792,13 @@ class UsageService {
});
}
async snapshot(version) {
// `persist` is false for the settings preview. snapshot() records applied
// custom CSS as a lifetime marker, which meant the "see exactly what would
// be sent" view changed what gets sent — a read with a write behind it, in
// the one place whose whole job is transparency. The reported value is
// unaffected: the marker is derived here either way, and the next real
// report persists it.
async snapshot(version, { persist = true } = {}) {
version = version || this.schemaVersion(await this.state());
const rows = await this.db('app_settings')
.whereIn('setting_key', SETTING_KEYS)
@@ -777,7 +909,7 @@ class UsageService {
// Applied CSS is already a capability in use; no visitor observation is
// needed. Remember its presence as a coarse lifetime marker after consent.
if (features.custom_css.configured) {
await this.markUsed(['custom_css']);
if (persist) await this.markUsed(['custom_css']);
features.custom_css.used = true;
}
const now = new Date(this.now()).toISOString();
@@ -797,7 +929,7 @@ class UsageService {
const state = await this.state();
if (state.status !== 'active')
throw new ConflictError('Usage participation is not active');
return this.snapshot();
return this.snapshot(null, { persist: false });
}
async command(action, payload) {
let receipt;
@@ -894,10 +1026,23 @@ class UsageService {
kind: 'export',
receipt_id: crypto.randomUUID(),
confirmed_at: new Date(this.now()).toISOString(),
// Reports only. Counting every packet — feedback, votes, portal
// sessions, the registration — and labelling the total "usage
// reports" made a privacy receipt state something untrue about
// its own contents, which is exactly the document that has to be
// exact. `packet_count` keeps the total available alongside it.
report_count: Array.isArray(result.packets)
? result.packets.filter(
(envelope) => envelope?.packet?.action === 'report'
).length
: 0,
packet_count: Array.isArray(result.packets)
? result.packets.length
: 0,
scope: ['unique accepted usage reports']
scope: [
'accepted usage reports',
'accepted participant operations'
]
}
})
});
+16 -4
View File
@@ -91,12 +91,24 @@ async function expandSnapshot(db, { features, flags, used, now }) {
result.gallery_expiration.configured = await exists('events', ['expires_at'], (query) => query.whereNotNull('expires_at'));
result.download_resolution_picker.configured = truth(settings.download_resolution_picker_enabled) ||
await enabled('events', 'download_resolution_picker_enabled');
result.gallery_image_protection.configured = ['standard', 'enhanced', 'maximum'].includes(settings.default_protection_level) ||
truth(settings.enable_devtools_protection) || truth(settings.enable_canvas_rendering);
for (const column of ['disable_right_click', 'enable_devtools_protection', 'use_canvas_rendering'])
// Only what an operator actually changed. PicPeak ships
// default_protection_level='standard' and enable_devtools_protection=true —
// globally and on every event row — so accepting either as evidence made
// this signal `true` on a bare install with no galleries at all. It reported
// fleet-wide 100% and could never separate a deliberate configuration from
// an untouched one, which is a field that costs consent budget and explains
// nothing. `enable_devtools_protection` is therefore not read at all: being
// on by default, its only informative state is off, which is the opposite
// of what this key claims. The remaining inputs each ship off ('standard'
// protection, no canvas rendering, right-click allowed), so a true here is
// always a decision someone made.
result.gallery_image_protection.configured =
['enhanced', 'maximum'].includes(settings.default_protection_level) ||
truth(settings.enable_canvas_rendering);
for (const column of ['disable_right_click', 'use_canvas_rendering'])
result.gallery_image_protection.configured ||= await enabled('events', column);
result.gallery_image_protection.configured ||= await exists('events', ['protection_level'], (query) =>
query.whereIn('protection_level', ['standard', 'enhanced', 'maximum']));
query.whereIn('protection_level', ['enhanced', 'maximum']));
for (const [suffix, column] of Object.entries({
likes: 'allow_likes', ratings: 'allow_ratings', comments: 'allow_comments',
favorites: 'allow_favorites', reactions: 'allow_reactions', color_labels: 'allow_color_labels'
+2 -2
View File
@@ -1252,8 +1252,8 @@
"de": "Bildschutz aktiviert"
},
"configured": {
"en": "Enabled in applicable gallery/global configuration; only existence across the installation, never gallery IDs or counts.",
"de": "In der betreffenden Galerie-/Globalkonfiguration aktiviert; nur installationsweite Existenz, niemals Galeriekennungen oder Anzahlen."
"en": "Enabled beyond the shipped defaults — a stronger protection level, canvas rendering, or right-click disabled — globally or on at least one gallery; only existence across the installation, never gallery IDs or counts.",
"de": "Über die Auslieferungsvorgaben hinaus aktiviert — höhere Schutzstufe, Canvas-Rendering oder deaktivierter Rechtsklick — global oder in mindestens einer Galerie; nur installationsweite Existenz, niemals Galeriekennungen oder Anzahlen."
},
"used": null
},