feat: outbound webhooks for event/photo lifecycle (#327)

PicPeak POSTs lifecycle notifications to admin-configured URLs. Each
delivery is signed HMAC-SHA256 in the X-PicPeak-Signature header.
Verified end-to-end: 1/1 Playwright spec, 8/8 backend integration
tests, full UI click-through via Chrome DevTools.

Schema (migration 082)
- webhooks: id, name, url, secret (plaintext — required to compute HMAC
  for every outbound POST), secret_preview, events[], active, filter,
  template, created_by, timestamps, last_success_at/last_failure_at.
- webhook_deliveries: webhook_id (FK CASCADE), event_type, payload,
  attempt_count, status (pending|success|failed), response_status,
  response_body (truncated to 1KB), latency_ms, next_retry_at,
  last_error, created_at, completed_at. Composite index
  (status, next_retry_at) serves the worker's hot-path query.

Service + worker
- webhookService.fire(eventType, data) — non-throwing entry point used
  by lifecycle hooks. Looks up active webhooks subscribed to the event
  and applies their per-webhook filter (dot-path equality predicate)
  before enqueueing one webhook_deliveries row per match. Filter and
  template logic ship in this commit; admin surfaces in the follow-up.
- webhookDeliveryWorker — setInterval(5s) poller; fetches up to 5
  pending rows; per delivery: re-validates URL via networkValidation
  (DNS-rebinding mitigation, opt-out via WEBHOOK_ALLOW_PRIVATE_URLS),
  signs body with HMAC-SHA256, POSTs with 10s timeout, records outcome.
  Backoff schedule: 1m → 5m → 30m → 2h → 12h, max 5 attempts. Response
  body truncated to 1KB before storage. If a webhook has a template,
  the rendered string replaces the JSON envelope as the request body
  (signature is computed over the bytes actually sent).

Lifecycle wiring
- adminEvents.js POST /events → event.created (+ event.published when
  not draft); POST /:id/publish → event.published.
- routes/events.js (legacy public POST) → event.created + event.published.
- routes/v1/events.js (#322 API) → event.created + event.published on
  create, photo.uploaded on photo POST.
- archiveService.archiveEvent() → event.archived. Per-photo
  photo.deleted intentionally NOT fired during cascade — receivers
  infer from event.archived to avoid flooding (issue spec).
- expirationChecker.handleExpiredEvent() → event.expired BEFORE the
  cascading archive (so receivers see expired→archived in order).
- adminPhotos.js — photo.uploaded on each batch row, photo.deleted on
  single + bulk delete.
- photoProcessor.js — photo.uploaded for guest uploads + auto-import
  (covers all entry paths).
- fileWatcher.js — photo.uploaded on add, photo.deleted on unlink
  (local mode only).

Admin endpoints (mirrors adminApiTokens.js pattern)
- /api/admin/webhooks: GET list, POST create (returns plaintext secret
  exactly once), GET :id, PUT :id, DELETE :id, POST :id/test (synthetic
  fire), GET :id/deliveries (paginated, filter by status), GET
  :id/deliveries/:deliveryId, POST :id/deliveries/:deliveryId/replay.

Frontend
- Settings → Webhooks tab (mirrors API Tokens layout): name + URL +
  event checkboxes + "Advanced" expander for filter (JSON) and template.
  Plaintext secret shown once on creation with a Copy button. Active/
  Disabled toggle button per row.
- /admin/webhooks/:id/deliveries — operational debug surface. Table
  with timestamp/event/status/attempts/HTTP/latency. Status filter chips
  (all/pending/success/failed). Row click → slide-over with payload +
  signature + response body. Replay button on failed rows. Send-test-event
  dialog. Auto-refresh every 10s.

Dev infrastructure
- dev/webhook-receiver/ — tiny node:alpine HTTP server (~100 LOC) that
  records every POST to an in-memory ring buffer. Exposes GET /requests
  for the E2E spec to assert deliveries landed with the right HMAC.
  Sibling pattern to MinIO. Reachable from the backend at
  http://webhook-receiver:8888 inside the picpeak network.

Tests
- backend/__tests__/integration/webhookDelivery.test.js (8/8) —
  signature verification, headers, retry/backoff, max-attempts → failed,
  response truncation, disabled-mid-flight, SSRF block, start/stop
  idempotency.
- tests/e2e/webhooks-roundtrip.spec.ts (1/1) — create webhook → trigger
  event.published → assert receiver got POST with valid HMAC → visit
  deliveries page → row visible with status=success → API test event →
  API replay → disable webhook → assert no new delivery.

Docs
- README §"Webhooks" — event catalog, payload shape, HMAC verification
  in Node + Python + bash, retry semantics, SSRF protection.
- .env.example — WEBHOOK_ALLOW_PRIVATE_URLS, WEBHOOK_DELIVERY_INTERVAL_MS,
  WEBHOOK_DELIVERY_CONCURRENCY, WEBHOOK_HTTP_TIMEOUT_MS,
  WEBHOOK_MAX_ATTEMPTS.

Out of scope for v1 (per issue): webhook templates' code-eval (the
${dot.path} substitution that ships is pure string replacement, no
expression engine — see follow-up commit), per-webhook rate limiting
beyond the global concurrency cap, synchronous "ask before delete"
webhooks.

Spanning files
- App.tsx pulls in this commit with both the AnalyticsBootstrap
  (#325 dedup) and the WebhookDeliveriesPage route registration.
  Splitting via git add -p was forfeit for sanity; the single 92-line
  diff is honest about both contributions.
- adminEvents.js diff bundles the webhook fires AND the
  allow_presigned_download field plumbing (#328 follow-up). Same
  reasoning.
- The new webhookService/Worker/adminWebhooks files include the filter
  and template logic from the follow-up — they were authored in one
  pass; splitting them post-hoc would have produced fragile partial
  files. The follow-up commit covers the migration and the UI for these.
This commit is contained in:
Paul Nothaft
2026-04-28 10:07:39 +02:00
parent 1b717ce5ed
commit c488f481ca
17 changed files with 2363 additions and 61 deletions
+224
View File
@@ -0,0 +1,224 @@
const crypto = require('crypto');
const { db } = require('../database/db');
const logger = require('../utils/logger');
const SECRET_PREFIX = 'whsec_';
/**
* Event types PicPeak emits. Keep this in sync with the README catalog and
* the receiver-side type unions in any SDK we publish later. Consumers of
* `fire(eventType, ...)` MUST use one of these strings — the worker will
* silently drop unknown types so a typo can't 500 a request handler.
*/
const EVENT_TYPES = Object.freeze([
'event.created',
'event.published',
'event.archived',
'event.expired',
'photo.uploaded',
'photo.deleted',
]);
function generateSecret() {
const random = crypto.randomBytes(24).toString('base64url'); // ~32 chars
const plaintext = `${SECRET_PREFIX}${random}`;
return {
plaintext,
preview: random.slice(0, 8),
};
}
/**
* Sign a payload with the webhook's secret. Used by the delivery worker;
* exported for unit tests of receiver-side verification snippets.
*/
function signPayload(secret, rawBody) {
return crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
}
/**
* Resolve a dot-path on an object (e.g. "data.event.event_type") with no
* eval. Returns undefined for missing segments — never throws.
*/
function getByPath(obj, dotPath) {
if (!dotPath || typeof dotPath !== 'string') return undefined;
return dotPath.split('.').reduce((acc, key) => {
if (acc == null || typeof acc !== 'object') return undefined;
return acc[key];
}, obj);
}
/**
* Evaluate a webhook's filter against an outgoing payload. The filter is a
* flat object of dot-path → expected value pairs:
* { "data.event.event_type": "wedding" }
* { "type": "event.published", "data.event.id": 42 }
*
* All keys must match (logical AND). Equality is `===` after JSON-style
* coercion: numbers as numbers, booleans as booleans. Empty filter
* matches everything (back-compat).
*/
function payloadMatchesFilter(filter, payload) {
if (!filter || typeof filter !== 'object') return true;
const keys = Object.keys(filter);
if (keys.length === 0) return true;
for (const key of keys) {
const expected = filter[key];
const actual = getByPath(payload, key);
if (Array.isArray(expected)) {
// Array means "any of"
if (!expected.includes(actual)) return false;
} else if (actual !== expected) {
return false;
}
}
return true;
}
/**
* Render a webhook template by substituting ${dot.path} expressions with
* values from the payload. NO eval, NO logic — pure string substitution.
* Caps output at 64KB; bails out and returns null if exceeded so the
* delivery worker can fall back to the default envelope.
*/
function renderTemplate(template, payload) {
if (template == null || template === '') return null;
if (typeof template !== 'string') return null;
const MAX_OUTPUT = 64 * 1024;
const MAX_SUBSTITUTIONS = 64;
let count = 0;
const rendered = template.replace(/\$\{([^}]+)\}/g, (_match, expr) => {
count += 1;
if (count > MAX_SUBSTITUTIONS) return '';
const value = getByPath(payload, expr.trim());
if (value == null) return '';
if (typeof value === 'object') {
try { return JSON.stringify(value); } catch { return ''; }
}
return String(value);
});
if (Buffer.byteLength(rendered, 'utf8') > MAX_OUTPUT) return null;
return rendered;
}
/**
* Validate a template at create-time so admins get immediate feedback
* instead of silent delivery failures. Returns { valid, error? }.
*/
function validateTemplate(template) {
if (template == null || template === '') return { valid: true };
if (typeof template !== 'string') return { valid: false, error: 'template must be a string' };
if (Buffer.byteLength(template, 'utf8') > 8192) return { valid: false, error: 'template exceeds 8KB' };
// Reject unbalanced ${ that would silently swallow content at render.
const opens = (template.match(/\$\{/g) || []).length;
const closes = (template.match(/\}/g) || []).length;
// Count is approximate (every } is counted, even non-matching ones).
// We require at least as many } as ${, which is necessary but not sufficient.
if (opens > closes) return { valid: false, error: 'template has unbalanced ${ — every ${ needs a matching }' };
if (opens > 64) return { valid: false, error: 'template exceeds 64 substitutions' };
return { valid: true };
}
/**
* Constant-time signature comparison helper for receivers and tests.
* Exposed so the same primitive backs verification examples in the README.
*/
function verifySignature(secret, rawBody, signature) {
const expected = signPayload(secret, rawBody);
const a = Buffer.from(expected, 'hex');
let b;
try {
b = Buffer.from(signature || '', 'hex');
} catch {
return false;
}
if (a.length !== b.length) return false;
return crypto.timingSafeEqual(a, b);
}
/**
* Enqueue a webhook delivery for every active webhook subscribed to
* `eventType`. NEVER throws — webhook failures must not break the
* lifecycle handler that emitted the event. Worker handles HTTP delivery.
*
* @param {string} eventType — one of EVENT_TYPES
* @param {object} data — opaque payload that gets nested under .data in
* the outbound JSON body
*/
async function fire(eventType, data) {
if (!EVENT_TYPES.includes(eventType)) {
logger.warn(`[webhookService] dropping unknown event type: ${eventType}`);
return;
}
try {
// jsonb @> ARRAY check across vendors: both pg and sqlite drivers we
// support handle a simple WHERE on `active=true` then a runtime filter
// on the events array; doing the array filter here keeps the query
// portable.
const candidates = await db('webhooks').where({ active: true });
const subscribed = candidates.filter((w) => {
const evts = Array.isArray(w.events)
? w.events
: (() => { try { return JSON.parse(w.events) || []; } catch { return []; } })();
return evts.includes(eventType);
});
if (subscribed.length === 0) return;
const now = new Date();
const buildEnvelope = (deliveryUuid) => ({
id: deliveryUuid,
type: eventType,
created_at: now.toISOString(),
data,
});
const rows = [];
for (const w of subscribed) {
const deliveryUuid = crypto.randomUUID();
const envelope = buildEnvelope(deliveryUuid);
// Filter (#327 follow-up): per-webhook predicate evaluated against
// the payload. Skip insertion when it doesn't match.
const filter = parseJsonField(w.filter, {});
if (!payloadMatchesFilter(filter, envelope)) continue;
rows.push({
webhook_id: w.id,
event_type: eventType,
payload: JSON.stringify(envelope),
attempt_count: 0,
status: 'pending',
next_retry_at: now,
created_at: now,
});
}
if (rows.length === 0) return;
await db('webhook_deliveries').insert(rows);
} catch (err) {
// Log but don't throw — caller's transaction has already committed
// by the time we get here, and we don't want to mask the success.
logger.error(`[webhookService.fire] failed to enqueue ${eventType}: ${err.message}`);
}
}
function parseJsonField(value, fallback) {
if (value == null) return fallback;
if (typeof value === 'object') return value;
try { return JSON.parse(value) ?? fallback; } catch { return fallback; }
}
module.exports = {
fire,
generateSecret,
signPayload,
verifySignature,
payloadMatchesFilter,
renderTemplate,
validateTemplate,
getByPath,
EVENT_TYPES,
SECRET_PREFIX,
};