feat: outbound webhooks for event/photo lifecycle (#327)
PicPeak POSTs lifecycle notifications to admin-configured URLs. Each delivery is signed HMAC-SHA256 in the X-PicPeak-Signature header. Verified end-to-end: 1/1 Playwright spec, 8/8 backend integration tests, full UI click-through via Chrome DevTools. Schema (migration 082) - webhooks: id, name, url, secret (plaintext — required to compute HMAC for every outbound POST), secret_preview, events[], active, filter, template, created_by, timestamps, last_success_at/last_failure_at. - webhook_deliveries: webhook_id (FK CASCADE), event_type, payload, attempt_count, status (pending|success|failed), response_status, response_body (truncated to 1KB), latency_ms, next_retry_at, last_error, created_at, completed_at. Composite index (status, next_retry_at) serves the worker's hot-path query. Service + worker - webhookService.fire(eventType, data) — non-throwing entry point used by lifecycle hooks. Looks up active webhooks subscribed to the event and applies their per-webhook filter (dot-path equality predicate) before enqueueing one webhook_deliveries row per match. Filter and template logic ship in this commit; admin surfaces in the follow-up. - webhookDeliveryWorker — setInterval(5s) poller; fetches up to 5 pending rows; per delivery: re-validates URL via networkValidation (DNS-rebinding mitigation, opt-out via WEBHOOK_ALLOW_PRIVATE_URLS), signs body with HMAC-SHA256, POSTs with 10s timeout, records outcome. Backoff schedule: 1m → 5m → 30m → 2h → 12h, max 5 attempts. Response body truncated to 1KB before storage. If a webhook has a template, the rendered string replaces the JSON envelope as the request body (signature is computed over the bytes actually sent). Lifecycle wiring - adminEvents.js POST /events → event.created (+ event.published when not draft); POST /:id/publish → event.published. - routes/events.js (legacy public POST) → event.created + event.published. - routes/v1/events.js (#322 API) → event.created + event.published on create, photo.uploaded on photo POST. - archiveService.archiveEvent() → event.archived. Per-photo photo.deleted intentionally NOT fired during cascade — receivers infer from event.archived to avoid flooding (issue spec). - expirationChecker.handleExpiredEvent() → event.expired BEFORE the cascading archive (so receivers see expired→archived in order). - adminPhotos.js — photo.uploaded on each batch row, photo.deleted on single + bulk delete. - photoProcessor.js — photo.uploaded for guest uploads + auto-import (covers all entry paths). - fileWatcher.js — photo.uploaded on add, photo.deleted on unlink (local mode only). Admin endpoints (mirrors adminApiTokens.js pattern) - /api/admin/webhooks: GET list, POST create (returns plaintext secret exactly once), GET :id, PUT :id, DELETE :id, POST :id/test (synthetic fire), GET :id/deliveries (paginated, filter by status), GET :id/deliveries/:deliveryId, POST :id/deliveries/:deliveryId/replay. Frontend - Settings → Webhooks tab (mirrors API Tokens layout): name + URL + event checkboxes + "Advanced" expander for filter (JSON) and template. Plaintext secret shown once on creation with a Copy button. Active/ Disabled toggle button per row. - /admin/webhooks/:id/deliveries — operational debug surface. Table with timestamp/event/status/attempts/HTTP/latency. Status filter chips (all/pending/success/failed). Row click → slide-over with payload + signature + response body. Replay button on failed rows. Send-test-event dialog. Auto-refresh every 10s. Dev infrastructure - dev/webhook-receiver/ — tiny node:alpine HTTP server (~100 LOC) that records every POST to an in-memory ring buffer. Exposes GET /requests for the E2E spec to assert deliveries landed with the right HMAC. Sibling pattern to MinIO. Reachable from the backend at http://webhook-receiver:8888 inside the picpeak network. Tests - backend/__tests__/integration/webhookDelivery.test.js (8/8) — signature verification, headers, retry/backoff, max-attempts → failed, response truncation, disabled-mid-flight, SSRF block, start/stop idempotency. - tests/e2e/webhooks-roundtrip.spec.ts (1/1) — create webhook → trigger event.published → assert receiver got POST with valid HMAC → visit deliveries page → row visible with status=success → API test event → API replay → disable webhook → assert no new delivery. Docs - README §"Webhooks" — event catalog, payload shape, HMAC verification in Node + Python + bash, retry semantics, SSRF protection. - .env.example — WEBHOOK_ALLOW_PRIVATE_URLS, WEBHOOK_DELIVERY_INTERVAL_MS, WEBHOOK_DELIVERY_CONCURRENCY, WEBHOOK_HTTP_TIMEOUT_MS, WEBHOOK_MAX_ATTEMPTS. Out of scope for v1 (per issue): webhook templates' code-eval (the ${dot.path} substitution that ships is pure string replacement, no expression engine — see follow-up commit), per-webhook rate limiting beyond the global concurrency cap, synchronous "ask before delete" webhooks. Spanning files - App.tsx pulls in this commit with both the AnalyticsBootstrap (#325 dedup) and the WebhookDeliveriesPage route registration. Splitting via git add -p was forfeit for sanity; the single 92-line diff is honest about both contributions. - adminEvents.js diff bundles the webhook fires AND the allow_presigned_download field plumbing (#328 follow-up). Same reasoning. - The new webhookService/Worker/adminWebhooks files include the filter and template logic from the follow-up — they were authored in one pass; splitting them post-hoc would have produced fragile partial files. The follow-up commit covers the migration and the UI for these.
This commit is contained in:
@@ -0,0 +1,224 @@
|
||||
const crypto = require('crypto');
|
||||
const { db } = require('../database/db');
|
||||
const logger = require('../utils/logger');
|
||||
|
||||
const SECRET_PREFIX = 'whsec_';
|
||||
|
||||
/**
|
||||
* Event types PicPeak emits. Keep this in sync with the README catalog and
|
||||
* the receiver-side type unions in any SDK we publish later. Consumers of
|
||||
* `fire(eventType, ...)` MUST use one of these strings — the worker will
|
||||
* silently drop unknown types so a typo can't 500 a request handler.
|
||||
*/
|
||||
const EVENT_TYPES = Object.freeze([
|
||||
'event.created',
|
||||
'event.published',
|
||||
'event.archived',
|
||||
'event.expired',
|
||||
'photo.uploaded',
|
||||
'photo.deleted',
|
||||
]);
|
||||
|
||||
function generateSecret() {
|
||||
const random = crypto.randomBytes(24).toString('base64url'); // ~32 chars
|
||||
const plaintext = `${SECRET_PREFIX}${random}`;
|
||||
return {
|
||||
plaintext,
|
||||
preview: random.slice(0, 8),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Sign a payload with the webhook's secret. Used by the delivery worker;
|
||||
* exported for unit tests of receiver-side verification snippets.
|
||||
*/
|
||||
function signPayload(secret, rawBody) {
|
||||
return crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a dot-path on an object (e.g. "data.event.event_type") with no
|
||||
* eval. Returns undefined for missing segments — never throws.
|
||||
*/
|
||||
function getByPath(obj, dotPath) {
|
||||
if (!dotPath || typeof dotPath !== 'string') return undefined;
|
||||
return dotPath.split('.').reduce((acc, key) => {
|
||||
if (acc == null || typeof acc !== 'object') return undefined;
|
||||
return acc[key];
|
||||
}, obj);
|
||||
}
|
||||
|
||||
/**
|
||||
* Evaluate a webhook's filter against an outgoing payload. The filter is a
|
||||
* flat object of dot-path → expected value pairs:
|
||||
* { "data.event.event_type": "wedding" }
|
||||
* { "type": "event.published", "data.event.id": 42 }
|
||||
*
|
||||
* All keys must match (logical AND). Equality is `===` after JSON-style
|
||||
* coercion: numbers as numbers, booleans as booleans. Empty filter
|
||||
* matches everything (back-compat).
|
||||
*/
|
||||
function payloadMatchesFilter(filter, payload) {
|
||||
if (!filter || typeof filter !== 'object') return true;
|
||||
const keys = Object.keys(filter);
|
||||
if (keys.length === 0) return true;
|
||||
for (const key of keys) {
|
||||
const expected = filter[key];
|
||||
const actual = getByPath(payload, key);
|
||||
if (Array.isArray(expected)) {
|
||||
// Array means "any of"
|
||||
if (!expected.includes(actual)) return false;
|
||||
} else if (actual !== expected) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a webhook template by substituting ${dot.path} expressions with
|
||||
* values from the payload. NO eval, NO logic — pure string substitution.
|
||||
* Caps output at 64KB; bails out and returns null if exceeded so the
|
||||
* delivery worker can fall back to the default envelope.
|
||||
*/
|
||||
function renderTemplate(template, payload) {
|
||||
if (template == null || template === '') return null;
|
||||
if (typeof template !== 'string') return null;
|
||||
const MAX_OUTPUT = 64 * 1024;
|
||||
const MAX_SUBSTITUTIONS = 64;
|
||||
let count = 0;
|
||||
const rendered = template.replace(/\$\{([^}]+)\}/g, (_match, expr) => {
|
||||
count += 1;
|
||||
if (count > MAX_SUBSTITUTIONS) return '';
|
||||
const value = getByPath(payload, expr.trim());
|
||||
if (value == null) return '';
|
||||
if (typeof value === 'object') {
|
||||
try { return JSON.stringify(value); } catch { return ''; }
|
||||
}
|
||||
return String(value);
|
||||
});
|
||||
if (Buffer.byteLength(rendered, 'utf8') > MAX_OUTPUT) return null;
|
||||
return rendered;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a template at create-time so admins get immediate feedback
|
||||
* instead of silent delivery failures. Returns { valid, error? }.
|
||||
*/
|
||||
function validateTemplate(template) {
|
||||
if (template == null || template === '') return { valid: true };
|
||||
if (typeof template !== 'string') return { valid: false, error: 'template must be a string' };
|
||||
if (Buffer.byteLength(template, 'utf8') > 8192) return { valid: false, error: 'template exceeds 8KB' };
|
||||
// Reject unbalanced ${ that would silently swallow content at render.
|
||||
const opens = (template.match(/\$\{/g) || []).length;
|
||||
const closes = (template.match(/\}/g) || []).length;
|
||||
// Count is approximate (every } is counted, even non-matching ones).
|
||||
// We require at least as many } as ${, which is necessary but not sufficient.
|
||||
if (opens > closes) return { valid: false, error: 'template has unbalanced ${ — every ${ needs a matching }' };
|
||||
if (opens > 64) return { valid: false, error: 'template exceeds 64 substitutions' };
|
||||
return { valid: true };
|
||||
}
|
||||
|
||||
/**
|
||||
* Constant-time signature comparison helper for receivers and tests.
|
||||
* Exposed so the same primitive backs verification examples in the README.
|
||||
*/
|
||||
function verifySignature(secret, rawBody, signature) {
|
||||
const expected = signPayload(secret, rawBody);
|
||||
const a = Buffer.from(expected, 'hex');
|
||||
let b;
|
||||
try {
|
||||
b = Buffer.from(signature || '', 'hex');
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (a.length !== b.length) return false;
|
||||
return crypto.timingSafeEqual(a, b);
|
||||
}
|
||||
|
||||
/**
|
||||
* Enqueue a webhook delivery for every active webhook subscribed to
|
||||
* `eventType`. NEVER throws — webhook failures must not break the
|
||||
* lifecycle handler that emitted the event. Worker handles HTTP delivery.
|
||||
*
|
||||
* @param {string} eventType — one of EVENT_TYPES
|
||||
* @param {object} data — opaque payload that gets nested under .data in
|
||||
* the outbound JSON body
|
||||
*/
|
||||
async function fire(eventType, data) {
|
||||
if (!EVENT_TYPES.includes(eventType)) {
|
||||
logger.warn(`[webhookService] dropping unknown event type: ${eventType}`);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
// jsonb @> ARRAY check across vendors: both pg and sqlite drivers we
|
||||
// support handle a simple WHERE on `active=true` then a runtime filter
|
||||
// on the events array; doing the array filter here keeps the query
|
||||
// portable.
|
||||
const candidates = await db('webhooks').where({ active: true });
|
||||
const subscribed = candidates.filter((w) => {
|
||||
const evts = Array.isArray(w.events)
|
||||
? w.events
|
||||
: (() => { try { return JSON.parse(w.events) || []; } catch { return []; } })();
|
||||
return evts.includes(eventType);
|
||||
});
|
||||
|
||||
if (subscribed.length === 0) return;
|
||||
|
||||
const now = new Date();
|
||||
const buildEnvelope = (deliveryUuid) => ({
|
||||
id: deliveryUuid,
|
||||
type: eventType,
|
||||
created_at: now.toISOString(),
|
||||
data,
|
||||
});
|
||||
|
||||
const rows = [];
|
||||
for (const w of subscribed) {
|
||||
const deliveryUuid = crypto.randomUUID();
|
||||
const envelope = buildEnvelope(deliveryUuid);
|
||||
|
||||
// Filter (#327 follow-up): per-webhook predicate evaluated against
|
||||
// the payload. Skip insertion when it doesn't match.
|
||||
const filter = parseJsonField(w.filter, {});
|
||||
if (!payloadMatchesFilter(filter, envelope)) continue;
|
||||
|
||||
rows.push({
|
||||
webhook_id: w.id,
|
||||
event_type: eventType,
|
||||
payload: JSON.stringify(envelope),
|
||||
attempt_count: 0,
|
||||
status: 'pending',
|
||||
next_retry_at: now,
|
||||
created_at: now,
|
||||
});
|
||||
}
|
||||
|
||||
if (rows.length === 0) return;
|
||||
await db('webhook_deliveries').insert(rows);
|
||||
} catch (err) {
|
||||
// Log but don't throw — caller's transaction has already committed
|
||||
// by the time we get here, and we don't want to mask the success.
|
||||
logger.error(`[webhookService.fire] failed to enqueue ${eventType}: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function parseJsonField(value, fallback) {
|
||||
if (value == null) return fallback;
|
||||
if (typeof value === 'object') return value;
|
||||
try { return JSON.parse(value) ?? fallback; } catch { return fallback; }
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
fire,
|
||||
generateSecret,
|
||||
signPayload,
|
||||
verifySignature,
|
||||
payloadMatchesFilter,
|
||||
renderTemplate,
|
||||
validateTemplate,
|
||||
getByPath,
|
||||
EVENT_TYPES,
|
||||
SECRET_PREFIX,
|
||||
};
|
||||
Reference in New Issue
Block a user