Files
picpeak/backend/src/routes/adminWebhooks.js
T
Paul Nothaft c488f481ca feat: outbound webhooks for event/photo lifecycle (#327)
PicPeak POSTs lifecycle notifications to admin-configured URLs. Each
delivery is signed HMAC-SHA256 in the X-PicPeak-Signature header.
Verified end-to-end: 1/1 Playwright spec, 8/8 backend integration
tests, full UI click-through via Chrome DevTools.

Schema (migration 082)
- webhooks: id, name, url, secret (plaintext — required to compute HMAC
  for every outbound POST), secret_preview, events[], active, filter,
  template, created_by, timestamps, last_success_at/last_failure_at.
- webhook_deliveries: webhook_id (FK CASCADE), event_type, payload,
  attempt_count, status (pending|success|failed), response_status,
  response_body (truncated to 1KB), latency_ms, next_retry_at,
  last_error, created_at, completed_at. Composite index
  (status, next_retry_at) serves the worker's hot-path query.

Service + worker
- webhookService.fire(eventType, data) — non-throwing entry point used
  by lifecycle hooks. Looks up active webhooks subscribed to the event
  and applies their per-webhook filter (dot-path equality predicate)
  before enqueueing one webhook_deliveries row per match. Filter and
  template logic ship in this commit; admin surfaces in the follow-up.
- webhookDeliveryWorker — setInterval(5s) poller; fetches up to 5
  pending rows; per delivery: re-validates URL via networkValidation
  (DNS-rebinding mitigation, opt-out via WEBHOOK_ALLOW_PRIVATE_URLS),
  signs body with HMAC-SHA256, POSTs with 10s timeout, records outcome.
  Backoff schedule: 1m → 5m → 30m → 2h → 12h, max 5 attempts. Response
  body truncated to 1KB before storage. If a webhook has a template,
  the rendered string replaces the JSON envelope as the request body
  (signature is computed over the bytes actually sent).

Lifecycle wiring
- adminEvents.js POST /events → event.created (+ event.published when
  not draft); POST /:id/publish → event.published.
- routes/events.js (legacy public POST) → event.created + event.published.
- routes/v1/events.js (#322 API) → event.created + event.published on
  create, photo.uploaded on photo POST.
- archiveService.archiveEvent() → event.archived. Per-photo
  photo.deleted intentionally NOT fired during cascade — receivers
  infer from event.archived to avoid flooding (issue spec).
- expirationChecker.handleExpiredEvent() → event.expired BEFORE the
  cascading archive (so receivers see expired→archived in order).
- adminPhotos.js — photo.uploaded on each batch row, photo.deleted on
  single + bulk delete.
- photoProcessor.js — photo.uploaded for guest uploads + auto-import
  (covers all entry paths).
- fileWatcher.js — photo.uploaded on add, photo.deleted on unlink
  (local mode only).

Admin endpoints (mirrors adminApiTokens.js pattern)
- /api/admin/webhooks: GET list, POST create (returns plaintext secret
  exactly once), GET :id, PUT :id, DELETE :id, POST :id/test (synthetic
  fire), GET :id/deliveries (paginated, filter by status), GET
  :id/deliveries/:deliveryId, POST :id/deliveries/:deliveryId/replay.

Frontend
- Settings → Webhooks tab (mirrors API Tokens layout): name + URL +
  event checkboxes + "Advanced" expander for filter (JSON) and template.
  Plaintext secret shown once on creation with a Copy button. Active/
  Disabled toggle button per row.
- /admin/webhooks/:id/deliveries — operational debug surface. Table
  with timestamp/event/status/attempts/HTTP/latency. Status filter chips
  (all/pending/success/failed). Row click → slide-over with payload +
  signature + response body. Replay button on failed rows. Send-test-event
  dialog. Auto-refresh every 10s.

Dev infrastructure
- dev/webhook-receiver/ — tiny node:alpine HTTP server (~100 LOC) that
  records every POST to an in-memory ring buffer. Exposes GET /requests
  for the E2E spec to assert deliveries landed with the right HMAC.
  Sibling pattern to MinIO. Reachable from the backend at
  http://webhook-receiver:8888 inside the picpeak network.

Tests
- backend/__tests__/integration/webhookDelivery.test.js (8/8) —
  signature verification, headers, retry/backoff, max-attempts → failed,
  response truncation, disabled-mid-flight, SSRF block, start/stop
  idempotency.
- tests/e2e/webhooks-roundtrip.spec.ts (1/1) — create webhook → trigger
  event.published → assert receiver got POST with valid HMAC → visit
  deliveries page → row visible with status=success → API test event →
  API replay → disable webhook → assert no new delivery.

Docs
- README §"Webhooks" — event catalog, payload shape, HMAC verification
  in Node + Python + bash, retry semantics, SSRF protection.
- .env.example — WEBHOOK_ALLOW_PRIVATE_URLS, WEBHOOK_DELIVERY_INTERVAL_MS,
  WEBHOOK_DELIVERY_CONCURRENCY, WEBHOOK_HTTP_TIMEOUT_MS,
  WEBHOOK_MAX_ATTEMPTS.

Out of scope for v1 (per issue): webhook templates' code-eval (the
${dot.path} substitution that ships is pure string replacement, no
expression engine — see follow-up commit), per-webhook rate limiting
beyond the global concurrency cap, synchronous "ask before delete"
webhooks.

Spanning files
- App.tsx pulls in this commit with both the AnalyticsBootstrap
  (#325 dedup) and the WebhookDeliveriesPage route registration.
  Splitting via git add -p was forfeit for sanity; the single 92-line
  diff is honest about both contributions.
- adminEvents.js diff bundles the webhook fires AND the
  allow_presigned_download field plumbing (#328 follow-up). Same
  reasoning.
- The new webhookService/Worker/adminWebhooks files include the filter
  and template logic from the follow-up — they were authored in one
  pass; splitting them post-hoc would have produced fragile partial
  files. The follow-up commit covers the migration and the UI for these.
2026-04-28 10:07:39 +02:00

390 lines
16 KiB
JavaScript

/**
* Admin endpoints for managing outbound webhooks (#327). Mirrors
* adminApiTokens.js — same permission gates, same "secret shown once"
* pattern.
*
* Routes mounted under /api/admin/webhooks:
* GET / — list
* POST / — create (returns plaintext secret once)
* GET /:id — detail (no secret)
* PUT /:id — update name/url/events/active
* DELETE /:id — delete (cascades to deliveries)
* POST /:id/test — fire a synthetic delivery now
* GET /:id/deliveries — list deliveries (paginated, filter)
* GET /:id/deliveries/:deliveryId — delivery detail (payload+response)
* POST /:id/deliveries/:deliveryId/replay — re-enqueue a delivery
*/
const express = require('express');
const { body, query, validationResult } = require('express-validator');
const { db, logActivity } = require('../database/db');
const { adminAuth } = require('../middleware/auth');
const { requirePermission } = require('../middleware/permissions');
const { validateExternalUrl } = require('../utils/networkValidation');
const webhookService = require('../services/webhookService');
const logger = require('../utils/logger');
const router = express.Router();
const ALLOW_PRIVATE_URLS = process.env.WEBHOOK_ALLOW_PRIVATE_URLS === 'true';
function publicWebhook(row) {
if (!row) return null;
return {
id: row.id,
name: row.name,
url: row.url,
events: typeof row.events === 'string' ? safeJson(row.events, []) : (row.events || []),
active: row.active,
secret_preview: row.secret_preview,
filter: typeof row.filter === 'string' ? safeJson(row.filter, {}) : (row.filter || {}),
template: row.template || null,
created_by: row.created_by,
created_at: row.created_at,
updated_at: row.updated_at,
last_success_at: row.last_success_at,
last_failure_at: row.last_failure_at,
};
}
function safeJson(s, fallback) {
try { return JSON.parse(s); } catch { return fallback; }
}
// ─── List ────────────────────────────────────────────────────────────────
router.get('/', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
const rows = await db('webhooks')
.leftJoin('admin_users', 'admin_users.id', 'webhooks.created_by')
.select(
'webhooks.*',
'admin_users.username as owner_username'
)
.orderBy('webhooks.created_at', 'desc');
res.json(rows.map((r) => ({
...publicWebhook(r),
owner_username: r.owner_username,
})));
} catch (err) {
logger.error('webhooks list failed', { error: err.message });
res.status(500).json({ error: 'Failed to list webhooks' });
}
});
// ─── Create ──────────────────────────────────────────────────────────────
router.post(
'/',
adminAuth,
requirePermission('settings.edit'),
[
body('name').isString().trim().isLength({ min: 1, max: 100 }),
body('url').isString().isLength({ max: 2048 }).custom((url) => {
if (ALLOW_PRIVATE_URLS) return true;
const check = validateExternalUrl(url);
if (!check.valid) throw new Error(check.error);
return true;
}),
body('events').isArray({ min: 1 }).custom((arr) => {
const ok = arr.every((e) => webhookService.EVENT_TYPES.includes(e));
if (!ok) throw new Error(`events must be a subset of: ${webhookService.EVENT_TYPES.join(', ')}`);
return true;
}),
body('active').optional().isBoolean(),
body('filter').optional().custom((v) => {
if (v == null) return true;
if (typeof v !== 'object' || Array.isArray(v)) {
throw new Error('filter must be an object of dot-path → value pairs');
}
return true;
}),
body('template').optional({ nullable: true }).custom((v) => {
const check = webhookService.validateTemplate(v);
if (!check.valid) throw new Error(check.error);
return true;
}),
],
async (req, res) => {
try {
const errors = validationResult(req);
if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() });
const { name, url, events, active = true, filter, template } = req.body;
const { plaintext, preview } = webhookService.generateSecret();
const insertResult = await db('webhooks').insert({
name,
url,
secret: plaintext,
secret_preview: preview,
events: JSON.stringify(events),
active,
filter: JSON.stringify(filter || {}),
template: template || null,
created_by: req.admin.id,
}).returning('id');
const id = insertResult[0]?.id || insertResult[0];
await logActivity('webhook_created', { name, events }, null, {
type: 'admin', id: req.admin.id, name: req.admin.username,
});
const row = await db('webhooks').where({ id }).first();
res.status(201).json({
...publicWebhook(row),
secret: plaintext,
notice: 'Save this signing secret now — it will not be shown again.',
});
} catch (err) {
logger.error('webhooks create failed', { error: err.message });
res.status(500).json({ error: 'Failed to create webhook' });
}
}
);
// ─── Detail ──────────────────────────────────────────────────────────────
router.get('/:id', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
const row = await db('webhooks').where({ id: req.params.id }).first();
if (!row) return res.status(404).json({ error: 'Webhook not found' });
res.json(publicWebhook(row));
} catch (err) {
logger.error('webhooks detail failed', { error: err.message });
res.status(500).json({ error: 'Failed to load webhook' });
}
});
// ─── Update ──────────────────────────────────────────────────────────────
router.put(
'/:id',
adminAuth,
requirePermission('settings.edit'),
[
body('name').optional().isString().trim().isLength({ min: 1, max: 100 }),
body('url').optional().isString().isLength({ max: 2048 }).custom((url) => {
if (ALLOW_PRIVATE_URLS) return true;
const check = validateExternalUrl(url);
if (!check.valid) throw new Error(check.error);
return true;
}),
body('events').optional().isArray({ min: 1 }).custom((arr) => {
const ok = arr.every((e) => webhookService.EVENT_TYPES.includes(e));
if (!ok) throw new Error(`events must be a subset of: ${webhookService.EVENT_TYPES.join(', ')}`);
return true;
}),
body('active').optional().isBoolean(),
body('filter').optional().custom((v) => {
if (v == null) return true;
if (typeof v !== 'object' || Array.isArray(v)) {
throw new Error('filter must be an object of dot-path → value pairs');
}
return true;
}),
body('template').optional({ nullable: true }).custom((v) => {
const check = webhookService.validateTemplate(v);
if (!check.valid) throw new Error(check.error);
return true;
}),
],
async (req, res) => {
try {
const errors = validationResult(req);
if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() });
const row = await db('webhooks').where({ id: req.params.id }).first();
if (!row) return res.status(404).json({ error: 'Webhook not found' });
const updates = { updated_at: new Date() };
if ('name' in req.body) updates.name = req.body.name;
if ('url' in req.body) updates.url = req.body.url;
if ('events' in req.body) updates.events = JSON.stringify(req.body.events);
if ('active' in req.body) updates.active = req.body.active;
if ('filter' in req.body) updates.filter = JSON.stringify(req.body.filter || {});
if ('template' in req.body) updates.template = req.body.template || null;
await db('webhooks').where({ id: req.params.id }).update(updates);
const updated = await db('webhooks').where({ id: req.params.id }).first();
await logActivity('webhook_updated', { changes: Object.keys(updates) }, null, {
type: 'admin', id: req.admin.id, name: req.admin.username,
});
res.json(publicWebhook(updated));
} catch (err) {
logger.error('webhooks update failed', { error: err.message });
res.status(500).json({ error: 'Failed to update webhook' });
}
}
);
// ─── Delete ──────────────────────────────────────────────────────────────
router.delete('/:id', adminAuth, requirePermission('settings.edit'), async (req, res) => {
try {
const row = await db('webhooks').where({ id: req.params.id }).first();
if (!row) return res.status(404).json({ error: 'Webhook not found' });
await db('webhooks').where({ id: req.params.id }).delete();
await logActivity('webhook_deleted', { name: row.name }, null, {
type: 'admin', id: req.admin.id, name: req.admin.username,
});
res.json({ id: Number(req.params.id), deleted: true });
} catch (err) {
logger.error('webhooks delete failed', { error: err.message });
res.status(500).json({ error: 'Failed to delete webhook' });
}
});
// ─── Send test event ─────────────────────────────────────────────────────
router.post(
'/:id/test',
adminAuth,
requirePermission('settings.edit'),
[body('event_type').optional().isIn(webhookService.EVENT_TYPES)],
async (req, res) => {
try {
const errors = validationResult(req);
if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() });
const row = await db('webhooks').where({ id: req.params.id }).first();
if (!row) return res.status(404).json({ error: 'Webhook not found' });
if (!row.active) return res.status(400).json({ error: 'Webhook is disabled' });
const eventType = req.body.event_type || (() => {
const subscribed = typeof row.events === 'string' ? safeJson(row.events, []) : (row.events || []);
return subscribed[0] || 'event.published';
})();
// Fire a synthetic event WITHOUT writing to webhooks table — the test
// bypasses subscription matching by inserting a delivery directly.
const crypto = require('crypto');
const deliveryId = crypto.randomUUID();
const payload = {
id: deliveryId,
type: eventType,
created_at: new Date().toISOString(),
data: { test: true, fired_by: req.admin.username, webhook_id: row.id },
};
await db('webhook_deliveries').insert({
webhook_id: row.id,
event_type: eventType,
payload: JSON.stringify(payload),
attempt_count: 0,
status: 'pending',
next_retry_at: new Date(),
created_at: new Date(),
});
res.status(202).json({ enqueued: true, event_type: eventType });
} catch (err) {
logger.error('webhook test failed', { error: err.message });
res.status(500).json({ error: 'Failed to enqueue test event' });
}
}
);
// ─── List deliveries ─────────────────────────────────────────────────────
router.get(
'/:id/deliveries',
adminAuth,
requirePermission('settings.view'),
[
query('status').optional().isIn(['pending', 'success', 'failed']),
query('page').optional().isInt({ min: 1 }),
query('limit').optional().isInt({ min: 1, max: 100 }),
],
async (req, res) => {
try {
const errors = validationResult(req);
if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() });
const webhookId = req.params.id;
const exists = await db('webhooks').where({ id: webhookId }).first();
if (!exists) return res.status(404).json({ error: 'Webhook not found' });
const page = parseInt(req.query.page || '1', 10);
const limit = parseInt(req.query.limit || '25', 10);
const offset = (page - 1) * limit;
let q = db('webhook_deliveries').where({ webhook_id: webhookId });
if (req.query.status) q = q.where({ status: req.query.status });
const totalRow = await q.clone().count('id as count').first();
const total = parseInt(totalRow?.count || 0, 10);
const rows = await q
.select(
'id', 'event_type', 'attempt_count', 'status', 'response_status',
'latency_ms', 'next_retry_at', 'created_at', 'completed_at', 'last_error'
)
.orderBy('created_at', 'desc')
.limit(limit)
.offset(offset);
res.json({ deliveries: rows, pagination: { page, limit, total } });
} catch (err) {
logger.error('deliveries list failed', { error: err.message });
res.status(500).json({ error: 'Failed to list deliveries' });
}
}
);
// ─── Delivery detail ─────────────────────────────────────────────────────
router.get(
'/:id/deliveries/:deliveryId',
adminAuth,
requirePermission('settings.view'),
async (req, res) => {
try {
const row = await db('webhook_deliveries')
.where({ id: req.params.deliveryId, webhook_id: req.params.id })
.first();
if (!row) return res.status(404).json({ error: 'Delivery not found' });
res.json({
...row,
payload: typeof row.payload === 'string' ? safeJson(row.payload, row.payload) : row.payload,
});
} catch (err) {
logger.error('delivery detail failed', { error: err.message });
res.status(500).json({ error: 'Failed to load delivery' });
}
}
);
// ─── Replay ──────────────────────────────────────────────────────────────
router.post(
'/:id/deliveries/:deliveryId/replay',
adminAuth,
requirePermission('settings.edit'),
async (req, res) => {
try {
const row = await db('webhook_deliveries')
.where({ id: req.params.deliveryId, webhook_id: req.params.id })
.first();
if (!row) return res.status(404).json({ error: 'Delivery not found' });
// Re-enqueue: copy the original payload + event_type into a new row
// marked pending. Preserves the audit log of the original attempt.
const crypto = require('crypto');
const newPayload = (() => {
const obj = typeof row.payload === 'string' ? safeJson(row.payload, {}) : row.payload || {};
// Replays get a fresh delivery id but keep the event payload data.
return JSON.stringify({ ...obj, id: crypto.randomUUID(), replayed_from: row.id });
})();
const insertResult = await db('webhook_deliveries').insert({
webhook_id: row.webhook_id,
event_type: row.event_type,
payload: newPayload,
attempt_count: 0,
status: 'pending',
next_retry_at: new Date(),
created_at: new Date(),
}).returning('id');
const newId = insertResult[0]?.id || insertResult[0];
res.status(202).json({ enqueued: true, original_id: row.id, replay_id: newId });
} catch (err) {
logger.error('delivery replay failed', { error: err.message });
res.status(500).json({ error: 'Failed to replay delivery' });
}
}
);
module.exports = router;