feat: outbound webhooks for event/photo lifecycle (#327)
PicPeak POSTs lifecycle notifications to admin-configured URLs. Each delivery is signed HMAC-SHA256 in the X-PicPeak-Signature header. Verified end-to-end: 1/1 Playwright spec, 8/8 backend integration tests, full UI click-through via Chrome DevTools. Schema (migration 082) - webhooks: id, name, url, secret (plaintext — required to compute HMAC for every outbound POST), secret_preview, events[], active, filter, template, created_by, timestamps, last_success_at/last_failure_at. - webhook_deliveries: webhook_id (FK CASCADE), event_type, payload, attempt_count, status (pending|success|failed), response_status, response_body (truncated to 1KB), latency_ms, next_retry_at, last_error, created_at, completed_at. Composite index (status, next_retry_at) serves the worker's hot-path query. Service + worker - webhookService.fire(eventType, data) — non-throwing entry point used by lifecycle hooks. Looks up active webhooks subscribed to the event and applies their per-webhook filter (dot-path equality predicate) before enqueueing one webhook_deliveries row per match. Filter and template logic ship in this commit; admin surfaces in the follow-up. - webhookDeliveryWorker — setInterval(5s) poller; fetches up to 5 pending rows; per delivery: re-validates URL via networkValidation (DNS-rebinding mitigation, opt-out via WEBHOOK_ALLOW_PRIVATE_URLS), signs body with HMAC-SHA256, POSTs with 10s timeout, records outcome. Backoff schedule: 1m → 5m → 30m → 2h → 12h, max 5 attempts. Response body truncated to 1KB before storage. If a webhook has a template, the rendered string replaces the JSON envelope as the request body (signature is computed over the bytes actually sent). Lifecycle wiring - adminEvents.js POST /events → event.created (+ event.published when not draft); POST /:id/publish → event.published. - routes/events.js (legacy public POST) → event.created + event.published. - routes/v1/events.js (#322 API) → event.created + event.published on create, photo.uploaded on photo POST. - archiveService.archiveEvent() → event.archived. Per-photo photo.deleted intentionally NOT fired during cascade — receivers infer from event.archived to avoid flooding (issue spec). - expirationChecker.handleExpiredEvent() → event.expired BEFORE the cascading archive (so receivers see expired→archived in order). - adminPhotos.js — photo.uploaded on each batch row, photo.deleted on single + bulk delete. - photoProcessor.js — photo.uploaded for guest uploads + auto-import (covers all entry paths). - fileWatcher.js — photo.uploaded on add, photo.deleted on unlink (local mode only). Admin endpoints (mirrors adminApiTokens.js pattern) - /api/admin/webhooks: GET list, POST create (returns plaintext secret exactly once), GET :id, PUT :id, DELETE :id, POST :id/test (synthetic fire), GET :id/deliveries (paginated, filter by status), GET :id/deliveries/:deliveryId, POST :id/deliveries/:deliveryId/replay. Frontend - Settings → Webhooks tab (mirrors API Tokens layout): name + URL + event checkboxes + "Advanced" expander for filter (JSON) and template. Plaintext secret shown once on creation with a Copy button. Active/ Disabled toggle button per row. - /admin/webhooks/:id/deliveries — operational debug surface. Table with timestamp/event/status/attempts/HTTP/latency. Status filter chips (all/pending/success/failed). Row click → slide-over with payload + signature + response body. Replay button on failed rows. Send-test-event dialog. Auto-refresh every 10s. Dev infrastructure - dev/webhook-receiver/ — tiny node:alpine HTTP server (~100 LOC) that records every POST to an in-memory ring buffer. Exposes GET /requests for the E2E spec to assert deliveries landed with the right HMAC. Sibling pattern to MinIO. Reachable from the backend at http://webhook-receiver:8888 inside the picpeak network. Tests - backend/__tests__/integration/webhookDelivery.test.js (8/8) — signature verification, headers, retry/backoff, max-attempts → failed, response truncation, disabled-mid-flight, SSRF block, start/stop idempotency. - tests/e2e/webhooks-roundtrip.spec.ts (1/1) — create webhook → trigger event.published → assert receiver got POST with valid HMAC → visit deliveries page → row visible with status=success → API test event → API replay → disable webhook → assert no new delivery. Docs - README §"Webhooks" — event catalog, payload shape, HMAC verification in Node + Python + bash, retry semantics, SSRF protection. - .env.example — WEBHOOK_ALLOW_PRIVATE_URLS, WEBHOOK_DELIVERY_INTERVAL_MS, WEBHOOK_DELIVERY_CONCURRENCY, WEBHOOK_HTTP_TIMEOUT_MS, WEBHOOK_MAX_ATTEMPTS. Out of scope for v1 (per issue): webhook templates' code-eval (the ${dot.path} substitution that ships is pure string replacement, no expression engine — see follow-up commit), per-webhook rate limiting beyond the global concurrency cap, synchronous "ask before delete" webhooks. Spanning files - App.tsx pulls in this commit with both the AnalyticsBootstrap (#325 dedup) and the WebhookDeliveriesPage route registration. Splitting via git add -p was forfeit for sanity; the single 92-line diff is honest about both contributions. - adminEvents.js diff bundles the webhook fires AND the allow_presigned_download field plumbing (#328 follow-up). Same reasoning. - The new webhookService/Worker/adminWebhooks files include the filter and template logic from the follow-up — they were authored in one pass; splitting them post-hoc would have produced fragile partial files. The follow-up commit covers the migration and the UI for these.
This commit is contained in:
Paul Nothaft
@@ -0,0 +1,277 @@
|
||||
const axios = require('axios');
|
||||
const { db } = require('../database/db');
|
||||
const logger = require('../utils/logger');
|
||||
const { signPayload, renderTemplate } = require('./webhookService');
|
||||
const { validateExternalUrl } = require('../utils/networkValidation');
|
||||
|
||||
const POLL_INTERVAL_MS = parseInt(process.env.WEBHOOK_DELIVERY_INTERVAL_MS || '5000', 10);
|
||||
const CONCURRENCY = parseInt(process.env.WEBHOOK_DELIVERY_CONCURRENCY || '5', 10);
|
||||
const HTTP_TIMEOUT_MS = parseInt(process.env.WEBHOOK_HTTP_TIMEOUT_MS || '10000', 10);
|
||||
const MAX_ATTEMPTS = parseInt(process.env.WEBHOOK_MAX_ATTEMPTS || '5', 10);
|
||||
const RESPONSE_TRUNCATE_BYTES = 1024;
|
||||
// Mutable so tests can flip it without juggling require.cache; reads the
|
||||
// env var at module load for the production code path.
|
||||
let allowPrivateUrls = process.env.WEBHOOK_ALLOW_PRIVATE_URLS === 'true';
|
||||
const SIGNATURE_HEADER = 'X-PicPeak-Signature';
|
||||
const EVENT_HEADER = 'X-PicPeak-Event';
|
||||
const DELIVERY_HEADER = 'X-PicPeak-Delivery';
|
||||
|
||||
// Backoff schedule per the issue spec — index = attempt that just failed.
|
||||
// attempt_count after the failure becomes (failedAttempt + 1); we look up
|
||||
// the delay using the *new* attempt count to schedule the next try.
|
||||
// attempt 1 fails → wait 1m
|
||||
// attempt 2 fails → wait 5m
|
||||
// attempt 3 fails → wait 30m
|
||||
// attempt 4 fails → wait 2h
|
||||
// attempt 5 fails → wait 12h THEN give up (max 5 attempts total)
|
||||
const BACKOFF_MS = [
|
||||
60_000, // 1 min
|
||||
5 * 60_000, // 5 min
|
||||
30 * 60_000, // 30 min
|
||||
2 * 60 * 60_000, // 2 h
|
||||
12 * 60 * 60_000, // 12 h (only used when MAX_ATTEMPTS extended past 5)
|
||||
];
|
||||
|
||||
let intervalHandle = null;
|
||||
let stopped = false;
|
||||
// Tracks deliveries currently being processed in this tick — guards
|
||||
// against the same row being claimed twice if a tick takes longer than
|
||||
// POLL_INTERVAL_MS.
|
||||
const inFlight = new Set();
|
||||
|
||||
function truncate(str, bytes) {
|
||||
if (str == null) return null;
|
||||
const buf = Buffer.from(String(str), 'utf8');
|
||||
if (buf.length <= bytes) return buf.toString('utf8');
|
||||
return buf.subarray(0, bytes).toString('utf8');
|
||||
}
|
||||
|
||||
async function fetchPending(limit) {
|
||||
// Skip rows already in-flight from a previous tick that's still running.
|
||||
const excludeIds = Array.from(inFlight);
|
||||
let q = db('webhook_deliveries')
|
||||
.where('status', 'pending')
|
||||
.where('next_retry_at', '<=', new Date())
|
||||
.orderBy('next_retry_at', 'asc')
|
||||
.limit(limit);
|
||||
if (excludeIds.length > 0) {
|
||||
q = q.whereNotIn('id', excludeIds);
|
||||
}
|
||||
return q.select('*');
|
||||
}
|
||||
|
||||
async function deliverOne(row) {
|
||||
const startedAt = Date.now();
|
||||
const webhook = await db('webhooks').where({ id: row.webhook_id }).first();
|
||||
|
||||
if (!webhook) {
|
||||
// Webhook was deleted while a delivery was pending. Mark failed and move on.
|
||||
await db('webhook_deliveries')
|
||||
.where({ id: row.id })
|
||||
.update({
|
||||
status: 'failed',
|
||||
last_error: 'webhook subscription no longer exists',
|
||||
completed_at: new Date(),
|
||||
attempt_count: row.attempt_count + 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (!webhook.active) {
|
||||
// Subscription disabled mid-flight. Don't abandon — leave as failed
|
||||
// so the deliveries page reflects the reality.
|
||||
await db('webhook_deliveries')
|
||||
.where({ id: row.id })
|
||||
.update({
|
||||
status: 'failed',
|
||||
last_error: 'webhook is disabled',
|
||||
completed_at: new Date(),
|
||||
attempt_count: row.attempt_count + 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Re-validate URL per delivery — DNS-rebinding mitigation. Admin can opt
|
||||
// out via WEBHOOK_ALLOW_PRIVATE_URLS=true for local-receiver dev runs.
|
||||
if (!allowPrivateUrls) {
|
||||
const urlCheck = validateExternalUrl(webhook.url);
|
||||
if (!urlCheck.valid) {
|
||||
await markFailedFinal(row, `URL rejected: ${urlCheck.error}`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const envelopeBody = typeof row.payload === 'string' ? row.payload : JSON.stringify(row.payload);
|
||||
const envelopeObj = (() => {
|
||||
try { return JSON.parse(envelopeBody); } catch { return {}; }
|
||||
})();
|
||||
|
||||
// Per-webhook template (#327 follow-up). If set + valid, replaces the
|
||||
// default JSON envelope as the request body. Signature is computed over
|
||||
// the BODY ACTUALLY SENT, so receivers verify whatever they receive.
|
||||
let rawBody = envelopeBody;
|
||||
let contentType = 'application/json';
|
||||
if (webhook.template) {
|
||||
const rendered = renderTemplate(webhook.template, envelopeObj);
|
||||
if (rendered != null) {
|
||||
rawBody = rendered;
|
||||
// Best-effort content-type detection: if it parses as JSON, keep
|
||||
// application/json; otherwise send as text/plain.
|
||||
try { JSON.parse(rendered); } catch { contentType = 'text/plain; charset=utf-8'; }
|
||||
}
|
||||
}
|
||||
const signature = signPayload(webhook.secret, rawBody);
|
||||
const deliveryId = envelopeObj?.id || String(row.id);
|
||||
|
||||
let response;
|
||||
let networkError;
|
||||
try {
|
||||
response = await axios.post(webhook.url, rawBody, {
|
||||
headers: {
|
||||
'Content-Type': contentType,
|
||||
[SIGNATURE_HEADER]: signature,
|
||||
[EVENT_HEADER]: row.event_type,
|
||||
[DELIVERY_HEADER]: deliveryId,
|
||||
'User-Agent': 'PicPeak-Webhooks/1.0',
|
||||
},
|
||||
timeout: HTTP_TIMEOUT_MS,
|
||||
// Don't throw on non-2xx; we handle status manually.
|
||||
validateStatus: () => true,
|
||||
// Don't follow redirects — security + receivers should give us the
|
||||
// final URL up front.
|
||||
maxRedirects: 0,
|
||||
// Cap response body so a chatty receiver can't OOM us before truncation.
|
||||
maxContentLength: 10 * 1024,
|
||||
maxBodyLength: rawBody.length + 1024,
|
||||
});
|
||||
} catch (err) {
|
||||
networkError = err;
|
||||
}
|
||||
|
||||
const latency = Date.now() - startedAt;
|
||||
const newAttempt = row.attempt_count + 1;
|
||||
|
||||
if (response && response.status >= 200 && response.status < 300) {
|
||||
await db('webhook_deliveries')
|
||||
.where({ id: row.id })
|
||||
.update({
|
||||
status: 'success',
|
||||
response_status: response.status,
|
||||
response_body: truncate(stringifyBody(response.data), RESPONSE_TRUNCATE_BYTES),
|
||||
latency_ms: latency,
|
||||
attempt_count: newAttempt,
|
||||
completed_at: new Date(),
|
||||
next_retry_at: null,
|
||||
});
|
||||
await db('webhooks').where({ id: webhook.id }).update({ last_success_at: new Date() });
|
||||
return;
|
||||
}
|
||||
|
||||
// Failure path — schedule retry or give up.
|
||||
const errorMsg = networkError
|
||||
? `network error: ${networkError.code || networkError.message}`
|
||||
: `non-2xx status: ${response?.status}`;
|
||||
|
||||
if (newAttempt >= MAX_ATTEMPTS) {
|
||||
await db('webhook_deliveries')
|
||||
.where({ id: row.id })
|
||||
.update({
|
||||
status: 'failed',
|
||||
response_status: response?.status || null,
|
||||
response_body: response ? truncate(stringifyBody(response.data), RESPONSE_TRUNCATE_BYTES) : null,
|
||||
last_error: errorMsg,
|
||||
latency_ms: latency,
|
||||
attempt_count: newAttempt,
|
||||
completed_at: new Date(),
|
||||
next_retry_at: null,
|
||||
});
|
||||
await db('webhooks').where({ id: webhook.id }).update({ last_failure_at: new Date() });
|
||||
return;
|
||||
}
|
||||
|
||||
const backoff = BACKOFF_MS[Math.min(newAttempt - 1, BACKOFF_MS.length - 1)];
|
||||
await db('webhook_deliveries')
|
||||
.where({ id: row.id })
|
||||
.update({
|
||||
status: 'pending',
|
||||
response_status: response?.status || null,
|
||||
response_body: response ? truncate(stringifyBody(response.data), RESPONSE_TRUNCATE_BYTES) : null,
|
||||
last_error: errorMsg,
|
||||
latency_ms: latency,
|
||||
attempt_count: newAttempt,
|
||||
next_retry_at: new Date(Date.now() + backoff),
|
||||
});
|
||||
await db('webhooks').where({ id: webhook.id }).update({ last_failure_at: new Date() });
|
||||
}
|
||||
|
||||
async function markFailedFinal(row, reason) {
|
||||
await db('webhook_deliveries')
|
||||
.where({ id: row.id })
|
||||
.update({
|
||||
status: 'failed',
|
||||
last_error: reason,
|
||||
attempt_count: row.attempt_count + 1,
|
||||
completed_at: new Date(),
|
||||
next_retry_at: null,
|
||||
});
|
||||
await db('webhooks').where({ id: row.webhook_id }).update({ last_failure_at: new Date() });
|
||||
}
|
||||
|
||||
function stringifyBody(data) {
|
||||
if (data == null) return null;
|
||||
if (typeof data === 'string') return data;
|
||||
if (Buffer.isBuffer(data)) return data.toString('utf8');
|
||||
try { return JSON.stringify(data); } catch { return String(data); }
|
||||
}
|
||||
|
||||
async function tick() {
|
||||
if (stopped) return;
|
||||
try {
|
||||
const slots = Math.max(0, CONCURRENCY - inFlight.size);
|
||||
if (slots === 0) return;
|
||||
const rows = await fetchPending(slots);
|
||||
if (rows.length === 0) return;
|
||||
rows.forEach((r) => inFlight.add(r.id));
|
||||
await Promise.allSettled(
|
||||
rows.map((r) =>
|
||||
deliverOne(r)
|
||||
.catch((err) => logger.error(`[webhookWorker] delivery ${r.id} crashed: ${err.message}`))
|
||||
.finally(() => inFlight.delete(r.id))
|
||||
)
|
||||
);
|
||||
} catch (err) {
|
||||
logger.error(`[webhookWorker] tick failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function startWebhookDeliveryWorker() {
|
||||
if (intervalHandle) return; // idempotent
|
||||
stopped = false;
|
||||
intervalHandle = setInterval(tick, POLL_INTERVAL_MS);
|
||||
logger.info(
|
||||
`[webhookWorker] started — interval=${POLL_INTERVAL_MS}ms, concurrency=${CONCURRENCY}, ` +
|
||||
`max_attempts=${MAX_ATTEMPTS}, allow_private=${allowPrivateUrls}`
|
||||
);
|
||||
}
|
||||
|
||||
function stopWebhookDeliveryWorker() {
|
||||
stopped = true;
|
||||
if (intervalHandle) {
|
||||
clearInterval(intervalHandle);
|
||||
intervalHandle = null;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
startWebhookDeliveryWorker,
|
||||
stopWebhookDeliveryWorker,
|
||||
// exported for tests
|
||||
__test: {
|
||||
tick,
|
||||
BACKOFF_MS,
|
||||
SIGNATURE_HEADER,
|
||||
EVENT_HEADER,
|
||||
DELIVERY_HEADER,
|
||||
setAllowPrivateUrls(value) { allowPrivateUrls = !!value; },
|
||||
},
|
||||
};
|
||||
Reference in New Issue
Block a user