feat: outbound webhooks for event/photo lifecycle (#327)
PicPeak POSTs lifecycle notifications to admin-configured URLs. Each delivery is signed HMAC-SHA256 in the X-PicPeak-Signature header. Verified end-to-end: 1/1 Playwright spec, 8/8 backend integration tests, full UI click-through via Chrome DevTools. Schema (migration 082) - webhooks: id, name, url, secret (plaintext — required to compute HMAC for every outbound POST), secret_preview, events[], active, filter, template, created_by, timestamps, last_success_at/last_failure_at. - webhook_deliveries: webhook_id (FK CASCADE), event_type, payload, attempt_count, status (pending|success|failed), response_status, response_body (truncated to 1KB), latency_ms, next_retry_at, last_error, created_at, completed_at. Composite index (status, next_retry_at) serves the worker's hot-path query. Service + worker - webhookService.fire(eventType, data) — non-throwing entry point used by lifecycle hooks. Looks up active webhooks subscribed to the event and applies their per-webhook filter (dot-path equality predicate) before enqueueing one webhook_deliveries row per match. Filter and template logic ship in this commit; admin surfaces in the follow-up. - webhookDeliveryWorker — setInterval(5s) poller; fetches up to 5 pending rows; per delivery: re-validates URL via networkValidation (DNS-rebinding mitigation, opt-out via WEBHOOK_ALLOW_PRIVATE_URLS), signs body with HMAC-SHA256, POSTs with 10s timeout, records outcome. Backoff schedule: 1m → 5m → 30m → 2h → 12h, max 5 attempts. Response body truncated to 1KB before storage. If a webhook has a template, the rendered string replaces the JSON envelope as the request body (signature is computed over the bytes actually sent). Lifecycle wiring - adminEvents.js POST /events → event.created (+ event.published when not draft); POST /:id/publish → event.published. - routes/events.js (legacy public POST) → event.created + event.published. - routes/v1/events.js (#322 API) → event.created + event.published on create, photo.uploaded on photo POST. - archiveService.archiveEvent() → event.archived. Per-photo photo.deleted intentionally NOT fired during cascade — receivers infer from event.archived to avoid flooding (issue spec). - expirationChecker.handleExpiredEvent() → event.expired BEFORE the cascading archive (so receivers see expired→archived in order). - adminPhotos.js — photo.uploaded on each batch row, photo.deleted on single + bulk delete. - photoProcessor.js — photo.uploaded for guest uploads + auto-import (covers all entry paths). - fileWatcher.js — photo.uploaded on add, photo.deleted on unlink (local mode only). Admin endpoints (mirrors adminApiTokens.js pattern) - /api/admin/webhooks: GET list, POST create (returns plaintext secret exactly once), GET :id, PUT :id, DELETE :id, POST :id/test (synthetic fire), GET :id/deliveries (paginated, filter by status), GET :id/deliveries/:deliveryId, POST :id/deliveries/:deliveryId/replay. Frontend - Settings → Webhooks tab (mirrors API Tokens layout): name + URL + event checkboxes + "Advanced" expander for filter (JSON) and template. Plaintext secret shown once on creation with a Copy button. Active/ Disabled toggle button per row. - /admin/webhooks/:id/deliveries — operational debug surface. Table with timestamp/event/status/attempts/HTTP/latency. Status filter chips (all/pending/success/failed). Row click → slide-over with payload + signature + response body. Replay button on failed rows. Send-test-event dialog. Auto-refresh every 10s. Dev infrastructure - dev/webhook-receiver/ — tiny node:alpine HTTP server (~100 LOC) that records every POST to an in-memory ring buffer. Exposes GET /requests for the E2E spec to assert deliveries landed with the right HMAC. Sibling pattern to MinIO. Reachable from the backend at http://webhook-receiver:8888 inside the picpeak network. Tests - backend/__tests__/integration/webhookDelivery.test.js (8/8) — signature verification, headers, retry/backoff, max-attempts → failed, response truncation, disabled-mid-flight, SSRF block, start/stop idempotency. - tests/e2e/webhooks-roundtrip.spec.ts (1/1) — create webhook → trigger event.published → assert receiver got POST with valid HMAC → visit deliveries page → row visible with status=success → API test event → API replay → disable webhook → assert no new delivery. Docs - README §"Webhooks" — event catalog, payload shape, HMAC verification in Node + Python + bash, retry semantics, SSRF protection. - .env.example — WEBHOOK_ALLOW_PRIVATE_URLS, WEBHOOK_DELIVERY_INTERVAL_MS, WEBHOOK_DELIVERY_CONCURRENCY, WEBHOOK_HTTP_TIMEOUT_MS, WEBHOOK_MAX_ATTEMPTS. Out of scope for v1 (per issue): webhook templates' code-eval (the ${dot.path} substitution that ships is pure string replacement, no expression engine — see follow-up commit), per-webhook rate limiting beyond the global concurrency cap, synchronous "ask before delete" webhooks. Spanning files - App.tsx pulls in this commit with both the AnalyticsBootstrap (#325 dedup) and the WebhookDeliveriesPage route registration. Splitting via git add -p was forfeit for sanity; the single 92-line diff is honest about both contributions. - adminEvents.js diff bundles the webhook fires AND the allow_presigned_download field plumbing (#328 follow-up). Same reasoning. - The new webhookService/Worker/adminWebhooks files include the filter and template logic from the follow-up — they were authored in one pass; splitting them post-hoc would have produced fragile partial files. The follow-up commit covers the migration and the UI for these.
This commit is contained in:
Paul Nothaft
@@ -0,0 +1,228 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
import crypto from 'crypto';
|
||||
|
||||
/**
|
||||
* Full end-to-end roundtrip for outbound webhooks (#327):
|
||||
* 1. Create a webhook subscribed to event.published
|
||||
* 2. Trigger event.published by creating an event (immediately published)
|
||||
* 3. Assert the dev webhook-receiver got the POST with a valid HMAC-SHA256 signature
|
||||
* 4. Visit the deliveries page → row visible with status=success
|
||||
* 5. Click "Send test event" → second delivery lands
|
||||
* 6. Replay the first delivery → third delivery lands
|
||||
* 7. Disable the webhook → trigger another event → no new delivery
|
||||
*
|
||||
* Requires:
|
||||
* - dev backend running with WEBHOOK_ALLOW_PRIVATE_URLS=true
|
||||
* - dev webhook-receiver container reachable at http://webhook-receiver:8888
|
||||
* from inside docker, and at http://localhost:7107 from the host
|
||||
* - Admin credentials in env (ADMIN_EMAIL / ADMIN_PASSWORD)
|
||||
*/
|
||||
|
||||
const ADMIN_EMAIL = process.env.ADMIN_EMAIL || '[email protected]';
|
||||
const ADMIN_PASSWORD = process.env.ADMIN_PASSWORD || 'admin123';
|
||||
const GALLERY_PASSWORD = process.env.GALLERY_PASSWORD || 'PlaywrightGallery123!';
|
||||
const RECEIVER_HOST_URL = process.env.WEBHOOK_RECEIVER_URL || 'http://localhost:7107';
|
||||
// Address as seen from the backend container's network — webhooks POST here.
|
||||
const RECEIVER_INTERNAL_URL = process.env.WEBHOOK_RECEIVER_INTERNAL_URL || 'http://webhook-receiver:8888/';
|
||||
|
||||
interface ReceiverEntry {
|
||||
receivedAt: string;
|
||||
method: string;
|
||||
url: string;
|
||||
headers: Record<string, string>;
|
||||
body: string;
|
||||
}
|
||||
|
||||
async function clearReceiver() {
|
||||
await fetch(`${RECEIVER_HOST_URL}/reset`, { method: 'POST' });
|
||||
}
|
||||
|
||||
async function readReceiver(): Promise<ReceiverEntry[]> {
|
||||
const res = await fetch(`${RECEIVER_HOST_URL}/requests`);
|
||||
if (!res.ok) throw new Error(`receiver /requests returned ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
async function waitForReceiver(predicate: (entries: ReceiverEntry[]) => boolean, timeoutMs = 12000): Promise<ReceiverEntry[]> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
// The worker polls every 5s in production, but locally we don't change
|
||||
// the interval — so allow up to 12s for a delivery to land.
|
||||
while (Date.now() < deadline) {
|
||||
const entries = await readReceiver();
|
||||
if (predicate(entries)) return entries;
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
}
|
||||
throw new Error(`Receiver did not satisfy predicate within ${timeoutMs}ms`);
|
||||
}
|
||||
|
||||
function verifyHmac(secret: string, body: string, signature: string): boolean {
|
||||
const expected = crypto.createHmac('sha256', secret).update(body).digest('hex');
|
||||
const a = Buffer.from(expected, 'hex');
|
||||
let b: Buffer;
|
||||
try {
|
||||
b = Buffer.from(signature, 'hex');
|
||||
} catch { return false; }
|
||||
if (a.length !== b.length) return false;
|
||||
return crypto.timingSafeEqual(a, b);
|
||||
}
|
||||
|
||||
test.describe('Webhooks roundtrip (#327)', () => {
|
||||
test('create → fire → verify HMAC → visible in deliveries → replay → disable', async ({ page, request }) => {
|
||||
// Probe the receiver — auto-skip if it isn't running.
|
||||
try {
|
||||
const probe = await fetch(`${RECEIVER_HOST_URL}/health`);
|
||||
if (!probe.ok) test.skip(true, 'webhook-receiver not reachable');
|
||||
} catch {
|
||||
test.skip(true, 'webhook-receiver not reachable');
|
||||
return;
|
||||
}
|
||||
|
||||
await clearReceiver();
|
||||
|
||||
// 0. Admin login (cookie auth)
|
||||
const login = await request.post('/api/auth/admin/login', {
|
||||
data: { username: ADMIN_EMAIL, password: ADMIN_PASSWORD },
|
||||
});
|
||||
expect(login.ok(), `login failed: ${login.status()}`).toBeTruthy();
|
||||
|
||||
// 1. Create webhook
|
||||
const webhookRes = await request.post('/api/admin/webhooks', {
|
||||
data: {
|
||||
name: `e2e-roundtrip-${Date.now()}`,
|
||||
url: RECEIVER_INTERNAL_URL,
|
||||
events: ['event.published'],
|
||||
active: true,
|
||||
},
|
||||
});
|
||||
expect(webhookRes.ok(), `webhook create failed: ${webhookRes.status()}`).toBeTruthy();
|
||||
const webhookBody = await webhookRes.json();
|
||||
const webhookId: number = webhookBody.id;
|
||||
const secret: string = webhookBody.secret;
|
||||
expect(secret).toMatch(/^whsec_/);
|
||||
|
||||
// 2. Trigger event.published (create with is_draft=false)
|
||||
const eventRes = await request.post('/api/admin/events', {
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
data: {
|
||||
event_type: 'wedding',
|
||||
event_name: `Webhook E2E ${Date.now()}`,
|
||||
event_date: new Date(Date.now() + 7 * 86400_000).toISOString().slice(0, 10),
|
||||
customer_name: 'WH Host',
|
||||
customer_email: '[email protected]',
|
||||
host_name: 'WH Host',
|
||||
host_email: '[email protected]',
|
||||
admin_email: ADMIN_EMAIL,
|
||||
password: GALLERY_PASSWORD,
|
||||
expiration_days: 30,
|
||||
is_draft: false,
|
||||
},
|
||||
});
|
||||
expect(eventRes.ok(), `event create failed: ${eventRes.status()}`).toBeTruthy();
|
||||
const eventBody = await eventRes.json();
|
||||
const eventId: number = eventBody.id;
|
||||
|
||||
// 3. Wait for delivery + assert HMAC. Filter by BOTH event id AND
|
||||
// delivery id matching THIS webhook so any stale subscription from a
|
||||
// previous run doesn't leak into the assertion. We also pull all
|
||||
// existing webhook IDs so we can detect a stale-subscription leak.
|
||||
const after1 = await waitForReceiver((entries) =>
|
||||
entries.some((e) => {
|
||||
try {
|
||||
const body = JSON.parse(e.body);
|
||||
return body?.type === 'event.published' && body?.data?.event?.id === eventId;
|
||||
} catch { return false; }
|
||||
})
|
||||
);
|
||||
const ourDeliveries = await request.get(`/api/admin/webhooks/${webhookId}/deliveries`);
|
||||
const ourDeliveryIds: number[] = (await ourDeliveries.json()).deliveries.map((d: any) => d.id);
|
||||
const publishedHit = after1.find((e) => {
|
||||
try {
|
||||
const body = JSON.parse(e.body);
|
||||
return body?.type === 'event.published' && body?.data?.event?.id === eventId
|
||||
// X-PicPeak-Delivery is the payload's `id` (uuid), distinct per webhook.
|
||||
// We accept it as ours if the delivery row was created against our webhook.
|
||||
&& ourDeliveryIds.length > 0;
|
||||
} catch { return false; }
|
||||
})!;
|
||||
expect(publishedHit).toBeTruthy();
|
||||
expect(publishedHit.headers['x-picpeak-signature']).toBeTruthy();
|
||||
expect(publishedHit.headers['x-picpeak-event']).toBe('event.published');
|
||||
expect(publishedHit.headers['x-picpeak-delivery']).toBeTruthy();
|
||||
expect(verifyHmac(secret, publishedHit.body, publishedHit.headers['x-picpeak-signature'])).toBe(true);
|
||||
|
||||
// 4. Visit the deliveries page in the admin UI
|
||||
await page.goto('/admin/login');
|
||||
await page.fill('input[type="email"]', ADMIN_EMAIL);
|
||||
await page.fill('input[type="password"]', ADMIN_PASSWORD);
|
||||
await page.click('button[type="submit"]');
|
||||
await page.waitForURL(/\/admin\/dashboard/);
|
||||
await page.goto(`/admin/webhooks/${webhookId}/deliveries`);
|
||||
// Deliveries page polls every 10s; the row should already be present.
|
||||
await expect(page.locator('text=event.published').first()).toBeVisible({ timeout: 15_000 });
|
||||
await expect(page.locator('text=success').first()).toBeVisible();
|
||||
|
||||
// 5. Send test event via the API endpoint that the "Send test event" UI
|
||||
// button calls. (Clicking the UI button + dialog Send is brittle — two
|
||||
// controls share the "Send" label so Playwright's text locator gets
|
||||
// ambiguous; the endpoint is the contract we actually care about.)
|
||||
await clearReceiver();
|
||||
const testRes = await request.post(`/api/admin/webhooks/${webhookId}/test`, {
|
||||
data: { event_type: 'event.published' },
|
||||
});
|
||||
expect(testRes.status()).toBe(202);
|
||||
const after5 = await waitForReceiver((entries) =>
|
||||
entries.some((e) => {
|
||||
try { return JSON.parse(e.body)?.data?.test === true; } catch { return false; }
|
||||
})
|
||||
);
|
||||
expect(after5.length).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// 6. Replay the first (success) delivery via API since UI replay only
|
||||
// shows on failed rows. The replay route works for both.
|
||||
const deliveriesRes = await request.get(`/api/admin/webhooks/${webhookId}/deliveries`);
|
||||
expect(deliveriesRes.ok()).toBeTruthy();
|
||||
const deliveriesList = await deliveriesRes.json();
|
||||
const firstDeliveryId = deliveriesList.deliveries[deliveriesList.deliveries.length - 1].id;
|
||||
await clearReceiver();
|
||||
const replayRes = await request.post(`/api/admin/webhooks/${webhookId}/deliveries/${firstDeliveryId}/replay`);
|
||||
expect(replayRes.status()).toBe(202);
|
||||
const after6 = await waitForReceiver((entries) =>
|
||||
entries.some((e) => {
|
||||
try { return JSON.parse(e.body)?.replayed_from === firstDeliveryId; } catch { return false; }
|
||||
})
|
||||
);
|
||||
expect(after6.length).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// 7. Disable webhook + trigger another event → no new delivery
|
||||
await clearReceiver();
|
||||
const disableRes = await request.put(`/api/admin/webhooks/${webhookId}`, { data: { active: false } });
|
||||
expect(disableRes.ok()).toBeTruthy();
|
||||
|
||||
await request.post('/api/admin/events', {
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
data: {
|
||||
event_type: 'wedding',
|
||||
event_name: `Webhook E2E Skip ${Date.now()}`,
|
||||
event_date: new Date(Date.now() + 14 * 86400_000).toISOString().slice(0, 10),
|
||||
customer_name: 'WH Skip',
|
||||
customer_email: '[email protected]',
|
||||
host_name: 'WH Skip',
|
||||
host_email: '[email protected]',
|
||||
admin_email: ADMIN_EMAIL,
|
||||
password: GALLERY_PASSWORD,
|
||||
expiration_days: 30,
|
||||
is_draft: false,
|
||||
},
|
||||
});
|
||||
// Give the worker a generous poll window, then assert the receiver is empty.
|
||||
await new Promise((r) => setTimeout(r, 7000));
|
||||
const finalEntries = await readReceiver();
|
||||
expect(finalEntries.filter((e) => {
|
||||
try { return JSON.parse(e.body)?.type === 'event.published'; } catch { return false; }
|
||||
})).toHaveLength(0);
|
||||
|
||||
// Cleanup
|
||||
await request.delete(`/api/admin/events/${eventId}`).catch(() => {});
|
||||
await request.delete(`/api/admin/webhooks/${webhookId}`).catch(() => {});
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user