From b69dd134d0f5b1570ace261af4541d36771e62cd Mon Sep 17 00:00:00 2001 From: Paul Nothaft <53005142+the-luap@users.noreply.github.com> Date: Tue, 18 Aug 2026 22:37:28 +0200 Subject: [PATCH] =?UTF-8?q?feat(faces):=20People=20in=20this=20gallery=20?= =?UTF-8?q?=E2=80=94=20face=20recognition=20via=20an=20optional=20ML=20sid?= =?UTF-8?q?ecar=20(#1074)=20(#1075)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(ml): optional face-detection sidecar, opt-in and inert by default (#1074) First of four PRs for "People in this gallery". This one ships only the sidecar, its wiring and its CI — no schema, no backend code, no UI. Nothing in PicPeak calls it yet. picpeak-ml is a single FastAPI + onnxruntime container: three endpoints (/health, /info, /faces), no database, no volumes, no egress, no model download at runtime. Clustering, person identity and every privacy decision stay in the backend where the data already lives. Models are YuNet (detection) + FaceNet-512 (embedding), both MIT, both pinned by URL and SHA-256 and verified at build time. The licence analysis is in ml/LICENSES.md: the more accurate InsightFace weights are non-commercial-only and PicPeak's users are working photographers, so they are never baked into an image we publish. Two things worth review attention: - Alignment uses a least-squares similarity transform (Umeyama), NOT cv2.estimateAffinePartial2D. RANSAC and LMEDS exist to reject outliers among many correspondences; given five landmarks and no outliers they fit a three-point subset exactly and let the rest drift. Measured on a real off-frontal portrait: eyes and nose pinned to 0.11px, mouth corners 11.8px out on a 160px crop. Umeyama distributes it (max 6.5px, rms 5.1 vs 7.4). The failure mode is silent — a bad warp still yields 512 confident floats — so tests/test_pipeline.py pins it numerically. - FACENET_ONNX_URL has no default and the build fails loudly without it. deepface distributes FaceNet-512 as Keras .h5 only, so the ONNX is produced once by tools/convert_facenet.py and published as a release asset. Converting inside the build would drag TensorFlow through both architecture legs of every build to produce a byte-identical file. The CI jobs are gated on the FACENET_ONNX_URL repository variable and skip cleanly until it is set. Off by default, twice over: the sidecar is behind the `faces` compose profile, and the backend will gate on a `faces` feature flag that defaults to false. FACE_ML_URL defaults to http://picpeak-ml:8000 so the standard deployment needs no configuration — nothing dials that host while the flag is off, which is why a non-resolving default is harmless. Verified: 27 pytest tests green; YuNet loads and detects against a real portrait with its landmark order matching the alignment template index-for-index; both compose files validate and the faces profile is correctly excluded from a default `up`; workflow YAML parses and the job graph resolves. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(ml): pin the converter toolchain, verify parity, drop a false reproducibility claim (#1074) Ran the FaceNet-512 conversion for real and corrected what the previous commit assumed about it. The conversion works: 23,497,424 parameters, 89.6 MB ONNX, and the converted graph matches the Keras original to 2.086e-06 absolute / cosine 1.0000000000. That check is now part of the script rather than something I did once by hand — a subtly wrong graph still returns 512 plausible floats, so it refuses to leave the file on disk if parity fails. Also ran the full pipeline against both real models end to end. The embedding is L2-normalized to 1.000000, and the same face survives being re-rendered: half scale 0.973, double scale 0.984, JPEG q40 0.987, rotated 8 degrees 0.984, brightness +40 0.988. Scale invariance in particular is evidence the alignment warp is doing its job. Corrected claim: the conversion is NOT byte-reproducible. Two runs with the same pinned versions on the same machine gave different SHA-256s. The graphs are functionally identical — same 336 nodes, same 271 initializers, every weight matching to 0.000e+00 — but a few initializer names differ because tf2onnx's traced-op naming is not deterministic (Keras layer naming is deterministic; I checked). The previous commit message and README both claimed byte-identical output. They were wrong, and it matters: anyone re-running the conversion gets a different hash, and without this note that reads like tampering. The build-time SHA-256 pins one published artifact so its URL cannot start serving different bytes; validating a fresh conversion is the parity check's job. requirements-convert.txt now pins the exact set that produced the artifact, including transitive keras/protobuf/numpy, and documents that the converter needs Python 3.11 while the image runs 3.12. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): schema, queue, clustering and API for People in this gallery (#1074) Backend half of the feature. Migration 177, a face-detection queue, the clustering engine, the gallery and admin APIs, and the privacy wiring. No UI yet; nothing is reachable until the `faces` feature flag is on, which defaults to false. The flag is the gate, not FACE_ML_URL. That variable now has a working default (the compose service name), so its presence proves nothing about intent — if it were the gate, every install would poll a hostname that does not resolve. faceQueue re-checks the flag every tick, so turning it off stops the workers without a restart. Visibility scoping is the part worth reviewing closely. Face rows have no concept of photo visibility, but guests are restricted to photos.visibility='visible'. A raw count leaks how many hidden photos someone appears in, and an unscoped cover face renders a crop of a photo the guest may not open — with the best-scoring face being the likeliest pick, so it would happen often rather than rarely. facePeopleService recomputes both per request against the caller's own scope, and event_people.face_count_total is named to be conspicuous in a guest path. Six tests cover it, including the case where a person's photos are ALL hidden and they must vanish entirely. Face data is excluded from backups and .picpeak exports, per the decision in the thread: it is derived, so a restore re-scans rather than carrying biometrics between operators. Three separate mechanisms, because the engines cannot be filtered alike — EXCLUDED_TABLES for export, --exclude-table-data (not --exclude-table; the CREATE TABLE must survive or restore breaks on the first query) for Postgres, and DELETE + VACUUM on the temp copy for SQLite, which has no way to exclude a table from a whole-file .backup. The VACUUM is not cosmetic: without it the pages stay in the file and the claim is false on disk. Archiving now purges face data explicitly. photo_faces cascades off photos, but archive deletes neither the photo rows nor the event, so without this an archived gallery kept its biometrics indefinitely. Other decisions: clustering keeps names across a re-cluster by majority inheritance (without it, one button click silently discards every name the photographer typed); consolidation refuses to merge two people who were named differently; assignment never compares across model_version, since embeddings from two pipelines are not comparable; low-quality faces are stored but left unassigned so they show in "this photo contains" without spawning junk people. Migration is 177, not 174 — 174/175/176 landed on main while this branch was open. 29 tests green: 7 migration (idempotency, down(), cascade, and that installing it enqueues NOTHING), 11 clustering, 11 privacy/visibility. Lint clean; the pre-existing error counts in databaseBackup.js and server.js are unchanged. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): People strip, face filter and admin controls (#1074) Frontend half. Renders nothing anywhere unless the `faces` feature flag is on AND the photographer enabled detection for the gallery — the whole guest surface hangs off one boolean, event.people_enabled, which the server computes from the flag, the per-event toggle and the show-to-guests toggle together. Guest side: a People strip between the filter bar and the grid, circular crops from each person's cover face, an active-filter chip row, and a "Show all" bottom sheet. The face filter composes with category, search, media type and the liked/saved/rated filters in the same useMemo rather than replacing them, so "photos of Anna that I liked" works. Two people selected means AND by default — that is what picking a second face almost always asks for — with a toggle to OR that appears only once a second person is picked. Unnamed people show a photo count and never "Person 7". A number is honest about what the system knows; an invented name is not. There is a test asserting we don't do it. The strip renders nothing below two people, collapses to one line when dismissed (persisted per slug, so dismissing one gallery says nothing about the next), and appears mid-backfill with a progress line rather than blocking the gallery behind a spinner. Avatar crops are computed in ratios of the source dimensions so they survive whatever rendition the browser gets; without width/height they fall back to an uncropped thumbnail, since a wrongly-offset crop is worse than no crop. No new download endpoint: "download these N" rides the existing photoIds path, which already enforces access level and per-category permissions server-side. Adding a person_id selector would have been a second thing to authorize for no gain. Guest-facing copy never says "biometric" or "recognition" — those words describe our implementation, not the guest's experience. The sheet's footnote answers the first question every guest has (where does this go?) inline. The admin card, by contrast, is explicit: it states the controller obligation next to the toggle, and warns that scanning materializes the preview tier on galleries that never generated one, which is real CPU and disk an admin should know about before a 2,000-photo backfill. EN + DE translations. 140 frontend tests green (8 new), tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): measured match threshold, working build defaults, 89MB smaller image (#1074) Ran the Phase 0 spike that had been outstanding, published the model, and fixed what both turned up. THRESHOLD IS NOW MEASURED, NOT GUESSED. LFW's standard 1000-pair protocol run through this exact pipeline (YuNet -> Umeyama alignment -> FaceNet-512 ONNX), 100% detection on 2000 images: same person cosine 0.6958 +/- 0.1415 diff person cosine 0.0849 +/- 0.1674 separation 0.6109 peak accuracy 96.60% @ 0.405 So the pipeline separates people well — the thing I could not previously claim, since every earlier number was the same face re-rendered. Default moves 0.62 -> 0.50. The old value was a placeholder and a bad one: it gave 0% false merges but 22.4% false splits, i.e. roughly one in four same-person pairs failing to join, which fragments a gallery badly. 0.50 gives 1.0% false merge / 8.2% false split. Peak accuracy (0.405) is deliberately NOT chosen: for clustering the two errors do not cost the same. A false split is a duplicate row the photographer can merge away; a false merge puts a stranger into someone's "download my photos" — and until the Phase 2 merge/split UI ships, there is no way to undo one. So this sits on the conservative side of the optimum. The spike is committed as ml/tools/benchmark_threshold.py rather than thrown away, so "why 0.50?" has an answer in six months and a re-tune is one command. BUILD DEFAULTS. FACENET_ONNX_URL/_SHA256 now default to the published ml-models-v1 release asset, so `docker build ml/` and `docker compose --profile faces up` work with no arguments. Blanking either still fails loudly — a URL without a checksum is never acceptable, since the checksum is what makes the URL safe to trust. Found by running compose for real: it failed exactly as designed, which was correct behaviour and a bad out-of-box experience now that a canonical artifact exists. IMAGE SIZE. 389MB -> 300MB single-arch. `chown -R` after COPY rewrote every copied file into a fresh layer, duplicating the 90MB model for nothing; the user is now created before the copies and ownership set via COPY --chown. Also drops pip/setuptools from the runtime image. Measured RSS is 186MiB idle, and the container answers /faces end-to-end in well under the 80-150ms/photo the issue budgeted. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): threshold 0.50 -> 0.60 from real clustering, theme-aware People strip (#1074) Both fixes come from running the feature on an actual gallery — 61 photos, 5 real identities — rather than reasoning about it. THRESHOLD. The LFW pairwise sweep in the previous commit said 0.50, and it was wrong. On a real gallery at 0.50, three of six visible clusters were contaminated: two different people merged into one strip entry, which is the exact failure that puts a stranger into someone's "download my photos". Pairwise error rates do not predict cluster purity. Greedy assignment compounds — one wrong face drags the centroid toward the midpoint between two identities, making the next wrong face likelier. A 1% pairwise false-merge rate is not a 1% chance of a clean gallery, and no amount of staring at an ROC curve would have shown that. Sweep against ground truth (5 identities): 0.50 -> 6 clusters, 3 contaminated 0.56 -> 6 clusters, 0 contaminated 0.60 -> 5 clusters, 0 contaminated <- exactly right 0.64 -> 5 clusters, 0 contaminated, fewer faces assigned 0.60 recovers the right number of people with no contamination; higher only loses coverage. Migration 177 carries the full reasoning so the next person to touch this knows why the obvious pairwise answer is the wrong one. THEME. The People strip hardcoded `text-neutral-800` for named people. On a dark gallery — which the screenshot immediately showed — that renders a named person's label almost invisibly, while UNNAMED people stayed legible. Exactly backwards. Labels, headings, the collapsed summary, the scan line and the filter chip row now read the gallery's own theme tokens (--color-text / --color-muted-text / --color-accent / --color-surface-border) like the rest of the gallery surface. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): keep the mobile filter row inside the viewport (#1074) At 390px the photo count and Clear link were pushed against the right edge by ml-auto and clipped. Only apply it from the sm breakpoint up, where there is room; below that they flow after the chips. Found by screenshotting the real thing on an iPhone-sized viewport. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): complete Phase 1, add People management and auto-categories (#1074) Closes the two Phase 1 gaps, then builds Phase 2 and Phase 3. PHASE 1 GAPS. "Download these N" was specified, described as done in an earlier summary, and never actually built — I had verified the backend needed no new endpoint and let that stand as if the button existed. It now hands the filtered photo ids to the same path as a manual selection, so the server re-applies access level and per-category permissions on the way through. Photos in a downloads-disabled category are excluded client-side too, so the number on the button is the number the guest receives. Hidden entirely when downloads are off for the gallery. Lightbox person chips ("In this photo: Anna") are the second way into the face filter — a guest looking at a photo of themselves can act on it without scrolling back to the strip. Tapping one closes the lightbox and filters the grid behind it. PHASE 2. A People management modal over the endpoints that already existed and were already tested: rename inline, merge (multi-select, first pick is the target so the name a photographer typed survives), split via a face picker, hide, ignore. This matters more than it sounds — clustering deliberately errs toward splitting because a wrong merge puts a stranger into someone's download, and that trade only works if merging is easy. PHASE 3. Rule engine over face_count plus face-area ratio: 0 -> Details, 1 large -> Portraits, 2-5 -> Small groups, >5 -> Groups. The area ratio is what separates "a portrait of someone" from "someone is in this landscape". Three guarantees, all tested: it only ever fills an EMPTY category (enforced in the query AND re-checked in the UPDATE, so a photographer setting one mid-run still wins), everything it touches is marked auto_categorized so undo is exact, and it is a no-op unless separately enabled. Migration 178 adds the column — separate from 177, which has already run wherever this branch is deployed. Verified on the real gallery: 61 photos -> 48 portraits + 13 small groups, undo cleared exactly 61 and left the manual ones alone. Merge moved faces and removed the source. Both confirmed against the database, not just the UI. TWO BUGS THE BROWSER CAUGHT, both invisible to tsc: - The lightbox destructure never landed — my patch targeted a line that has a default value, matched nothing, and failed silently. `people` resolved to something else entirely and the chips would never have rendered. eslint's "outer scope value" warning is what surfaced it. - Admin face thumbnails 403'd because attaches whatever gallery token is in session storage; an admin who has also opened one of their own galleries sends a type:"gallery" bearer to an admin route. Admin routes authenticate from the httpOnly cookie, which a plain same-origin sends by itself. Worth noting AdminPhotoGrid has the same latent shape; not touched here. Also: the admin card now reports "N people (M shown to guests)" when those differ, so the settings page and the gallery stop disagreeing without explanation. 45 backend tests (8 new) and 140 frontend tests green; tsc and eslint clean. EN + DE for every new string. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * perf(faces): batch migration DDL and drop the face stack from server.js import (#1074) CI's backend job timed out at 10 minutes on the first run of this branch. Nothing failed — 132 of 182 suites passed and the wall clock ran out. Main does the same 182 in 124s, and where main has 12 suites slow enough for jest to print a duration, this branch had 77. Two changes, both worth making regardless of how much of the gap they close: - Migration 177 added its columns one ALTER TABLE at a time (four on photos, three on events, plus a separate index statement) and seeded settings with a SELECT and an INSERT per key. It now uses one alterTable per table and one SELECT plus one bulk INSERT. 178 folds its index into the same statement as its column. That chain replays in ~90 suites, so statement count there is multiplied by 90. - server.js required faceQueue at module scope, which pulls in axios and — through imageProcessor — sharp. Every supertest suite that imports server.js was paying for a module graph it never uses. Now required inside the startup block, next to the call that needs it. Honest about the evidence: locally the migration delta measures at zero (1.15s vs 1.13s for the same suite, three runs each), so batching alone does not explain an eight-minute regression. A fast local disk and many cores mask per-statement and per-import costs that a two-core runner with a shared disk does not. These are the two real costs this branch added to a path that runs in almost every suite; whether they are sufficient is a question for CI, not for another round of local speculation. 37 face tests still green after the change. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * i18n(faces): complete EN and DE coverage for the face feature (#1074) The admin card and the Features toggle were rendering entirely from inline English `defaultValue` fallbacks — 22 keys existed in no locale file at all, so a German admin saw an English consent notice, English toggles and English buttons. The gallery side was already translated; the admin side was not, and nothing in the toolchain flags this because a `defaultValue` always renders something. Adds the missing `admin.faces.*` (19), `settings.features.faces.*` (2) and shared `common.clear/saved/saveFailed` in both languages. Existing keys are left alone (setdefault, not overwrite), so the shared `common` strings other features rely on are untouched. Committed the audit as frontend/scripts/i18n-faces-audit.py rather than throwing it away: it extracts every t() key the face components actually use and diffs it against each locale, and it also reports German values that are byte-identical to English, which is the usual shape of an untranslated copy-paste. Currently: 69 keys in use, EN complete, DE complete, no identical pairs. Verified in the browser, not just in the JSON — the German card reads "61 / 61 Fotos durchsucht · 16 Personen (5 für Gäste sichtbar)" end to end. Also checked the components for hardcoded user-facing text (JSX nodes, title/aria-label/placeholder attributes) outside t(); there is none. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 13 defects from external review — coordinates, counts, erasure, races (#1074) Codex reviewed the branch against main. Thirteen findings, nine P1. I checked every one against the code and could not dismiss a single one as a false positive, so all thirteen are fixed here. THE WORST ONE: bounding boxes were stored in the wrong coordinate system. The sidecar reports coordinates in the space of the image it was HANDED — which is the ≤1920px preview, not the original — while every consumer compares them against photos.width/height, the original dimensions. A 6000px photo therefore produced boxes ~3x too small and areas ~9x too small: avatar crops landed in the wrong place and the Portraits rule could never fire. It is invisible on any photo already under 1920px, which is exactly why the demo gallery and every screenshot looked correct. Now scaled once in faceProcessor so everything downstream can assume original-image coordinates. ERASURE. The FK cascade on photo_faces is decorative on SQLite: PicPeak never enables `PRAGMA foreign_keys`, so deleting a photo left its embeddings behind. I first enabled the pragma globally and reverted it — six unrelated suites immediately failed on pre-existing dangling references, and switching it on would start rejecting inserts on every existing install. That is a real change worth making, but it is its own PR, not a rider on this one. Instead deletion purges explicitly: purgePhotoFaces in the photo paths (single, bulk, service) and photo_faces/event_people in deleteEventCascade. Tests assert this with the pragma explicitly OFF, so they can only pass if the code does the work. COUNTS. A re-scan deleted the old face rows without undoing their contribution to event_people, so counts inflated on every re-scan and ghost people survived. Now the affected people are recomputed before the replacements are assigned. My own "must not double its faces" test only checked photo_faces rows, which is why it passed throughout. RACES. A worker that finished after an admin purged the event committed its rows anyway — erasure reported success and the data reappeared. The commit is now conditional on the row still being 'processing'. And assignFaces is read-modify-write over an event's people, so two workers lost each other's updates; it is now serialised per event with an in-process mutex plus a Postgres advisory lock for the multi-pod case the queue advertises. METADATA LOSS. Merging discarded the source's name and suppression flags, so a merge could erase a typed name or un-hide someone. Reclustering remembered only people with a label, so an unnamed-but-hidden bystander came back guest-visible after one "Re-group people" — and suppression now propagates to every descendant cluster, not just the majority one. Also: export reset face_status so a restored gallery re-scans instead of claiming to be scanned forever; manual category edits clear auto_categorized so "undo automatic" cannot delete a photographer's own choice; external photos are skipped rather than failed (resolvePhotoStorageKey returns null for them by design); the gallery refetches photo memberships as a scan progresses so filtering is not stale; a failed VACUUM now fails the backup rather than publishing one that may retain biometric pages; and the ML Dockerfile's `|| true` is scoped to the uninstall — as written it was `(install && uninstall) || true`, so a failed dependency install produced a green layer and an image with no onnxruntime. Four new regression tests. Full backend suite failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 12 more defects from review round 2 — cross-event purge, leaks, lifecycle (#1074) Second Codex round on the same diff, now including round 1's fixes. Twelve findings, seven P1. Again none were false positives. SECURITY, AND MINE FROM ROUND 1: the bulk-delete face purge iterated the raw `photoIds` from the request instead of the event-scoped `photos` rows the handler had already validated. purgePhotoFaces has no event scope of its own, so an editor could pass another gallery's photo id and delete its face data — even though the photo deletion right below it was correctly scoped. Fixing one thing and introducing another is exactly why the second round was worth running. ANOTHER VISIBILITY LEAK, same class as the one round 1 fixed: /people returns scan progress, and getScanStatus counted every photo with a face_status — including hidden ones. Guests could read the hidden-photo count off the progress bar while the people list and covers beside it were properly scoped. Now scoped by the same predicate, with the caller passing its audience. RECLUSTER, ROUND 1'S FIX WAS INCOMPLETE. I made suppression follow every descendant but still copied the flags from the majority ANCESTOR. When reclustering merges a visible named person with a hidden one, the majority ancestor is often the visible one — republishing the hidden person's photos. Suppression is now OR-ed across every ancestor contributing faces. The name also now goes to the genuine largest descendant; the previous code took whichever cluster came first in map order, which the comment already claimed it did not. LIFECYCLE. Face data is excluded from backups and exports, but photos. face_status came across intact, so a restored install claimed every photo was scanned while holding no faces — and the worker only claims 'pending', so it stayed that way forever. Now: the SQLite backup requeues in the dump, restore requeues after the pool reinit (the Postgres path cannot rewrite rows inside pg_dump), the portable importer purges LOCAL face tables (they were excluded from the replace list, so another instance's embeddings survived an import with FK checks suspended) and requeues, and archiving disables detection so a restored archive is honestly off rather than enabled-and-empty. WRITE PATHS. Only processPhoto enqueued. The synchronous upload path (chunked-upload completion, watch-folder) left photos unscanned, and replacePhoto kept the OLD image's faces on a row now pointing at a different picture — stale identities shown on the new photo. FRONTEND. PeopleSheet and the admin manager rendered centred thumbnails and ignored the bbox, so on group photos the avatar showed whoever stood in the middle and two people from one photo were indistinguishable — in the manager whose entire job is telling faces apart. The crop maths is now one shared helper (faceCrop.ts) so the three surfaces cannot drift again. Full-page layouts (gallery-premium, gallery-story) render their own lightbox and never received the people props. Backend failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): round 3 — five of round 2's fixes were wrong or no-ops (#1074) Third and final Codex round. Eight findings, four P1 — and the important part is that FIVE of them are defects in round 2's fixes, not in the original code. - The sync-upload enqueue I added was a silent no-op. It queried through `trx` after the transaction had already been committed, which throws "Transaction query already complete" straight into the catch I had wrapped it in. Chunked uploads and watch-folder imports were still never scanned, and the code read as though they were. Uses `db` now. - The post-restore requeue ran BEFORE the files were restored, in both the portable importer and the native restore. The face worker is live during a restore, so it could claim those rows and scan the previous instance's files, or fail them for originals not yet on disk — with nothing to requeue them afterwards. Both now run after file restoration; the native one is extracted into requeueFaceScans() and called from the full and database-only paths. - The admin face crop mixed coordinate spaces: an original-pixel bbox scaled against the THUMBNAIL's natural size. The API now returns the source dimensions alongside the box, so there is one space to reason about. - Forwarding people props through layoutProps did not make them work — the full-page layouts never destructured them. GalleryStoryLayout now threads them to its own lightbox. Genuinely new findings, all in the same class as ones already fixed: - releaseToPending updated unconditionally, so a photo purged while its sidecar request was in flight came back as 'pending' and was rescanned — biometric rows reappearing after the purge reported success. Round 2 fixed exactly this on the COMMIT path and I did not carry it to the retry path. Now guarded on 'processing'. - purgePhotoFaces left face_status alone, so a worker mid-scan still satisfied its commit guard and could write fresh faces into a photo being deleted — orphans, since the FK cascade is inert on SQLite. It now clears the claim as part of the purge. - Phase 3 was unreachable: the migration seeds face_auto_categorize_enabled false and nothing could ever write it, so the rule engine and its undo endpoint returned "disabled" in every real flow. Added GET/PUT and a toggle on the admin card, EN + DE. NOT fixed, deliberately: GalleryPremiumLayout uses yet-another-react-lightbox rather than the shared PhotoLightbox, so person chips there are a real port rather than a prop forward. Recorded as open rather than bodged. Backend failure set identical to origin/main; 41 face tests and 140 frontend tests green; i18n audit reports EN and DE complete at 71 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): block face recognition on the all-in-one image (#1074, #1042) The single-container image cannot run this feature, so it is refused there rather than left to degrade. WHY, since the reason is not obvious from the code: the AIO image runs the backend, the frontend, SQLite and every background worker inside one container aimed at "one photographer plus guests browsing". It has no Redis, SQLite gives it a single writer, and it contains no ML sidecar to talk to. Face detection would add a second image-processing pipeline competing with Sharp for the same CPU and memory. That failure is not loud — the install just becomes slow and looks broken, which is the worst possible shape for a deployment whose whole promise is one container and no decisions. Gated on an explicit PICPEAK_SINGLE_CONTAINER marker, NOT inferred from SERVE_FRONTEND or a SQLite path: plenty of legitimate multi-container setups serve the frontend from the backend or run SQLite, and none of them should lose the feature by accident. Three layers, because the first is the only one that enforces: - faceSettings.isFeatureEnabled() returns false before consulting the flag, so a database restored from a full deployment with `faces` enabled still cannot switch it on here. - The feature-flag API forces `faces: false` in both directions, so the admin UI reflects reality instead of offering a switch that refuses to stay on. - The Features tab renders the card disabled with a plain-language reason, read from a new `single_container` field on /admin/system/version (an endpoint the admin UI already calls). Documented in ml/README.md and .env.example. Three tests pin the behaviour, including that the marker only accepts explicit truthy values. NOTE FOR PR #1068: this expects `Dockerfile.aio` to set `ENV PICPEAK_SINGLE_CONTAINER=true`. That one line lives on that branch and is not in this commit — until it lands, an AIO build would still offer the feature. Worth adding alongside the `Limits` section of docs/single-container.md. 44 face tests green; EN + DE complete at 72 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(faces): pin the bbox coordinate space with a real scale factor (#1074) The coordinate-space bug — boxes stored in preview space while every consumer reads them as original-image pixels — had no test, and could not have been caught by the ones that existed: every photo in the demo gallery is 750px, so the scale factor was always exactly 1.0 and the correction never executed. Verified by hand first, on a real 4000x3000 upload with the face placed off-centre so a wrong crop would be unmistakable. Before the fix the stored box was 1493,204 (preview space, face actually at x≈2850-3618); after, 3110,426 — a factor of 2.083, exactly 4000/1920, landing inside the face. The admin crop then resolved to left=-395px/top=-46px on a 64px window, which is the face centred. That verification is now a test rather than a memory. Three cases: a 4000px photo must scale by 4000/1920, a 1920px photo must NOT change (the case that hid the bug), and a row with no width must fall back to unscaled rather than storing NaN. Note for anyone extending these: jest hoists mock factories above the file, so anything they close over has to be `mock`-prefixed. Getting that wrong fails at transform time with a message that does not name the variable. 47 face tests green. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft --- .env.example | 49 ++ .github/workflows/docker-build.yml | 262 ++++++++- .github/workflows/tests.yml | 30 + .gitignore | 12 + .../integration/faceAutoCategories.test.js | 217 +++++++ .../integration/faceClustering.test.js | 325 +++++++++++ .../__tests__/integration/facePrivacy.test.js | 339 +++++++++++ .../integration/faceProcessorScaling.test.js | 146 +++++ .../177_add_face_recognition.test.js | 166 ++++++ .../core/177_add_face_recognition.js | 268 +++++++++ .../core/178_add_auto_categorized.js | 41 ++ backend/server.js | 11 + backend/src/routes/adminEvents/faces.js | 475 ++++++++++++++++ backend/src/routes/adminEvents/helpers.js | 8 + backend/src/routes/adminEvents/index.js | 1 + backend/src/routes/adminFeatureFlags.js | 23 + backend/src/routes/adminPhotos.js | 38 ++ backend/src/routes/adminSystem.js | 10 +- backend/src/routes/gallery.js | 98 +++- backend/src/services/archiveService.js | 29 + backend/src/services/databaseBackup.js | 79 ++- backend/src/services/faceAutoCategories.js | 178 ++++++ backend/src/services/faceClient.js | 134 +++++ backend/src/services/faceClustering.js | 533 ++++++++++++++++++ backend/src/services/facePeopleService.js | 247 ++++++++ backend/src/services/faceProcessor.js | 325 +++++++++++ backend/src/services/faceQueue.js | 210 +++++++ backend/src/services/faceSettings.js | 170 ++++++ backend/src/services/photoProcessor.js | 36 ++ .../src/services/photoReplacementService.js | 20 + backend/src/services/photoService.js | 10 + backend/src/services/picpeakExportService.js | 29 +- backend/src/services/picpeakImportService.js | 38 ++ backend/src/services/restoreService.js | 46 ++ docker-compose.production.yml | 37 ++ docker-compose.yml | 52 ++ frontend/scripts/i18n-faces-audit.py | 90 +++ .../components/admin/FaceRecognitionCard.tsx | 357 ++++++++++++ .../components/admin/PeopleManagerModal.tsx | 436 ++++++++++++++ frontend/src/components/admin/VersionInfo.tsx | 2 + .../src/components/gallery/GalleryView.tsx | 247 +++++++- .../src/components/gallery/PeopleSheet.tsx | 172 ++++++ .../src/components/gallery/PeopleStrip.tsx | 246 ++++++++ .../gallery/PhotoGridWithLayouts.tsx | 16 +- .../src/components/gallery/PhotoLightbox.tsx | 57 +- .../gallery/__tests__/PeopleStrip.test.tsx | 134 +++++ frontend/src/components/gallery/faceCrop.ts | 52 ++ .../gallery/layouts/BaseGalleryLayout.tsx | 7 +- .../gallery/layouts/GalleryStoryLayout.tsx | 8 + .../features/settings/tabs/FeaturesTab.tsx | 40 ++ frontend/src/i18n/locales/de.json | 81 ++- frontend/src/i18n/locales/en.json | 81 ++- .../pages/admin/event-details/OverviewTab.tsx | 9 + frontend/src/services/featureFlags.service.ts | 12 +- frontend/src/services/gallery.service.ts | 14 +- frontend/src/types/index.ts | 42 ++ ml/.dockerignore | 16 + ml/Dockerfile | 123 ++++ ml/LICENSES.md | 60 ++ ml/README.md | 149 +++++ ml/app/__init__.py | 0 ml/app/config.py | 72 +++ ml/app/main.py | 133 +++++ ml/app/pipeline.py | 313 ++++++++++ ml/app/schemas.py | 41 ++ ml/requirements.txt | 24 + ml/tests/test_api.py | 136 +++++ ml/tests/test_pipeline.py | 156 +++++ ml/tools/benchmark_threshold.py | 113 ++++ ml/tools/convert_facenet.py | 158 ++++++ ml/tools/requirements-convert.txt | 22 + 71 files changed, 8287 insertions(+), 24 deletions(-) create mode 100644 backend/__tests__/integration/faceAutoCategories.test.js create mode 100644 backend/__tests__/integration/faceClustering.test.js create mode 100644 backend/__tests__/integration/facePrivacy.test.js create mode 100644 backend/__tests__/integration/faceProcessorScaling.test.js create mode 100644 backend/__tests__/migrations/177_add_face_recognition.test.js create mode 100644 backend/migrations/core/177_add_face_recognition.js create mode 100644 backend/migrations/core/178_add_auto_categorized.js create mode 100644 backend/src/routes/adminEvents/faces.js create mode 100644 backend/src/services/faceAutoCategories.js create mode 100644 backend/src/services/faceClient.js create mode 100644 backend/src/services/faceClustering.js create mode 100644 backend/src/services/facePeopleService.js create mode 100644 backend/src/services/faceProcessor.js create mode 100644 backend/src/services/faceQueue.js create mode 100644 backend/src/services/faceSettings.js create mode 100644 frontend/scripts/i18n-faces-audit.py create mode 100644 frontend/src/components/admin/FaceRecognitionCard.tsx create mode 100644 frontend/src/components/admin/PeopleManagerModal.tsx create mode 100644 frontend/src/components/gallery/PeopleSheet.tsx create mode 100644 frontend/src/components/gallery/PeopleStrip.tsx create mode 100644 frontend/src/components/gallery/__tests__/PeopleStrip.test.tsx create mode 100644 frontend/src/components/gallery/faceCrop.ts create mode 100644 ml/.dockerignore create mode 100644 ml/Dockerfile create mode 100644 ml/LICENSES.md create mode 100644 ml/README.md create mode 100644 ml/app/__init__.py create mode 100644 ml/app/config.py create mode 100644 ml/app/main.py create mode 100644 ml/app/pipeline.py create mode 100644 ml/app/schemas.py create mode 100644 ml/requirements.txt create mode 100644 ml/tests/test_api.py create mode 100644 ml/tests/test_pipeline.py create mode 100644 ml/tools/benchmark_threshold.py create mode 100644 ml/tools/convert_facenet.py create mode 100644 ml/tools/requirements-convert.txt diff --git a/.env.example b/.env.example index cb805631..22b8d81a 100644 --- a/.env.example +++ b/.env.example @@ -223,6 +223,55 @@ LOGS=./logs # attempts is exponential: 1m, 5m, 30m, 2h, 12h. # WEBHOOK_MAX_ATTEMPTS=5 +# ----------------------------------------------------------------------------- +# Face recognition — "People in this gallery" (#1074, optional) +# ----------------------------------------------------------------------------- +# Requires the optional picpeak-ml sidecar container: +# docker compose --profile faces up -d +# +# NONE of these variables do anything until the `faces` feature flag is +# enabled in Admin → Settings, AND the per-event "Detect people in this +# gallery" toggle is switched on. Both default to OFF. With the flag off the +# backend never contacts the sidecar, so leaving these at their defaults on an +# install without the container is completely inert. +# +# Face embeddings are biometric data (GDPR Art. 9 special category in the EU). +# The photographer is the controller and needs a lawful basis for the people +# in their photos — see docs/feature-face-recognition.md before enabling. +# +# NOT AVAILABLE ON THE ALL-IN-ONE IMAGE. The single-container build sets +# PICPEAK_SINGLE_CONTAINER=true and the backend refuses to enable face +# recognition there regardless of these variables or the feature flag: that +# image runs the backend, frontend, database and every worker in one +# container, with no ML sidecar to talk to, and face detection would compete +# with image processing for the same CPU and memory. Use the standard +# multi-container deployment if you want this feature. +# +# FACE_ML_TOKEN (no default — REQUIRED to run the sidecar) +# Shared secret between the backend and the sidecar. The sidecar refuses to +# start without it rather than serving anonymously, so an accidentally +# published port is never a free face-detection API. Generate with: +# openssl rand -hex 32 +# FACE_ML_TOKEN= +# +# FACE_ML_URL (default: http://picpeak-ml:8000) +# Defaults to the sidecar's compose service name, so the standard +# deployment needs no configuration here. Only change it if you run the +# sidecar outside the default compose network. +# FACE_ML_URL=http://picpeak-ml:8000 +# +# FACE_PROCESSOR_CONCURRENCY (default: 1) +# Face-detection workers in the backend. Defaults to 1 deliberately: face +# scanning shares a host with Sharp image processing, which is the real +# memory pressure (see UPLOAD_PROCESSOR_CONCURRENCY). Raise only on hosts +# with headroom to spare. +# FACE_PROCESSOR_CONCURRENCY=1 +# +# FACE_ORT_THREADS (default: 1) +# ONNX Runtime threads inside the sidecar. More threads mean faster +# per-photo inference and higher RSS. +# FACE_ORT_THREADS=1 + # Note on FRONTEND_API_URL (documentation only): # When using pre-built frontend images, runtime env vars cannot override the built JS. # Do NOT rely on FRONTEND_API_URL in Compose. Instead, keep VITE_API_URL=/api and diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index ea8657ff..b250941f 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -979,8 +979,245 @@ jobs: if: failure() run: docker logs aio 2>&1 | tail -200 + # ----------------------------------------------------------------------------- + # ML sidecar (#1074): per-arch build, then merge into a multi-arch manifest + # ----------------------------------------------------------------------------- + # Gated on the FACENET_ONNX_URL repository *variable* (Settings → Variables, + # not Secrets — it's a public release-asset URL). While it is unset, both ML + # jobs skip and the workflow behaves exactly as it did before this feature. + # + # Why a gate at all: deepface distributes FaceNet-512 as Keras .h5 only, so + # the ONNX has to be produced once by ml/tools/convert_facenet.py and + # published as a release asset before anything can build. Converting inside + # this workflow would drag TensorFlow (~600MB) through BOTH architecture legs + # of EVERY build to produce a file that is byte-identical each time. + # + # To activate, set two repository variables: + # FACENET_ONNX_URL https://github.com/PicPeak/picpeak/releases/download//facenet512.onnx + # FACENET_ONNX_SHA256 + # See ml/README.md for producing them. + build-ml: + if: vars.FACENET_ONNX_URL != '' + strategy: + fail-fast: false + matrix: + include: + - platform: linux/amd64 + runner: ubuntu-latest + - platform: linux/arm64 + runner: ubuntu-24.04-arm + runs-on: ${{ matrix.runner }} + permissions: + contents: read + packages: write + security-events: write + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Compute image names (lowercase for GHCR) + run: | + repo_lc="${GITHUB_REPOSITORY,,}" + echo "ML_IMAGE_NAME=${repo_lc}/ml" >> "$GITHUB_ENV" + + - name: Prepare platform pair + run: | + platform="${{ matrix.platform }}" + echo "PLATFORM_PAIR=${platform//\//-}" >> "$GITHUB_ENV" + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Container Registry + if: github.event_name != 'pull_request' || github.event.inputs.push == 'true' + id: login-ghcr + continue-on-error: true + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Determine if pushing + id: push-decision + run: | + if [[ "${{ github.event_name }}" == "pull_request" && "${{ github.event.inputs.push }}" != "true" ]]; then + echo "push=false" >> "$GITHUB_OUTPUT" + elif [[ "${{ steps.login-ghcr.outcome }}" != "success" ]]; then + echo "push=false" >> "$GITHUB_OUTPUT" + else + echo "push=true" >> "$GITHUB_OUTPUT" + fi + + - name: Extract metadata for ML (labels only) + id: meta-ml + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ env.ML_IMAGE_NAME }} + labels: | + org.opencontainers.image.title=PicPeak ML + org.opencontainers.image.description=PicPeak face detection and embedding sidecar + org.opencontainers.image.vendor=PicPeak + maintainer=${{ github.repository_owner }} + + - name: Build ML image (push by digest) + id: build + uses: docker/build-push-action@v5 + with: + context: ./ml + file: ./ml/Dockerfile + platforms: ${{ matrix.platform }} + labels: ${{ steps.meta-ml.outputs.labels }} + cache-from: type=gha,scope=ml-${{ env.PLATFORM_PAIR }} + cache-to: type=gha,mode=max,scope=ml-${{ env.PLATFORM_PAIR }},ignore-error=true + outputs: ${{ steps.push-decision.outputs.push == 'true' && format('type=image,name={0}/{1},push-by-digest=true,name-canonical=true,push=true', env.REGISTRY, env.ML_IMAGE_NAME) || 'type=cacheonly' }} + build-args: | + CACHEBUST=${{ github.run_number }} + BUILD_DATE=${{ github.event.head_commit.timestamp }} + VCS_REF=${{ github.sha }} + VERSION=${{ steps.meta-ml.outputs.version }} + FACENET_ONNX_URL=${{ vars.FACENET_ONNX_URL }} + FACENET_ONNX_SHA256=${{ vars.FACENET_ONNX_SHA256 }} + + - name: Export digest + if: steps.push-decision.outputs.push == 'true' + run: | + mkdir -p /tmp/digests + digest="${{ steps.build.outputs.digest }}" + touch "/tmp/digests/${digest#sha256:}" + + - name: Upload digest artifact + if: steps.push-decision.outputs.push == 'true' + uses: actions/upload-artifact@v4 + with: + name: digests-ml-${{ env.PLATFORM_PAIR }} + path: /tmp/digests/* + if-no-files-found: error + retention-days: 1 + + # Per-arch scan by digest, same reasoning as the backend leg (#476). + # This image carries a Python/Debian dependency surface the other two + # don't, so it gets its own Security-tab category. + - name: Run Trivy vulnerability scanner (per-arch, by digest) + if: steps.push-decision.outputs.push == 'true' + uses: aquasecurity/trivy-action@v0.36.0 + env: + TRIVY_PLATFORM: ${{ matrix.platform }} + with: + image-ref: ${{ env.REGISTRY }}/${{ env.ML_IMAGE_NAME }}@${{ steps.build.outputs.digest }} + format: 'sarif' + output: 'trivy-ml-${{ env.PLATFORM_PAIR }}.sarif' + severity: 'CRITICAL,HIGH' + timeout: '10m' + + - name: Upload Trivy scan results to GitHub Security tab + if: steps.push-decision.outputs.push == 'true' + uses: github/codeql-action/upload-sarif@v4 + with: + sarif_file: 'trivy-ml-${{ env.PLATFORM_PAIR }}.sarif' + category: 'ml-vulnerabilities-${{ env.PLATFORM_PAIR }}' + + merge-ml: + needs: build-ml + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + if: vars.FACENET_ONNX_URL != '' && (github.event_name != 'pull_request' || github.event.inputs.push == 'true') + + steps: + - name: Compute image names (lowercase for GHCR) + run: | + repo_lc="${GITHUB_REPOSITORY,,}" + echo "ML_IMAGE_NAME=${repo_lc}/ml" >> "$GITHUB_ENV" + if [[ "$GITHUB_REPOSITORY" == "PicPeak/picpeak" ]]; then + echo "DOCKERHUB_ENABLED=true" >> "$GITHUB_ENV" + else + echo "DOCKERHUB_ENABLED=false" >> "$GITHUB_ENV" + fi + + - name: Download digest artifacts + uses: actions/download-artifact@v4 + with: + path: /tmp/digests + pattern: digests-ml-* + merge-multiple: true + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Container Registry + id: login-ghcr + continue-on-error: true + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Determine build context + id: context + run: | + if [[ "${{ github.ref }}" == refs/tags/v*-beta* ]] || [[ "${{ github.ref }}" == refs/heads/main ]]; then + echo "channel=beta" >> $GITHUB_OUTPUT + echo "is_prerelease=true" >> $GITHUB_OUTPUT + else + echo "channel=stable" >> $GITHUB_OUTPUT + echo "is_prerelease=false" >> $GITHUB_OUTPUT + fi + + - name: Log in to Docker Hub + if: env.DOCKERHUB_ENABLED == 'true' + uses: docker/login-action@v3 + with: + registry: docker.io + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Extract metadata for ML + id: meta-ml + uses: docker/metadata-action@v5 + with: + images: | + ${{ env.REGISTRY }}/${{ env.ML_IMAGE_NAME }} + ${{ env.DOCKERHUB_ENABLED == 'true' && 'docker.io/picpeak/ml' || '' }} + labels: | + org.opencontainers.image.title=PicPeak ML + org.opencontainers.image.description=PicPeak face detection and embedding sidecar + org.opencontainers.image.vendor=PicPeak + maintainer=${{ github.repository_owner }} + # Identical tag scheme to backend/frontend: the sidecar's API contract + # is versioned with the backend that calls it, so PICPEAK_CHANNEL + # resolves the same string across all three images. + tags: | + type=ref,event=branch + type=ref,event=pr + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}},enable=${{ steps.context.outputs.is_prerelease == 'false' }} + type=semver,pattern={{major}},enable=${{ steps.context.outputs.is_prerelease == 'false' }} + type=ref,event=tag + type=sha,format=short + type=raw,value=latest,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }} + type=raw,value=stable,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }} + + - name: Create and push multi-arch manifest + working-directory: /tmp/digests + run: | + docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \ + $(printf "${{ env.REGISTRY }}/${{ env.ML_IMAGE_NAME }}@sha256:%s " *) + + - name: Inspect manifest (GHCR) + run: | + docker buildx imagetools inspect ${{ env.REGISTRY }}/${{ env.ML_IMAGE_NAME }}:${{ steps.meta-ml.outputs.version }} + + - name: Inspect manifest (Docker Hub) + if: env.DOCKERHUB_ENABLED == 'true' + run: | + docker buildx imagetools inspect docker.io/picpeak/ml:${{ steps.meta-ml.outputs.version }} + summary: - needs: [build-backend, merge-backend, build-frontend, merge-frontend, build-aio, merge-aio, smoke-aio] + needs: [build-backend, merge-backend, build-frontend, merge-frontend, build-aio, merge-aio, smoke-aio, build-ml, merge-ml] if: always() runs-on: ubuntu-latest permissions: @@ -992,6 +1229,7 @@ jobs: repo_lc="${GITHUB_REPOSITORY,,}" echo "BACKEND_IMAGE_NAME=${repo_lc}/backend" >> "$GITHUB_ENV" echo "FRONTEND_IMAGE_NAME=${repo_lc}/frontend" >> "$GITHUB_ENV" + echo "ML_IMAGE_NAME=${repo_lc}/ml" >> "$GITHUB_ENV" echo "AIO_IMAGE_NAME=${repo_lc}/aio" >> "$GITHUB_ENV" # Mirror manifests to Docker Hub (picpeak/{backend,frontend}) only on the # canonical org repo, where the DOCKERHUB_* secrets live. Forks (and any @@ -1056,10 +1294,32 @@ jobs: echo "❌ **AIO boot smoke**: ${{ needs.smoke-aio.result }}" >> $GITHUB_STEP_SUMMARY fi + # The ML sidecar (#1074) is optional and only builds once the + # FACENET_ONNX_URL repository variable is set — "skipped" is the + # expected state, not a failure, so report it as such. + if [[ "${{ needs.build-ml.result }}" == "success" ]]; then + echo "✅ **ML sidecar build (per-arch)**: Successfully built" >> $GITHUB_STEP_SUMMARY + elif [[ "${{ needs.build-ml.result }}" == "skipped" ]]; then + echo "ℹ️ **ML sidecar build**: Skipped (FACENET_ONNX_URL repository variable not set — see ml/README.md)" >> $GITHUB_STEP_SUMMARY + else + echo "❌ **ML sidecar build (per-arch)**: ${{ needs.build-ml.result }}" >> $GITHUB_STEP_SUMMARY + fi + + if [[ "${{ needs.merge-ml.result }}" == "success" ]]; then + echo "✅ **ML sidecar manifest merge**: Successfully published" >> $GITHUB_STEP_SUMMARY + elif [[ "${{ needs.merge-ml.result }}" == "skipped" ]]; then + echo "ℹ️ **ML sidecar manifest merge**: Skipped" >> $GITHUB_STEP_SUMMARY + else + echo "❌ **ML sidecar manifest merge**: ${{ needs.merge-ml.result }}" >> $GITHUB_STEP_SUMMARY + fi + echo "" >> $GITHUB_STEP_SUMMARY echo "### 📦 Images" >> $GITHUB_STEP_SUMMARY echo "- Backend: \`${{ env.REGISTRY }}/${{ env.BACKEND_IMAGE_NAME }}\`" >> $GITHUB_STEP_SUMMARY echo "- Frontend: \`${{ env.REGISTRY }}/${{ env.FRONTEND_IMAGE_NAME }}\`" >> $GITHUB_STEP_SUMMARY + if [[ "${{ needs.merge-ml.result }}" == "success" ]]; then + echo "- ML sidecar (optional): \`${{ env.REGISTRY }}/${{ env.ML_IMAGE_NAME }}\`" >> $GITHUB_STEP_SUMMARY + fi echo "- All-in-one: \`${{ env.REGISTRY }}/${{ env.AIO_IMAGE_NAME }}\` (GHCR only — Docker Hub mirror pending)" >> $GITHUB_STEP_SUMMARY if [[ "$DOCKERHUB_ENABLED" == "true" ]]; then echo "- Backend (Docker Hub): \`docker.io/picpeak/backend\`" >> $GITHUB_STEP_SUMMARY diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 6f47727d..2164569f 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -113,3 +113,33 @@ jobs: - name: Run Vitest suite working-directory: ./frontend run: npm test -- --run + + # Optional face-detection sidecar (#1074). Runs on every PR regardless of + # whether the feature is enabled anywhere — these tests need no model + # weights (they stub the pipeline out) and cover the auth boundary, the + # request guards and the alignment geometry, which is where a mistake is a + # security problem or a silent accuracy problem rather than a visible bug. + ml: + runs-on: ubuntu-latest + timeout-minutes: 10 + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + # Matches ml/Dockerfile's base image, so a wheel that resolves here + # resolves in the image too. + python-version: '3.12' + cache: 'pip' + cache-dependency-path: ml/requirements.txt + + - name: Install ml deps + working-directory: ./ml + run: pip install -r requirements.txt pytest httpx + + - name: Run pytest suite + working-directory: ./ml + run: python -m pytest tests/ -q diff --git a/.gitignore b/.gitignore index d066ec46..a836589c 100644 --- a/.gitignore +++ b/.gitignore @@ -133,3 +133,15 @@ new-layouts/ # Backend runtime storage (generated media, previews, thumbnails, # CRM/accounting documents) — never commit backend/storage/ + +# Python artifacts — the picpeak-ml sidecar (#1074) is the only Python in +# this tree, but bytecode and virtualenvs must never be committed. +__pycache__/ +*.py[cod] +.pytest_cache/ +ml/.venv/ +ml/venv/ +# Locally produced model weights. The image fetches these by pinned URL and +# SHA-256 at build time; a 90MB blob must not end up in git history. +ml/*.onnx +ml/*.h5 diff --git a/backend/__tests__/integration/faceAutoCategories.test.js b/backend/__tests__/integration/faceAutoCategories.test.js new file mode 100644 index 00000000..f70d5a70 --- /dev/null +++ b/backend/__tests__/integration/faceAutoCategories.test.js @@ -0,0 +1,217 @@ +/** + * Auto-category rule engine (#1074 phase 3). + * + * The rules themselves are simple enough to read. What needs testing is the + * promise around them: this engine may only ever fill an EMPTY category, and + * everything it touches must be reversible. A photographer's own assignment + * is a decision; this is a heuristic, and the heuristic never wins. + */ + +const path = require('path'); +const fs = require('fs'); +const os = require('os'); + +process.env.NODE_ENV = 'test'; +process.env.TEST_DATABASE_PATH = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-autocat-')), 'db.sqlite', +); +process.env.JWT_SECRET = process.env.JWT_SECRET || 'autocat-test-secret'; + +const { bootCrmDb } = require('./helpers/crmDb'); + +let db; let cleanup; let engine; + +async function seedEvent(slug) { + const [row] = await db('events').insert({ + slug, + event_type: 'wedding', + event_name: slug, + event_date: '2026-01-01', + host_email: 'h@example.com', + admin_email: 'a@example.com', + password_hash: 'x', + share_link: `${slug}-share`, + expires_at: new Date().toISOString(), + face_recognition_enabled: true, + }).returning('id'); + return typeof row === 'object' ? row.id : row; +} + +/** A scanned photo with `faceCount` faces, each `faceSide` px square. */ +async function addScannedPhoto(eventId, faceCount, { faceSide = 400, categoryId = null } = {}) { + const [p] = await db('photos').insert({ + event_id: eventId, + filename: `${Math.random()}.jpg`, + path: '/tmp/x.jpg', + type: 'individual', + width: 1000, + height: 1000, + processing_status: 'complete', + face_status: 'done', + face_count: faceCount, + category_id: categoryId, + }).returning('id'); + const photoId = typeof p === 'object' ? p.id : p; + + for (let i = 0; i < faceCount; i++) { + await db('photo_faces').insert({ + photo_id: photoId, + event_id: eventId, + bbox_x: 10, bbox_y: 10, bbox_w: faceSide, bbox_h: faceSide, + det_score: 0.95, + model_version: 'test-v1', + created_at: new Date().toISOString(), + }); + } + return photoId; +} + +async function enable(on) { + const existing = await db('app_settings') + .where('setting_key', 'face_auto_categorize_enabled').first(); + if (existing) { + await db('app_settings') + .where('setting_key', 'face_auto_categorize_enabled') + .update({ setting_value: JSON.stringify(on) }); + } +} + +async function categoryOf(photoId) { + const photo = await db('photos').where({ id: photoId }).first(); + if (!photo.category_id) return null; + const cat = await db('photo_categories').where({ id: photo.category_id }).first(); + return cat?.slug ?? null; +} + +describe('faceAutoCategories (#1074 phase 3)', () => { + beforeAll(async () => { + ({ db, cleanup } = await bootCrmDb()); + engine = require('../../src/services/faceAutoCategories'); + await enable(true); + }, 120000); + + afterAll(async () => { if (cleanup) await cleanup(); }); + + describe('rules', () => { + it('sorts by face count, and by face size for portraits', async () => { + const eventId = await seedEvent('rules'); + // 400px face in a 1000x1000 frame = 16% of the frame, over the 8% floor. + const portrait = await addScannedPhoto(eventId, 1, { faceSide: 400 }); + const details = await addScannedPhoto(eventId, 0); + const small = await addScannedPhoto(eventId, 3); + const group = await addScannedPhoto(eventId, 9); + + await engine.categorizeEvent(eventId); + + expect(await categoryOf(details)).toBe('details'); + expect(await categoryOf(portrait)).toBe('portraits'); + expect(await categoryOf(small)).toBe('small-groups'); + expect(await categoryOf(group)).toBe('groups'); + }); + + it('does not call a distant single face a portrait', async () => { + // One person in a wide landscape is not a portrait of them. 60px in a + // 1000x1000 frame is 0.36% — far below the 8% floor. + const eventId = await seedEvent('small-face'); + const distant = await addScannedPhoto(eventId, 1, { faceSide: 60 }); + + await engine.categorizeEvent(eventId); + + expect(await categoryOf(distant)).toBeNull(); + }); + + it('ignores photos that have not been scanned', async () => { + const eventId = await seedEvent('unscanned'); + const [p] = await db('photos').insert({ + event_id: eventId, filename: 'u.jpg', path: '/tmp/u.jpg', type: 'individual', + processing_status: 'complete', face_status: 'pending', + }).returning('id'); + const photoId = typeof p === 'object' ? p.id : p; + + await engine.categorizeEvent(eventId); + expect(await categoryOf(photoId)).toBeNull(); + }); + }); + + describe('the promise', () => { + it('NEVER overwrites a category a person chose', async () => { + // The single most important behaviour in this file. + const eventId = await seedEvent('no-overwrite'); + const [c] = await db('photo_categories').insert({ + name: 'Ceremony', slug: 'ceremony', is_global: false, event_id: eventId, + created_at: new Date().toISOString(), + }).returning('id'); + const ceremonyId = typeof c === 'object' ? c.id : c; + + // 9 faces — the rules would call this "groups" if they were allowed to. + const claimed = await addScannedPhoto(eventId, 9, { categoryId: ceremonyId }); + + await engine.categorizeEvent(eventId); + + expect(await categoryOf(claimed)).toBe('ceremony'); + const row = await db('photos').where({ id: claimed }).first(); + expect(row.auto_categorized).toBeFalsy(); + }); + + it('marks only what it assigned, so undo is exact', async () => { + const eventId = await seedEvent('undo'); + const [c] = await db('photo_categories').insert({ + name: 'Ceremony', slug: 'ceremony-2', is_global: false, event_id: eventId, + created_at: new Date().toISOString(), + }).returning('id'); + const ceremonyId = typeof c === 'object' ? c.id : c; + + const manual = await addScannedPhoto(eventId, 4, { categoryId: ceremonyId }); + const auto = await addScannedPhoto(eventId, 4); + + await engine.categorizeEvent(eventId); + expect(await categoryOf(auto)).toBe('small-groups'); + + const result = await engine.undoEvent(eventId); + + expect(result.cleared).toBe(1); + // The automatic one is cleared... + expect(await categoryOf(auto)).toBeNull(); + // ...and the photographer's own choice survives untouched. + expect(await categoryOf(manual)).toBe('ceremony-2'); + }); + + it('is a no-op while the setting is off', async () => { + const eventId = await seedEvent('disabled'); + const photoId = await addScannedPhoto(eventId, 0); + + await enable(false); + const result = await engine.categorizeEvent(eventId); + await enable(true); + + expect(result.skipped).toBe(true); + expect(await categoryOf(photoId)).toBeNull(); + }); + + it('is idempotent — a second run assigns nothing new', async () => { + const eventId = await seedEvent('idempotent'); + await addScannedPhoto(eventId, 0); + await addScannedPhoto(eventId, 7); + + const first = await engine.categorizeEvent(eventId); + const second = await engine.categorizeEvent(eventId); + + expect(first.assigned).toBe(2); + expect(second.assigned).toBe(0); + }); + + it('reuses one category per slug rather than creating duplicates', async () => { + const eventId = await seedEvent('reuse'); + await addScannedPhoto(eventId, 0); + await addScannedPhoto(eventId, 0); + await addScannedPhoto(eventId, 0); + + await engine.categorizeEvent(eventId); + + const details = await db('photo_categories') + .where({ slug: 'details' }) + .where(function () { this.where('event_id', eventId).orWhere('is_global', true); }); + expect(details).toHaveLength(1); + }); + }); +}); diff --git a/backend/__tests__/integration/faceClustering.test.js b/backend/__tests__/integration/faceClustering.test.js new file mode 100644 index 00000000..75804c49 --- /dev/null +++ b/backend/__tests__/integration/faceClustering.test.js @@ -0,0 +1,325 @@ +/** + * Clustering engine (#1074). + * + * Uses synthetic embeddings with known identities rather than real faces: the + * question here is whether the ALGORITHM groups vectors correctly, which is + * separable from whether the model produces good vectors. Model quality is + * the spike's job. + */ + +const path = require('path'); +const fs = require('fs'); +const os = require('os'); + +process.env.NODE_ENV = 'test'; +process.env.TEST_DATABASE_PATH = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-faceclust-')), 'db.sqlite', +); +process.env.JWT_SECRET = process.env.JWT_SECRET || 'faceclust-test-secret'; + +const { bootCrmDb } = require('./helpers/crmDb'); + +let db; let cleanup; let clustering; + +/** Deterministic unit vector for identity `id`, jittered by `variant`. */ +function makeEmbedding(id, variant = 0, dim = 64) { + const vec = new Float32Array(dim); + for (let i = 0; i < dim; i++) { + vec[i] = Math.sin((i + 1) * (id + 1) * 0.7) + variant * 0.02 * Math.cos(i * 3.1); + } + let norm = 0; + for (let i = 0; i < dim; i++) norm += vec[i] * vec[i]; + norm = Math.sqrt(norm); + for (let i = 0; i < dim; i++) vec[i] /= norm; + return vec; +} + +async function seedEvent(slug) { + const [row] = await db('events').insert({ + slug, + event_type: 'wedding', + event_name: slug, + event_date: '2026-01-01', + host_email: 'h@example.com', + admin_email: 'a@example.com', + password_hash: 'x', + share_link: `${slug}-share`, + expires_at: new Date().toISOString(), + }).returning('id'); + return typeof row === 'object' ? row.id : row; +} + +async function insertFace(eventId, embedding, overrides = {}) { + const [p] = await db('photos').insert({ + event_id: eventId, + filename: `${Math.random()}.jpg`, + path: '/tmp/x.jpg', + type: 'individual', + }).returning('id'); + const photoId = typeof p === 'object' ? p.id : p; + + const row = { + photo_id: photoId, + event_id: eventId, + bbox_x: 0, bbox_y: 0, bbox_w: 200, bbox_h: 200, + det_score: 0.99, + embedding: clustering.packEmbedding(embedding), + model_version: 'test-v1', + created_at: new Date().toISOString(), + ...overrides, + }; + const [f] = await db('photo_faces').insert(row).returning('id'); + return { ...row, id: typeof f === 'object' ? f.id : f }; +} + +describe('faceClustering (#1074)', () => { + beforeAll(async () => { + ({ db, cleanup } = await bootCrmDb()); + clustering = require('../../src/services/faceClustering'); + }, 120000); + + afterAll(async () => { if (cleanup) await cleanup(); }); + + describe('embedding round-trip', () => { + it('survives pack/unpack through the BLOB column exactly', async () => { + const original = makeEmbedding(1); + const eventId = await seedEvent('roundtrip'); + const face = await insertFace(eventId, original); + + const stored = await db('photo_faces').where({ id: face.id }).first(); + const restored = clustering.unpackEmbedding(stored.embedding); + + expect(restored).toHaveLength(original.length); + for (let i = 0; i < original.length; i++) { + expect(restored[i]).toBeCloseTo(original[i], 6); + } + }); + + it('returns null for a corrupt blob rather than throwing', () => { + expect(clustering.unpackEmbedding(Buffer.from([1, 2, 3]))).toBeNull(); + expect(clustering.unpackEmbedding(null)).toBeNull(); + }); + }); + + describe('assignment', () => { + it('groups the same identity and separates different ones', async () => { + const eventId = await seedEvent('grouping'); + const faces = []; + // Three identities, four shots each, interleaved so assignment order + // is not conveniently grouped. + for (let variant = 0; variant < 4; variant++) { + for (const identity of [1, 2, 3]) { + faces.push(await insertFace(eventId, makeEmbedding(identity, variant))); + } + } + + await clustering.assignFaces(eventId, faces); + + const people = await db('event_people').where({ event_id: eventId }); + expect(people).toHaveLength(3); + + // Every face of one identity must share a person id. + const rows = await db('photo_faces').where({ event_id: eventId }).select('id', 'person_id'); + const byPerson = new Map(); + for (const r of rows) { + byPerson.set(r.person_id, (byPerson.get(r.person_id) || 0) + 1); + } + expect([...byPerson.values()].sort()).toEqual([4, 4, 4]); + }); + + it('leaves low-quality faces unassigned instead of spawning junk people', async () => { + const eventId = await seedEvent('quality-floor'); + const good = await insertFace(eventId, makeEmbedding(5)); + // Tiny bbox — below the 40px floor. + const tiny = await insertFace(eventId, makeEmbedding(6), { bbox_w: 12, bbox_h: 12 }); + // Weak detection score. + const weak = await insertFace(eventId, makeEmbedding(7), { det_score: 0.2 }); + + await clustering.assignFaces(eventId, [good, tiny, weak]); + + const rows = await db('photo_faces') + .whereIn('id', [good.id, tiny.id, weak.id]) + .select('id', 'person_id'); + const map = Object.fromEntries(rows.map((r) => [r.id, r.person_id])); + + expect(map[good.id]).not.toBeNull(); + // Still stored — they show in "this photo contains" — just unassigned. + expect(map[tiny.id]).toBeNull(); + expect(map[weak.id]).toBeNull(); + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(1); + }); + + it('never mixes embedding spaces from different model versions', async () => { + const eventId = await seedEvent('model-version'); + const a = await insertFace(eventId, makeEmbedding(9), { model_version: 'v1' }); + await clustering.assignFaces(eventId, [a]); + + // Same vector, different model. Comparable numerically, meaningless + // semantically — it must NOT join the v1 cluster. + const b = await insertFace(eventId, makeEmbedding(9), { model_version: 'v2' }); + await clustering.assignFaces(eventId, [b]); + + const people = await db('event_people').where({ event_id: eventId }); + expect(people).toHaveLength(2); + }); + }); + + describe('merge and split', () => { + it('merge moves every face and removes the source person', async () => { + const eventId = await seedEvent('merge'); + const f1 = await insertFace(eventId, makeEmbedding(11)); + const f2 = await insertFace(eventId, makeEmbedding(21)); + await clustering.assignFaces(eventId, [f1, f2]); + + const people = await db('event_people').where({ event_id: eventId }).orderBy('id'); + expect(people).toHaveLength(2); + + await clustering.mergePeople(eventId, [people[1].id], people[0].id); + + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(1); + const remaining = await db('event_people').where({ event_id: eventId }).first(); + expect(remaining.face_count_total).toBe(2); + const orphaned = await db('photo_faces') + .where({ event_id: eventId }).whereNull('person_id'); + expect(orphaned).toHaveLength(0); + }); + + it('split pulls the named faces into a new person', async () => { + const eventId = await seedEvent('split'); + const faces = []; + for (let v = 0; v < 4; v++) faces.push(await insertFace(eventId, makeEmbedding(13, v))); + await clustering.assignFaces(eventId, faces); + + const person = await db('event_people').where({ event_id: eventId }).first(); + expect(person.face_count_total).toBe(4); + + const newId = await clustering.splitPerson(eventId, person.id, [faces[0].id, faces[1].id]); + expect(newId).toBeTruthy(); + + const original = await db('event_people').where({ id: person.id }).first(); + const created = await db('event_people').where({ id: newId }).first(); + expect(original.face_count_total).toBe(2); + expect(created.face_count_total).toBe(2); + }); + + it('deletes a person left with no faces rather than keeping a ghost', async () => { + const eventId = await seedEvent('empty-person'); + const f = await insertFace(eventId, makeEmbedding(15)); + await clustering.assignFaces(eventId, [f]); + const person = await db('event_people').where({ event_id: eventId }).first(); + + await db('photo_faces').where({ id: f.id }).update({ person_id: null }); + await clustering.recomputeCentroid(person.id); + + expect(await db('event_people').where({ id: person.id }).first()).toBeUndefined(); + }); + }); + + describe('regressions from external review', () => { + it('merge carries a name and suppression onto the survivor', async () => { + // A merge used to move the faces and delete the source outright, so a + // photographer-entered name vanished and a person they had hidden came + // back guest-visible. + const eventId = await seedEvent('merge-metadata'); + const a = await insertFace(eventId, makeEmbedding(61)); + const b = await insertFace(eventId, makeEmbedding(62)); + await clustering.assignFaces(eventId, [a, b]); + + const [p1, p2] = await db('event_people').where({ event_id: eventId }).orderBy('id'); + // Target is unnamed and visible; the SOURCE carries the human state. + await db('event_people').where({ id: p2.id }).update({ label: 'Anna', is_hidden: true }); + + await clustering.mergePeople(eventId, [p2.id], p1.id); + + const survivor = await db('event_people').where({ id: p1.id }).first(); + expect(survivor.label).toBe('Anna'); + expect(!!survivor.is_hidden).toBe(true); + }); + + it('recluster keeps hidden/ignored on people that were never named', async () => { + // The old query remembered only rows with a label, so a suppressed + // bystander came back visible after one "Re-group people". + const eventId = await seedEvent('recluster-suppression'); + const faces = []; + for (let v = 0; v < 3; v++) faces.push(await insertFace(eventId, makeEmbedding(71, v))); + await clustering.assignFaces(eventId, faces); + + const person = await db('event_people').where({ event_id: eventId }).first(); + expect(person.label).toBeNull(); + await db('event_people').where({ id: person.id }).update({ is_ignored: true }); + + await clustering.recluster(eventId); + + const after = await db('event_people').where({ event_id: eventId }); + expect(after.length).toBeGreaterThan(0); + expect(after.every((p) => !!p.is_ignored)).toBe(true); + }); + }); + + describe('recluster', () => { + it('re-derives clusters and preserves photographer-assigned names', async () => { + // This is the property that makes re-clustering safe to offer as a + // button: without it, one click silently discards every typed name. + const eventId = await seedEvent('recluster'); + const faces = []; + for (let v = 0; v < 3; v++) { + faces.push(await insertFace(eventId, makeEmbedding(31, v))); + faces.push(await insertFace(eventId, makeEmbedding(32, v))); + } + await clustering.assignFaces(eventId, faces); + + const people = await db('event_people').where({ event_id: eventId }).orderBy('id'); + expect(people).toHaveLength(2); + await db('event_people').where({ id: people[0].id }).update({ label: 'Anna' }); + await db('event_people').where({ id: people[1].id }).update({ label: 'Ben' }); + + const count = await clustering.recluster(eventId); + expect(count).toBe(2); + + const after = await db('event_people').where({ event_id: eventId }); + const labels = after.map((p) => p.label).filter(Boolean).sort(); + expect(labels).toEqual(['Anna', 'Ben']); + }); + + it('is stable across repeated runs', async () => { + const eventId = await seedEvent('recluster-stable'); + const faces = []; + for (let v = 0; v < 3; v++) { + for (const id of [41, 42]) faces.push(await insertFace(eventId, makeEmbedding(id, v))); + } + await clustering.assignFaces(eventId, faces); + + const first = await clustering.recluster(eventId); + const second = await clustering.recluster(eventId); + expect(second).toBe(first); + }); + }); + + describe('consolidate', () => { + it('refuses to merge two people the photographer named differently', async () => { + // A human assertion this heuristic does not get to overrule. + const eventId = await seedEvent('consolidate-labels'); + const a = await insertFace(eventId, makeEmbedding(51)); + await clustering.assignFaces(eventId, [a]); + const first = await db('event_people').where({ event_id: eventId }).first(); + + // A near-identical centroid that would otherwise merge. + const [inserted] = await db('event_people').insert({ + event_id: eventId, + centroid: clustering.packEmbedding(makeEmbedding(51, 0.01)), + face_count_total: 1, + model_version: 'test-v1', + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }).returning('id'); + const secondId = typeof inserted === 'object' ? inserted.id : inserted; + + await db('event_people').where({ id: first.id }).update({ label: 'Anna' }); + await db('event_people').where({ id: secondId }).update({ label: 'Ben' }); + + await clustering.consolidate(eventId); + + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(2); + }); + }); +}); diff --git a/backend/__tests__/integration/facePrivacy.test.js b/backend/__tests__/integration/facePrivacy.test.js new file mode 100644 index 00000000..d0a97a25 --- /dev/null +++ b/backend/__tests__/integration/facePrivacy.test.js @@ -0,0 +1,339 @@ +/** + * Privacy and visibility guarantees for face recognition (#1074). + * + * These are the tests that matter most in this feature. Two of them cover + * defects that would be invisible in normal use: + * + * - The people strip is computed from face rows, which have no concept of + * photo visibility. Handing a guest a raw count leaks how many hidden + * photos someone appears in, and a cover face picked without scoping + * renders a crop of a photo the guest may not open. + * + * - Face embeddings are biometric data. They must not ride along in a + * .picpeak export, which gets handed to clients and moved between + * operators. + */ + +const path = require('path'); +const fs = require('fs'); +const os = require('os'); + +process.env.NODE_ENV = 'test'; +process.env.TEST_DATABASE_PATH = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-faceprivacy-')), 'db.sqlite', +); +process.env.JWT_SECRET = process.env.JWT_SECRET || 'faceprivacy-test-secret'; + +const { bootCrmDb } = require('./helpers/crmDb'); + +let db; let cleanup; let clustering; let peopleService; let faceProcessor; + +function makeEmbedding(id, variant = 0, dim = 64) { + const vec = new Float32Array(dim); + for (let i = 0; i < dim; i++) { + vec[i] = Math.sin((i + 1) * (id + 1) * 0.7) + variant * 0.02 * Math.cos(i * 3.1); + } + let norm = 0; + for (let i = 0; i < dim; i++) norm += vec[i] * vec[i]; + norm = Math.sqrt(norm); + for (let i = 0; i < dim; i++) vec[i] /= norm; + return vec; +} + +async function seedEvent(slug) { + const [row] = await db('events').insert({ + slug, + event_type: 'wedding', + event_name: slug, + event_date: '2026-01-01', + host_email: 'h@example.com', + admin_email: 'a@example.com', + password_hash: 'x', + share_link: `${slug}-share`, + expires_at: new Date().toISOString(), + face_recognition_enabled: true, + }).returning('id'); + return typeof row === 'object' ? row.id : row; +} + +async function addPhotoWithFace(eventId, embedding, { visibility = 'visible', score = 0.99 } = {}) { + const [p] = await db('photos').insert({ + event_id: eventId, + filename: `${Math.random()}.jpg`, + path: '/tmp/x.jpg', + type: 'individual', + visibility, + processing_status: 'complete', + }).returning('id'); + const photoId = typeof p === 'object' ? p.id : p; + + const row = { + photo_id: photoId, + event_id: eventId, + bbox_x: 0, bbox_y: 0, bbox_w: 200, bbox_h: 200, + det_score: score, + embedding: clustering.packEmbedding(embedding), + model_version: 'test-v1', + created_at: new Date().toISOString(), + }; + const [f] = await db('photo_faces').insert(row).returning('id'); + return { photoId, face: { ...row, id: typeof f === 'object' ? f.id : f } }; +} + +describe('face privacy and visibility (#1074)', () => { + beforeAll(async () => { + ({ db, cleanup } = await bootCrmDb()); + clustering = require('../../src/services/faceClustering'); + peopleService = require('../../src/services/facePeopleService'); + faceProcessor = require('../../src/services/faceProcessor'); + }, 120000); + + afterAll(async () => { if (cleanup) await cleanup(); }); + + describe('visibility scoping', () => { + it('counts only photos the audience can actually see', async () => { + const eventId = await seedEvent('visibility-count'); + const faces = []; + // Same person: 3 visible photos, 4 hidden ones. + for (let v = 0; v < 3; v++) { + faces.push((await addPhotoWithFace(eventId, makeEmbedding(1, v))).face); + } + for (let v = 3; v < 7; v++) { + faces.push((await addPhotoWithFace(eventId, makeEmbedding(1, v), { visibility: 'hidden' })).face); + } + await clustering.assignFaces(eventId, faces); + + const guestView = await peopleService.listPeople(eventId, { isClient: false, minClusterSize: 1 }); + const clientView = await peopleService.listPeople(eventId, { isClient: true, minClusterSize: 1 }); + + expect(guestView).toHaveLength(1); + // The leak this test exists to prevent: 3, never 7. + expect(guestView[0].face_count).toBe(3); + expect(clientView[0].face_count).toBe(7); + }); + + it('never returns face_count_total to a guest', async () => { + const eventId = await seedEvent('no-total-leak'); + const { face } = await addPhotoWithFace(eventId, makeEmbedding(2)); + await clustering.assignFaces(eventId, [face]); + + const [person] = await peopleService.listPeople(eventId, { isClient: false, minClusterSize: 1 }); + expect(person).not.toHaveProperty('total_face_count'); + expect(person).not.toHaveProperty('is_hidden'); + }); + + it('picks a cover face from a photo the guest may open', async () => { + const eventId = await seedEvent('cover-scoping'); + // The BEST face (highest score) is in a hidden photo — a naive + // implementation would hand its crop to the guest. + const hidden = await addPhotoWithFace(eventId, makeEmbedding(3, 0), { + visibility: 'hidden', score: 0.99, + }); + const visible = await addPhotoWithFace(eventId, makeEmbedding(3, 1), { + visibility: 'visible', score: 0.80, + }); + await clustering.assignFaces(eventId, [hidden.face, visible.face]); + + const [guestPerson] = await peopleService.listPeople(eventId, { isClient: false, minClusterSize: 1 }); + expect(guestPerson.cover.photo_id).toBe(visible.photoId); + expect(guestPerson.cover.photo_id).not.toBe(hidden.photoId); + }); + + it('drops a person entirely when all their photos are hidden', async () => { + const eventId = await seedEvent('all-hidden'); + const faces = []; + for (let v = 0; v < 3; v++) { + faces.push((await addPhotoWithFace(eventId, makeEmbedding(4, v), { visibility: 'hidden' })).face); + } + await clustering.assignFaces(eventId, faces); + + const guestView = await peopleService.listPeople(eventId, { isClient: false, minClusterSize: 1 }); + expect(guestView).toHaveLength(0); + const clientView = await peopleService.listPeople(eventId, { isClient: true, minClusterSize: 1 }); + expect(clientView).toHaveLength(1); + }); + + it('omits hidden and ignored people from the guest response', async () => { + const eventId = await seedEvent('hidden-people'); + const a = (await addPhotoWithFace(eventId, makeEmbedding(5))).face; + const b = (await addPhotoWithFace(eventId, makeEmbedding(6))).face; + await clustering.assignFaces(eventId, [a, b]); + + const people = await db('event_people').where({ event_id: eventId }).orderBy('id'); + await db('event_people').where({ id: people[0].id }).update({ is_hidden: true }); + await db('event_people').where({ id: people[1].id }).update({ is_ignored: true }); + + const guestView = await peopleService.listPeople(eventId, { isClient: false, minClusterSize: 1 }); + expect(guestView).toHaveLength(0); + const adminView = await peopleService.listPeople(eventId, { isClient: true, forAdmin: true }); + expect(adminView).toHaveLength(2); + }); + + it('does not attach a hidden person to a photo a guest can see', async () => { + const eventId = await seedEvent('person-ids-hidden'); + const { photoId, face } = await addPhotoWithFace(eventId, makeEmbedding(7)); + await clustering.assignFaces(eventId, [face]); + const person = await db('event_people').where({ event_id: eventId }).first(); + await db('event_people').where({ id: person.id }).update({ is_hidden: true }); + + const guestMap = await peopleService.getPersonIdsByPhoto(eventId, [photoId], { forAdmin: false }); + expect(guestMap.get(photoId)).toBeUndefined(); + + const adminMap = await peopleService.getPersonIdsByPhoto(eventId, [photoId], { forAdmin: true }); + expect(adminMap.get(photoId)).toEqual([person.id]); + }); + + it('respects the minimum cluster size so one-off bystanders stay out', async () => { + const eventId = await seedEvent('min-cluster'); + const solo = (await addPhotoWithFace(eventId, makeEmbedding(8))).face; + const crowd = []; + for (let v = 0; v < 4; v++) { + crowd.push((await addPhotoWithFace(eventId, makeEmbedding(9, v))).face); + } + await clustering.assignFaces(eventId, [solo, ...crowd]); + + const people = await peopleService.listPeople(eventId, { isClient: false, minClusterSize: 3 }); + expect(people).toHaveLength(1); + expect(people[0].face_count).toBe(4); + }); + }); + + describe('erasure', () => { + it('purgeEvent removes every face row and resets the photos', async () => { + const eventId = await seedEvent('purge'); + const faces = []; + for (let v = 0; v < 3; v++) { + faces.push((await addPhotoWithFace(eventId, makeEmbedding(10, v))).face); + } + await clustering.assignFaces(eventId, faces); + await db('photos').where({ event_id: eventId }).update({ face_status: 'done', face_count: 1 }); + + expect(await db('photo_faces').where({ event_id: eventId })).not.toHaveLength(0); + expect(await db('event_people').where({ event_id: eventId })).not.toHaveLength(0); + + await faceProcessor.purgeEvent(eventId); + + expect(await db('photo_faces').where({ event_id: eventId })).toHaveLength(0); + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(0); + const photos = await db('photos').where({ event_id: eventId }); + expect(photos.every((p) => p.face_status === null && p.face_count === null)).toBe(true); + }); + + it('purgePhotoFaces removes face rows WITHOUT relying on the FK cascade', async () => { + // The regression this guards: PicPeak does not enable + // `PRAGMA foreign_keys` on SQLite, so ON DELETE CASCADE never fires + // there and biometric embeddings outlived the photo. The pragma is + // explicitly OFF here so the assertion can only pass if the deletion + // path purges the rows itself. + await db.raw('PRAGMA foreign_keys = OFF'); + + const eventId = await seedEvent('purge-no-cascade'); + const faces = []; + for (let v = 0; v < 3; v++) { + faces.push((await addPhotoWithFace(eventId, makeEmbedding(20, v))).face); + } + await clustering.assignFaces(eventId, faces); + + const person = await db('event_people').where({ event_id: eventId }).first(); + expect(person.face_count_total).toBe(3); + + const victim = faces[0]; + await faceProcessor.purgePhotoFaces(victim.photo_id); + + expect(await db('photo_faces').where({ photo_id: victim.photo_id })).toHaveLength(0); + // …and the person it belonged to was rebuilt, not left with a stale count. + const after = await db('event_people').where({ id: person.id }).first(); + expect(after.face_count_total).toBe(2); + }); + + it('purging the last face of a person removes the person too', async () => { + await db.raw('PRAGMA foreign_keys = OFF'); + const eventId = await seedEvent('purge-last-face'); + const { face, photoId } = await addPhotoWithFace(eventId, makeEmbedding(21)); + await clustering.assignFaces(eventId, [face]); + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(1); + + await faceProcessor.purgePhotoFaces(photoId); + + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(0); + }); + + it('deleting an event removes its people and faces', async () => { + await db.raw('PRAGMA foreign_keys = ON'); + const eventId = await seedEvent('event-delete'); + const { face } = await addPhotoWithFace(eventId, makeEmbedding(11)); + await clustering.assignFaces(eventId, [face]); + + await db('photos').where({ event_id: eventId }).del(); + await db('events').where({ id: eventId }).del(); + + expect(await db('photo_faces').where({ event_id: eventId })).toHaveLength(0); + expect(await db('event_people').where({ event_id: eventId })).toHaveLength(0); + }); + }); + + describe('all-in-one image block (#1042 / PR #1068)', () => { + // Blocked for performance: the AIO image runs backend, frontend, SQLite + // and every worker in one container, with no ML sidecar to talk to. The + // failure there would not be loud — just a slow install that looks + // broken — so the gate is asserted rather than assumed. + const faceSettings = require('../../src/services/faceSettings'); + + afterEach(() => { delete process.env.PICPEAK_SINGLE_CONTAINER; }); + + it('reports the feature off regardless of the flag row', async () => { + process.env.PICPEAK_SINGLE_CONTAINER = 'true'; + expect(faceSettings.isSingleContainerImage()).toBe(true); + // Even with the flag ON in the database. + await db('feature_flags').insert({ key: 'faces', value: true }) + .onConflict('key').merge() + .catch(async () => { + await db('feature_flags').where({ key: 'faces' }).update({ value: true }); + }); + expect(await faceSettings.isFeatureEnabled()).toBe(false); + }); + + it('refuses per-event detection too', async () => { + process.env.PICPEAK_SINGLE_CONTAINER = 'true'; + const eventId = await seedEvent('aio-block'); + const event = await db('events').where({ id: eventId }).first(); + expect(event.face_recognition_enabled).toBeTruthy(); + expect(await faceSettings.isEnabledForEvent(event)).toBe(false); + }); + + it('accepts only explicit truthy markers', () => { + for (const v of ['true', '1', 'yes', 'TRUE']) { + process.env.PICPEAK_SINGLE_CONTAINER = v; + expect(faceSettings.isSingleContainerImage()).toBe(true); + } + for (const v of ['false', '0', '', 'no']) { + process.env.PICPEAK_SINGLE_CONTAINER = v; + expect(faceSettings.isSingleContainerImage()).toBe(false); + } + delete process.env.PICPEAK_SINGLE_CONTAINER; + expect(faceSettings.isSingleContainerImage()).toBe(false); + }); + }); + + describe('export and backup exclusion', () => { + it('excludes both face tables from .picpeak exports', () => { + const { EXCLUDED_TABLES } = require('../../src/services/picpeakExportService'); + expect(EXCLUDED_TABLES.has('photo_faces')).toBe(true); + expect(EXCLUDED_TABLES.has('event_people')).toBe(true); + }); + + it('excludes both face tables from the database backup table list', async () => { + const databaseBackup = require('../../src/services/databaseBackup'); + const service = databaseBackup.DatabaseBackupService + ? new databaseBackup.DatabaseBackupService() + : databaseBackup; + if (typeof service.getTables !== 'function') return; // shape differs; covered by the export test + + const tables = await service.getTables(); + expect(tables).not.toContain('photo_faces'); + expect(tables).not.toContain('event_people'); + // Sanity: the filter didn't eat everything. + expect(tables).toContain('events'); + }); + }); +}); diff --git a/backend/__tests__/integration/faceProcessorScaling.test.js b/backend/__tests__/integration/faceProcessorScaling.test.js new file mode 100644 index 00000000..a0853820 --- /dev/null +++ b/backend/__tests__/integration/faceProcessorScaling.test.js @@ -0,0 +1,146 @@ +/** + * Bounding-box coordinate space (#1074). + * + * The sidecar reports boxes in the pixel space of the image it was HANDED — + * the ≤1920px preview — while every consumer (the strip's avatar crop, the + * admin manager, the auto-category portrait rule) compares them against + * photos.width/height, the ORIGINAL dimensions. faceProcessor scales once so + * everything downstream can assume original-image coordinates. + * + * This is the defect that survived longest in review, and it is invisible on + * any photo already under 1920px — the entire demo gallery was 750px, so the + * scale factor was always exactly 1.0 and the correction never ran. Verified + * by hand afterwards on a real 4000x3000 upload (stored box moved from + * 1493,204 to 3110,426 — a factor of 2.083, exactly 4000/1920). This test + * exists so that verification does not have to be repeated by hand. + */ + +const path = require('path'); +const fs = require('fs'); +const os = require('os'); + +process.env.NODE_ENV = 'test'; +process.env.TEST_DATABASE_PATH = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-facescale-')), 'db.sqlite', +); +process.env.JWT_SECRET = process.env.JWT_SECRET || 'facescale-test-secret'; + +// A 1920x1440 JPEG standing in for the preview rendition. faceProcessor reads +// its dimensions with sharp to derive the scale, so it must be a real image. +const sharp = require('sharp'); + +let mockPreviewBuffer; +const mockSidecarBox = [1493, 204, 131, 161]; // what the sidecar sees on the preview + +jest.mock('../../src/services/imageProcessor', () => ({ + ...jest.requireActual('../../src/services/imageProcessor'), + ensurePreviewImage: jest.fn(async () => 'previews/preview_test.jpg'), +})); + +jest.mock('../../src/services/storage', () => ({ + getStorage: () => ({ get: async () => mockPreviewBuffer }), +})); + +jest.mock('../../src/services/faceClient', () => ({ + detectFaces: jest.fn(async () => ({ + model_version: 'test-v1', + faces: [{ + bbox: mockSidecarBox, + score: 0.99, + landmarks: [[0, 0], [0, 0], [0, 0], [0, 0], [0, 0]], + yaw: 0, pitch: 0, blur: 500, + embedding: Array.from({ length: 64 }, (_, i) => (i === 0 ? 1 : 0)), + }], + })), + SidecarUnavailableError: class extends Error {}, +})); + +const { bootCrmDb } = require('./helpers/crmDb'); + +let db; let cleanup; let faceProcessor; + +async function seedPhoto(width, height) { + const [e] = await db('events').insert({ + slug: `scale-${width}-${Math.random().toString(36).slice(2, 8)}`, + event_type: 'wedding', + event_name: 'scale', + event_date: '2026-01-01', + host_email: 'h@example.com', + admin_email: 'a@example.com', + password_hash: 'x', + share_link: `scale-${Math.random()}`, + expires_at: new Date().toISOString(), + face_recognition_enabled: true, + }).returning('id'); + const eventId = typeof e === 'object' ? e.id : e; + + const [p] = await db('photos').insert({ + event_id: eventId, + filename: 'big.jpg', + path: '/tmp/big.jpg', + type: 'individual', + width, + height, + processing_status: 'complete', + face_status: 'processing', + }).returning('id'); + return { eventId, photoId: typeof p === 'object' ? p.id : p }; +} + +describe('face bbox coordinate space (#1074)', () => { + beforeAll(async () => { + mockPreviewBuffer = await sharp({ + create: { width: 1920, height: 1440, channels: 3, background: { r: 20, g: 40, b: 80 } }, + }).jpeg().toBuffer(); + + ({ db, cleanup } = await bootCrmDb()); + // The faces flag gates everything; turn it on for this suite. + await db('feature_flags').insert({ key: 'faces', value: true }) + .onConflict('key').merge() + .catch(async () => { await db('feature_flags').where({ key: 'faces' }).update({ value: true }); }); + faceProcessor = require('../../src/services/faceProcessor'); + }, 120000); + + afterAll(async () => { if (cleanup) await cleanup(); }); + + it('scales preview-space boxes into ORIGINAL image coordinates', async () => { + // 4000px original, 1920px preview -> every coordinate must grow by 4000/1920. + const { photoId } = await seedPhoto(4000, 3000); + await faceProcessor.processPhotoFaces(photoId); + + const face = await db('photo_faces').where({ photo_id: photoId }).first(); + const scale = 4000 / 1920; + + expect(face.bbox_x).toBeCloseTo(mockSidecarBox[0] * scale, 1); + expect(face.bbox_y).toBeCloseTo(mockSidecarBox[1] * scale, 1); + expect(face.bbox_w).toBeCloseTo(mockSidecarBox[2] * scale, 1); + expect(face.bbox_h).toBeCloseTo(mockSidecarBox[3] * scale, 1); + + // The regression this guards: the raw preview-space value being stored. + expect(face.bbox_x).not.toBeCloseTo(mockSidecarBox[0], 1); + // And a sanity check that it lands inside the original frame. + expect(face.bbox_x + face.bbox_w).toBeLessThanOrEqual(4000); + }); + + it('leaves boxes untouched when the photo is already preview-sized', async () => { + // The case that hid the bug: no downscale, so scale is exactly 1 and the + // stored box equals what the sidecar reported. + const { photoId } = await seedPhoto(1920, 1440); + await faceProcessor.processPhotoFaces(photoId); + + const face = await db('photo_faces').where({ photo_id: photoId }).first(); + expect(face.bbox_x).toBeCloseTo(mockSidecarBox[0], 1); + expect(face.bbox_w).toBeCloseTo(mockSidecarBox[2], 1); + }); + + it('falls back to unscaled rather than corrupting when width is unknown', async () => { + // Pre-dimension-migration rows have no width. Storing a box scaled by + // NaN/0 would be worse than storing an unscaled one. + const { photoId } = await seedPhoto(null, null); + await faceProcessor.processPhotoFaces(photoId); + + const face = await db('photo_faces').where({ photo_id: photoId }).first(); + expect(Number.isFinite(face.bbox_x)).toBe(true); + expect(face.bbox_x).toBeCloseTo(mockSidecarBox[0], 1); + }); +}); diff --git a/backend/__tests__/migrations/177_add_face_recognition.test.js b/backend/__tests__/migrations/177_add_face_recognition.test.js new file mode 100644 index 00000000..b2ac443f --- /dev/null +++ b/backend/__tests__/migrations/177_add_face_recognition.test.js @@ -0,0 +1,166 @@ +/** + * Migration 177 (#1074) — face recognition schema. + * + * The acceptance criteria for #1074 name three properties explicitly, so + * they get tests rather than a manual check: + * + * - idempotent on re-run, + * - a working down(), + * - and — the one that matters most — installing it must NOT enqueue + * anything. A `face_status` column defaulting to 'pending' would put + * every existing photo on every install into a queue the operator never + * asked for, on installs with no sidecar at all. + */ + +const path = require('path'); +const fs = require('fs'); +const os = require('os'); + +process.env.NODE_ENV = 'test'; +process.env.TEST_DATABASE_PATH = path.join( + fs.mkdtempSync(path.join(os.tmpdir(), 'picpeak-mig177-')), 'db.sqlite', +); +process.env.JWT_SECRET = process.env.JWT_SECRET || 'mig177-test-secret'; + +const { bootCrmDb } = require('../integration/helpers/crmDb'); +const migration = require('../../migrations/core/177_add_face_recognition'); + +describe('migration 177 — face recognition schema', () => { + let db; let cleanup; + + beforeAll(async () => { + ({ db, cleanup } = await bootCrmDb()); + }, 120000); + + afterAll(async () => { if (cleanup) await cleanup(); }); + + it('creates both tables with the columns the pipeline writes', async () => { + expect(await db.schema.hasTable('photo_faces')).toBe(true); + expect(await db.schema.hasTable('event_people')).toBe(true); + + for (const col of [ + 'photo_id', 'event_id', 'bbox_x', 'bbox_y', 'bbox_w', 'bbox_h', + 'det_score', 'yaw', 'pitch', 'blur', 'embedding', 'model_version', + 'person_id', 'created_at', + ]) { + expect(await db.schema.hasColumn('photo_faces', col)).toBe(true); + } + + for (const col of [ + 'event_id', 'label', 'cover_face_id', 'centroid', 'face_count_total', + 'model_version', 'is_hidden', 'is_ignored', + ]) { + expect(await db.schema.hasColumn('event_people', col)).toBe(true); + } + }); + + it('adds the photos and events columns', async () => { + for (const col of ['face_status', 'face_count', 'face_started_at', 'face_error']) { + expect(await db.schema.hasColumn('photos', col)).toBe(true); + } + for (const col of [ + 'face_recognition_enabled', 'faces_visible_to_guests', 'faces_last_scan_at', + ]) { + expect(await db.schema.hasColumn('events', col)).toBe(true); + } + }); + + it('enqueues nothing — face_status has no default', async () => { + // The whole "zero behaviour change by default" guarantee rests on this. + const [{ id: eventId }] = await db('events').insert({ + slug: 'mig177-event', + event_type: 'wedding', + event_name: 'Migration 177', + event_date: '2026-01-01', + host_email: 'h@example.com', + admin_email: 'a@example.com', + password_hash: 'x', + share_link: 'mig177-share', + expires_at: new Date().toISOString(), + }).returning('id'); + + const eid = typeof eventId === 'object' ? eventId.id : eventId; + await db('photos').insert({ + event_id: eid, filename: 'a.jpg', path: '/tmp/a.jpg', type: 'individual', + }); + + const row = await db('photos').where({ event_id: eid }).first(); + expect(row.face_status).toBeNull(); + expect(await db('photo_faces').count({ c: '*' }).first()).toMatchObject({ c: 0 }); + }); + + it('seeds the tunable thresholds rather than hardcoding them', async () => { + // Immich's clustering guide exists because no single threshold survives + // contact with every library — these must be operator-reachable. + const keys = [ + 'face_match_threshold', 'face_min_cluster_size', + 'face_quality_min_score', 'face_quality_min_px', + ]; + const rows = await db('app_settings').whereIn('setting_key', keys); + expect(rows).toHaveLength(keys.length); + expect(rows.every((r) => r.setting_type === 'faces')).toBe(true); + }); + + it('is idempotent on re-run', async () => { + await expect(migration.up(db)).resolves.not.toThrow(); + // And did not duplicate the settings rows. + const rows = await db('app_settings').where('setting_key', 'face_match_threshold'); + expect(rows).toHaveLength(1); + }); + + it('down() removes everything it added, and up() restores it', async () => { + await migration.down(db); + + expect(await db.schema.hasTable('photo_faces')).toBe(false); + expect(await db.schema.hasTable('event_people')).toBe(false); + expect(await db.schema.hasColumn('photos', 'face_status')).toBe(false); + expect(await db.schema.hasColumn('events', 'face_recognition_enabled')).toBe(false); + expect(await db('app_settings').where('setting_key', 'face_match_threshold')).toHaveLength(0); + + await migration.up(db); + expect(await db.schema.hasTable('photo_faces')).toBe(true); + expect(await db.schema.hasColumn('photos', 'face_status')).toBe(true); + }); + + it('cascades face rows when a photo is deleted', async () => { + // #1074 acceptance criterion: deleting a photo removes its face rows. + // + // SQLite ignores foreign keys unless the pragma is on, and PicPeak does + // NOT enable it globally (a large amount of existing data and fixtures + // would start failing). So the cascade below proves only that the schema + // declares it correctly — the code does not RELY on it. Deletion paths + // purge face rows explicitly; see faceProcessor.purgeEvent / + // purgePhotoFaces and the erasure tests in facePrivacy.test.js. + await db.raw('PRAGMA foreign_keys = ON'); + + const [{ id: eventId }] = await db('events').insert({ + slug: 'mig177-cascade', + event_type: 'wedding', + event_name: 'Cascade', + event_date: '2026-01-01', + host_email: 'h@example.com', + admin_email: 'a@example.com', + password_hash: 'x', + share_link: 'mig177-cascade-share', + expires_at: new Date().toISOString(), + }).returning('id'); + const eid = typeof eventId === 'object' ? eventId.id : eventId; + + const [{ id: photoId }] = await db('photos') + .insert({ event_id: eid, filename: 'c.jpg', path: '/tmp/c.jpg', type: 'individual' }) + .returning('id'); + const pid = typeof photoId === 'object' ? photoId.id : photoId; + + await db('photo_faces').insert({ + photo_id: pid, + event_id: eid, + bbox_x: 1, bbox_y: 2, bbox_w: 3, bbox_h: 4, + model_version: 'test', + created_at: new Date().toISOString(), + }); + expect(await db('photo_faces').where({ photo_id: pid })).toHaveLength(1); + + await db('photos').where({ id: pid }).del(); + expect(await db('photo_faces').where({ photo_id: pid })).toHaveLength(0); + }); +}); diff --git a/backend/migrations/core/177_add_face_recognition.js b/backend/migrations/core/177_add_face_recognition.js new file mode 100644 index 00000000..cb26c785 --- /dev/null +++ b/backend/migrations/core/177_add_face_recognition.js @@ -0,0 +1,268 @@ +/** + * Migration 177: Face recognition — "People in this gallery" (#1074). + * + * Adds two tables and four columns for per-gallery face detection and + * clustering. People are clustered INSIDE one event; there is deliberately no + * cross-event person identity, so `event_people` hangs off `events` and every + * query is naturally scoped to one gallery. + * + * Three decisions worth knowing before reading the schema: + * + * 1. Embeddings are a plain binary column, not a vector type. 512 float32 = + * 2KB per face, and a 2,000-photo wedding at ~3 faces/photo is ~12MB — + * small enough that no pgvector extension is needed and the SQLite path + * behaves identically. Clustering runs in JS over hundreds to low + * thousands of rows per event. + * + * 2. `photos.face_status` is NULLABLE with no default, and this migration + * enqueues NOTHING. A default of 'pending' would silently queue every + * photo on every install the moment this ran — including installs with + * no sidecar and no intention of using the feature. Rows are enqueued + * only by an explicit per-event opt-in. + * + * 3. `event_people.face_count_total` is named for what it is: a count over + * ALL faces, including photos a guest cannot see. Guests are restricted + * to `photos.visibility = 'visible'` (gallery.js), so a guest-facing + * count and cover face MUST be computed per request against that same + * predicate. The name is a deterrent: anything reading + * `face_count_total` in a guest path is a bug. + * + * Face embeddings are biometric data — GDPR Art. 9 special category. Both + * tables are excluded from backups and from .picpeak exports (see + * picpeakExportService and databaseBackup): the data is derived, so a restore + * re-scans rather than carrying biometrics between deployments. + */ + +const GLOBAL_DEFAULTS = [ + // Cosine similarity above which a face joins an existing person. + // + // 0.60, measured twice — and the second measurement overruled the first, + // which is the whole reason this comment is long. + // + // LFW's standard 1000-pair protocol through this exact pipeline gives + // same-person cosine 0.696 +/- 0.142, different-person 0.085 +/- 0.167, + // peak accuracy 96.6% at 0.405. Reading that as a clustering threshold + // suggests ~0.50 (1.0% false merge, 8.2% false split pairwise). + // + // Running it on an actual gallery says otherwise. At 0.50, three of six + // visible clusters were contaminated — two different people merged into + // one strip entry. PAIRWISE ERROR RATES DO NOT PREDICT CLUSTER PURITY: + // greedy assignment compounds, because one wrong face drags the centroid + // toward the midpoint between two identities, which makes the next wrong + // face likelier. A 1% pairwise false-merge rate is not a 1% chance of a + // clean gallery. + // + // Sweep on a 61-photo / 5-identity gallery (ml/tools/benchmark_threshold.py + // covers the pairwise half; the cluster half is the `recluster` endpoint): + // 0.50 -> 6 clusters, 3 contaminated + // 0.56 -> 6 clusters, 0 contaminated + // 0.60 -> 5 clusters, 0 contaminated <- ground truth is 5 + // 0.64 -> 5 clusters, 0 contaminated, fewer faces assigned + // + // 0.60 recovers the exact right number of people with no contamination. + // Higher only drops coverage. A false merge puts a stranger into someone's + // "download my photos" and cannot be undone until the Phase 2 merge/split + // UI ships, so contamination is the constraint being optimised against, + // not accuracy. + ['face_match_threshold', 0.60], + // Faces a cluster needs before it appears in the guest-facing strip. Keeps + // one-off bystanders out of "People in this gallery". + ['face_min_cluster_size', 3], + // Quality floor. Faces below any of these are still stored (so "this photo + // contains" stays accurate) but are left unassigned, so they cannot spawn + // junk people. + ['face_quality_min_score', 0.7], + ['face_quality_min_px', 40], + // Phase 3 rule engine. Present here so the settings block has one home. + ['face_auto_categorize_enabled', false], +]; + +exports.up = async function (knex) { + // --- event_people ------------------------------------------------------- + // Created before photo_faces: photo_faces.person_id references it. The + // reverse direction (event_people.cover_face_id → photo_faces.id) is a + // forward reference and is deliberately left WITHOUT a foreign key — + // Postgres rejects an FK to a table that doesn't exist yet, and adding it + // afterwards buys nothing here because the column is nullable and the + // clustering code always writes an id it just inserted. Migration 001 hit + // exactly this with events.hero_photo_id (#484). + if (!(await knex.schema.hasTable('event_people'))) { + await knex.schema.createTable('event_people', (table) => { + table.increments('id').primary(); + table.integer('event_id').unsigned().notNullable() + .references('id').inTable('events').onDelete('CASCADE'); + // NULL until a photographer names them. The UI shows a photo count + // instead — a number is honest, an invented "Person 7" is not. + table.string('label', 255); + table.integer('cover_face_id'); + // Running-mean centroid of the cluster's embeddings, 512 float32. + table.binary('centroid'); + // Count over ALL faces — see note 3 in the header. NEVER guest-facing. + table.integer('face_count_total').notNullable().defaultTo(0); + table.string('model_version', 64); + // Photographer-only: hidden people never reach a guest response. + table.boolean('is_hidden').notNullable().defaultTo(false); + // Bystanders and false positives — excluded from the strip entirely. + table.boolean('is_ignored').notNullable().defaultTo(false); + table.timestamp('created_at').defaultTo(knex.fn.now()); + table.timestamp('updated_at').defaultTo(knex.fn.now()); + table.index(['event_id'], 'event_people_event_idx'); + }); + } + + // --- photo_faces -------------------------------------------------------- + if (!(await knex.schema.hasTable('photo_faces'))) { + await knex.schema.createTable('photo_faces', (table) => { + table.increments('id').primary(); + table.integer('photo_id').unsigned().notNullable() + .references('id').inTable('photos').onDelete('CASCADE'); + // Denormalized from photos.event_id. Every clustering and people query + // is per-event; without this they would all join through photos. + table.integer('event_id').unsigned().notNullable() + .references('id').inTable('events').onDelete('CASCADE'); + + // Bounding box in ORIGINAL image pixels (the sidecar reports original + // coordinates even though it detects on a downscaled copy), so cover + // avatars can be cropped from any rendition by ratio. + table.float('bbox_x').notNullable(); + table.float('bbox_y').notNullable(); + table.float('bbox_w').notNullable(); + table.float('bbox_h').notNullable(); + + table.float('det_score'); + table.float('yaw'); + table.float('pitch'); + table.float('blur'); + + // 512 × float32, L2-normalized. See note 1. + table.binary('embedding'); + // Which detector+embedder+alignment produced this. Embeddings from two + // pipelines are not comparable, so a model change re-derives rather + // than silently mixing two spaces. + table.string('model_version', 64); + + // SET NULL rather than CASCADE: deleting a person must un-assign its + // faces, never delete the detections themselves. + table.integer('person_id').unsigned() + .references('id').inTable('event_people').onDelete('SET NULL'); + + table.timestamp('created_at').defaultTo(knex.fn.now()); + + table.index(['photo_id'], 'photo_faces_photo_idx'); + table.index(['event_id', 'person_id'], 'photo_faces_event_person_idx'); + }); + } + + // --- photos ------------------------------------------------------------- + // + // Every column added in ONE alterTable, not one call each. This migration + // replays in ~90 test suites that boot a database from scratch, and each + // ALTER is its own statement and its own fsync — the per-column version of + // this was measurably slower on CI's shared disk than the whole rest of the + // chain. Same reason the settings seed below is one round trip. + if (await knex.schema.hasTable('photos')) { + const wanted = { + // NULL = never queued. See note 2 — this migration enqueues nothing. + // pending | processing | done | failed | skipped + face_status: (t) => t.string('face_status', 16), + face_count: (t) => t.integer('face_count'), + // Set when a row is claimed, so the janitor can recover it after a + // worker dies mid-photo. Mirrors photos.processing_started_at. + face_started_at: (t) => t.timestamp('face_started_at'), + face_error: (t) => t.text('face_error'), + }; + const missing = []; + for (const name of Object.keys(wanted)) { + if (!(await knex.schema.hasColumn('photos', name))) missing.push(name); + } + if (missing.length) { + await knex.schema.alterTable('photos', (table) => { + for (const name of missing) wanted[name](table); + // Index for the queue claim, added in the same statement. The worker + // polls face_status='pending' constantly; without it that is a full + // scan of `photos` on every tick. + if (missing.includes('face_status')) { + table.index(['face_status'], 'photos_face_status_idx'); + } + }); + } + } + + // --- events ------------------------------------------------------------- + if (await knex.schema.hasTable('events')) { + const wanted = { + // Per-event opt-in. NULL and false both mean off; the column is + // nullable only so an existing row doesn't need backfilling. + face_recognition_enabled: (t) => t.boolean('face_recognition_enabled'), + // When detection is on, does the GUEST see the people strip? Off means + // the photographer gets the tool and guests see an unchanged gallery. + // Defaults to on, but only matters once the above is enabled. + faces_visible_to_guests: (t) => t.boolean('faces_visible_to_guests'), + faces_last_scan_at: (t) => t.timestamp('faces_last_scan_at'), + }; + const missing = []; + for (const name of Object.keys(wanted)) { + if (!(await knex.schema.hasColumn('events', name))) missing.push(name); + } + if (missing.length) { + await knex.schema.alterTable('events', (table) => { + for (const name of missing) wanted[name](table); + }); + } + } + + // --- app_settings ------------------------------------------------------- + if (!(await knex.schema.hasTable('app_settings'))) return; + + // One SELECT and at most one INSERT, rather than a SELECT+INSERT per key. + const keys = GLOBAL_DEFAULTS.map(([k]) => k); + const present = new Set( + (await knex('app_settings').whereIn('setting_key', keys).select('setting_key')) + .map((r) => r.setting_key) + ); + const rows = GLOBAL_DEFAULTS + .filter(([key]) => !present.has(key)) + .map(([key, value]) => ({ + setting_key: key, + // JSON-stringified so SQLite (TEXT) and Postgres (JSONB) round-trip + // the same shape — matches migrations 104 and 173. + setting_value: JSON.stringify(value), + setting_type: 'faces', + // ISO string, not a Date: under Jest, Date objects handed to the + // sqlite3 binding store as the literal "[object Object]". + updated_at: new Date().toISOString(), + })); + if (rows.length) await knex('app_settings').insert(rows); +}; + +exports.down = async function (knex) { + // photo_faces first — it holds the FK into event_people. + await knex.schema.dropTableIfExists('photo_faces'); + await knex.schema.dropTableIfExists('event_people'); + + if (await knex.schema.hasTable('photos')) { + for (const name of ['face_status', 'face_count', 'face_started_at', 'face_error']) { + if (await knex.schema.hasColumn('photos', name)) { + await knex.schema.alterTable('photos', (table) => table.dropColumn(name)); + } + } + } + + if (await knex.schema.hasTable('events')) { + for (const name of [ + 'face_recognition_enabled', + 'faces_visible_to_guests', + 'faces_last_scan_at', + ]) { + if (await knex.schema.hasColumn('events', name)) { + await knex.schema.alterTable('events', (table) => table.dropColumn(name)); + } + } + } + + if (await knex.schema.hasTable('app_settings')) { + await knex('app_settings') + .whereIn('setting_key', GLOBAL_DEFAULTS.map(([k]) => k)) + .del(); + } +}; diff --git a/backend/migrations/core/178_add_auto_categorized.js b/backend/migrations/core/178_add_auto_categorized.js new file mode 100644 index 00000000..7e7941f1 --- /dev/null +++ b/backend/migrations/core/178_add_auto_categorized.js @@ -0,0 +1,41 @@ +/** + * Migration 178: auto-categorised flag (#1074 phase 3). + * + * Marks photos whose category was assigned by the face-count rule engine + * rather than by a person. That distinction is what makes "undo all automatic + * categories" a single query instead of an archaeology exercise, and it is + * why the engine can promise never to touch a photographer's own choices. + * + * Separate from 177 rather than folded into it: 177 has already run wherever + * this branch has been deployed, so editing it would be a no-op there and a + * silent schema divergence between installs. + * + * Nullable with no default. NULL and false both mean "a human chose this (or + * nothing did)", which is the safe reading for every pre-existing row — + * backfilling `false` would say the same thing at the cost of rewriting the + * whole table. + */ + +exports.up = async function (knex) { + if (!(await knex.schema.hasTable('photos'))) return; + if (await knex.schema.hasColumn('photos', 'auto_categorized')) return; + + // Column and index in ONE statement — this replays in ~90 test suites that + // build a database from scratch, and each separate ALTER costs a statement + // and an fsync there. See the note in 177. + await knex.schema.alterTable('photos', (table) => { + table.boolean('auto_categorized'); + // The undo path filters on this alone across a whole event, so it is + // worth an index on installs where most photos are NOT auto-categorised. + table.index(['auto_categorized'], 'photos_auto_categorized_idx'); + }); +}; + +exports.down = async function (knex) { + if (!(await knex.schema.hasTable('photos'))) return; + if (!(await knex.schema.hasColumn('photos', 'auto_categorized'))) return; + + await knex.schema.alterTable('photos', (table) => { + table.dropColumn('auto_categorized'); + }); +}; diff --git a/backend/server.js b/backend/server.js index 8675a7f1..cf205c9c 100644 --- a/backend/server.js +++ b/backend/server.js @@ -1186,6 +1186,17 @@ async function startServer() { // sharp/ffmpeg/EXIF pipeline off the request thread. backgroundProcessor.start(); + // Face detection (#1074). Starts alongside the photo processor but stays + // idle — every worker tick re-checks the `faces` feature flag, which is + // off by default. It is safe to start unconditionally precisely because + // it never touches FACE_ML_URL until that flag is on. + // + // Required HERE rather than at module scope: the face stack pulls in + // axios and (via imageProcessor) sharp, and server.js is imported by a + // large number of supertest suites that never start a worker. Keeping it + // lazy means they don't pay for a module graph they never use. + require('./src/services/faceQueue').start(); + app.listen(PORT, () => { logger.info(`Server running on port ${PORT}`); logger.info(`Admin interface: ${process.env.ADMIN_URL || 'http://localhost:3000'}`); diff --git a/backend/src/routes/adminEvents/faces.js b/backend/src/routes/adminEvents/faces.js new file mode 100644 index 00000000..d1ddfb09 --- /dev/null +++ b/backend/src/routes/adminEvents/faces.js @@ -0,0 +1,475 @@ +// Per-event face recognition — "People in this gallery" (#1074). Same +// sub-router shape as ./downloadResolutions.js — see ./index.js for the +// registration-order contract. +// +// Every route here is gated on the `faces` feature flag as well as the usual +// auth/ownership chain. The frontend hides these surfaces when the flag is +// off, but a direct API call must be refused too — this feature touches +// biometric data, so "the UI doesn't show it" is not a control. + +const { body, validationResult } = require('express-validator'); +const { db, logActivity } = require('../../database/db'); +const { adminAuth } = require('../../middleware/auth'); +const { requirePermission } = require('../../middleware/permissions'); +const { requireFeatureFlag } = require('../../middleware/requireFeatureFlag'); +const { requireEventOwnership } = require('../../middleware/ownership'); +const { errorResponse } = require('../../utils/routeHelpers'); +const { parseBooleanInput } = require('../../utils/parsers'); +const logger = require('../../utils/logger'); + +const faceSettings = require('../../services/faceSettings'); +const faceProcessor = require('../../services/faceProcessor'); +const faceClustering = require('../../services/faceClustering'); +const facePeopleService = require('../../services/facePeopleService'); +const faceClient = require('../../services/faceClient'); + +const requireFaces = requireFeatureFlag('faces', 'FACES_DISABLED'); + +async function loadOwnedEvent(req) { + let q = db('events').where('id', req.params.id); + if (req.admin.roleName === 'editor') { + q = q.where('created_by', req.admin.id); + } + return q.first(); +} + +module.exports = (router) => { + /** + * Sidecar connection test for the global settings panel. + * + * Registered FIRST because it is the only literal-prefix route in this + * module — see ./index.js on why registration order is load-bearing. It + * does not currently collide with '/:id/faces' (the second segment differs) + * but relying on that would be one refactor away from a silent 404. + */ + router.get('/faces/health', adminAuth, requirePermission('events.view'), requireFaces, async (req, res) => { + try { + const result = await faceClient.checkHealth(); + res.json({ url: faceSettings.getSidecarUrl(), ...result }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to check sidecar health'); + } + }); + + /** + * Settings + scan status for the Event → Settings → People panel. + */ + router.get('/:id/faces', adminAuth, requirePermission('events.view'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const status = await facePeopleService.getScanStatus(event.id); + res.json({ + enabled: event.face_recognition_enabled === true || event.face_recognition_enabled === 1, + visible_to_guests: faceSettings.areFacesVisibleToGuests(event), + last_scan_at: event.faces_last_scan_at || null, + status, + }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to load face settings'); + } + }); + + /** + * Flip the two per-event toggles. + * + * Enabling detection backfills existing photos. Disabling it stops there — + * it does NOT delete anything, because "stop scanning" and "destroy the + * clusters I already named" are different intentions and the destructive + * one gets its own explicit endpoint. + */ + router.patch('/:id/faces', + adminAuth, + requirePermission('events.edit'), + requireFaces, + requireEventOwnership, + [ + body('enabled').optional().isBoolean(), + body('visible_to_guests').optional().isBoolean(), + ], + async (req, res) => { + const errors = validationResult(req); + if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() }); + + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const update = {}; + const wasEnabled = event.face_recognition_enabled === true || event.face_recognition_enabled === 1; + let nowEnabled = wasEnabled; + + if (req.body.enabled !== undefined) { + nowEnabled = parseBooleanInput(req.body.enabled, false); + update.face_recognition_enabled = nowEnabled; + } + if (req.body.visible_to_guests !== undefined) { + update.faces_visible_to_guests = parseBooleanInput(req.body.visible_to_guests, true); + } + if (!Object.keys(update).length) { + return res.status(400).json({ error: 'Nothing to update' }); + } + + await db('events').where({ id: event.id }).update(update); + + let queued = 0; + if (!wasEnabled && nowEnabled) { + queued = await faceProcessor.enqueueEvent(event.id, { onlyUnscanned: true }); + await db('events').where({ id: event.id }) + .update({ faces_last_scan_at: new Date().toISOString() }); + } + + await logActivity('event_faces_updated', { + actorType: 'admin', + actorId: req.admin.id, + eventId: event.id, + metadata: { ...update, queued }, + }).catch(() => {}); + + res.json({ success: true, queued }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to update face settings'); + } + }); + + /** + * People management grid. Unlike the gallery endpoint this returns hidden + * and ignored people, plus the total (all-visibility) face count — the + * photographer is allowed to see their own hidden photos. + */ + router.get('/:id/people', adminAuth, requirePermission('events.view'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const people = await facePeopleService.listPeople(event.id, { + isClient: true, + forAdmin: true, + }); + res.json({ people }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to fetch people'); + } + }); + + /** + * Rename / hide / ignore / set cover. + */ + router.patch('/:id/people/:personId', + adminAuth, + requirePermission('events.edit'), + requireFaces, + requireEventOwnership, + [ + body('label').optional({ nullable: true }).isString().isLength({ max: 255 }), + body('is_hidden').optional().isBoolean(), + body('is_ignored').optional().isBoolean(), + body('cover_face_id').optional({ nullable: true }).isInt(), + ], + async (req, res) => { + const errors = validationResult(req); + if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() }); + + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const person = await db('event_people') + .where({ id: req.params.personId, event_id: event.id }).first(); + if (!person) return res.status(404).json({ error: 'Person not found' }); + + const update = { updated_at: new Date().toISOString() }; + if (req.body.label !== undefined) { + const label = req.body.label === null ? null : String(req.body.label).trim(); + update.label = label || null; + } + if (req.body.is_hidden !== undefined) update.is_hidden = parseBooleanInput(req.body.is_hidden, false); + if (req.body.is_ignored !== undefined) update.is_ignored = parseBooleanInput(req.body.is_ignored, false); + + if (req.body.cover_face_id !== undefined && req.body.cover_face_id !== null) { + // The cover must belong to THIS person in THIS event — otherwise a + // crafted id could point the avatar at any face in the database. + const face = await db('photo_faces') + .where({ id: req.body.cover_face_id, event_id: event.id, person_id: person.id }) + .first(); + if (!face) return res.status(400).json({ error: 'cover_face_id does not belong to this person' }); + update.cover_face_id = face.id; + } + + await db('event_people').where({ id: person.id }).update(update); + res.json({ success: true }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to update person'); + } + }); + + /** + * Merge people. The two operations that make automatic clustering survive + * contact with reality are this and split — a well-behaved wedding gallery + * still produces "Anna in daylight" and "Anna at the party". + */ + router.post('/:id/people/merge', + adminAuth, + requirePermission('events.edit'), + requireFaces, + requireEventOwnership, + [body('source_ids').isArray({ min: 1 }), body('target_id').isInt()], + async (req, res) => { + const errors = validationResult(req); + if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() }); + + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + // Every id must belong to this event — never merge across galleries. + const ids = [...req.body.source_ids.map(Number), Number(req.body.target_id)]; + const owned = await db('event_people').where({ event_id: event.id }).whereIn('id', ids).pluck('id'); + if (owned.length !== new Set(ids).size) { + return res.status(400).json({ error: 'One or more people do not belong to this event' }); + } + + const result = await faceClustering.mergePeople( + event.id, req.body.source_ids.map(Number), Number(req.body.target_id) + ); + res.json({ success: true, ...result }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to merge people'); + } + }); + + router.post('/:id/people/:personId/split', + adminAuth, + requirePermission('events.edit'), + requireFaces, + requireEventOwnership, + [body('face_ids').isArray({ min: 1 })], + async (req, res) => { + const errors = validationResult(req); + if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() }); + + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const newPersonId = await faceClustering.splitPerson( + event.id, Number(req.params.personId), req.body.face_ids.map(Number) + ); + if (!newPersonId) return res.status(400).json({ error: 'No matching faces to split' }); + res.json({ success: true, person_id: newPersonId }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to split person'); + } + }); + + /** + * Faces of one person, for the split picker. + */ + router.get('/:id/people/:personId/faces', adminAuth, requirePermission('events.view'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const faces = await db('photo_faces') + .where({ event_id: event.id, person_id: req.params.personId }) + .orderBy('det_score', 'desc') + .limit(500) + .join('photos', 'photos.id', 'photo_faces.photo_id') + .select( + 'photo_faces.id', 'photo_faces.photo_id', + 'photo_faces.bbox_x', 'photo_faces.bbox_y', + 'photo_faces.bbox_w', 'photo_faces.bbox_h', + 'photo_faces.det_score', 'photo_faces.blur', + // Needed to crop the box — it is in original-image pixels. + 'photos.width as photo_width', 'photos.height as photo_height' + ); + + res.json({ + faces: faces.map((f) => ({ + id: f.id, + photo_id: f.photo_id, + bbox: [f.bbox_x, f.bbox_y, f.bbox_w, f.bbox_h], + photo_width: f.photo_width ?? null, + photo_height: f.photo_height ?? null, + score: f.det_score, + blur: f.blur, + })), + }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to fetch faces'); + } + }); + + /** + * Re-scan: re-runs detection through the sidecar. Expensive. + */ + router.post('/:id/faces/rescan', adminAuth, requirePermission('events.edit'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + if (!(await faceSettings.isEnabledForEvent(event))) { + return res.status(400).json({ error: 'Face detection is not enabled for this event' }); + } + + const queued = await faceProcessor.enqueueEvent(event.id, { onlyUnscanned: false }); + await db('events').where({ id: event.id }) + .update({ faces_last_scan_at: new Date().toISOString() }); + + await logActivity('event_faces_rescan', { + actorType: 'admin', actorId: req.admin.id, eventId: event.id, metadata: { queued }, + }).catch(() => {}); + + res.json({ success: true, queued }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to queue re-scan'); + } + }); + + /** + * Re-cluster: re-derives people from stored embeddings. Cheap — no sidecar + * call — and the thing to run after changing the match threshold. + */ + router.post('/:id/faces/recluster', adminAuth, requirePermission('events.edit'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const people = await faceClustering.recluster(event.id); + await logActivity('event_faces_recluster', { + actorType: 'admin', actorId: req.admin.id, eventId: event.id, metadata: { people }, + }).catch(() => {}); + + res.json({ success: true, people }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to recluster'); + } + }); + + /** + * Read/write the global auto-category setting. + * + * The migration seeds it false and nothing else ever wrote it, so the whole + * rule engine — and its undo endpoint — were unreachable in normal product + * flows: every call returned `skipped: disabled`. A feature with no way to + * turn it on is not shipped. + */ + router.get('/faces/auto-categories', adminAuth, requirePermission('settings.view'), requireFaces, + async (req, res) => { + try { + const thresholds = await faceSettings.getThresholds(); + res.json({ enabled: thresholds.face_auto_categorize_enabled === true }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to read the auto-category setting'); + } + }); + + router.put('/faces/auto-categories', + adminAuth, + requirePermission('settings.edit'), + requireFaces, + [body('enabled').isBoolean()], + async (req, res) => { + const errors = validationResult(req); + if (!errors.isEmpty()) return res.status(400).json({ errors: errors.array() }); + + try { + const enabled = parseBooleanInput(req.body.enabled, false); + const existing = await db('app_settings') + .where('setting_key', 'face_auto_categorize_enabled').first(); + + if (existing) { + await db('app_settings').where('setting_key', 'face_auto_categorize_enabled') + .update({ setting_value: JSON.stringify(enabled), updated_at: new Date().toISOString() }); + } else { + await db('app_settings').insert({ + setting_key: 'face_auto_categorize_enabled', + setting_value: JSON.stringify(enabled), + setting_type: 'faces', + updated_at: new Date().toISOString(), + }); + } + + await logActivity('face_auto_categories_toggled', { + actorType: 'admin', actorId: req.admin.id, metadata: { enabled }, + }).catch(() => {}); + + res.json({ success: true, enabled }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to update the auto-category setting'); + } + }); + + /** + * Run the auto-category rules over this event now. + * + * Normally happens automatically after each photo is scanned; this is for + * applying the rules to a gallery that was scanned before the setting was + * turned on. + */ + router.post('/:id/faces/categorize', adminAuth, requirePermission('events.edit'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const { categorizeEvent } = require('../../services/faceAutoCategories'); + const result = await categorizeEvent(event.id); + res.json({ success: true, ...result }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to apply automatic categories'); + } + }); + + /** + * Undo every automatic category assignment. Only clears what the rule + * engine set — categories chosen by a person are untouched. + */ + router.delete('/:id/faces/categorize', adminAuth, requirePermission('events.edit'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const { undoEvent } = require('../../services/faceAutoCategories'); + const result = await undoEvent(event.id); + + await logActivity('event_faces_categories_undone', { + actorType: 'admin', actorId: req.admin.id, eventId: event.id, metadata: result, + }).catch(() => {}); + + res.json({ success: true, ...result }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to undo automatic categories'); + } + }); + + /** + * Delete all face data for this event. The GDPR erasure path. + */ + router.delete('/:id/faces', adminAuth, requirePermission('events.edit'), requireFaces, requireEventOwnership, + async (req, res) => { + try { + const event = await loadOwnedEvent(req); + if (!event) return res.status(404).json({ error: 'Event not found' }); + + const result = await faceProcessor.purgeEvent(event.id); + await db('events').where({ id: event.id }).update({ faces_last_scan_at: null }); + + await logActivity('event_faces_purged', { + actorType: 'admin', actorId: req.admin.id, eventId: event.id, metadata: result, + }).catch(() => {}); + + logger.info(`adminEvents: face data purged for event ${event.id} by admin ${req.admin.id}`); + res.json({ success: true, ...result }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to delete face data'); + } + }); + +}; diff --git a/backend/src/routes/adminEvents/helpers.js b/backend/src/routes/adminEvents/helpers.js index 9d42655f..5b24a345 100644 --- a/backend/src/routes/adminEvents/helpers.js +++ b/backend/src/routes/adminEvents/helpers.js @@ -232,6 +232,14 @@ async function deleteEventCascade(eventId, adminContext) { // 3. Delete email queue entries await trx('email_queue').where('event_id', eventId).del(); // 4. Delete photos (also handles hero_photo_id foreign key) + // Face data (#1074). The FK declares ON DELETE CASCADE, but SQLite only + // honours that when `PRAGMA foreign_keys = ON`, which PicPeak does not set + // — so on the SQLite path the cascade is inert and biometric embeddings + // would outlive the gallery they belong to. Delete explicitly, before the + // photos, so the guarantee holds on both engines. + await trx('photo_faces').where('event_id', eventId).del(); + await trx('event_people').where('event_id', eventId).del(); + await trx('photos').where('event_id', eventId).del(); // 5. Finally delete the event row await trx('events').where('id', eventId).del(); diff --git a/backend/src/routes/adminEvents/index.js b/backend/src/routes/adminEvents/index.js index fa5774b1..f29e023b 100644 --- a/backend/src/routes/adminEvents/index.js +++ b/backend/src/routes/adminEvents/index.js @@ -15,5 +15,6 @@ require('./resets')(router); require('./archiveBulk')(router); require('./logo')(router); require('./qr')(router); +require('./faces')(router); module.exports = router; diff --git a/backend/src/routes/adminFeatureFlags.js b/backend/src/routes/adminFeatureFlags.js index d488c482..d93603d2 100644 --- a/backend/src/routes/adminFeatureFlags.js +++ b/backend/src/routes/adminFeatureFlags.js @@ -98,6 +98,18 @@ const KNOWN_FLAGS = [ // kill-switch for the Workflows admin area AND the engine's runtime side // effects (no run is created/resumed while off). 'workflows', + // Face recognition — "People in this gallery" (migration 177, #1074). + // Requires the optional picpeak-ml sidecar container. THIS FLAG IS THE + // GATE for the whole feature: FACE_ML_URL has a working default (the + // compose service name), so the variable's presence proves nothing and + // cannot be used to detect intent. While this is off the backend never + // contacts the sidecar, the face queue idles, no face UI renders anywhere + // and no face_status is ever written. + // + // Face embeddings are biometric data (GDPR Art. 9). Turning this on is only + // the first of two deliberate actions — detection still has to be enabled + // per event. Strictly opt-in. + 'faces', ]; // Spec defaults for any flag missing from the DB (e.g. a row added by a @@ -129,6 +141,8 @@ const DEFAULT_FLAGS = { slideshow: false, transfers: false, workflows: false, + // #1074 — off by default is the whole "zero behaviour change" guarantee. + faces: false, }; async function readAllFlags() { @@ -146,6 +160,15 @@ function applyDependencyRules(flags) { const out = { ...flags }; // Galleries is the foundation — never off. out.galleries = true; + // Face recognition is unavailable on the all-in-one single-container image + // (#1042 / PR #1068) for performance reasons — see + // faceSettings.isSingleContainerImage. Forced false in BOTH directions: + // GET reports it off so the UI can show it as unavailable rather than a + // switch that silently does nothing, and PUT cannot turn it on. The backend + // gate refuses independently, so this is presentation plus defence in + // depth, not the enforcement itself. + const { isSingleContainerImage } = require('../services/faceSettings'); + if (isSingleContainerImage()) out.faces = false; // Sub-features can't outlive their parents. if (out.quotes === false) out.bills = false; if (out.calendar === false) out.calendarBooking = false; diff --git a/backend/src/routes/adminPhotos.js b/backend/src/routes/adminPhotos.js index 7c4ffe87..fd1aab8b 100644 --- a/backend/src/routes/adminPhotos.js +++ b/backend/src/routes/adminPhotos.js @@ -684,6 +684,15 @@ router.delete('/:eventId/photos/:photoId', adminAuth, requirePermission('photos. } // Remove from database + // Face data (#1074): the FK cascade is inert on SQLite, and cannot fix up + // event_people counts anyway. See faceProcessor.purgePhotoFaces. + try { + const { purgePhotoFaces } = require('../services/faceProcessor'); + await purgePhotoFaces(photoId); + } catch (err) { + logger.warn(`deletePhoto: face purge failed for photo ${photoId}`, { error: err.message }); + } + await db('photos').where({ id: photoId }).delete(); // Log activity (event was fetched above for storage key resolution) @@ -752,6 +761,12 @@ router.patch('/:eventId/photos/:photoId', adminAuth, requirePermission('photos.e } } + // A human just set (or cleared) this category, so it is no longer an + // automatic assignment (#1074 phase 3). Without resetting the flag, + // "undo automatic categories" would later wipe the photographer's own + // choice — exactly the guarantee the rule engine advertises. + updateData.auto_categorized = false; + // Update photo await db('photos') .where({ id: photoId, event_id: eventId }) @@ -828,6 +843,23 @@ router.post('/:eventId/photos/bulk-delete', adminAuth, requirePermission('photos } } + // Face data (#1074), bulk path. Same reasoning as the single delete: the + // SQLite FK cascade never fires, and event_people counts need rebuilding + // regardless of engine. + // Iterate the VALIDATED rows, not the raw request ids. `photos` is already + // scoped to this event; `photoIds` is user input, and purgePhotoFaces has + // no event scope of its own — so looping the raw ids let an editor delete + // face data (and recompute people) in a gallery they do not own, even + // though the photo deletion below is correctly scoped. + for (const photo of photos) { + try { + const { purgePhotoFaces } = require('../services/faceProcessor'); + await purgePhotoFaces(photo.id); + } catch (err) { + logger.warn(`bulk delete: face purge failed for photo ${photo.id}`, { error: err.message }); + } + } + // Delete from database await db('photos') .whereIn('id', photoIds) @@ -908,6 +940,12 @@ router.post('/:eventId/photos/bulk-update', adminAuth, requirePermission('photos updateData.category_id = null; } } + + // A human just set (or cleared) this category, so it is no longer an + // automatic assignment (#1074 phase 3). Without resetting the flag, + // "undo automatic categories" would later wipe the photographer's own + // choice — exactly the guarantee the rule engine advertises. + updateData.auto_categorized = false; } await db('photos') diff --git a/backend/src/routes/adminSystem.js b/backend/src/routes/adminSystem.js index 18dd492f..68ee2130 100644 --- a/backend/src/routes/adminSystem.js +++ b/backend/src/routes/adminSystem.js @@ -35,12 +35,20 @@ router.get('/version', adminAuth, requirePermission(['settings.view', 'system.vi const channel = getCurrentChannel(backendVersion); + // `single_container` lets the admin UI explain why a feature is + // unavailable instead of rendering a switch that silently refuses to + // stay on. Currently only face recognition is gated this way (#1074 on + // the all-in-one image, #1042 / PR #1068) — see + // faceSettings.isSingleContainerImage for the reasoning. + const { isSingleContainerImage } = require('../services/faceSettings'); + res.json({ backend: backendVersion, frontend: '1.0.0', // This will be set by frontend node: process.version, environment: process.env.NODE_ENV || 'production', - channel: channel + channel: channel, + single_container: isSingleContainerImage() }); } catch (error) { logger.error('Error fetching version:', error); diff --git a/backend/src/routes/gallery.js b/backend/src/routes/gallery.js index d194f012..e0a64f7f 100644 --- a/backend/src/routes/gallery.js +++ b/backend/src/routes/gallery.js @@ -809,7 +809,36 @@ router.get('/:slug/photos', verifyGalleryAccess, resolveGuest, async (req, res) const likedRows = await likeQuery.select('photo_id'); likedRows.forEach(row => likedPhotoIds.add(row.photo_id)); } - + + // People in each photo (#1074). Two independent gates: the feature must + // be on for this event AND, for a plain guest, the photographer must have + // left the strip visible. A client (PIN access) is the photographer's own + // view, so faces_visible_to_guests doesn't restrict them. + // + // `photos` is already visibility-filtered above, and this only ever asks + // about ids in that set, so it cannot widen what the caller sees. + let peopleEnabled = false; + let personIdsByPhoto = new Map(); + try { + const { isEnabledForEvent, areFacesVisibleToGuests } = require('../services/faceSettings'); + if (photos.length > 0 && await isEnabledForEvent(req.event)) { + peopleEnabled = isClient || areFacesVisibleToGuests(req.event); + if (peopleEnabled) { + const { getPersonIdsByPhoto } = require('../services/facePeopleService'); + personIdsByPhoto = await getPersonIdsByPhoto( + req.event.id, + photos.map(p => p.id), + { forAdmin: isClient } + ); + } + } + } catch (err) { + // A face-feature failure must never take down the gallery payload. + logger.warn(`gallery: person_ids lookup failed for event ${req.event.id}`, { error: err.message }); + peopleEnabled = false; + personIdsByPhoto = new Map(); + } + // Get actual categories used by photos in this event // This includes both global categories and event-specific ones const usedCategoryIds = hiddenForGuest ? [] : await db('photos') @@ -961,6 +990,11 @@ router.get('/:slug/photos', verifyGalleryAccess, resolveGuest, async (req, res) // Mirror of the admin-side toggle so the lightbox can decide // whether to surface original camera filenames (#508). use_original_filenames: useOriginalFilenames, + // "People in this gallery" (#1074). False whenever the global flag + // is off, detection is off for this event, or the photographer chose + // to keep the strip to themselves — the frontend renders no face UI + // at all in that case. + people_enabled: peopleEnabled, ...protectionSettings }, // Reveal mode (#838): the guest UI switches to the upload-only view @@ -1050,6 +1084,12 @@ router.get('/:slug/photos', verifyGalleryAccess, resolveGuest, async (req, res) // grid seed its lifted likedPhotoIds correctly on hard refresh. is_liked: showFeedbackToGuests ? likedPhotoIds.has(photo.id) : false, favorite_count: showFeedbackToGuests ? (photo.favorite_count || 0) : 0, + // People in this photo (#1074). Empty array when the feature is + // off for this event or hidden from guests, so the frontend has + // one shape to handle. Riding along on this payload is what keeps + // face filtering client-side and instant, like the category and + // liked/rated filters. + person_ids: personIdsByPhoto.get(photo.id) || [], // Visibility (only included for clients) ...(isClient ? { visibility: photo.visibility || 'visible' } : {}) }; @@ -1060,6 +1100,62 @@ router.get('/:slug/photos', verifyGalleryAccess, resolveGuest, async (req, res) } }); +/** + * People in this gallery (#1074). + * + * Returns [] rather than 403 whenever the feature is unavailable — a guest + * must not be able to tell "this gallery has no people" from "this gallery + * has the feature switched off". Same reasoning as reveal mode returning an + * empty photo set rather than an error. + * + * Counts and cover faces are computed against the caller's own visibility + * scope inside facePeopleService; nothing here reads face_count_total. + */ +router.get('/:slug/people', verifyGalleryAccess, resolveGuest, async (req, res) => { + try { + const isClient = req.accessLevel === 'client'; + const { isEnabledForEvent, areFacesVisibleToGuests, getThresholds } = + require('../services/faceSettings'); + + if (!(await isEnabledForEvent(req.event))) { + return res.json({ people: [] }); + } + if (!isClient && !areFacesVisibleToGuests(req.event)) { + return res.json({ people: [] }); + } + // While a gallery is hidden behind reveal mode (#838), a plain guest sees + // no photos — so they see no people either. + if (guestBlockedByReveal(req)) { + return res.json({ people: [] }); + } + + const { listPeople, getScanStatus } = require('../services/facePeopleService'); + const thresholds = await getThresholds(); + + const people = await listPeople(req.event.id, { + isClient, + forAdmin: false, + minClusterSize: thresholds.face_min_cluster_size, + }); + + // Drives the "Finding people… 240/1200" progress line during a backfill. + // Scoped to what this viewer may see — an unscoped total would leak the + // number of hidden photos through the progress bar. + const status = await getScanStatus(req.event.id, { isClient }); + + res.json({ + people, + scan: { + in_progress: status.in_progress, + scanned: status.scanned, + total: status.total, + }, + }); + } catch (error) { + errorResponse(res, error, 500, 'Failed to fetch people'); + } +}); + // Toggle photo visibility (client-only) router.patch('/:slug/photos/:photoId/visibility', verifyGalleryAccess, async (req, res) => { try { diff --git a/backend/src/services/archiveService.js b/backend/src/services/archiveService.js index 87158fde..331838f6 100644 --- a/backend/src/services/archiveService.js +++ b/backend/src/services/archiveService.js @@ -225,6 +225,35 @@ async function archiveEvent(event) { } } + // Purge face data (#1074). photo_faces cascades off photos, but archiving + // does NOT delete the photo rows — and event_people hangs off the event, + // which also survives. So neither would go without an explicit purge, and + // an archived gallery would keep its biometric data indefinitely. + // + // Face data is derived: if the event is ever restored, re-enabling + // detection re-scans. Nothing irreplaceable is lost except assigned + // names, which is the same trade already accepted for backups/exports. + try { + const { purgeEvent } = require('./faceProcessor'); + await purgeEvent(event.id); + + // Turn detection OFF as well. purgeEvent clears the rows but leaves the + // toggle on, so restoring the archive would bring back a gallery that + // claims face detection is enabled while having no people and no queued + // work — indistinguishable from a broken scan. Off is the honest state: + // the photographer re-enables it and gets a fresh backfill, which is + // exactly the flow the toggle already implements. + await db('events').where({ id: event.id }) + .update({ face_recognition_enabled: false, faces_last_scan_at: null }); + } catch (err) { + // Never fail an archive over this — but say so loudly, because it + // means biometric data outlived the gallery. + logger.error( + `Archive: failed to purge face data for event ${event.slug} — ` + + `face rows may remain. ${err.message}` + ); + } + // Queue completion email — admin_email is nullable on events (migration 073); // skip queueing rather than violating email_queue.recipient_email NOT NULL. // diff --git a/backend/src/services/databaseBackup.js b/backend/src/services/databaseBackup.js index 649bb2a2..63883f5f 100644 --- a/backend/src/services/databaseBackup.js +++ b/backend/src/services/databaseBackup.js @@ -16,6 +16,20 @@ const packageJson = require('../../package.json'); const CHUNK_SIZE = 1024 * 1024; // 1MB chunks for streaming const PROGRESS_INTERVAL = 100; // Report progress every 100 rows +// Face recognition tables (#1074). Their SCHEMA is backed up, their CONTENTS +// are not: embeddings are biometric data (GDPR Art. 9) and fully derived from +// the photos, so a restore re-scans instead of carrying biometrics forward. +// +// The schema must survive, which is why Postgres uses --exclude-table-DATA +// rather than --exclude-table: knex_migrations records 177 as applied, so a +// restore whose dump lacked the CREATE TABLE would fail on the first query +// instead of merely coming back empty. +// +// SQLite cannot filter at all — `sqlite3 .backup` is a whole-file binary copy +// — so the rows are deleted from the temp copy before it is finalised. See +// createSQLiteBackup below. +const FACE_TABLES = ['photo_faces', 'event_people']; + /** * Database Backup Service * Supports both SQLite and PostgreSQL with proper escaping, @@ -149,18 +163,20 @@ class DatabaseBackupService { AND name != 'knex_migrations_lock' ORDER BY name `); - return result.map(row => row.name); + return result.map(row => row.name).filter((t) => !FACE_TABLES.includes(t)); } else { // PostgreSQL const result = await db.raw(` - SELECT table_name - FROM information_schema.tables - WHERE table_schema = 'public' + SELECT table_name + FROM information_schema.tables + WHERE table_schema = 'public' AND table_type = 'BASE TABLE' AND table_name NOT IN ('knex_migrations', 'knex_migrations_lock') ORDER BY table_name `); - return result.rows.map(row => row.table_name); + return result.rows + .map(row => row.table_name) + .filter((t) => !FACE_TABLES.includes(t)); } } @@ -175,6 +191,48 @@ class DatabaseBackupService { // Use SQLite's backup API for consistency await spawnAsync('sqlite3', [dbPath, `.backup '${tempPath}'`]); + // Strip face data from the COPY (#1074). `.backup` is a whole-file + // binary copy with no way to exclude a table, so the rows come out and + // are deleted here — the live database is never touched. + // + // VACUUM is not cosmetic: without it the deleted pages remain in the + // file and "not backed up" would be false on disk, which is the exact + // claim this code exists to make true. + for (const table of FACE_TABLES) { + await spawnAsync('sqlite3', [ + tempPath, + `DELETE FROM ${table} WHERE 1=1;`, + ]).catch(() => { + // Table absent on installs that predate migration 177 — fine. + }); + } + + // Reset the DERIVED state on photos as well. Without this the restored + // database says every photo is scanned ('done') while the face tables + // are empty, and the worker only ever claims 'pending' — so the gallery + // reports a finished scan and shows nobody, permanently, until an admin + // works out that a manual re-scan is needed. Requeue instead. + await spawnAsync('sqlite3', [ + tempPath, + "UPDATE photos SET face_status = CASE WHEN face_status IS NULL THEN NULL ELSE 'pending' END, " + + 'face_count = NULL, face_started_at = NULL, face_error = NULL;', + ]).catch(() => {}); + // FATAL, not a warning. Deleting rows leaves their pages in the file + // until VACUUM rewrites it, so a backup that skipped the VACUUM can + // still contain recoverable face embeddings. Publishing it would break + // the explicit promise that biometric data is never backed up — better + // to fail the backup and say so than to hand over an artifact that + // quietly violates it. + try { + await spawnAsync('sqlite3', [tempPath, 'VACUUM;']); + } catch (err) { + await fs.unlink(tempPath).catch(() => {}); + throw new Error( + 'SQLite backup aborted: could not VACUUM after removing face data, so the ' + + `backup could still contain recoverable biometric pages (${err.message})` + ); + } + // Verify the backup const verifyResult = await spawnAsync('sqlite3', [tempPath, 'PRAGMA integrity_check']); if (!verifyResult.stdout.includes('ok')) { @@ -232,6 +290,17 @@ class DatabaseBackupService { // callers (scheduled cron, dedicated admin DB-backup page) hit // the same failure but on installs that had never exercised them. + // Face data (#1074): schema yes, rows no. --exclude-table-DATA, not + // --exclude-table — see the FACE_TABLES comment at the top of this file + // for why dropping the CREATE TABLE would break restore outright. + for (const table of FACE_TABLES) { + pgDumpOptions.push(`--exclude-table-data=public.${table}`); + } + // NOTE: the Postgres path cannot rewrite rows inside pg_dump the way the + // SQLite path can, so photos.face_status is restored as-is here. The + // restore path compensates — see resetDerivedFaceState in restoreService. + // Both engines must end up requeued, not "done with no people". + // Add compression if not doing it separately if (options.compress && !options.separateCompression) { pgDumpOptions.push('--compress=6'); diff --git a/backend/src/services/faceAutoCategories.js b/backend/src/services/faceAutoCategories.js new file mode 100644 index 00000000..9b7a1a8d --- /dev/null +++ b/backend/src/services/faceAutoCategories.js @@ -0,0 +1,178 @@ +/** + * Rule-based auto-categorisation from face counts (#1074 phase 3). + * + * Once faces are detected, `photos.face_count` and the landmark yaw are free + * inputs for the sorting @bluecow94 asked for in discussion #1069. This is a + * small rule list, not a classifier — the point is that it is predictable and + * explainable, which a photographer can work with. + * + * THREE NON-NEGOTIABLE RULES, in order of importance: + * + * 1. It only ever fills an EMPTY category. It never overwrites a category + * a photographer set, because a human assignment is a decision and this + * is a heuristic. There is no configuration that changes this. + * + * 2. Everything it does is marked `auto_categorized = true`, so "undo all + * automatic categories" is one query rather than an archaeology + * exercise. + * + * 3. It is a separate step from detection and separately enableable. Some + * photographers want the sorting and not the people strip. + */ + +const { db } = require('../database/db'); +const logger = require('../utils/logger'); +const { getThresholds } = require('./faceSettings'); + +/** + * Category slugs this engine manages, in evaluation order. First match wins. + * + * `minFaceAreaRatio` on Portraits is what separates "a portrait of someone" + * from "someone happens to be in this landscape" — one face in the frame + * means nothing without knowing how much of the frame it occupies. + */ +const RULES = [ + { + slug: 'details', + name: 'Details', + match: (photo) => photo.face_count === 0, + }, + { + slug: 'portraits', + name: 'Portraits', + match: (photo, ctx) => photo.face_count === 1 && ctx.largestFaceAreaRatio >= 0.08, + }, + { + slug: 'small-groups', + name: 'Small groups', + match: (photo) => photo.face_count >= 2 && photo.face_count <= 5, + }, + { + slug: 'groups', + name: 'Groups', + match: (photo) => photo.face_count > 5, + }, +]; + +/** + * Find or create the event-scoped category for a rule. + * + * Event-scoped rather than global: these are generated categories, and + * polluting every gallery's category list with them would be worse than the + * duplication. An existing GLOBAL category with the same slug is reused when + * present, so an operator who already has "Portraits" keeps one. + */ +async function resolveCategory(eventId, rule, cache, trx = db) { + if (cache.has(rule.slug)) return cache.get(rule.slug); + + let category = await trx('photo_categories') + .where('slug', rule.slug) + .where(function () { + this.where('event_id', eventId).orWhere('is_global', true); + }) + .first(); + + if (!category) { + const [inserted] = await trx('photo_categories').insert({ + name: rule.name, + slug: rule.slug, + is_global: false, + event_id: eventId, + created_at: new Date().toISOString(), + }).returning('id'); + const id = typeof inserted === 'object' ? inserted.id : inserted; + category = { id, slug: rule.slug }; + } + + cache.set(rule.slug, category); + return category; +} + +/** + * Apply the rules to one event. + * + * Only touches photos that have been face-scanned (`face_status = 'done'`) + * and have no category. Returns a per-slug tally. + */ +async function categorizeEvent(eventId) { + const thresholds = await getThresholds(); + if (!thresholds.face_auto_categorize_enabled) { + return { skipped: true, reason: 'disabled' }; + } + + const photos = await db('photos') + .where({ event_id: eventId, face_status: 'done' }) + // Rule 1, enforced in the query rather than trusted to a later branch. + .whereNull('category_id') + .select('id', 'face_count', 'width', 'height'); + + if (!photos.length) return { assigned: 0, byCategory: {} }; + + // One query for every face box in the event, rather than one per photo. + const faceRows = await db('photo_faces') + .where({ event_id: eventId }) + .whereIn('photo_id', photos.map((p) => p.id)) + .select('photo_id', 'bbox_w', 'bbox_h'); + + const largestByPhoto = new Map(); + for (const row of faceRows) { + const area = (row.bbox_w || 0) * (row.bbox_h || 0); + if (area > (largestByPhoto.get(row.photo_id) || 0)) { + largestByPhoto.set(row.photo_id, area); + } + } + + const cache = new Map(); + const byCategory = {}; + let assigned = 0; + + for (const photo of photos) { + const frameArea = (photo.width || 0) * (photo.height || 0); + const ctx = { + largestFaceAreaRatio: frameArea > 0 + ? (largestByPhoto.get(photo.id) || 0) / frameArea + : 0, + }; + + const rule = RULES.find((r) => r.match(photo, ctx)); + if (!rule) continue; + + const category = await resolveCategory(eventId, rule, cache); + + // Guard the UPDATE on category_id still being NULL. Between the SELECT + // above and here a photographer may have set one by hand, and their + // choice wins — rule 1 is not a best-effort. + const updated = await db('photos') + .where({ id: photo.id }) + .whereNull('category_id') + .update({ category_id: category.id, auto_categorized: true }); + + if (updated > 0) { + assigned += updated; + byCategory[rule.slug] = (byCategory[rule.slug] || 0) + updated; + } + } + + logger.info( + `faceAutoCategories: assigned ${assigned} photo(s) in event ${eventId} ` + + `(${Object.entries(byCategory).map(([k, v]) => `${k}=${v}`).join(', ') || 'none'})` + ); + return { assigned, byCategory }; +} + +/** + * Undo every automatic assignment for an event. + * + * Clears only rows this engine set — a photographer's own categories are + * untouched, which is the entire reason `auto_categorized` exists. + */ +async function undoEvent(eventId) { + const cleared = await db('photos') + .where({ event_id: eventId, auto_categorized: true }) + .update({ category_id: null, auto_categorized: false }); + + logger.info(`faceAutoCategories: cleared ${cleared} automatic category assignment(s) in event ${eventId}`); + return { cleared }; +} + +module.exports = { RULES, categorizeEvent, undoEvent }; diff --git a/backend/src/services/faceClient.js b/backend/src/services/faceClient.js new file mode 100644 index 00000000..dbc510b4 --- /dev/null +++ b/backend/src/services/faceClient.js @@ -0,0 +1,134 @@ +/** + * HTTP client for the picpeak-ml sidecar (#1074). + * + * The important behaviour here is the error taxonomy, because the queue + * treats the two classes completely differently: + * + * SidecarUnavailableError — the sidecar is down, unreachable, timing out, + * or returned 5xx. The photo goes BACK to 'pending' and is retried later. + * Turning the container off for a week must not require a manual re-scan. + * + * Everything else (4xx) — this image is a lost cause. The photo is marked + * 'failed' and never retried, because retrying an undecodable file + * forever is just a busy loop. + * + * Log volume matters too: an hour of downtime at a 1s poll is 3,600 identical + * warnings. Unavailability is logged at most once per LOG_INTERVAL_MS. + */ + +const axios = require('axios'); +const logger = require('../utils/logger'); +const { getSidecarUrl, getSidecarToken } = require('./faceSettings'); + +const REQUEST_TIMEOUT_MS = parseInt(process.env.FACE_ML_TIMEOUT_MS || '30000', 10); +const LOG_INTERVAL_MS = 5 * 60 * 1000; + +let lastUnavailableLogAt = 0; + +class SidecarUnavailableError extends Error { + constructor(message) { + super(message); + this.name = 'SidecarUnavailableError'; + } +} + +class SidecarRejectedError extends Error { + constructor(message, status) { + super(message); + this.name = 'SidecarRejectedError'; + this.status = status; + } +} + +function logUnavailable(message) { + const now = Date.now(); + if (now - lastUnavailableLogAt < LOG_INTERVAL_MS) return; + lastUnavailableLogAt = now; + logger.warn( + `faceClient: ML sidecar unavailable (${message}). Photos stay queued and will ` + + 'be retried; no action needed unless this persists. Further identical warnings ' + + 'are suppressed for 5 minutes.' + ); +} + +function authHeaders() { + const token = getSidecarToken(); + return token ? { 'X-Face-ML-Token': token } : {}; +} + +/** + * Translate an axios failure into our two-class taxonomy. + */ +function classify(err) { + const status = err.response?.status; + + if (status && status >= 400 && status < 500) { + // 401 is a configuration error, not a bad image — but it is also not + // something retrying fixes, so it surfaces loudly and stops the photo. + if (status === 401) { + logger.error( + 'faceClient: sidecar rejected our token (401). FACE_ML_TOKEN must match ' + + 'on both the backend and the picpeak-ml container.' + ); + } + return new SidecarRejectedError( + err.response?.data?.detail || `Sidecar rejected the request (${status})`, + status + ); + } + + logUnavailable(err.code || err.message || `HTTP ${status}`); + return new SidecarUnavailableError(err.message || 'Sidecar unreachable'); +} + +/** + * Detect faces in an image. `buffer` is the preview rendition's bytes. + * Returns the sidecar's `{ model_version, faces: [...] }`. + */ +async function detectFaces(buffer, filename = 'photo.jpg') { + const form = new FormData(); + // Node 18+ ships FormData/Blob globally, so no multipart dependency is + // needed for the one endpoint that uploads anything. + form.append('image', new Blob([buffer]), filename); + + try { + const response = await axios.post(`${getSidecarUrl()}/faces`, form, { + headers: authHeaders(), + timeout: REQUEST_TIMEOUT_MS, + // A 45MP preview is ~2MB; the cap is generous but not unbounded. + maxBodyLength: 64 * 1024 * 1024, + maxContentLength: 64 * 1024 * 1024, + }); + return response.data; + } catch (err) { + throw classify(err); + } +} + +/** + * Sidecar identity + liveness, for the admin connection test. + * Returns { ok: true, info } or { ok: false, error } — never throws, because + * the caller is a UI button and a stack trace helps nobody there. + */ +async function checkHealth() { + try { + const { data } = await axios.get(`${getSidecarUrl()}/info`, { + headers: authHeaders(), + timeout: 5000, + }); + return { ok: true, info: data }; + } catch (err) { + const status = err.response?.status; + if (status === 401) { + return { ok: false, error: 'Sidecar rejected the token (check FACE_ML_TOKEN on both containers)' }; + } + return { ok: false, error: err.message || 'Sidecar unreachable' }; + } +} + +module.exports = { + detectFaces, + checkHealth, + SidecarUnavailableError, + SidecarRejectedError, +}; diff --git a/backend/src/services/faceClustering.js b/backend/src/services/faceClustering.js new file mode 100644 index 00000000..36ced23e --- /dev/null +++ b/backend/src/services/faceClustering.js @@ -0,0 +1,533 @@ +/** + * Per-event face clustering (#1074). + * + * Plain JS over a few thousand rows. A gallery is hundreds to low-thousands + * of faces, so this needs no vector database and no native extension — the + * embeddings live in a BLOB column and the maths is a dot product. + * + * Algorithm: greedy incremental assignment. A new face is compared against + * every existing person centroid in the event; above the match threshold it + * joins the nearest one and updates that centroid as a running mean, + * otherwise it opens a new person. + * + * Greedy assignment is order-dependent by construction — the same photos + * uploaded in a different order can produce different clusters. That is + * accepted here (it is what Immich, PhotoPrism and Ente all do) and mitigated + * by `consolidate()`, which merges centroid pairs that have drifted together, + * plus the admin merge/split tools. What it buys is that a photo can be + * clustered the moment it is scanned, so the strip fills in during a backfill + * instead of after it. + * + * Embeddings arrive L2-normalized from the sidecar, so cosine similarity is a + * plain dot product. Centroids are re-normalized after every update to keep + * that true. + */ + +const { db } = require('../database/db'); +const logger = require('../utils/logger'); +const { getThresholds } = require('./faceSettings'); + +const FLOAT_BYTES = 4; + +// -- serialization ----------------------------------------------------------- + +/** + * Float32Array → Buffer for the BLOB column. Little-endian on every platform + * we ship (x86_64, aarch64), and the value never leaves the deployment, so + * no byte-order header is needed. + */ +function packEmbedding(vec) { + const arr = vec instanceof Float32Array ? vec : Float32Array.from(vec); + return Buffer.from(arr.buffer, arr.byteOffset, arr.byteLength); +} + +function unpackEmbedding(buf) { + if (!buf) return null; + // Postgres bytea comes back as Buffer; SQLite may hand back a Uint8Array. + const b = Buffer.isBuffer(buf) ? buf : Buffer.from(buf); + if (b.length % FLOAT_BYTES !== 0) return null; + // Copy rather than aliasing: a Buffer from the driver may be a view into a + // larger pooled allocation, and Float32Array over an unaligned offset + // throws. + const copy = Buffer.from(b); + return new Float32Array(copy.buffer, copy.byteOffset, copy.length / FLOAT_BYTES); +} + +// -- vector maths ------------------------------------------------------------ + +function dot(a, b) { + let sum = 0; + const n = Math.min(a.length, b.length); + for (let i = 0; i < n; i++) sum += a[i] * b[i]; + return sum; +} + +function normalize(vec) { + let sumSq = 0; + for (let i = 0; i < vec.length; i++) sumSq += vec[i] * vec[i]; + const norm = Math.sqrt(sumSq); + if (norm === 0) return vec; + const out = new Float32Array(vec.length); + for (let i = 0; i < vec.length; i++) out[i] = vec[i] / norm; + return out; +} + +/** + * Running mean of `count` existing vectors with one new vector, re-normalized. + */ +function updateCentroid(centroid, count, incoming) { + const out = new Float32Array(centroid.length); + for (let i = 0; i < centroid.length; i++) { + out[i] = (centroid[i] * count + incoming[i]) / (count + 1); + } + return normalize(out); +} + +// -- quality ----------------------------------------------------------------- + +/** + * Is this face good enough to define a person? + * + * Low-quality faces are still STORED and still shown in "this photo contains" + * — they just don't get assigned, so they can't spawn junk people or drag a + * good centroid off course. A blurry profile at 30px is a real detection and + * a terrible identity. + */ +function meetsQualityFloor(face, thresholds) { + if (face.det_score != null && face.det_score < thresholds.face_quality_min_score) return false; + const size = Math.min(face.bbox_w ?? 0, face.bbox_h ?? 0); + if (size < thresholds.face_quality_min_px) return false; + return true; +} + +// -- clustering -------------------------------------------------------------- + +/** + * Assign a set of freshly-inserted faces to people within one event. + * + * Loads the event's people once, mutates centroids in memory across the whole + * batch, then writes back — so a photo with five faces costs one read and one + * write pass rather than five of each. + */ +/** + * Per-event serialization for cluster assignment. + * + * assignFaces is read-modify-write over an event's people: it loads every + * centroid, mutates them in memory across the batch, then writes back. Two + * workers on the same event therefore lose updates — both read the same + * snapshot, and the second write clobbers the first (or both open a duplicate + * person for the same face). The default concurrency is 1, but the queue + * advertises multi-pod safety and the tunable exists, so this cannot rely on + * there only ever being one writer. + * + * In-process mutex + (on Postgres) a transaction-scoped advisory lock keyed on + * the event. The advisory lock is what covers multiple pods; the mutex avoids + * pointless lock round-trips within one process. SQLite is single-writer + * anyway, so the mutex alone is sufficient there. + */ +const eventLocks = new Map(); + +async function withEventLock(eventId, trx, fn) { + const previous = eventLocks.get(eventId) || Promise.resolve(); + let release; + const current = new Promise((resolve) => { release = resolve; }); + eventLocks.set(eventId, previous.then(() => current)); + + await previous; + try { + // pg_advisory_xact_lock is released automatically when the transaction + // ends, including on rollback — no leak if the caller throws. + if (trx && typeof trx.raw === 'function') { + const client = db.client.config.client; + const isPg = client === 'pg' || (typeof client === 'string' && client.includes('postgres')); + if (isPg) { + await trx.raw('SELECT pg_advisory_xact_lock(?, ?)', [1074, Number(eventId)]); + } + } + return await fn(); + } finally { + release(); + if (eventLocks.get(eventId) === current) eventLocks.delete(eventId); + } +} + +async function assignFaces(eventId, faceRows, options = {}) { + const thresholds = options.thresholds || (await getThresholds()); + const trx = options.trx || db; + + return withEventLock(eventId, options.trx, () => assignFacesLocked(eventId, faceRows, thresholds, trx)); +} + +async function assignFacesLocked(eventId, faceRows, thresholds, trx) { + const people = await trx('event_people') + .where({ event_id: eventId }) + .select('id', 'centroid', 'face_count_total', 'model_version'); + + const state = people.map((p) => ({ + id: p.id, + centroid: unpackEmbedding(p.centroid), + count: p.face_count_total || 0, + dirty: false, + modelVersion: p.model_version, + })).filter((p) => p.centroid); + + const assignments = []; + + for (const face of faceRows) { + const embedding = unpackEmbedding(face.embedding); + if (!embedding) continue; + + if (!meetsQualityFloor(face, thresholds)) { + assignments.push({ faceId: face.id, personId: null }); + continue; + } + + let best = null; + let bestScore = -Infinity; + for (const person of state) { + // Never compare across embedding spaces — a model change makes old + // centroids meaningless rather than merely stale. + if (person.modelVersion && face.model_version && person.modelVersion !== face.model_version) { + continue; + } + const score = dot(embedding, person.centroid); + if (score > bestScore) { + bestScore = score; + best = person; + } + } + + if (best && bestScore >= thresholds.face_match_threshold) { + best.centroid = updateCentroid(best.centroid, best.count, embedding); + best.count += 1; + best.dirty = true; + assignments.push({ faceId: face.id, personId: best.id }); + } else { + const [inserted] = await trx('event_people').insert({ + event_id: eventId, + centroid: packEmbedding(embedding), + face_count_total: 1, + model_version: face.model_version, + cover_face_id: face.id, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }).returning('id'); + const personId = typeof inserted === 'object' ? inserted.id : inserted; + + state.push({ + id: personId, + centroid: embedding, + count: 1, + dirty: false, + modelVersion: face.model_version, + }); + assignments.push({ faceId: face.id, personId }); + } + } + + for (const { faceId, personId } of assignments) { + await trx('photo_faces').where({ id: faceId }).update({ person_id: personId }); + } + + for (const person of state) { + if (!person.dirty) continue; + await trx('event_people').where({ id: person.id }).update({ + centroid: packEmbedding(person.centroid), + face_count_total: person.count, + updated_at: new Date().toISOString(), + }); + } + + return assignments; +} + +/** + * Merge people whose centroids have drifted together. + * + * Greedy assignment can open "Anna in daylight" and "Anna at the party" as + * two clusters when the first few photos of each were dissimilar. Once both + * have absorbed enough faces their centroids converge, and this pass catches + * that. Runs at a slightly stricter threshold than initial assignment: + * merging two established clusters is a bigger claim than adding one face to + * one of them, and an over-eager merge is much harder for a photographer to + * unpick than a missed one. + */ +async function consolidate(eventId, options = {}) { + const thresholds = options.thresholds || (await getThresholds()); + const mergeThreshold = Math.min(0.95, thresholds.face_match_threshold + 0.08); + + const people = await db('event_people') + .where({ event_id: eventId }) + .select('id', 'centroid', 'face_count_total', 'model_version', 'label'); + + const state = people + .map((p) => ({ ...p, vec: unpackEmbedding(p.centroid) })) + .filter((p) => p.vec); + + const merged = []; + const absorbed = new Set(); + + for (let i = 0; i < state.length; i++) { + if (absorbed.has(state[i].id)) continue; + for (let j = i + 1; j < state.length; j++) { + if (absorbed.has(state[j].id)) continue; + const a = state[i]; + const b = state[j]; + if (a.model_version !== b.model_version) continue; + + // Never silently merge two people the photographer has NAMED + // differently — that is a human assertion this heuristic does not get + // to overrule. + if (a.label && b.label && a.label !== b.label) continue; + + if (dot(a.vec, b.vec) >= mergeThreshold) { + await mergePeople(eventId, [b.id], a.id); + absorbed.add(b.id); + merged.push({ from: b.id, into: a.id }); + } + } + } + + if (merged.length) { + logger.info(`faceClustering: consolidated ${merged.length} person pair(s) in event ${eventId}`); + } + return merged; +} + +/** + * Move every face from `sourceIds` onto `targetId` and delete the sources. + * Recomputes the target centroid from its actual members rather than + * averaging the two centroids — cheap at this scale, and exact. + */ +async function mergePeople(eventId, sourceIds, targetId) { + const ids = sourceIds.filter((id) => id !== targetId); + if (!ids.length) return { moved: 0 }; + + return db.transaction(async (trx) => { + // Carry metadata forward before the sources are deleted. Without this a + // merge silently discards a photographer-entered name, or un-hides a + // person they had suppressed — the target keeps its own values where it + // has them, and inherits from a source only where it does not. + const target = await trx('event_people').where({ id: targetId }).first(); + const sources = await trx('event_people').whereIn('id', ids).select('label', 'is_hidden', 'is_ignored'); + + const inherited = {}; + if (target && !target.label) { + const named = sources.find((p) => p.label); + if (named) inherited.label = named.label; + } + // Suppression is one-way on merge: if ANY party was hidden or ignored, + // the survivor stays that way. Re-exposing someone by merging is the + // failure that matters; leaving them hidden is trivially reversible. + if (sources.some((p) => p.is_hidden) || target?.is_hidden) inherited.is_hidden = true; + if (sources.some((p) => p.is_ignored) || target?.is_ignored) inherited.is_ignored = true; + + const moved = await trx('photo_faces') + .where({ event_id: eventId }) + .whereIn('person_id', ids) + .update({ person_id: targetId }); + + await trx('event_people').where({ event_id: eventId }).whereIn('id', ids).del(); + + if (Object.keys(inherited).length) { + await trx('event_people').where({ id: targetId }) + .update({ ...inherited, updated_at: new Date().toISOString() }); + } + + await recomputeCentroid(targetId, trx); + return { moved }; + }); +} + +/** + * Pull `faceIds` out of their cluster into a brand-new person. + */ +async function splitPerson(eventId, personId, faceIds) { + if (!faceIds?.length) return null; + + return db.transaction(async (trx) => { + const faces = await trx('photo_faces') + .where({ event_id: eventId, person_id: personId }) + .whereIn('id', faceIds) + .select('id', 'embedding', 'model_version'); + + if (!faces.length) return null; + + const [inserted] = await trx('event_people').insert({ + event_id: eventId, + face_count_total: 0, + model_version: faces[0].model_version, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }).returning('id'); + const newPersonId = typeof inserted === 'object' ? inserted.id : inserted; + + await trx('photo_faces') + .whereIn('id', faces.map((f) => f.id)) + .update({ person_id: newPersonId }); + + await recomputeCentroid(newPersonId, trx); + await recomputeCentroid(personId, trx); + return newPersonId; + }); +} + +/** + * Recompute one person's centroid and count from its member faces. + * Deletes the person if it has no members left. + */ +async function recomputeCentroid(personId, trx = db) { + const faces = await trx('photo_faces') + .where({ person_id: personId }) + .select('id', 'embedding', 'det_score'); + + if (!faces.length) { + await trx('event_people').where({ id: personId }).del(); + return; + } + + const vectors = faces.map((f) => unpackEmbedding(f.embedding)).filter(Boolean); + if (!vectors.length) return; + + const mean = new Float32Array(vectors[0].length); + for (const vec of vectors) { + for (let i = 0; i < mean.length; i++) mean[i] += vec[i]; + } + for (let i = 0; i < mean.length; i++) mean[i] /= vectors.length; + + // Cover face: the highest-scoring detection in the cluster, so the avatar + // is the sharpest available crop of that person rather than whichever face + // happened to arrive first. + const cover = faces.reduce((a, b) => ((b.det_score ?? 0) > (a.det_score ?? 0) ? b : a)); + + await trx('event_people').where({ id: personId }).update({ + centroid: packEmbedding(normalize(mean)), + face_count_total: faces.length, + cover_face_id: cover.id, + updated_at: new Date().toISOString(), + }); +} + +/** + * Re-derive every cluster in an event from stored embeddings, without + * touching the sidecar. Needed after threshold tuning — the expensive part + * (inference) is already done and cached in the rows. + * + * Preserves labels by re-attaching them to whichever new cluster inherited + * the most faces from the old one. Without this, re-clustering a gallery + * would silently discard every name the photographer typed. + */ +async function recluster(eventId) { + const thresholds = await getThresholds(); + + // Remember every person carrying HUMAN state — a name, or a hidden/ignored + // decision. Keying this on `label` alone silently dropped the privacy flags + // of unnamed people: a bystander the photographer had suppressed came back + // guest-visible after one "Re-group people". + const previousLabels = await db('event_people') + .where({ event_id: eventId }) + .where(function () { + this.whereNotNull('label').orWhere('is_hidden', true).orWhere('is_ignored', true); + }) + .select('id', 'label', 'is_hidden', 'is_ignored'); + + const priorMembership = new Map(); + if (previousLabels.length) { + const rows = await db('photo_faces') + .where({ event_id: eventId }) + .whereIn('person_id', previousLabels.map((p) => p.id)) + .select('id', 'person_id'); + for (const row of rows) priorMembership.set(row.id, row.person_id); + } + + await db.transaction(async (trx) => { + await trx('photo_faces').where({ event_id: eventId }).update({ person_id: null }); + await trx('event_people').where({ event_id: eventId }).del(); + }); + + const faces = await db('photo_faces') + .where({ event_id: eventId }) + .orderBy('id', 'asc') + .select('id', 'embedding', 'model_version', 'det_score', 'bbox_w', 'bbox_h'); + + const assignments = await assignFaces(eventId, faces, { thresholds }); + + // Re-attach labels by majority inheritance. + if (previousLabels.length) { + const tally = new Map(); // newPersonId -> Map + for (const { faceId, personId } of assignments) { + if (personId == null) continue; + const oldId = priorMembership.get(faceId); + if (oldId == null) continue; + if (!tally.has(personId)) tally.set(personId, new Map()); + const inner = tally.get(personId); + inner.set(oldId, (inner.get(oldId) || 0) + 1); + } + + // Suppression is OR-ed across EVERY ancestor that contributed faces to a + // new cluster — not copied from the majority one. Reclustering can merge a + // visible named person with a hidden one; taking the majority ancestor's + // flags would then publish the hidden person's photos. Erring toward + // staying hidden is trivially reversible; erring toward visible is not. + const suppression = new Map(); // newId -> { is_hidden, is_ignored } + for (const [newId, inner] of tally) { + let hidden = false; + let ignored = false; + for (const oldId of inner.keys()) { + const old = previousLabels.find((p) => p.id === oldId); + if (!old) continue; + if (old.is_hidden) hidden = true; + if (old.is_ignored) ignored = true; + } + suppression.set(newId, { is_hidden: hidden, is_ignored: ignored }); + } + + // The NAME goes to exactly one cluster: the descendant that inherited the + // MOST of that old person's faces, chosen globally. Iterating `tally` and + // taking the first match gave the name to whichever cluster happened to + // come first — an early outlier could take it while the real majority + // cluster ended up unnamed. + const bestDescendant = new Map(); // oldId -> { newId, count } + for (const [newId, inner] of tally) { + for (const [oldId, count] of inner) { + const current = bestDescendant.get(oldId); + if (!current || count > current.count) bestDescendant.set(oldId, { newId, count }); + } + } + + const labelFor = new Map(); // newId -> label + for (const [oldId, { newId }] of bestDescendant) { + const old = previousLabels.find((p) => p.id === oldId); + if (old?.label && !labelFor.has(newId)) labelFor.set(newId, old.label); + } + + for (const [newId, flags] of suppression) { + const update = { ...flags, updated_at: new Date().toISOString() }; + if (labelFor.has(newId)) update.label = labelFor.get(newId); + await db('event_people').where({ id: newId }).update(update); + } + } + + for (const personId of new Set(assignments.map((a) => a.personId).filter(Boolean))) { + await recomputeCentroid(personId); + } + await consolidate(eventId, { thresholds }); + + const count = await db('event_people').where({ event_id: eventId }).count({ c: '*' }).first(); + logger.info(`faceClustering: reclustered event ${eventId} → ${count?.c ?? 0} people`); + return Number(count?.c ?? 0); +} + +module.exports = { + packEmbedding, + unpackEmbedding, + dot, + normalize, + meetsQualityFloor, + assignFaces, + consolidate, + mergePeople, + splitPerson, + recomputeCentroid, + recluster, +}; diff --git a/backend/src/services/facePeopleService.js b/backend/src/services/facePeopleService.js new file mode 100644 index 00000000..414205e7 --- /dev/null +++ b/backend/src/services/facePeopleService.js @@ -0,0 +1,247 @@ +/** + * People queries for the gallery and admin surfaces (#1074). + * + * THE RULE THIS MODULE EXISTS TO ENFORCE: a guest's view of "who is in this + * gallery" must be computed over the photos that guest can actually see. + * + * Guests are restricted to `photos.visibility = 'visible'` (gallery.js); + * PIN-clients see everything. `event_people.face_count_total` counts ALL + * faces, so handing it to a guest leaks the existence and volume of hidden + * photos — and a cover face chosen from a hidden photo would render a crop of + * an image the guest was never allowed to open. Neither is theoretical: a + * photographer hides shots precisely because someone should not see them. + * + * So the guest-facing count and cover are recomputed per request against the + * same predicate the photo query uses. `face_count_total` is never returned + * to a guest. + */ + +const { db } = require('../database/db'); + +/** + * Apply the visibility predicate a given audience is allowed to see. + * Mirrors gallery.js: guests get 'visible' (or NULL, pre-migration rows), + * clients get everything. Also excludes photos still being processed, which + * are invisible in the gallery payload too. + */ +function applyVisibilityScope(query, { isClient }) { + query.where(function () { + this.where('photos.processing_status', 'complete').orWhereNull('photos.processing_status'); + }); + if (!isClient) { + query.where(function () { + this.where('photos.visibility', 'visible').orWhereNull('photos.visibility'); + }); + } + return query; +} + +/** + * People in an event, scoped to what this audience can see. + * + * Returns [{ id, label, face_count, cover: { photo_id, bbox } }] ordered by + * visible photo count. People whose visible count falls below `minClusterSize` + * are dropped — a person who appears in three hidden photos and one visible + * one is not "in this gallery" from the guest's point of view. + */ +async function listPeople(eventId, { isClient = false, forAdmin = false, minClusterSize = 3 } = {}) { + const people = await db('event_people') + .where({ event_id: eventId }) + .modify((q) => { + if (!forAdmin) { + // Hidden = photographer-only. Ignored = bystander/false positive. + // Neither ever reaches a guest response. + q.where('is_hidden', false).where('is_ignored', false); + } + }) + .select('id', 'label', 'is_hidden', 'is_ignored', 'face_count_total', 'cover_face_id'); + + if (!people.length) return []; + + // One grouped query for the audience-scoped counts, rather than N queries. + const counts = await applyVisibilityScope( + db('photo_faces') + .join('photos', 'photos.id', 'photo_faces.photo_id') + .where('photo_faces.event_id', eventId) + .whereNotNull('photo_faces.person_id'), + { isClient } + ) + .groupBy('photo_faces.person_id') + .select('photo_faces.person_id') + .countDistinct({ visible_photos: 'photos.id' }); + + const countByPerson = new Map( + counts.map((r) => [r.person_id, Number(r.visible_photos) || 0]) + ); + + // Cover faces, also scoped: the stored cover_face_id may point at a face in + // a photo this audience cannot see. Pick the best VISIBLE face instead. + const covers = await applyVisibilityScope( + db('photo_faces') + .join('photos', 'photos.id', 'photo_faces.photo_id') + .where('photo_faces.event_id', eventId) + .whereNotNull('photo_faces.person_id'), + { isClient } + ) + .orderBy('photo_faces.det_score', 'desc') + .select( + 'photo_faces.id', + 'photo_faces.person_id', + 'photo_faces.photo_id', + 'photo_faces.bbox_x', + 'photo_faces.bbox_y', + 'photo_faces.bbox_w', + 'photo_faces.bbox_h', + // The bbox is in ORIGINAL image pixels, so any consumer cropping it has + // to know what those pixels were measured against. Without these the + // admin manager was scaling an original-space box by a THUMBNAIL's + // natural size and rendering the wrong region entirely. + 'photos.width as photo_width', + 'photos.height as photo_height' + ); + + const coverByPerson = new Map(); + for (const row of covers) { + // Rows arrive best-score-first, so the first hit per person wins. + if (!coverByPerson.has(row.person_id)) coverByPerson.set(row.person_id, row); + } + + const out = []; + for (const person of people) { + const count = countByPerson.get(person.id) || 0; + const cover = coverByPerson.get(person.id); + + // A person with no visible photos, or too few to be worth a face in the + // strip, simply does not exist for this audience. + if (!forAdmin && (count < minClusterSize || !cover)) continue; + if (forAdmin && !cover && count === 0) { + out.push({ + id: person.id, + label: person.label || null, + face_count: 0, + total_face_count: person.face_count_total, + is_hidden: !!person.is_hidden, + is_ignored: !!person.is_ignored, + cover: null, + }); + continue; + } + + const entry = { + id: person.id, + label: person.label || null, + face_count: count, + cover: cover + ? { + face_id: cover.id, + photo_id: cover.photo_id, + bbox: [cover.bbox_x, cover.bbox_y, cover.bbox_w, cover.bbox_h], + photo_width: cover.photo_width ?? null, + photo_height: cover.photo_height ?? null, + } + : null, + }; + + if (forAdmin) { + // face_count_total is admin-only by design — see the module header. + entry.total_face_count = person.face_count_total; + entry.is_hidden = !!person.is_hidden; + entry.is_ignored = !!person.is_ignored; + } + + out.push(entry); + } + + return out.sort((a, b) => b.face_count - a.face_count); +} + +/** + * Map of photoId → person ids, for the photos in `photoIds`. + * + * Only ever called with ids already present in the gallery payload, which is + * itself visibility-filtered — so this cannot widen what a caller sees. It + * still filters hidden/ignored people, so a hidden person leaves no trace on + * a photo the guest CAN see. + */ +async function getPersonIdsByPhoto(eventId, photoIds, { forAdmin = false } = {}) { + if (!photoIds?.length) return new Map(); + + const rows = await db('photo_faces') + .join('event_people', 'event_people.id', 'photo_faces.person_id') + .where('photo_faces.event_id', eventId) + .whereIn('photo_faces.photo_id', photoIds) + .whereNotNull('photo_faces.person_id') + .modify((q) => { + if (!forAdmin) { + q.where('event_people.is_hidden', false).where('event_people.is_ignored', false); + } + }) + .select('photo_faces.photo_id', 'photo_faces.person_id'); + + const map = new Map(); + for (const row of rows) { + if (!map.has(row.photo_id)) map.set(row.photo_id, []); + const list = map.get(row.photo_id); + if (!list.includes(row.person_id)) list.push(row.person_id); + } + return map; +} + +/** + * Scan progress for the admin status line and the gallery's "Finding + * people… 240/1200" indicator. + */ +async function getScanStatus(eventId, { isClient = true } = {}) { + // `isClient` defaults to TRUE (the admin/photographer view) because every + // existing caller is an admin surface. A guest must pass isClient:false: + // counting every photo with a face_status would otherwise tell them how + // many hidden photos the gallery holds — the same leak the people list and + // covers are already scoped against, arriving through the progress bar. + const rows = await applyVisibilityScope( + db('photos').where({ event_id: eventId }).whereNotNull('face_status'), + { isClient } + ) + .groupBy('face_status') + .select('face_status') + .count({ c: '*' }); + + const byStatus = Object.fromEntries(rows.map((r) => [r.face_status, Number(r.c) || 0])); + const done = byStatus.done || 0; + const total = Object.values(byStatus).reduce((a, b) => a + b, 0); + + const peopleRow = await db('event_people') + .where({ event_id: eventId }) + .count({ c: '*' }) + .first(); + + // Two counts, deliberately. `people` is every cluster that exists; + // `people_visible_to_guests` applies the minimum-cluster-size floor and the + // hidden/ignored flags, i.e. what the gallery strip actually shows. An + // admin comparing the settings page against their own gallery will + // otherwise see two numbers that disagree with no explanation. + const { getThresholds } = require('./faceSettings'); + const thresholds = await getThresholds(); + const visible = await listPeople(eventId, { + isClient: false, + forAdmin: false, + minClusterSize: thresholds.face_min_cluster_size, + }); + + return { + scanned: done, + total, + pending: (byStatus.pending || 0) + (byStatus.processing || 0), + failed: byStatus.failed || 0, + skipped: byStatus.skipped || 0, + people: Number(peopleRow?.c ?? 0), + people_visible_to_guests: visible.length, + in_progress: ((byStatus.pending || 0) + (byStatus.processing || 0)) > 0, + }; +} + +module.exports = { + listPeople, + getPersonIdsByPhoto, + getScanStatus, + applyVisibilityScope, +}; diff --git a/backend/src/services/faceProcessor.js b/backend/src/services/faceProcessor.js new file mode 100644 index 00000000..482dda54 --- /dev/null +++ b/backend/src/services/faceProcessor.js @@ -0,0 +1,325 @@ +/** + * Per-photo face detection (#1074). + * + * Runs on the PREVIEW rendition, not the original. A 1920px JPEG is ~40x + * cheaper to decode than a 45MP original and costs nothing in recall at the + * face sizes an event gallery actually contains. + * + * Note the side effect that has: on an install with `lightbox_preview_enabled` + * off, no previews exist, so a face backfill generates the whole preview tier + * as it goes — a real Sharp workload and real disk. That is why the admin + * toggle warns about it and why the face queue defaults to concurrency 1. + */ + +const sharp = require('sharp'); +const { db } = require('../database/db'); +const logger = require('../utils/logger'); +const { getStorage } = require('./storage'); +const { ensurePreviewImage } = require('./imageProcessor'); +// SidecarUnavailableError is deliberately NOT caught here — it propagates to +// faceQueue, which is the only layer that knows to retry rather than fail. +const { detectFaces } = require('./faceClient'); +const { assignFaces, packEmbedding, recomputeCentroid } = require('./faceClustering'); +const { getThresholds, isEnabledForEvent } = require('./faceSettings'); + +/** + * Thrown when a scan finishes but the photo is no longer the row we claimed — + * the event was purged, archived or re-queued mid-flight. Not a failure of + * the photo, so the queue must not mark it 'failed'. + */ +class StaleScanError extends Error { + constructor(photoId) { + super(`Face scan for photo ${photoId} was superseded`); + this.name = 'StaleScanError'; + } +} + +async function streamToBuffer(stream) { + if (Buffer.isBuffer(stream)) return stream; + const chunks = []; + for await (const chunk of stream) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + return Buffer.concat(chunks); +} + +/** + * Detect and cluster the faces in one photo. + * + * Throws SidecarUnavailableError when the sidecar is down — the caller must + * return the row to 'pending' rather than failing it. + */ +async function processPhotoFaces(photoId) { + const photo = await db('photos').where({ id: photoId }).first(); + if (!photo) { + logger.warn(`faceProcessor: photo ${photoId} disappeared before processing`); + return { status: 'skipped' }; + } + + const event = await db('events').where({ id: photo.event_id }).first(); + if (!(await isEnabledForEvent(event))) { + // The toggle was switched off between enqueue and claim. Not an error. + await db('photos').where({ id: photoId }).update({ + face_status: 'skipped', face_started_at: null, face_error: null, + }); + return { status: 'skipped' }; + } + + // Videos have no preview tier and no meaningful single frame to scan. + const isVideo = photo.media_type === 'video' + || (photo.mime_type && photo.mime_type.startsWith('video/')); + if (isVideo) { + await db('photos').where({ id: photoId }).update({ + face_status: 'skipped', face_started_at: null, face_count: 0, + }); + return { status: 'skipped' }; + } + + // External / reference photos live outside managed storage, and + // resolvePhotoStorageKey returns null for them by design (photoResolver.js). + // ensurePreviewImage therefore cannot build a preview, so scanning them is + // unsupported rather than broken — 'skipped', not 'failed', so an external + // gallery does not report every photo as an error. + if (photo.source_origin === 'external' || photo.source_origin === 'reference') { + await db('photos').where({ id: photoId }).update({ + face_status: 'skipped', face_started_at: null, face_error: null, + }); + return { status: 'skipped' }; + } + + const previewKey = await ensurePreviewImage(photo); + if (!previewKey) { + // No preview and no way to make one — the source is missing or corrupt. + // That is a property of this photo, so it fails rather than retries. + await db('photos').where({ id: photoId }).update({ + face_status: 'failed', + face_started_at: null, + face_error: 'Could not generate a preview rendition to scan', + }); + return { status: 'failed' }; + } + + const storage = getStorage(); + const buffer = await streamToBuffer(await storage.get(previewKey)); + + // Throws SidecarUnavailableError upward on a down sidecar — deliberately + // not caught here, so the queue can distinguish "retry" from "failed". + const result = await detectFaces(buffer, photo.filename || 'photo.jpg'); + const faces = result?.faces || []; + const modelVersion = result?.model_version || null; + + // Bounding boxes come back in the pixel space of the image the SIDECAR was + // given — which is the preview (≤1920px long edge), not the original. Every + // consumer compares them against photos.width/height, which are the + // ORIGINAL dimensions: the strip's avatar crop works in ratios of them, and + // the auto-category portrait rule divides face area by frame area. + // + // Left unscaled, a 6000px photo yields boxes ~3x too small (and areas ~9x + // too small), so avatars crop to the wrong place and "Portraits" never + // fires. It is invisible on any photo already under 1920px, which is why + // the demo gallery looked correct. + // + // Scale here, once, so everything downstream can assume original-image + // coordinates. + let boxScale = 1; + try { + const previewMeta = await sharp(buffer).metadata(); + if (previewMeta?.width && photo.width) { + boxScale = photo.width / previewMeta.width; + } + } catch (err) { + logger.warn(`faceProcessor: could not read preview dimensions for photo ${photoId}`, { + error: err.message, + }); + } + + const thresholds = await getThresholds(); + + try { + await db.transaction(async (trx) => { + // Replace rather than append: a re-scan of the same photo must not + // double its faces. Detaching first keeps the FK to event_people clean. + // + // Remember which people the OLD faces belonged to. Deleting the rows does + // not undo their contribution to event_people.face_count_total or to the + // running-mean centroid, so without recomputing those afterwards a + // re-scan inflates every count (typically doubling it) and leaves ghost + // people behind when a face is no longer detected. + const affectedPeople = await trx('photo_faces') + .where({ photo_id: photoId }) + .whereNotNull('person_id') + .distinct('person_id') + .pluck('person_id'); + + await trx('photo_faces').where({ photo_id: photoId }).del(); + + const inserted = []; + for (const face of faces) { + const [bx, by, bw, bh] = (face.bbox || [0, 0, 0, 0]).map((v) => v * boxScale); + const row = { + photo_id: photoId, + event_id: photo.event_id, + bbox_x: bx, bbox_y: by, bbox_w: bw, bbox_h: bh, + det_score: face.score ?? null, + yaw: face.yaw ?? null, + pitch: face.pitch ?? null, + blur: face.blur ?? null, + embedding: face.embedding ? packEmbedding(face.embedding) : null, + model_version: modelVersion, + // ISO string, not a Date — under Jest a Date handed to the sqlite3 + // binding stores as the literal "[object Object]" (see CLAUDE.md). + created_at: new Date().toISOString(), + }; + const [id] = await trx('photo_faces').insert(row).returning('id'); + inserted.push({ ...row, id: typeof id === 'object' ? id.id : id }); + } + + // Rebuild the people the removed faces belonged to BEFORE assigning the + // replacements, so assignment compares against honest centroids. + for (const personId of affectedPeople) { + await recomputeCentroid(personId, trx); + } + + if (inserted.length) { + await assignFaces(photo.event_id, inserted, { thresholds, trx }); + } + + // Commit the photo only if it is STILL the row we claimed. An admin who + // purges or archives the event while this worker was waiting on the + // sidecar has already cleared face_status; without this guard the worker + // would write its rows back moments after "all face data deleted" + // reported success, so an erasure request would silently not stick. + const committed = await trx('photos') + .where({ id: photoId, face_status: 'processing' }) + .update({ + face_status: 'done', + face_count: faces.length, + face_started_at: null, + face_error: null, + }); + + if (committed === 0) { + logger.info( + `faceProcessor: photo ${photoId} was purged or re-queued while scanning — discarding results` + ); + // Undo this transaction's inserts rather than leaving orphans behind. + throw new StaleScanError(photoId); + } + }); + } catch (err) { + // Losing a race with a purge is an expected outcome, not a photo that + // failed to scan. The transaction has already rolled back, so nothing was + // written; leave face_status exactly as the purge left it. + if (err instanceof StaleScanError) return { status: 'skipped' }; + throw err; + } + + // Auto-categories (#1074 phase 3) run as a distinct step AFTER detection, + // outside the transaction: they are a convenience, and a rule-engine + // failure must never roll back the faces we just paid the sidecar for. + // No-ops unless separately enabled. + try { + const { categorizeEvent } = require('./faceAutoCategories'); + await categorizeEvent(photo.event_id); + } catch (err) { + logger.warn(`faceProcessor: auto-categorisation failed for event ${photo.event_id}`, { + error: err.message, + }); + } + + return { status: 'done', faceCount: faces.length }; +} + +/** + * Queue every eligible photo in an event for (re-)scanning. + * + * `onlyUnscanned` is the backfill case — enabling the toggle on a gallery + * that already has photos. Without it, this is a full re-scan. + */ +async function enqueueEvent(eventId, { onlyUnscanned = false } = {}) { + const query = db('photos') + .where({ event_id: eventId }) + // Photos still being processed have no preview and no dimensions yet; + // photoProcessor enqueues them itself on completion. + .where(function () { + this.where('processing_status', 'complete').orWhereNull('processing_status'); + }); + + if (onlyUnscanned) { + query.where(function () { + this.whereNull('face_status').orWhereIn('face_status', ['failed', 'skipped']); + }); + } + + const updated = await query.update({ + face_status: 'pending', + face_started_at: null, + face_error: null, + }); + + logger.info(`faceProcessor: queued ${updated} photo(s) for face detection in event ${eventId}`); + return updated; +} + +/** + * Remove every face row for an event. Used by "delete all face data", by the + * per-event toggle going off, and by archival. + */ +async function purgeEvent(eventId) { + return db.transaction(async (trx) => { + // Detach before deleting people so the SET NULL FK never fires against + // rows that are about to go anyway. + await trx('photo_faces').where({ event_id: eventId }).update({ person_id: null }); + const faces = await trx('photo_faces').where({ event_id: eventId }).del(); + const people = await trx('event_people').where({ event_id: eventId }).del(); + await trx('photos').where({ event_id: eventId }).update({ + face_status: null, face_count: null, face_started_at: null, face_error: null, + }); + logger.info(`faceProcessor: purged ${faces} face(s) and ${people} person(s) from event ${eventId}`); + return { faces, people }; + }); +} + +/** + * Remove the face rows belonging to one photo, and rebuild the people that + * lose members as a result. + * + * Called from every photo-deletion path. The schema declares ON DELETE + * CASCADE, but SQLite only enforces foreign keys when `PRAGMA foreign_keys` + * is on and PicPeak does not enable it globally — so on SQLite the cascade + * never fires and biometric embeddings would outlive the photo. Even where it + * does fire (Postgres), the cascade cannot fix up event_people counts or + * centroids, which is the other half of this. + * + * Safe to call for photos that were never scanned: it simply deletes nothing. + */ +async function purgePhotoFaces(photoId, trx = db) { + const affected = await trx('photo_faces') + .where({ photo_id: photoId }) + .whereNotNull('person_id') + .distinct('person_id') + .pluck('person_id'); + + // Drop any in-flight claim as well. A worker holding this photo would + // otherwise still satisfy its `face_status = 'processing'` commit guard and + // write fresh faces straight after the purge — and with FK enforcement off + // on SQLite, the subsequent photo delete cannot cascade them away, leaving + // orphaned biometric rows. + await trx('photos').where({ id: photoId }).update({ + face_status: null, face_count: null, face_started_at: null, face_error: null, + }); + + const removed = await trx('photo_faces').where({ photo_id: photoId }).del(); + if (!removed) return { removed: 0 }; + + const { recomputeCentroid: recompute } = require('./faceClustering'); + for (const personId of affected) { + // Deletes the person outright when it has no members left. + await recompute(personId, trx); + } + return { removed, peopleTouched: affected.length }; +} + +module.exports = { + processPhotoFaces, enqueueEvent, purgeEvent, purgePhotoFaces, StaleScanError, +}; diff --git a/backend/src/services/faceQueue.js b/backend/src/services/faceQueue.js new file mode 100644 index 00000000..722b24a9 --- /dev/null +++ b/backend/src/services/faceQueue.js @@ -0,0 +1,210 @@ +/** + * Background face-detection worker pool (#1074). + * + * Deliberately a near-copy of backgroundProcessor.js rather than a shared + * abstraction: the two differ in exactly one behaviour (below), and inventing + * a generic queue framework to save forty lines would make both harder to + * read. Claim semantics, the janitor and the tunable shape are identical, so + * anyone who understands one understands the other. + * + * THE ONE DIFFERENCE — a sidecar that is unreachable puts the photo back to + * 'pending' with backoff, never 'failed'. Turning the ML container off for a + * week must not require a manual re-scan afterwards. Only a 4xx (a genuinely + * unprocessable image) marks a row failed. + * + * Nothing here starts unless the `faces` feature flag is on. That matters + * more than usual because FACE_ML_URL now has a working default, so "is the + * variable set" is no longer a signal — without the flag check, every install + * would poll a hostname that does not resolve. + * + * Tunables (env, all optional): + * FACE_PROCESSOR_CONCURRENCY default 1. Face scanning shares a host + * with Sharp; 1 is the safe default on + * the 2GB VPS class this project supports. + * FACE_PROCESSOR_POLL_MS default 2000 + * FACE_PROCESSOR_STUCK_TIMEOUT_MS default 600000 (10 minutes) + * FACE_PROCESSOR_DISABLED default false ('true' to opt out, e.g. CI) + */ + +const { db } = require('../database/db'); +const logger = require('../utils/logger'); +const { processPhotoFaces } = require('./faceProcessor'); +const { SidecarUnavailableError } = require('./faceClient'); +const { isFeatureEnabled } = require('./faceSettings'); + +const POLL_INTERVAL_MS = parseInt(process.env.FACE_PROCESSOR_POLL_MS || '2000', 10); +const CONCURRENCY = Math.max(1, parseInt(process.env.FACE_PROCESSOR_CONCURRENCY || '1', 10)); +const STUCK_TIMEOUT_MS = parseInt(process.env.FACE_PROCESSOR_STUCK_TIMEOUT_MS || '600000', 10); +const JANITOR_INTERVAL_MS = 60 * 1000; + +// Backoff after the sidecar goes away. Without it a down sidecar turns into a +// hot loop: claim, fail, release, claim again, thousands of times a minute. +const UNAVAILABLE_BACKOFF_MS = parseInt(process.env.FACE_PROCESSOR_BACKOFF_MS || '30000', 10); + +let running = false; +let workerHandles = []; +let janitorHandle = null; + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +function isPostgres() { + const c = db.client.config.client; + return c === 'pg' || (typeof c === 'string' && c.includes('postgres')); +} + +/** + * Atomically claim the oldest pending photo. Returns the row or null. + * Same two-path approach as backgroundProcessor: SKIP LOCKED on Postgres so + * multiple pods race cleanly, a status-guarded UPDATE on SQLite. + */ +async function claimNextPhoto() { + if (isPostgres()) { + return db.transaction(async (trx) => { + const row = await trx('photos') + .where('face_status', 'pending') + .orderBy('id', 'asc') + .forUpdate() + .skipLocked() + .first(); + if (!row) return null; + await trx('photos').where('id', row.id).update({ + face_status: 'processing', + face_started_at: new Date().toISOString(), + }); + return row; + }); + } + + return db.transaction(async (trx) => { + const row = await trx('photos') + .where('face_status', 'pending') + .orderBy('id', 'asc') + .first(); + if (!row) return null; + const updated = await trx('photos') + .where({ id: row.id, face_status: 'pending' }) + .update({ + face_status: 'processing', + face_started_at: new Date().toISOString(), + }); + return updated > 0 ? row : null; + }); +} + +async function releaseToPending(photoId) { + // Guarded on 'processing' — the state THIS worker put the row in. If the + // event was purged while the sidecar request was in flight, purgeEvent has + // already set face_status to NULL, and an unconditional update would put it + // back to 'pending' and rescan it: biometric rows reappearing after the + // purge endpoint reported success. Same reasoning as the commit guard in + // faceProcessor; the retry path needed it too. + await db('photos') + .where({ id: photoId, face_status: 'processing' }) + .update({ face_status: 'pending', face_started_at: null }); +} + +async function workerLoop(workerIdx) { + while (running) { + // Re-checked every tick, not once at startup: an admin turning the flag + // off must stop the workers without a restart. + if (!(await isFeatureEnabled())) { + await sleep(POLL_INTERVAL_MS * 5); + continue; + } + + let claimed; + try { + claimed = await claimNextPhoto(); + } catch (e) { + logger.warn(`faceQueue[${workerIdx}]: claim error`, { error: e.message }); + await sleep(POLL_INTERVAL_MS); + continue; + } + + if (!claimed) { + await sleep(POLL_INTERVAL_MS); + continue; + } + + try { + await processPhotoFaces(claimed.id); + } catch (err) { + if (err instanceof SidecarUnavailableError) { + // Retry, don't fail. faceClient already rate-limits the log line. + await releaseToPending(claimed.id).catch(() => {}); + await sleep(UNAVAILABLE_BACKOFF_MS); + continue; + } + + logger.error(`faceQueue[${workerIdx}]: photo ${claimed.id} failed`, { + error: err.message, + stack: err.stack, + }); + try { + await db('photos').where({ id: claimed.id }).update({ + face_status: 'failed', + face_started_at: null, + face_error: String(err.message || err).slice(0, 1000), + }); + } catch (updateErr) { + logger.error(`faceQueue[${workerIdx}]: failed to mark photo ${claimed.id} as failed`, { + error: updateErr.message, + }); + } + } + } +} + +async function janitorLoop() { + while (running) { + try { + const cutoff = new Date(Date.now() - STUCK_TIMEOUT_MS).toISOString(); + const reset = await db('photos') + .where('face_status', 'processing') + .where('face_started_at', '<', cutoff) + .update({ face_status: 'pending', face_started_at: null }); + if (reset > 0) { + logger.warn(`faceQueue: janitor reset ${reset} stuck photo(s) from 'processing' to 'pending'`); + } + } catch (e) { + logger.warn('faceQueue: janitor error', { error: e.message }); + } + await sleep(JANITOR_INTERVAL_MS); + } +} + +function start() { + if (running) return; + if (process.env.FACE_PROCESSOR_DISABLED === 'true') { + logger.info('faceQueue: disabled via FACE_PROCESSOR_DISABLED'); + return; + } + + running = true; + workerHandles = []; + for (let i = 0; i < CONCURRENCY; i++) { + workerHandles.push( + workerLoop(i).catch((e) => + logger.error(`faceQueue[${i}]: crashed`, { error: e.message, stack: e.stack }) + ) + ); + } + janitorHandle = janitorLoop().catch((e) => + logger.error('faceQueue: janitor crashed', { error: e.message, stack: e.stack }) + ); + + logger.info( + `faceQueue: started ${CONCURRENCY} worker(s), poll=${POLL_INTERVAL_MS}ms, stuck=${STUCK_TIMEOUT_MS}ms ` + + '(idle until the `faces` feature flag is enabled)' + ); +} + +async function stop() { + if (!running) return; + running = false; + await Promise.all([...workerHandles, janitorHandle].filter(Boolean)); + workerHandles = []; + janitorHandle = null; +} + +module.exports = { start, stop, claimNextPhoto }; diff --git a/backend/src/services/faceSettings.js b/backend/src/services/faceSettings.js new file mode 100644 index 00000000..1d4ebc9b --- /dev/null +++ b/backend/src/services/faceSettings.js @@ -0,0 +1,170 @@ +/** + * Face-recognition settings and the feature gate (#1074). + * + * One module owns the answer to "is this feature on?", because the answer has + * three independent parts and getting any of them wrong is either a privacy + * problem or a log-spam problem: + * + * 1. The `faces` feature flag (global, default OFF). + * 2. `FACE_ML_URL` — which now has a DEFAULT (the compose service name), + * so its presence proves nothing. This is exactly why the flag is the + * gate: if the URL were the gate, every install would try to reach a + * hostname that doesn't resolve. + * 3. The per-event `face_recognition_enabled` toggle. + * + * Nothing may contact the sidecar unless (1) is true. `isFeatureEnabled()` is + * the only correct way to ask. + */ + +const { db } = require('../database/db'); +const { + isFeatureEnabled: isFlagEnabled, + invalidateFeatureFlagCache, +} = require('../middleware/requireFeatureFlag'); +const logger = require('../utils/logger'); + +// Canonical flag key. Must match the entry in adminFeatureFlags KNOWN_FLAGS +// and the FeatureKey union on the frontend. +const FLAG_KEY = 'faces'; + +/** + * Is this the all-in-one single-container image (#1042 / PR #1068)? + * + * Face recognition is BLOCKED there, on performance grounds. The AIO image + * runs the backend, the frontend, SQLite and every background worker inside + * one container sized for "one photographer plus guests browsing" — it has no + * Redis and SQLite gives it a single writer. Face detection would add a + * second image-processing pipeline competing with Sharp for the same CPU and + * RAM, on top of an ML sidecar the image does not contain and cannot start. + * + * Enabling it there would not fail loudly; it would just make the whole + * install slow and appear broken, which is the worst shape for a deployment + * aimed at people who want one container and no decisions. + * + * Detected from an explicit marker rather than inferred from SERVE_FRONTEND + * or a SQLite path: plenty of legitimate multi-container setups serve the + * frontend from the backend or run SQLite, and none of them should lose the + * feature by accident. + */ +function isSingleContainerImage() { + const v = String(process.env.PICPEAK_SINGLE_CONTAINER || '').toLowerCase(); + return v === 'true' || v === '1' || v === 'yes'; +} + +const DEFAULTS = { + // Measured on a real gallery, not just on LFW pairs — see migration 177 + // for why those two disagree and why cluster purity wins. + face_match_threshold: 0.60, + face_min_cluster_size: 3, + face_quality_min_score: 0.7, + face_quality_min_px: 40, + face_auto_categorize_enabled: false, +}; + +const SETTING_KEYS = Object.keys(DEFAULTS); + +function parseSetting(raw, fallback) { + if (raw === undefined || raw === null) return fallback; + try { + return JSON.parse(raw); + } catch { + // Pre-JSON rows (and plain strings written by hand) — coerce by the + // shape of the default so callers always get the type they expect. + if (typeof fallback === 'number') { + const n = Number(raw); + return Number.isFinite(n) ? n : fallback; + } + if (typeof fallback === 'boolean') return raw === 'true' || raw === '1'; + return raw; + } +} + +/** + * Is the `faces` feature flag on? This is THE gate — see the header. + * + * Delegates to the shared `feature_flags` reader (already cached, already + * invalidated by PUT /admin/feature-flags) rather than keeping a second + * cache that could disagree with the middleware guarding the same routes. + */ +async function isFeatureEnabled() { + // Hard block on the all-in-one image, ahead of the flag. Even if the row + // says true — restored from a backup taken on a full deployment, say — the + // feature stays off here. + if (isSingleContainerImage()) return false; + + try { + return await isFlagEnabled(FLAG_KEY); + } catch (e) { + // A missing table (pre-migration install) means off, not a crash. + logger.debug?.('faceSettings: feature flag read failed', { error: e.message }); + return false; + } +} + +/** + * Effective sidecar URL. Callers must check isFeatureEnabled() first — this + * always returns a string, because the default is a hostname rather than an + * absence. + */ +function getSidecarUrl() { + return (process.env.FACE_ML_URL || 'http://picpeak-ml:8000').replace(/\/+$/, ''); +} + +function getSidecarToken() { + return process.env.FACE_ML_TOKEN || ''; +} + +/** + * Clustering/quality thresholds, with defaults. Deliberately settings rather + * than constants: no single threshold survives contact with every library. + */ +async function getThresholds() { + const out = { ...DEFAULTS }; + try { + const rows = await db('app_settings') + .whereIn('setting_key', SETTING_KEYS) + .select('setting_key', 'setting_value'); + for (const row of rows) { + out[row.setting_key] = parseSetting(row.setting_value, DEFAULTS[row.setting_key]); + } + } catch (e) { + logger.debug?.('faceSettings: threshold read failed, using defaults', { error: e.message }); + } + return out; +} + +/** + * Is detection enabled for this specific event? Requires BOTH the global flag + * and the per-event toggle — the "two deliberate actions" rule from #1074 §6. + */ +async function isEnabledForEvent(event) { + if (!(await isFeatureEnabled())) return false; + if (!event) return false; + return event.face_recognition_enabled === true || event.face_recognition_enabled === 1; +} + +/** + * Do GUESTS of this event see the people strip? Separate from the above so a + * photographer can use clustering as a private tool. + * + * Defaults to true when detection is on, because the column is nullable and + * NULL means "not explicitly set" rather than "off" — matching the tri-state + * convention used by show_watermark / show_qr. + */ +function areFacesVisibleToGuests(event) { + if (!event) return false; + return event.faces_visible_to_guests !== false && event.faces_visible_to_guests !== 0; +} + +module.exports = { + DEFAULTS, + FLAG_KEY, + isSingleContainerImage, + isFeatureEnabled, + invalidateFeatureFlagCache, + getSidecarUrl, + getSidecarToken, + getThresholds, + isEnabledForEvent, + areFacesVisibleToGuests, +}; diff --git a/backend/src/services/photoProcessor.js b/backend/src/services/photoProcessor.js index 07b0055d..4bc2e36c 100644 --- a/backend/src/services/photoProcessor.js +++ b/backend/src/services/photoProcessor.js @@ -283,6 +283,25 @@ async function processUploadedPhotos(files, eventId, uploadedBy = 'admin', categ }); } catch (e) { /* non-fatal */ } + // Face detection (#1074). processPhoto() — the ASYNC path — enqueues on + // completion, but this synchronous path (chunked-upload completion, + // watch-folder import) writes a finished photo row directly and never + // reaches it, so those photos stayed unscanned in a face-enabled event + // until someone ran a manual re-scan. + try { + const { isEnabledForEvent } = require('./faceSettings'); + if (!isVideo && await isEnabledForEvent(event)) { + // `db`, NOT `trx`: the transaction is committed above, so a query + // through it throws "Transaction query already complete" — which the + // catch below swallowed, making this whole enqueue a silent no-op. + await db('photos').where({ id: photoId }).update({ face_status: 'pending' }); + } + } catch (err) { + logger.warn(`processUploadedPhotos: face enqueue failed for photo ${photoId}`, { + error: err.message, + }); + } + uploadedPhotos.push({ id: photoId, filename: newFilename, @@ -541,6 +560,23 @@ async function processPhoto(photoId) { // Mark complete updateData.processing_status = 'complete'; updateData.processing_error = null; + + // Face detection (#1074): this is the only correct place to enqueue. + // Earlier and there is no preview rendition to scan; later and there is no + // hook at all. Same UPDATE rather than a follow-up write, so a crash + // between the two can't leave a complete photo permanently unqueued. + // Guarded on BOTH the global flag and the per-event toggle, so installs + // without the feature never write a face_status at all. + try { + const { isEnabledForEvent } = require('./faceSettings'); + if (!isVideo && await isEnabledForEvent(event)) { + updateData.face_status = 'pending'; + } + } catch (err) { + // Never let the face feature block a photo from completing. + logger.warn(`processPhoto: face enqueue check failed for ${photoId}`, { error: err.message }); + } + await db('photos').where({ id: photoId }).update(updateData); // Side effects (best-effort, never fail the photo if these break) diff --git a/backend/src/services/photoReplacementService.js b/backend/src/services/photoReplacementService.js index 1944aeae..9fc918c9 100644 --- a/backend/src/services/photoReplacementService.js +++ b/backend/src/services/photoReplacementService.js @@ -122,6 +122,26 @@ async function replacePhoto(existingPhoto, newFileTempPath, { originalFilename, media_type: mimeType?.startsWith('video/') ? 'video' : 'image', }; + // Face data (#1074): the row keeps its id but now points at a DIFFERENT + // image, so every stored face describes the old picture. Left alone the + // gallery would keep showing the previous subject's identities on the new + // photo. Drop them, then re-queue if the event has detection on. + try { + const { purgePhotoFaces } = require('./faceProcessor'); + const { isEnabledForEvent } = require('./faceSettings'); + await purgePhotoFaces(existingPhoto.id); + + const event = await db('events').where({ id: existingPhoto.event_id }).first(); + updates.face_status = (await isEnabledForEvent(event)) ? 'pending' : null; + updates.face_count = null; + updates.face_started_at = null; + updates.face_error = null; + } catch (err) { + logger.warn(`replacePhoto: face reset failed for photo ${existingPhoto.id}`, { + error: err.message, + }); + } + await db('photos').where({ id: existingPhoto.id }).update(updates); const updatedPhoto = await db('photos').where({ id: existingPhoto.id }).first(); diff --git a/backend/src/services/photoService.js b/backend/src/services/photoService.js index 1bd8511a..a4cf7db8 100644 --- a/backend/src/services/photoService.js +++ b/backend/src/services/photoService.js @@ -6,6 +6,7 @@ */ const path = require('path'); +const logger = require('../utils/logger'); const fs = require('fs').promises; const { db } = require('../database/db'); const { formatBoolean } = require('../utils/dbCompat'); @@ -120,6 +121,15 @@ const deletePhoto = async (photoId, options = {}) => { } } + // Face data (#1074) — see purgePhotoFaces on why the FK cascade is not + // relied on. Must run BEFORE the photo row goes. + try { + const { purgePhotoFaces } = require('./faceProcessor'); + await purgePhotoFaces(photoId); + } catch (err) { + logger.warn(`deletePhoto: face purge failed for photo ${photoId}`, { error: err.message }); + } + // Delete from database await db('photos').where('id', photoId).delete(); } diff --git a/backend/src/services/picpeakExportService.js b/backend/src/services/picpeakExportService.js index 43c6aab0..4595b581 100644 --- a/backend/src/services/picpeakExportService.js +++ b/backend/src/services/picpeakExportService.js @@ -29,7 +29,20 @@ const packageJson = require('../../package.json'); const PICPEAK_FORMAT_VERSION = 1; // Never exported as data — the target owns these (its own migrations set them). -const EXCLUDED_TABLES = new Set(['knex_migrations', 'knex_migrations_lock']); +// +// photo_faces / event_people (#1074) are excluded for a different reason: +// face embeddings are biometric data (GDPR Art. 9), and this export gets +// handed to clients and moved between operators. The data is fully derived +// from the photos, so the target re-scans rather than receiving biometrics it +// has no lawful basis for. NOTE the cost of that decision: any names the +// photographer assigned to people are lost too, since they live in +// event_people. That is accepted — see ml/README.md and the #1074 thread. +const EXCLUDED_TABLES = new Set([ + 'knex_migrations', + 'knex_migrations_lock', + 'photo_faces', + 'event_people', +]); // Storage subdirs holding non-recalculable blobs — always included. const DOC_DIRS = ['business-docs', 'uploads']; @@ -90,7 +103,19 @@ async function writeTableNdjson(table, dataDir) { const hash = crypto.createHash('sha256'); const rows = await db(table).select('*'); const lines = rows.map((row) => { - const line = JSON.stringify(row); + // photo_faces / event_people are excluded from the export (#1074), so a + // restored install has no face data — but `photos.face_status = 'done'` + // would come across intact and the worker only ever claims 'pending'. + // The gallery would then report itself fully scanned while showing no + // people at all, permanently, with no way to tell why. + // + // Reset the derived state so the target simply re-scans once the operator + // enables the feature there. + const line = JSON.stringify( + table === 'photos' && (row.face_status !== null && row.face_status !== undefined) + ? { ...row, face_status: null, face_count: null, face_started_at: null, face_error: null } + : row + ); hash.update(`${line}\n`); return line; }); diff --git a/backend/src/services/picpeakImportService.js b/backend/src/services/picpeakImportService.js index dec5cd57..ee9c38ef 100644 --- a/backend/src/services/picpeakImportService.js +++ b/backend/src/services/picpeakImportService.js @@ -351,6 +351,21 @@ async function replaceAllTables(tables, dataDir, currentAdmin, roleSnapshot, { c for (const table of tables) { await trx(table).del(); } + + // Face data (#1074) is excluded from the archive, which also excludes it + // from `tables` — so the LOCAL rows would survive a whole-DB replace. + // FK enforcement is deliberately suspended during import, so those + // orphans can end up attached to reused photo/event ids from the incoming + // archive: one instance's biometric data silently adopted by another's + // galleries. Purge them explicitly. + for (const faceTable of ['photo_faces', 'event_people']) { + try { + await trx(faceTable).del(); + } catch (err) { + // Absent on targets that predate migration 177 — nothing to purge. + } + } + for (const table of tables) { const rows = parseNdjson(path.join(dataDir, `${table}.ndjson`)); if (!rows.length) continue; @@ -493,7 +508,30 @@ async function importFromPicpeak({ picpeakPath, currentAdminId }) { await resyncSequences(tables); await setSessionsValidAfter(Math.floor(Date.now() / 1000)); + const filesRestored = await restoreFiles(staging); + + // Face data (#1074): queue ONLY once the files are on disk. The archive + // carries no face rows and the export blanked photos.face_status, but the + // event toggles come across enabled, so the "enable" transition that + // normally triggers a backfill never happens here. + // + // Ordering matters: the worker is live during a restore. Queued before + // restoreFiles, it races the copy and either scans the PREVIOUS + // instance's files or marks photos failed for originals that are not + // there yet — and nothing re-queues them afterwards. + try { + const requeued = await db('photos') + .whereIn('event_id', db('events').select('id').where('face_recognition_enabled', true)) + .update({ + face_status: 'pending', face_count: null, face_started_at: null, face_error: null, + }); + if (requeued > 0) { + logger.info(`picpeakImport: queued ${requeued} photo(s) for face detection after import`); + } + } catch (err) { + logger.debug?.(`picpeakImport: face requeue skipped: ${err.message}`); + } const usesExternalMedia = await detectExternalMedia(); logger.info( diff --git a/backend/src/services/restoreService.js b/backend/src/services/restoreService.js index 70e7909c..ebba330e 100644 --- a/backend/src/services/restoreService.js +++ b/backend/src/services/restoreService.js @@ -212,9 +212,15 @@ class RestoreService { switch (options.restoreType) { case 'full': restoreResult = await this.performFullRestore(localBackupPath, manifest, options); + // After the FILES too — see requeueFaceScans on why the ordering + // matters with a live worker. + await this.requeueFaceScans(); break; case 'database': restoreResult = await this.performDatabaseRestore(localBackupPath, manifest, options); + // Database-only restore: the existing files stay in place, so there + // is nothing to race. + await this.requeueFaceScans(); break; case 'files': restoreResult = await this.performFilesRestore(localBackupPath, manifest, options); @@ -836,6 +842,45 @@ class RestoreService { /** * Perform full restore (database + files) */ + /** + * Requeue face detection after a restore (#1074). + * + * Face data is deliberately excluded from backups — it is derived, and + * biometric data should not travel in an archive. But photos.face_status + * DOES restore, so without this the restored install claims every photo is + * scanned while photo_faces is empty, and the worker never picks them up + * because it only claims 'pending'. The gallery shows a finished scan and + * no people, forever, with nothing to indicate why. + * + * MUST run after the FILES are restored, not merely after the database. + * The face worker is live throughout a restore; queued earlier it races the + * file copy and either scans the previous instance's originals or marks + * photos failed for files that are not there yet — and nothing re-queues + * them afterwards. + * + * Only touches rows that had been scanned; NULL stays NULL, so this never + * switches the feature on for anyone. + */ + async requeueFaceScans() { + try { + const { db: restoredDb } = require('../database/db'); + const requeued = await restoredDb('photos') + .whereNotNull('face_status') + .update({ + face_status: 'pending', + face_count: null, + face_started_at: null, + face_error: null, + }); + if (requeued > 0) { + this.log('info', `Requeued ${requeued} photo(s) for face detection after restore`); + } + } catch (err) { + // Pre-migration-177 backups have no such column; not an error. + this.log('info', `Face state reset skipped: ${err.message}`); + } + } + async performFullRestore(backupPath, manifest, options) { const result = { databaseRestored: false, @@ -1166,6 +1211,7 @@ END $$;` await reinitPool(); this.log('info', 'Knex pool re-initialized'); + // NOTE: we deliberately do NOT call `db.migrate.latest()` here. // // The picpeak migrations directory contains `helpers.js` (a diff --git a/docker-compose.production.yml b/docker-compose.production.yml index 73a2605f..4b2faeb7 100644 --- a/docker-compose.production.yml +++ b/docker-compose.production.yml @@ -102,6 +102,13 @@ services: - PICPEAK_RELEASE_CHANNEL=${PICPEAK_CHANNEL:-stable} # Watch-folder auto-import: max photos processed in parallel (default 2). - FILE_WATCHER_CONCURRENCY=${FILE_WATCHER_CONCURRENCY:-2} + # Face recognition (#1074). Defaults to the sidecar's compose service + # name; nothing touches it until the `faces` feature flag is enabled in + # admin settings, so installs without the picpeak-ml container are + # unaffected. Start the sidecar with `--profile faces`. + - FACE_ML_URL=${FACE_ML_URL:-http://picpeak-ml:8000} + - FACE_ML_TOKEN=${FACE_ML_TOKEN:-} + - FACE_PROCESSOR_CONCURRENCY=${FACE_PROCESSOR_CONCURRENCY:-} volumes: - ${APP_STORAGE}:/app/storage - ${LOGS}:/app/logs @@ -140,6 +147,36 @@ services: timeout: 10s retries: 3 + # Optional face-detection sidecar (#1074). Gated behind the `faces` profile: + # `docker compose --profile faces up -d`. Nothing depends on it, and the + # backend never calls it while the `faces` feature flag is off. + # + # The service name is `picpeak-ml` because it doubles as the hostname in + # FACE_ML_URL's default. Renaming it breaks that default for every install + # that never set the variable. + picpeak-ml: + image: ghcr.io/picpeak/picpeak/ml:${PICPEAK_CHANNEL:-stable} + container_name: picpeak-ml + profiles: + - faces + environment: + # The container refuses to start without this rather than serving + # anonymously — it must match the backend's FACE_ML_TOKEN. + - FACE_ML_TOKEN=${FACE_ML_TOKEN:-} + - FACE_ORT_THREADS=${FACE_ORT_THREADS:-1} + - TZ=${TZ:-UTC} + # No volumes and no published ports: stateless, and reachable only from + # the backend on picpeak-network. + networks: + - picpeak-network + restart: unless-stopped + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + frontend: # Use pre-built image from GitHub Container Registry # Uses same channel as backend for consistency diff --git a/docker-compose.yml b/docker-compose.yml index ceea2dcb..cf1ca216 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -73,6 +73,14 @@ services: - STORAGE_PATH=/app/storage # Watch-folder auto-import: max photos processed in parallel (default 2). - FILE_WATCHER_CONCURRENCY=${FILE_WATCHER_CONCURRENCY:-2} + # Face recognition (#1074). The URL defaults to the sidecar's compose + # service name, so the common case needs no configuration. None of this + # is touched until the `faces` feature flag is enabled in admin + # settings — an install without the picpeak-ml container never attempts + # a connection. + - FACE_ML_URL=${FACE_ML_URL:-http://picpeak-ml:8000} + - FACE_ML_TOKEN=${FACE_ML_TOKEN:-} + - FACE_PROCESSOR_CONCURRENCY=${FACE_PROCESSOR_CONCURRENCY:-} # No `user:` directive — as of #484, the container starts as root, # chowns the bind mounts to nodejs (UID 1001), then drops privileges # via su-exec. PUID/PGID env vars are no longer read; if you need @@ -167,6 +175,50 @@ services: networks: - picpeak-network + # Optional face-detection sidecar (#1074). Gated behind the `faces` profile + # so a plain `docker compose up -d` does NOT start it — opt in with + # `docker compose --profile faces up -d`. Nothing depends on it: the backend + # only ever calls it when the `faces` feature flag is on, so an install that + # skips this service behaves exactly as it did before the feature existed. + # + # The service name is `picpeak-ml` (not `ml`) because it doubles as the + # hostname in FACE_ML_URL's default, `http://picpeak-ml:8000`. Renaming this + # service silently breaks that default for every install that never set the + # variable. + # + # Requires FACENET_ONNX_URL / FACENET_ONNX_SHA256 at build time — see + # ml/README.md. The image publishes no host port and mounts no volumes; it + # is reachable only from the backend on picpeak-network. + picpeak-ml: + build: + context: ./ml + dockerfile: Dockerfile + args: + # Defaults live in ml/Dockerfile and point at the canonical published + # model. These pass an override through from .env when set; an empty + # value here would BLANK the Dockerfile default and fail the build, + # so the fallbacks repeat it deliberately. + - FACENET_ONNX_URL=${FACENET_ONNX_URL:-https://github.com/PicPeak/picpeak/releases/download/ml-models-v1/facenet512.onnx} + - FACENET_ONNX_SHA256=${FACENET_ONNX_SHA256:-a1c06dcb79dc17a42af01d5bcbce4822caa148b9c24bf7eb8b8e556b4fd0d5db} + container_name: picpeak-ml + restart: unless-stopped + profiles: + - faces + environment: + # Shared secret with the backend. The container refuses to start + # without it rather than serving anonymously. + - FACE_ML_TOKEN=${FACE_ML_TOKEN:-} + - FACE_ORT_THREADS=${FACE_ORT_THREADS:-1} + - TZ=${TZ:-UTC} + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + networks: + - picpeak-network + frontend: build: context: ./frontend diff --git a/frontend/scripts/i18n-faces-audit.py b/frontend/scripts/i18n-faces-audit.py new file mode 100644 index 00000000..0d45ba4d --- /dev/null +++ b/frontend/scripts/i18n-faces-audit.py @@ -0,0 +1,90 @@ +"""Audit i18n coverage for the face-recognition feature (#1074). + +Extracts every t('key', ...) used by the face components and reports which +are missing from each locale file. A key that resolves only via its inline +`defaultValue` renders ENGLISH to a German user — which is exactly the gap +this looks for, and which nothing else in the toolchain would flag. +""" +import json +import re +import sys +from pathlib import Path + +ROOT = Path('/Users/paul/Development/picpeak/frontend/src') + +FILES = [ + 'components/gallery/PeopleStrip.tsx', + 'components/gallery/PeopleSheet.tsx', + 'components/gallery/GalleryView.tsx', + 'components/gallery/PhotoLightbox.tsx', + 'components/admin/FaceRecognitionCard.tsx', + 'components/admin/PeopleManagerModal.tsx', + 'features/settings/tabs/FeaturesTab.tsx', +] + +# Only keys belonging to this feature (plus shared keys the new components use). +RELEVANT = re.compile(r'^(gallery\.people\.|admin\.people\.|admin\.faces\.|settings\.features\.faces\.|common\.)') + +KEY_RE = re.compile(r"\bt\(\s*'([a-zA-Z0-9_.]+)'") + + +def load(lang): + with open(ROOT / 'i18n' / 'locales' / f'{lang}.json') as fh: + return json.load(fh) + + +def has(data, dotted): + node = data + for part in dotted.split('.'): + if not isinstance(node, dict) or part not in node: + return False + node = node[part] + return isinstance(node, str) + + +used = {} +for rel in FILES: + path = ROOT / rel + if not path.exists(): + print(f'!! missing file {rel}') + continue + for key in KEY_RE.findall(path.read_text()): + if RELEVANT.match(key): + used.setdefault(key, set()).add(rel.split('/')[-1]) + +print(f'{len(used)} face-related keys in use\n') + +exit_code = 0 +for lang in ('en', 'de'): + data = load(lang) + missing = sorted(k for k in used if not has(data, k)) + status = 'COMPLETE' if not missing else f'{len(missing)} MISSING' + print(f'--- {lang.upper()}: {status} ---') + for k in missing: + print(f' {k} ({", ".join(sorted(used[k]))})') + if missing: + exit_code = 1 + print() + +# Also flag DE values that are byte-identical to EN — usually an untranslated +# copy-paste rather than a word that genuinely matches in both languages. +en, de = load('en'), load('de') + + +def get(data, dotted): + node = data + for part in dotted.split('.'): + node = node[part] + return node + + +same = [] +for k in sorted(used): + if has(en, k) and has(de, k) and get(en, k) == get(de, k): + same.append((k, get(en, k))) +if same: + print(f'--- DE identical to EN ({len(same)}) — check each is intentional ---') + for k, v in same: + print(f' {k} = {v!r}') + +sys.exit(exit_code) diff --git a/frontend/src/components/admin/FaceRecognitionCard.tsx b/frontend/src/components/admin/FaceRecognitionCard.tsx new file mode 100644 index 00000000..10cea24f --- /dev/null +++ b/frontend/src/components/admin/FaceRecognitionCard.tsx @@ -0,0 +1,357 @@ +/** + * + * + * Per-event "People in this gallery" controls (#1074). Only rendered when the + * `faces` feature flag is on — see OverviewTab. + * + * Two toggles, deliberately separate: + * - Detect people: the actual processing switch. Off by default. + * - Show to guests: whether the people bar reaches the gallery. On by + * default once detection is on, but a photographer may want clustering + * as a private sorting tool without changing what clients see. + * + * Face embeddings are biometric data (GDPR Art. 9 in the EU, where much of + * this user base operates). The consent paragraph is not boilerplate — we + * provide the switch, the photographer provides the lawful basis — so it + * renders next to the toggle rather than behind a "learn more". + */ +import React, { useEffect, useState } from 'react'; +import { useQuery } from '@tanstack/react-query'; +import { useTranslation } from 'react-i18next'; +import { toast } from 'react-toastify'; +import { Users, RefreshCw, Trash2, AlertTriangle, ShieldCheck, SlidersHorizontal } from 'lucide-react'; +import { PeopleManagerModal } from './PeopleManagerModal'; + +import { Button, Card, Loading } from '../common'; +import { api } from '../../config/api'; + +interface FacesPayload { + enabled: boolean; + visible_to_guests: boolean; + last_scan_at: string | null; + status: { + scanned: number; + total: number; + pending: number; + failed: number; + people: number; + // What guests actually see: the minimum-cluster-size floor and the + // hidden/ignored flags applied. Usually lower than `people`. + people_visible_to_guests?: number; + in_progress: boolean; + }; +} + +interface FaceRecognitionCardProps { + eventId: number; + isArchived?: boolean; +} + +export const FaceRecognitionCard: React.FC = ({ eventId, isArchived }) => { + const { t } = useTranslation(); + const [saving, setSaving] = useState(false); + const [managerOpen, setManagerOpen] = useState(false); + const [autoCategories, setAutoCategories] = useState(false); + + const { data, isLoading, refetch } = useQuery({ + queryKey: ['admin-event-faces', eventId], + queryFn: async () => (await api.get(`/admin/events/${eventId}/faces`)).data, + // Poll while a scan is running so the status line advances without a + // manual refresh; stop entirely once it settles. + refetchInterval: (query) => (query.state.data?.status?.in_progress ? 5000 : false), + }); + + useEffect(() => { + if (!data?.enabled) return; + api.get('/admin/events/faces/auto-categories') + .then((r) => setAutoCategories(!!r.data?.enabled)) + .catch(() => { /* pre-migration or no permission — leave off */ }); + }, [data?.enabled]); + + const patch = async (body: Record) => { + setSaving(true); + try { + const { data: result } = await api.patch(`/admin/events/${eventId}/faces`, body); + await refetch(); + if (result?.queued) { + toast.success(t('admin.faces.queued', { + count: result.queued, + defaultValue: `Scanning ${result.queued} photos for people…`, + })); + } else { + toast.success(t('common.saved', { defaultValue: 'Saved' })); + } + } catch (err: any) { + toast.error(err?.response?.data?.error || t('common.saveFailed', { defaultValue: 'Save failed' })); + } finally { + setSaving(false); + } + }; + + const action = async (path: string, successKey: string, fallback: string) => { + setSaving(true); + try { + await api.post(`/admin/events/${eventId}/faces/${path}`); + await refetch(); + toast.success(t(successKey, { defaultValue: fallback })); + } catch (err: any) { + toast.error(err?.response?.data?.error || fallback); + } finally { + setSaving(false); + } + }; + + const purge = async () => { + // Irreversible and covers biometric data — a plain confirm is the least + // this deserves. + const ok = window.confirm(t('admin.faces.confirmDelete', { + defaultValue: 'Delete all detected people and face data for this gallery? This cannot be undone. Photos are not affected.', + })); + if (!ok) return; + + setSaving(true); + try { + await api.delete(`/admin/events/${eventId}/faces`); + await refetch(); + toast.success(t('admin.faces.deleted', { defaultValue: 'Face data deleted' })); + } catch (err: any) { + toast.error(err?.response?.data?.error || 'Delete failed'); + } finally { + setSaving(false); + } + }; + + if (isLoading) { + return ; + } + if (!data) return null; + + const { status } = data; + + return ( + +
+ +
+

+ {t('admin.faces.title', { defaultValue: 'People in this gallery' })} +

+

+ {t('admin.faces.subtitle', { + defaultValue: 'Group photos by the people in them, so guests can find and download their own.', + })} +

+
+
+ + {/* Consent obligation. Stated plainly and up front, because by the time + someone has switched this on they have already processed the data. */} +
+ +

+ {t('admin.faces.consentNotice', { + defaultValue: + 'Detected faces are personal data, and in the EU they count as a special category. You are the controller for this gallery: make sure you have a lawful basis for the people in these photos before switching this on. Nothing leaves your server — detection runs in your own container.', + })} +

+
+ + + + {data.enabled && ( + + )} + + {/* The preview-tier cost. Scanning generates the lightbox preview for + every photo that lacks one, which is real CPU and real disk — an + admin deserves to know that before starting a 2,000-photo backfill + rather than discovering it in their storage graph. */} + {data.enabled && ( +
+ +

+ {t('admin.faces.previewNotice', { + defaultValue: 'Scanning works on the preview-sized copy of each photo. Galleries that have not generated previews yet will create them during the first scan, which uses additional CPU and disk space.', + })} +

+
+ )} + + {/* Auto-categories (#1074 phase 3). Global, not per-event, which is why + it sits apart from the two toggles above. Without a control here the + rule engine had no way to be switched on at all. */} + {data.enabled && ( + + )} + + {data.enabled && ( + <> +
+ {status.in_progress ? ( +

+ + {t('admin.faces.scanning', { + scanned: status.scanned, + total: status.total, + defaultValue: `Scanning… ${status.scanned} of ${status.total} photos`, + })} +

+ ) : ( +

+ {t('admin.faces.status', { + scanned: status.scanned, + total: status.total, + people: status.people, + defaultValue: `${status.scanned} / ${status.total} photos scanned · ${status.people} people`, + })} + {/* The gallery shows fewer: one-off appearances stay out of + the strip. Without both numbers an admin sees the settings + page and their own gallery disagree, with no explanation. */} + {typeof status.people_visible_to_guests === 'number' + && status.people_visible_to_guests !== status.people && ( + + {' '} + {t('admin.faces.visibleToGuests', { + count: status.people_visible_to_guests, + defaultValue: `(${status.people_visible_to_guests} shown to guests)`, + })} + + )} + {status.failed > 0 && ( + + {' · '} + {t('admin.faces.failed', { + count: status.failed, + defaultValue: `${status.failed} failed`, + })} + + )} +

+ )} +
+ +
+ + + + + {/* Cheap — re-derives people from data we already have, with no + sidecar call. The button to reach for after changing the + match threshold. */} + + + +
+ + setManagerOpen(false)} + onChanged={() => refetch()} + /> + + )} + + ); +}; + +export default FaceRecognitionCard; diff --git a/frontend/src/components/admin/PeopleManagerModal.tsx b/frontend/src/components/admin/PeopleManagerModal.tsx new file mode 100644 index 00000000..dcfc9636 --- /dev/null +++ b/frontend/src/components/admin/PeopleManagerModal.tsx @@ -0,0 +1,436 @@ +/** + * — manage the people detected in one gallery (#1074). + * + * Merge and split are the two operations that make automatic clustering + * survive contact with reality. A well-behaved wedding gallery still produces + * "Anna in daylight" and "Anna at the party" as separate people, and the + * clustering deliberately errs toward splitting rather than merging (a + * duplicate entry is an annoyance; a wrong merge puts a stranger into + * someone's download). That trade only works if merging is easy, which is + * what this screen is for. + * + * Every action here hits an endpoint that already re-checks event ownership + * and that the ids belong to this gallery — the UI is a convenience, never + * the control. + */ +import React, { useMemo, useState } from 'react'; +import { useQuery } from '@tanstack/react-query'; +import { useTranslation } from 'react-i18next'; +import { toast } from 'react-toastify'; +import { X, Check, Merge, Scissors, EyeOff, Ban, Loader2 } from 'lucide-react'; + +import { Button, Loading } from '../common'; +import { api } from '../../config/api'; +import { faceCropStyle } from '../gallery/faceCrop'; + +interface AdminPerson { + id: number; + label: string | null; + face_count: number; + total_face_count?: number; + is_hidden?: boolean; + is_ignored?: boolean; + cover: { + face_id: number; + photo_id: number; + bbox: [number, number, number, number]; + photo_width: number | null; + photo_height: number | null; + } | null; +} + +interface PersonFace { + id: number; + photo_id: number; + bbox: [number, number, number, number]; + photo_width: number | null; + photo_height: number | null; + score: number | null; + blur: number | null; +} + +interface PeopleManagerModalProps { + eventId: number; + open: boolean; + onClose: () => void; + onChanged?: () => void; +} + +/** + * Square crop of a face, served by the admin thumbnail route (which applies + * its own auth + event-ownership checks). + * + * A plain , deliberately NOT . That component + * attaches whatever gallery token it finds in session storage, and an admin + * who has also opened one of their own galleries in the same browser then + * sends a `type: "gallery"` bearer token to an admin route — which rejects + * it as insufficient permissions, so every thumbnail 403s. The admin routes + * authenticate from the httpOnly admin_token cookie, which a same-origin + * sends on its own. + */ +const FaceThumb: React.FC<{ + eventId: number; + photoId: number; + bbox?: [number, number, number, number] | null; + photoWidth?: number | null; + photoHeight?: number | null; + size?: number; + dim?: boolean; +}> = ({ eventId, photoId, bbox, photoWidth, photoHeight, size = 64, dim }) => { + // Crop to the face rather than showing the centred thumbnail. On a group + // photo the uncropped version shows whoever stands in the middle, so two + // different people whose cover is the same photo looked IDENTICAL here — + // in the manager whose whole job is telling faces apart. + // + // COORDINATE SPACE: the bbox is in ORIGINAL image pixels, so it must be + // scaled against the ORIGINAL dimensions (now supplied by the API), never + // against the thumbnail's own natural size. Mixing the two renders the + // wrong region entirely — the box ends up a fraction of its true size and + // offset toward the top-left. + const style = faceCropStyle(bbox ? { bbox } : null, photoWidth, photoHeight, size); + + return ( + + + + ); +}; + +export const PeopleManagerModal: React.FC = ({ + eventId, open, onClose, onChanged, +}) => { + const { t } = useTranslation(); + const [selected, setSelected] = useState([]); + const [renaming, setRenaming] = useState(null); + const [draftLabel, setDraftLabel] = useState(''); + const [splitting, setSplitting] = useState(null); + const [splitFaceIds, setSplitFaceIds] = useState([]); + const [busy, setBusy] = useState(false); + + const { data, isLoading, refetch } = useQuery<{ people: AdminPerson[] }>({ + queryKey: ['admin-event-people', eventId], + queryFn: async () => (await api.get(`/admin/events/${eventId}/people`)).data, + enabled: open, + }); + + const { data: faceData, isLoading: facesLoading } = useQuery<{ faces: PersonFace[] }>({ + queryKey: ['admin-person-faces', eventId, splitting?.id], + queryFn: async () => + (await api.get(`/admin/events/${eventId}/people/${splitting!.id}/faces`)).data, + enabled: !!splitting, + }); + + const people = useMemo(() => data?.people || [], [data]); + + const after = async (message: string) => { + await refetch(); + onChanged?.(); + setSelected([]); + toast.success(message); + }; + + const run = async (fn: () => Promise, successMessage: string) => { + setBusy(true); + try { + await fn(); + await after(successMessage); + } catch (err: any) { + toast.error(err?.response?.data?.error || t('common.saveFailed', { defaultValue: 'Failed' })); + } finally { + setBusy(false); + } + }; + + const toggleSelect = (id: number) => { + setSelected((prev) => (prev.includes(id) ? prev.filter((x) => x !== id) : [...prev, id])); + }; + + const saveLabel = (person: AdminPerson) => { + const label = draftLabel.trim(); + setRenaming(null); + if (label === (person.label || '')) return; + run( + async () => { + await api.patch(`/admin/events/${eventId}/people/${person.id}`, { label: label || null }); + }, + t('admin.people.renamed', { defaultValue: 'Name saved' }) + ); + }; + + const doMerge = () => { + // The first selected person is the target — it keeps its name, which is + // almost always the one the photographer already bothered to type. + const [target, ...sources] = selected; + run( + async () => { + await api.post(`/admin/events/${eventId}/people/merge`, { + source_ids: sources, target_id: target, + }); + }, + t('admin.people.merged', { count: sources.length, defaultValue: 'People merged' }) + ); + }; + + const doSplit = () => { + if (!splitting || !splitFaceIds.length) return; + const personId = splitting.id; + const ids = splitFaceIds; + setSplitting(null); + setSplitFaceIds([]); + run( + async () => { + await api.post(`/admin/events/${eventId}/people/${personId}/split`, { face_ids: ids }); + }, + t('admin.people.split', { defaultValue: 'Split into a new person' }) + ); + }; + + const setFlag = (person: AdminPerson, field: 'is_hidden' | 'is_ignored', value: boolean) => + run( + async () => { + await api.patch(`/admin/events/${eventId}/people/${person.id}`, { [field]: value }); + }, + t('admin.people.updated', { defaultValue: 'Updated' }) + ); + + if (!open) return null; + + return ( +
+ + ); +}; + +export default PeopleManagerModal; diff --git a/frontend/src/components/admin/VersionInfo.tsx b/frontend/src/components/admin/VersionInfo.tsx index b99d5e41..db198b28 100644 --- a/frontend/src/components/admin/VersionInfo.tsx +++ b/frontend/src/components/admin/VersionInfo.tsx @@ -20,6 +20,8 @@ interface SystemVersion { node: string; environment: string; channel?: 'stable' | 'beta'; + // True on the all-in-one image; some features are unavailable there. + single_container?: boolean; } interface UpdateInfo { diff --git a/frontend/src/components/gallery/GalleryView.tsx b/frontend/src/components/gallery/GalleryView.tsx index 82536567..3bb6eff2 100644 --- a/frontend/src/components/gallery/GalleryView.tsx +++ b/frontend/src/components/gallery/GalleryView.tsx @@ -17,12 +17,14 @@ import { PhotoFilterBar } from './PhotoFilterBar'; import { UserPhotoUpload } from './UserPhotoUpload'; import { GuestNamePromptModal } from './GuestNamePromptModal'; import { GuestRecoveryModal } from './GuestRecoveryModal'; +import { PeopleStrip } from './PeopleStrip'; +import { PeopleSheet } from './PeopleSheet'; import { GuestIdentityProvider } from '../../contexts/GuestIdentityContext'; import type { FilterType, FeedbackFilterType } from './GalleryFilter'; import { analyticsService } from '../../services/analytics.service'; import { useDevToolsProtection } from '../../hooks/useDevToolsProtection'; import { api } from '../../config/api'; -import { Upload, Menu, Eye, EyeOff, Shield } from 'lucide-react'; +import { Upload, Menu, Eye, EyeOff, Shield, X, Download } from 'lucide-react'; import { galleryService } from '../../services/gallery.service'; import { feedbackService } from '../../services/feedback.service'; import { useWatermarkSettings } from '../../hooks/useWatermarkSettings'; @@ -104,6 +106,41 @@ export const GalleryView: React.FC = ({ slug, event }) => { // Multi-select feedback filters (#889): OR-combined; empty = "All". // Clicking a filter toggles it, clicking "All" clears the set. const [activeFilters, setActiveFilters] = useState([]); + + // People filter (#1074). Multi-select, AND by default — see the filter + // block below. `peopleMatchAny` only becomes reachable once a second + // person is picked, since the toggle is meaningless for one. + const [selectedPersonIds, setSelectedPersonIds] = useState([]); + const [peopleMatchAny, setPeopleMatchAny] = useState(false); + const [showPeopleSheet, setShowPeopleSheet] = useState(false); + // Dismissal is per gallery: a guest who hides the bar in one gallery has + // said nothing about the next one. + const [peopleCollapsed, setPeopleCollapsed] = useState(() => { + try { + return localStorage.getItem(`picpeak_people_collapsed_${slug}`) === '1'; + } catch { + return false; + } + }); + const handlePeopleCollapsedChange = (collapsed: boolean) => { + setPeopleCollapsed(collapsed); + try { + localStorage.setItem(`picpeak_people_collapsed_${slug}`, collapsed ? '1' : '0'); + } catch { + // Private-mode Safari throws on setItem; the in-memory state still works. + } + }; + const togglePerson = (personId: number) => { + setSelectedPersonIds((prev) => { + const next = prev.includes(personId) + ? prev.filter((id) => id !== personId) + : [...prev, personId]; + // Dropping back below two people makes the any/all toggle meaningless; + // reset it so it doesn't silently persist into the next selection. + if (next.length < 2) setPeopleMatchAny(false); + return next; + }); + }; const handleFilterChange = (filter: FilterType) => { if (filter === 'all') { setActiveFilters([]); @@ -263,6 +300,41 @@ export const GalleryView: React.FC = ({ slug, event }) => { enabled: !!event.id, }); + // People in this gallery (#1074). + // + // Gated on people_enabled so an install without the feature never fires the + // request at all. Polls only while a backfill is running — a finished + // gallery has a stable people list, and polling it forever would be a + // request per guest per interval for no new information. + // From the /photos payload, not the prop: the prop's event shape comes + // from /info, which does not carry this flag. + const peopleEnabled = data?.event?.people_enabled === true; + const { data: peopleData } = useQuery({ + queryKey: ['gallery-people', slug], + queryFn: () => galleryService.getPeople(slug), + enabled: peopleEnabled, + refetchInterval: (query) => (query.state.data?.scan?.in_progress ? 5000 : false), + staleTime: 30_000, + }); + const people = peopleData?.people || []; + + // The strip comes from /people, but FILTERING uses photo.person_ids, which + // rides on the one-shot /photos response. During a backfill those drift + // apart: new faces appear in the strip while the photo memberships behind + // them are still the set fetched on page load, so tapping a person yields + // zero or a partial result until a manual reload — including after the scan + // has finished. + // + // Refetch the photos whenever the scan's progress changes, and once more on + // the transition to finished. + const scanProgress = peopleData?.scan + ? `${peopleData.scan.in_progress}:${peopleData.scan.scanned}` + : null; + useEffect(() => { + if (!peopleEnabled || !scanProgress) return; + queryClient.invalidateQueries({ queryKey: ['gallery-photos', slug] }); + }, [scanProgress, peopleEnabled, slug, queryClient]); + // Update feedbackEnabled when settings change useEffect(() => { if (feedbackSettings) { @@ -551,7 +623,22 @@ export const GalleryView: React.FC = ({ slug, event }) => { }; photos = photos.filter(photo => activeFilters.some(filter => matchers[filter](photo))); } - + + // Apply people filter (#1074). Composes with every filter above rather + // than replacing them, so "photos of Anna that I liked" works. + // + // Two people selected means AND by default ("photos with both Anna and + // Ben") — that is what someone picking a second face is almost always + // asking for. `peopleMatchAny` flips it to OR for the couple-shots case. + if (selectedPersonIds.length > 0) { + photos = photos.filter(photo => { + const ids = photo.person_ids || []; + return peopleMatchAny + ? selectedPersonIds.some(id => ids.includes(id)) + : selectedPersonIds.every(id => ids.includes(id)); + }); + } + // Apply sorting // Each comparator defaults to its natural order (desc for dates/size/rating, asc for name). // The flip multiplier reverses that when sortDesc differs from the natural order. @@ -592,7 +679,7 @@ export const GalleryView: React.FC = ({ slug, event }) => { } return photos; - }, [data?.photos, selectedCategoryId, searchTerm, sortBy, sortDesc, watermarkEnabled, slug, activeFilters, mediaFilter, isGuestIdentityMode, myFeedbackPhotoIds]); + }, [data?.photos, selectedCategoryId, searchTerm, sortBy, sortDesc, watermarkEnabled, slug, activeFilters, mediaFilter, isGuestIdentityMode, myFeedbackPhotoIds, selectedPersonIds, peopleMatchAny]); // Counts shown in the filter chips ("Liked (N)", etc.). In guest // mode these need to mirror the per-guest filter behaviour above — @@ -682,6 +769,40 @@ export const GalleryView: React.FC = ({ slug, event }) => { setIsSelectionMode(false); }; + // "Download these N" (#1074) — the payoff of the people filter. + // + // Deliberately NO new endpoint or person_id selector: the filtered photo + // ids go through the same path as a manual selection, and the server + // re-applies the access level and per-category permissions on the way + // through. One less thing to authorize. + // + // Photos in a category with downloads disabled (#640) are excluded HERE as + // well as server-side, so the number on the button is the number the guest + // actually receives rather than an optimistic one. + const peopleDownloadableIds = useMemo(() => { + if (selectedPersonIds.length === 0) return []; + return filteredPhotos + .filter((photo) => photo.category_allow_downloads !== false) + .map((photo) => photo.id); + }, [filteredPhotos, selectedPersonIds]); + + const handleDownloadPeopleFiltered = async () => { + if (!allowDownloads || peopleDownloadableIds.length === 0) return; + + // Same resolution-picker behaviour as every other multi-photo download. + if (downloadChoices.length > 1) { + setResolutionPickerIds(peopleDownloadableIds); + return; + } + + analyticsService.trackGalleryEvent('bulk_download', { + gallery: slug, + photo_count: peopleDownloadableIds.length, + }); + + await galleryService.downloadSelectedPhotos(slug, peopleDownloadableIds); + }; + // Calculate photo counts per category const photoCounts = useMemo(() => { if (!data?.photos) return {}; @@ -847,6 +968,8 @@ export const GalleryView: React.FC = ({ slug, event }) => { { refetch(); @@ -1118,6 +1241,107 @@ export const GalleryView: React.FC = ({ slug, event }) => {
) : null} + {/* People in this gallery (#1074). Sits between the filter bar and + the grid. Renders nothing at all unless the photographer enabled + detection AND left it visible to guests — people_enabled carries + both decisions plus the global feature flag. */} + {peopleEnabled && ( +
+ setShowPeopleSheet(true)} + scan={peopleData?.scan} + collapsed={peopleCollapsed} + onCollapsedChange={handlePeopleCollapsedChange} + /> + + {/* Active people filter. The chip row is the single place the + current selection is stated, so "why am I seeing 97 photos" + is always answerable at a glance. */} + {selectedPersonIds.length > 0 && ( +
+ {selectedPersonIds.map((id) => { + const person = people.find((p) => p.id === id); + if (!person) return null; + return ( + + ); + })} + + {/* Only meaningful with two or more people picked. */} + {selectedPersonIds.length > 1 && ( + + )} + + {/* ml-auto only once there's room for it — at 390px the count + and Clear were pushed against the right edge and clipped. */} + + {t('gallery.people.matchCount', { + count: filteredPhotos.length, + total: totalCount, + defaultValue: `${filteredPhotos.length} of ${totalCount} photos`, + })} + + + {/* Hidden entirely when downloads are off for the gallery, + rather than shown-and-failing. */} + {allowDownloads && peopleDownloadableIds.length > 0 && ( + + )} + + +
+ )} +
+ )} + {/* Photo Grid — when the hero header sits directly under the filter bar, double the wrapper margin (mt-12) so the hero's decorative `-mt-6` bleed leaves a visible gap instead of gluing the filter @@ -1125,7 +1349,9 @@ export const GalleryView: React.FC = ({ slug, event }) => {
{ refetch(); @@ -1201,6 +1427,19 @@ export const GalleryView: React.FC = ({ slug, event }) => { }} /> )} + + {/* "Show all" people (#1074) — a bottom sheet on mobile. */} + {peopleEnabled && ( + setShowPeopleSheet(false)} + people={people} + photos={data?.photos || []} + slug={slug} + selectedPersonIds={selectedPersonIds} + onToggle={togglePerson} + /> + )} diff --git a/frontend/src/components/gallery/PeopleSheet.tsx b/frontend/src/components/gallery/PeopleSheet.tsx new file mode 100644 index 00000000..a3f9464e --- /dev/null +++ b/frontend/src/components/gallery/PeopleSheet.tsx @@ -0,0 +1,172 @@ +import React, { useMemo, useState } from 'react'; +import { Search, X, Info } from 'lucide-react'; +import { useTranslation } from 'react-i18next'; + +import { AuthenticatedImage } from '../common/AuthenticatedImage'; +import type { GalleryPerson, Photo } from '../../types'; +import { faceCropStyle } from './faceCrop'; + +/** + * "Show all" people (#1074). + * + * A bottom sheet on mobile, a centred panel on desktop. The footnote is not + * decoration — "where does this data go?" is the first question every guest + * has, and answering it inline is cheaper than losing their trust. + */ + +interface PeopleSheetProps { + open: boolean; + onClose: () => void; + people: GalleryPerson[]; + photos: Photo[]; + slug: string; + selectedPersonIds: number[]; + onToggle: (personId: number) => void; +} + +export const PeopleSheet: React.FC = ({ + open, onClose, people, photos, slug, selectedPersonIds, onToggle, +}) => { + const { t } = useTranslation(); + const [query, setQuery] = useState(''); + + const photoById = useMemo(() => { + const map = new Map(); + for (const photo of photos) map.set(photo.id, photo); + return map; + }, [photos]); + + const filtered = useMemo(() => { + const q = query.trim().toLowerCase(); + if (!q) return people; + // Only named people are searchable — there is nothing to match an + // unnamed cluster against, and pretending otherwise (matching the count, + // say) would be a puzzle rather than a feature. + return people.filter((p) => p.label?.toLowerCase().includes(q)); + }, [people, query]); + + if (!open) return null; + + return ( +
+ + ); +}; + +export default PeopleSheet; diff --git a/frontend/src/components/gallery/PeopleStrip.tsx b/frontend/src/components/gallery/PeopleStrip.tsx new file mode 100644 index 00000000..8f735b4f --- /dev/null +++ b/frontend/src/components/gallery/PeopleStrip.tsx @@ -0,0 +1,246 @@ +import React, { useMemo, useState } from 'react'; +import { ChevronRight, Users, X } from 'lucide-react'; +import { useTranslation } from 'react-i18next'; + +import { AuthenticatedImage } from '../common/AuthenticatedImage'; +import type { GalleryPerson, Photo } from '../../types'; +import { faceCropStyle } from './faceCrop'; + +/** + * "People in this gallery" (#1074). + * + * The design goal is that a guest on a phone finds themselves in two taps and + * never learns the words "face recognition". So: circular crops, first names + * where the photographer supplied them, a photo count where they didn't, and + * no vocabulary from the implementation anywhere in the UI. + * + * Unnamed people show ONLY a count. Never "Person 7" — a number is honest + * about what the system knows; an invented name is not. + */ + +interface PersonAvatarProps { + person: GalleryPerson; + photo: Photo | undefined; + slug: string; + size: number; + selected: boolean; + onClick: () => void; +} + +/** + * A face crop, produced by scaling the source photo inside a round window so + * the stored bbox lands centred. + * + * The bbox is in ORIGINAL image pixels but the thumbnail is served at some + * other size, so everything is done in RATIOS of the source dimensions — + * which survives whatever rendition the browser actually gets. Without the + * photo's width/height we can't compute those ratios, so the component falls + * back to an un-cropped thumbnail rather than rendering a wrongly-offset crop. + */ +const PersonAvatar: React.FC = ({ + person, photo, slug, size, selected, onClick, +}) => { + const { t } = useTranslation(); + + const cropStyle = useMemo( + () => faceCropStyle(person.cover, photo?.width, photo?.height, size), + [person.cover, photo?.width, photo?.height, size], + ); + + const label = person.label + || t('gallery.people.unnamedCount', { count: person.face_count, defaultValue: `${person.face_count} photos` }); + + return ( + + ); +}; + +interface PeopleStripProps { + people: GalleryPerson[]; + photos: Photo[]; + slug: string; + selectedPersonIds: number[]; + onToggle: (personId: number) => void; + onShowAll: () => void; + scan?: { in_progress: boolean; scanned: number; total: number }; + /** Persisted per-slug so a guest who dismisses it stays dismissed. */ + collapsed: boolean; + onCollapsedChange: (collapsed: boolean) => void; + /** Avatars shown inline; the rest live behind "Show all". */ + maxInline?: number; +} + +export const PeopleStrip: React.FC = ({ + people, photos, slug, selectedPersonIds, onToggle, onShowAll, + scan, collapsed, onCollapsedChange, maxInline = 12, +}) => { + const { t } = useTranslation(); + const [avatarSize] = useState(() => (typeof window !== 'undefined' && window.innerWidth < 640 ? 56 : 64)); + + const photoById = useMemo(() => { + const map = new Map(); + for (const photo of photos) map.set(photo.id, photo); + return map; + }, [photos]); + + // Fewer than two people isn't a "people in this gallery" feature, it's a + // single face taking up a row of the screen. Don't render at all. + if (people.length < 2 && !scan?.in_progress) return null; + + if (collapsed) { + return ( +
+ + {t('gallery.people.collapsedSummary', { + count: people.length, + defaultValue: `${people.length} people found`, + })} + + +
+ ); + } + + const inline = people.slice(0, maxInline); + const hasMore = people.length > inline.length; + + return ( +
+
+

+ {t('gallery.people.title', { defaultValue: 'People in this gallery' })} +

+ +
+ {hasMore && ( + + )} + +
+
+ + {/* Backfill progress. The strip appears as soon as the first clusters + exist rather than blocking the gallery behind a spinner — a guest + who arrives mid-scan gets a working gallery and a growing strip. */} + {scan?.in_progress && ( +
+
+ + {t('gallery.people.scanning', { + scanned: scan.scanned, + total: scan.total, + defaultValue: `Finding people… ${scan.scanned}/${scan.total} photos`, + })} + +
+
+
+
+
+ )} + +
+ {inline.map((person) => ( +
+ onToggle(person.id)} + /> +
+ ))} +
+
+ ); +}; + +export default PeopleStrip; diff --git a/frontend/src/components/gallery/PhotoGridWithLayouts.tsx b/frontend/src/components/gallery/PhotoGridWithLayouts.tsx index eb0dade6..6b864071 100644 --- a/frontend/src/components/gallery/PhotoGridWithLayouts.tsx +++ b/frontend/src/components/gallery/PhotoGridWithLayouts.tsx @@ -3,7 +3,7 @@ import { Package } from 'lucide-react'; import { toast as toastify } from 'react-toastify'; import { useTranslation } from 'react-i18next'; -import type { Photo, DownloadResolutionChoice } from '../../types'; +import type { Photo, DownloadResolutionChoice, GalleryPerson } from '../../types'; import { useDownloadPhoto } from '../../hooks/useGallery'; import { PhotoLightbox } from './PhotoLightbox'; import { DownloadResolutionModal } from './DownloadResolutionModal'; @@ -79,6 +79,10 @@ interface PhotoGridWithLayoutsProps { // Mirror of the admin original-filename toggle (#508). When true, the // lightbox bottom toolbar surfaces each photo's original camera name. showOriginalFilename?: boolean; + // People in this gallery (#1074) — forwarded to the lightbox so it can + // show "In this photo: …". Undefined when the feature is off. + people?: GalleryPerson[]; + onSelectPerson?: (personId: number) => void; } export const PhotoGridWithLayouts: React.FC = ({ @@ -117,6 +121,8 @@ export const PhotoGridWithLayouts: React.FC = ({ isClient = false, onToggleVisibility, showOriginalFilename = false, + people, + onSelectPerson, }) => { const { t } = useTranslation(); const { theme } = useTheme(); @@ -232,6 +238,12 @@ export const PhotoGridWithLayouts: React.FC = ({ const layoutProps = { photos, slug, + // Face data (#1074) must reach the full-page layouts too — they render + // their OWN lightbox rather than the one below, so without this the + // "In this photo" chips silently vanish on gallery-premium and + // gallery-story even when the feature is fully enabled. + people, + onSelectPerson, // Full-page layouts own their bulk-download control, so the resolution // picker has to reach them too (#858) — otherwise premium/story galleries // silently skip the choice the admin enabled. @@ -407,6 +419,8 @@ export const PhotoGridWithLayouts: React.FC = ({ initialShowFeedback={openFeedbackInitially} onFeedbackChange={onFeedbackChange} showOriginalFilename={showOriginalFilename} + people={people} + onSelectPerson={onSelectPerson} /> )} diff --git a/frontend/src/components/gallery/PhotoLightbox.tsx b/frontend/src/components/gallery/PhotoLightbox.tsx index 20c1780a..db9e1ae3 100644 --- a/frontend/src/components/gallery/PhotoLightbox.tsx +++ b/frontend/src/components/gallery/PhotoLightbox.tsx @@ -1,7 +1,8 @@ -import React, { useState, useEffect, useLayoutEffect, useRef } from 'react'; +import React, { useState, useEffect, useLayoutEffect, useMemo, useRef } from 'react'; +import { useTranslation } from 'react-i18next'; import { useDevToolsProtection } from '../../hooks/useDevToolsProtection'; import { X, ChevronLeft, ChevronRight, Download, ZoomIn, ZoomOut, Minimize2, MessageSquare, Heart, Star } from 'lucide-react'; -import type { Photo } from '../../types'; +import type { Photo, GalleryPerson } from '../../types'; import { useSavePhotoToDevice } from '../../hooks/useGallery'; import { AuthenticatedImage } from '../common'; import { PhotoFeedback } from './PhotoFeedback'; @@ -31,6 +32,14 @@ interface PhotoLightboxProps { // back to source files (#508). Tied to the admin-side toggle that // also drives original-filename downloads (#493). showOriginalFilename?: boolean; + // People in this gallery (#1074). Passed only when the feature is on for + // the event AND visible to this viewer — an empty list means "scanned, + // nobody found", which the chips row handles by not rendering. + people?: GalleryPerson[]; + // Applying a person filter closes the lightbox and filters the grid + // behind it, so the guest lands on the result rather than paging through + // the old set. + onSelectPerson?: (personId: number) => void; } export const PhotoLightbox: React.FC = ({ @@ -48,6 +57,8 @@ export const PhotoLightbox: React.FC = ({ disableRightClick = false, enableDevtoolsProtection = false, showOriginalFilename = false, + people, + onSelectPerson, }) => { const [currentIndex, setCurrentIndex] = useState(initialIndex); const [zoom, setZoom] = useState(1); @@ -153,6 +164,17 @@ export const PhotoLightbox: React.FC = ({ // leaving currentIndex past the end. The effect below re-syncs the // index (or closes the lightbox when nothing is left). const currentPhoto = photos[currentIndex] ?? photos[photos.length - 1]; + + const { t } = useTranslation(); + + // People detected in the open photo (#1074). Resolved against the list the + // server returned rather than the raw ids, so a person the photographer + // hid or ignored has no entry to match and simply never appears. + const peopleInPhoto = useMemo(() => { + const ids = currentPhoto?.person_ids; + if (!ids?.length || !people?.length) return []; + return people.filter((person) => ids.includes(person.id)); + }, [currentPhoto?.person_ids, people]); // Per-category download permission (#640). AND'd with the event-level // allowDownloads — disabling at either level hides the download button. // Defaults true for uncategorised photos and pre-migration-135 categories. @@ -782,6 +804,37 @@ export const PhotoLightbox: React.FC = ({ {currentPhoto.original_filename || currentPhoto.filename}

)} + + {/* People in this photo (#1074). The second way into the face + filter: a guest looking at a photo of themselves can act on + it without scrolling back to the strip. + + Only people the server already returned are shown, so hidden + and ignored ones never appear here either. Unnamed people + show their photo count, never an invented name. */} + {peopleInPhoto.length > 0 && ( +
+ + {t('gallery.people.inThisPhoto', { defaultValue: 'In this photo:' })} + + {peopleInPhoto.map((person) => ( + + ))} +
+ )}
diff --git a/frontend/src/components/gallery/__tests__/PeopleStrip.test.tsx b/frontend/src/components/gallery/__tests__/PeopleStrip.test.tsx new file mode 100644 index 00000000..961d6e09 --- /dev/null +++ b/frontend/src/components/gallery/__tests__/PeopleStrip.test.tsx @@ -0,0 +1,134 @@ +import React from 'react'; +import { render, screen } from '@testing-library/react'; +import { describe, it, expect, vi } from 'vitest'; + +import { PeopleStrip } from '../PeopleStrip'; +import type { GalleryPerson, Photo } from '../../../types'; + +vi.mock('react-i18next', () => ({ + useTranslation: () => ({ + t: (_key: string, opts?: any) => opts?.defaultValue ?? _key, + }), +})); + +vi.mock('../../common/AuthenticatedImage', () => ({ + AuthenticatedImage: (props: any) => {props.alt}, +})); + +function person(id: number, over: Partial = {}): GalleryPerson { + return { + id, + label: null, + face_count: 10, + cover: { face_id: id, photo_id: id, bbox: [10, 10, 50, 50] }, + ...over, + }; +} + +const photos: Photo[] = [1, 2, 3].map((id) => ({ + id, + filename: `${id}.jpg`, + url: `/p/${id}`, + thumbnail_url: `/t/${id}`, + type: 'individual', + size: 100, + uploaded_at: '2026-01-01T00:00:00Z', + width: 1000, + height: 800, +})) as Photo[]; + +const baseProps = { + photos, + slug: 'test-gallery', + selectedPersonIds: [], + onToggle: vi.fn(), + onShowAll: vi.fn(), + collapsed: false, + onCollapsedChange: vi.fn(), +}; + +describe('PeopleStrip (#1074)', () => { + it('renders nothing for fewer than two people', () => { + // One face taking up a whole row is not a "people in this gallery" + // feature, it's clutter. + const { container } = render(); + expect(container).toBeEmptyDOMElement(); + }); + + it('still renders during a scan even with too few people yet', () => { + render( + + ); + expect(screen.getByText(/Finding people/)).toBeInTheDocument(); + }); + + it('shows a photo count for unnamed people, never an invented name', () => { + render(); + expect(screen.getByText('42 photos')).toBeInTheDocument(); + // The thing we specifically refuse to do. + expect(screen.queryByText(/Person \d/)).not.toBeInTheDocument(); + }); + + it('shows the name and the count when a person has been named', () => { + render( + + ); + expect(screen.getByText('Anna')).toBeInTheDocument(); + expect(screen.getByText('97')).toBeInTheDocument(); + }); + + it('marks the selected person as pressed for assistive tech', () => { + render( + + ); + const buttons = screen.getAllByRole('button', { pressed: true }); + expect(buttons).toHaveLength(1); + expect(buttons[0]).toHaveAttribute('aria-label', expect.stringContaining('Anna')); + }); + + it('collapses to a one-line summary when dismissed', () => { + render( + + ); + expect(screen.getByText('3 people found')).toBeInTheDocument(); + expect(screen.queryByTestId('avatar-img')).not.toBeInTheDocument(); + }); + + it('offers "show all" only when people overflow the inline strip', () => { + const many = Array.from({ length: 14 }, (_, i) => person(i + 1)); + const { rerender } = render(); + expect(screen.getByText('Show all 14')).toBeInTheDocument(); + + rerender(); + expect(screen.queryByText(/Show all/)).not.toBeInTheDocument(); + }); + + it('falls back to an uncropped thumbnail when photo dimensions are unknown', () => { + // Without width/height the bbox ratios can't be computed; rendering a + // wrongly-offset crop would be worse than not cropping. + const noDims = [{ ...photos[0], width: undefined, height: undefined }] as Photo[]; + render( + + ); + expect(screen.getAllByTestId('avatar-img').length).toBeGreaterThan(0); + }); +}); diff --git a/frontend/src/components/gallery/faceCrop.ts b/frontend/src/components/gallery/faceCrop.ts new file mode 100644 index 00000000..cd7210c0 --- /dev/null +++ b/frontend/src/components/gallery/faceCrop.ts @@ -0,0 +1,52 @@ +/** + * Shared face-crop geometry (#1074). + * + * Extracted because three surfaces need identical maths and two of them + * originally shipped without it: the "Show all" sheet and the admin People + * manager rendered centred full-photo thumbnails, so on any group photo the + * avatar showed whoever happened to be in the middle — often not the person + * it was labelling, and identical for two people from the same photo. + * + * The bbox is in ORIGINAL image pixels; the rendered thumbnail is some other + * size. Everything is therefore done in RATIOS of the source dimensions, which + * survives whatever rendition the browser is actually served. + */ + +export interface FaceBox { + bbox: [number, number, number, number]; +} + +/** + * Style for an inside a square, overflow-hidden, position-relative box + * of `size` px, such that `bbox` lands centred and filling it. + * + * Returns null when the source dimensions are unknown — the caller should then + * fall back to an uncropped thumbnail rather than render a wrongly-offset crop. + */ +export function faceCropStyle( + cover: FaceBox | null | undefined, + photoWidth: number | undefined, + photoHeight: number | undefined, + size: number, +): React.CSSProperties | null { + if (!cover || !photoWidth || !photoHeight) return null; + const [bx, by, bw, bh] = cover.bbox; + if (!bw || !bh) return null; + + // Detectors crop tight to the face; a little padding reads as a portrait + // rather than a specimen. + const pad = 0.45; + const cx = bx + bw / 2; + const cy = by + bh / 2; + const side = Math.max(bw, bh) * (1 + pad); + + const scale = size / side; + return { + width: `${photoWidth * scale}px`, + height: `${photoHeight * scale}px`, + maxWidth: 'none', + position: 'absolute', + left: `${size / 2 - cx * scale}px`, + top: `${size / 2 - cy * scale}px`, + }; +} diff --git a/frontend/src/components/gallery/layouts/BaseGalleryLayout.tsx b/frontend/src/components/gallery/layouts/BaseGalleryLayout.tsx index 1476af7b..4f4422f9 100644 --- a/frontend/src/components/gallery/layouts/BaseGalleryLayout.tsx +++ b/frontend/src/components/gallery/layouts/BaseGalleryLayout.tsx @@ -1,8 +1,13 @@ import React from 'react'; -import type { Photo, DownloadResolutionChoice } from '../../../types'; +import type { Photo, DownloadResolutionChoice, GalleryPerson } from '../../../types'; export interface BaseGalleryLayoutProps { photos: Photo[]; + // People in this gallery (#1074) — forwarded by PhotoGridWithLayouts so + // full-page layouts, which render their OWN lightbox, can still show the + // "In this photo" chips. + people?: GalleryPerson[]; + onSelectPerson?: (personId: number) => void; slug: string; onPhotoClick: (index: number) => void; // Optional: open the lightbox with feedback panel visible diff --git a/frontend/src/components/gallery/layouts/GalleryStoryLayout.tsx b/frontend/src/components/gallery/layouts/GalleryStoryLayout.tsx index a3580075..04c4ac20 100644 --- a/frontend/src/components/gallery/layouts/GalleryStoryLayout.tsx +++ b/frontend/src/components/gallery/layouts/GalleryStoryLayout.tsx @@ -62,6 +62,9 @@ export const GalleryStoryLayout: React.FC = ({ welcomeMessage, onLogout, showOriginalFilename = false, + + people, + onSelectPerson, }) => { // These props are passed by parent but we use our own feedback system, so mark as intentionally unused void _onPhotoClick; @@ -391,6 +394,11 @@ export const GalleryStoryLayout: React.FC = ({ useCanvasRendering={useCanvasRendering} onFeedbackChange={onFeedbackChange} showOriginalFilename={showOriginalFilename} + // #1074: this layout renders its own lightbox, so the people props + // have to be threaded through explicitly or the "In this photo" + // chips silently disappear on the Story theme. + people={people} + onSelectPerson={onSelectPerson} /> )} diff --git a/frontend/src/features/settings/tabs/FeaturesTab.tsx b/frontend/src/features/settings/tabs/FeaturesTab.tsx index 3f2a67f9..521c7d1f 100644 --- a/frontend/src/features/settings/tabs/FeaturesTab.tsx +++ b/frontend/src/features/settings/tabs/FeaturesTab.tsx @@ -26,8 +26,10 @@ import { Send, Workflow, } from 'lucide-react'; +import { useQuery } from '@tanstack/react-query'; import { useTranslation } from 'react-i18next'; import { Button, Card } from '../../../components/common'; +import { api } from '../../../config/api'; import { FeatureCard } from '../components/FeatureCard'; import { SidebarPreview } from '../components/SidebarPreview'; import { useFeatureFlags } from '../../../contexts/FeatureFlagsContext'; @@ -48,6 +50,17 @@ const Section: React.FC = ({ title, children }) => ( ); export const FeaturesTab: React.FC = () => { + // The all-in-one image cannot run face recognition (#1074) — see + // faceSettings.isSingleContainerImage. Read it from the version endpoint the + // admin UI already calls, so the card can say WHY rather than offering a + // switch that refuses to stay on. + const { data: systemVersion } = useQuery({ + queryKey: ['admin-system-version'], + queryFn: async () => (await api.get('/admin/system/version')).data, + staleTime: 5 * 60_000, + }); + const isSingleContainer = systemVersion?.single_container === true; + const { t } = useTranslation(); const { staged, setFlag, save, reset, isDirty, isSaving } = useFeatureFlags(); @@ -148,6 +161,33 @@ export const FeaturesTab: React.FC = () => { enabled={staged.transfers} onToggle={(next) => setFlag('transfers', next)} /> + + {/* Face recognition (#1074). Requires the optional picpeak-ml + sidecar container — with the flag on but no sidecar running, + photos simply stay queued and nothing breaks. Two deliberate + actions are still needed before any face is processed: this + toggle, and the per-event switch on each gallery. */} + setFlag('faces', next)} + disabled={isSingleContainer} + lockedReason={isSingleContainer + ? t( + 'settings.features.faces.lockedSingleContainer', + 'Not available on the all-in-one image. Face detection needs a separate ML container and competes with image processing for the same CPU and memory, which would slow the whole install down.', + ) + : undefined} + /> {/* Automation — the visual workflow engine. Master kill-switch for the diff --git a/frontend/src/i18n/locales/de.json b/frontend/src/i18n/locales/de.json index 5cd18164..f566e500 100644 --- a/frontend/src/i18n/locales/de.json +++ b/frontend/src/i18n/locales/de.json @@ -156,7 +156,10 @@ "show": "Einblenden", "poweredBy": "Bereitgestellt von", "on": "an", - "off": "aus" + "off": "aus", + "clear": "Zurücksetzen", + "saved": "Gespeichert", + "saveFailed": "Speichern fehlgeschlagen" }, "upload": { "photoCategory": "Fotokategorie", @@ -1061,7 +1064,27 @@ "downloadReady": "Ihr Download ist bereit", "downloadNow": "Herunterladen", "downloadPrepFailed": "Vorbereitung fehlgeschlagen", - "downloadPrepTimeout": "Das dauert länger als erwartet. Bitte erneut versuchen." + "downloadPrepTimeout": "Das dauert länger als erwartet. Bitte erneut versuchen.", + "people": { + "title": "Personen in dieser Galerie", + "showAll": "Alle {{count}} anzeigen", + "show": "Anzeigen", + "dismiss": "Personenleiste ausblenden", + "collapsedSummary": "{{count}} Personen gefunden", + "unnamedCount": "{{count}} Fotos", + "filterBy": "Fotos von {{name}} anzeigen", + "filterByUnnamed": "Fotos dieser Person anzeigen", + "scanning": "Personen werden gesucht… {{scanned}}/{{total}} Fotos", + "matchAll": "Beide Personen", + "matchAny": "Eine der Personen", + "matchCount": "{{count}} von {{total}} Fotos", + "clear": "Zurücksetzen", + "searchPlaceholder": "Person suchen", + "noMatches": "Niemand passt zu diesem Namen.", + "privacyNote": "Personen werden automatisch innerhalb dieser Galerie erkannt. Es werden keine Daten an externe Dienste gesendet.", + "inThisPhoto": "Auf diesem Foto:", + "downloadThese": "Diese {{count}} herunterladen" + } }, "categories": { "title": "Fotokategorien", @@ -2115,6 +2138,11 @@ "title": "Workflows", "description": "Visuelle Automatisierungen auf einer Canvas erstellen – Auslöser, Bedingungen, Verzweigungen, Schleifen und Freigabe-Gates für Admins. Deine Mahnstufen und Buchungsschritte werden zu bearbeitbaren Abläufen. Strikt optional.", "sidebar": "Workflows" + }, + "faces": { + "title": "Personen in Galerien", + "description": "Gruppiert die Fotos jeder Galerie nach den abgebildeten Personen, damit Gäste sich mit zwei Tipps selbst finden und nur ihre eigenen Fotos herunterladen. Läuft vollständig auf Ihrem eigenen Server in einem separaten, optionalen Container — es werden keine Daten übermittelt. Erkannte Gesichter sind personenbezogene Daten; die Funktion bleibt daher aus, bis Sie sie zusätzlich pro Galerie aktivieren.", + "lockedSingleContainer": "Im All-in-One-Image nicht verfügbar. Die Gesichtserkennung benötigt einen separaten ML-Container und konkurriert mit der Bildverarbeitung um dieselbe CPU und denselben Arbeitsspeicher, was die gesamte Installation verlangsamen würde." } }, "customerSurface": { @@ -3085,6 +3113,55 @@ "email_template_created": "E-Mail-Vorlage erstellt: {{template_key}}", "event_duplicated": "Event dupliziert aus {{source_event_name}}", "whatsapp_config_updated": "WhatsApp-Konfiguration aktualisiert" + }, + "people": { + "title": "Personen in dieser Galerie", + "subtitle": "Umbenennen, getrennte Personen zusammenführen oder jemanden vor Gästen verbergen.", + "select": "Zum Zusammenführen auswählen", + "unnamed": "Namen hinzufügen", + "namePlaceholder": "Namen hinzufügen", + "photoCount": "{{count}} Fotos", + "hidden": "für Gäste verborgen", + "ignored": "ignoriert", + "renamed": "Name gespeichert", + "merged": "Personen zusammengeführt", + "split": "Als neue Person abgetrennt", + "updated": "Aktualisiert", + "merge": "Zusammenführen", + "mergeHint": "Tippen Sie zwei oder mehr Gesichter an, um sie zu einer Person zusammenzuführen.", + "selectedCount": "{{count}} ausgewählt", + "splitAction": "Fotos abtrennen, die jemand anderes zeigen", + "hideAction": "Vor Gästen verbergen", + "ignoreAction": "Keine echte Person — ignorieren", + "splitHelp": "Wählen Sie die Fotos aus, die NICHT diese Person zeigen. Sie werden zu einem neuen Eintrag, alles andere bleibt.", + "splitSelected": "{{count}} ausgewählt", + "doSplit": "Abtrennen", + "empty": "Noch keine Personen erkannt." + }, + "faces": { + "manage": "Personen verwalten", + "visibleToGuests": "({{count}} für Gäste sichtbar)", + "title": "Personen in dieser Galerie", + "subtitle": "Fotos nach den abgebildeten Personen gruppieren, damit Gäste ihre eigenen finden und herunterladen können.", + "consentNotice": "Erkannte Gesichter sind personenbezogene Daten und gelten in der EU als besondere Kategorie. Sie sind für diese Galerie verantwortlich: Stellen Sie sicher, dass Sie eine Rechtsgrundlage für die abgebildeten Personen haben, bevor Sie dies aktivieren. Nichts verlässt Ihren Server — die Erkennung läuft in Ihrem eigenen Container.", + "enable": "Personen in dieser Galerie erkennen", + "enableHint": "Vorhandene Fotos werden im Hintergrund durchsucht. Gesichter und ihre numerischen Signaturen werden in Ihrer Datenbank gespeichert; sie sind niemals in Backups oder Exporten enthalten.", + "visible": "Personenleiste für Gäste anzeigen", + "visibleHint": "Ausgeschaltet erhalten nur Sie die Gruppierung als privates Werkzeug, und Gäste sehen eine unveränderte Galerie.", + "previewNotice": "Die Erkennung arbeitet mit der Vorschaugröße jedes Fotos. In Galerien ohne vorhandene Vorschaubilder werden diese beim ersten Durchlauf erzeugt, was zusätzliche CPU-Last und Speicherplatz benötigt.", + "scanning": "Wird durchsucht… {{scanned}} von {{total}} Fotos", + "status": "{{scanned}} / {{total}} Fotos durchsucht · {{people}} Personen", + "failed": "{{count}} fehlgeschlagen", + "queued": "{{count}} Fotos werden nach Personen durchsucht…", + "rescan": "Erneut durchsuchen", + "rescanQueued": "Erneute Suche eingeplant", + "recluster": "Personen neu gruppieren", + "reclustered": "Personen neu gruppiert", + "delete": "Alle Gesichtsdaten löschen", + "deleted": "Gesichtsdaten gelöscht", + "confirmDelete": "Alle erkannten Personen und Gesichtsdaten dieser Galerie löschen? Dies kann nicht rückgängig gemacht werden. Die Fotos selbst bleiben unverändert.", + "autoCategories": "Fotos automatisch in Kategorien einsortieren", + "autoCategoriesHint": "Nutzt die Anzahl der Gesichter, um Fotos als Details, Porträts, Kleine Gruppen oder Gruppen abzulegen. Gilt für alle Galerien, füllt ausschließlich leere Kategorien und ändert niemals eine von Ihnen gesetzte." } }, "acceptInvitation": { diff --git a/frontend/src/i18n/locales/en.json b/frontend/src/i18n/locales/en.json index 674d7d64..d6430e1f 100644 --- a/frontend/src/i18n/locales/en.json +++ b/frontend/src/i18n/locales/en.json @@ -156,7 +156,10 @@ "show": "Show", "poweredBy": "Powered by", "on": "on", - "off": "off" + "off": "off", + "clear": "Clear", + "saved": "Saved", + "saveFailed": "Save failed" }, "upload": { "photoCategory": "Photo Category", @@ -602,7 +605,27 @@ "downloadReady": "Your download is ready", "downloadNow": "Download", "downloadPrepFailed": "Preparation failed", - "downloadPrepTimeout": "This is taking longer than expected. Please try again." + "downloadPrepTimeout": "This is taking longer than expected. Please try again.", + "people": { + "title": "People in this gallery", + "showAll": "Show all {{count}}", + "show": "Show", + "dismiss": "Hide the people bar", + "collapsedSummary": "{{count}} people found", + "unnamedCount": "{{count}} photos", + "filterBy": "Show photos of {{name}}", + "filterByUnnamed": "Show photos of this person", + "scanning": "Finding people… {{scanned}}/{{total}} photos", + "matchAll": "Both people", + "matchAny": "Either person", + "matchCount": "{{count}} of {{total}} photos", + "clear": "Clear", + "searchPlaceholder": "Find a person", + "noMatches": "No one matches that name.", + "privacyNote": "People are detected automatically inside this gallery. Nothing is sent to any external service.", + "inThisPhoto": "In this photo:", + "downloadThese": "Download these {{count}}" + } }, "categories": { "title": "Photo Categories", @@ -1656,6 +1679,11 @@ "title": "Workflows", "description": "Build visual automations on a canvas — triggers, conditions, branches, loops and admin approval gates. Your reminder ladder and booking steps become editable flows. Strictly opt-in.", "sidebar": "Workflows" + }, + "faces": { + "title": "People in galleries", + "description": "Group each gallery's photos by the people in them, so guests can find themselves in two taps and download just their own photos. Runs entirely on your own server in a separate, optional container — nothing is sent anywhere. Detected faces are personal data, so this stays off until you enable it per gallery too.", + "lockedSingleContainer": "Not available on the all-in-one image. Face detection needs a separate ML container and competes with image processing for the same CPU and memory, which would slow the whole install down." } }, "customerSurface": { @@ -2657,6 +2685,55 @@ "email_template_created": "Email template created: {{template_key}}", "event_duplicated": "Event duplicated from {{source_event_name}}", "whatsapp_config_updated": "WhatsApp configuration updated" + }, + "people": { + "title": "People in this gallery", + "subtitle": "Rename, merge people who were split apart, or hide someone from guests.", + "select": "Select for merging", + "unnamed": "Add a name", + "namePlaceholder": "Add a name", + "photoCount": "{{count}} photos", + "hidden": "hidden from guests", + "ignored": "ignored", + "renamed": "Name saved", + "merged": "People merged", + "split": "Split into a new person", + "updated": "Updated", + "merge": "Merge", + "mergeHint": "Tap two or more faces to merge them into one person.", + "selectedCount": "{{count}} selected", + "splitAction": "Split out photos that are someone else", + "hideAction": "Hide from guests", + "ignoreAction": "Not a real person — ignore", + "splitHelp": "Pick the photos that are NOT this person. They become a new entry, and everything else stays.", + "splitSelected": "{{count}} selected", + "doSplit": "Split out", + "empty": "No people detected yet." + }, + "faces": { + "manage": "Manage people", + "visibleToGuests": "({{count}} shown to guests)", + "title": "People in this gallery", + "subtitle": "Group photos by the people in them, so guests can find and download their own.", + "consentNotice": "Detected faces are personal data, and in the EU they count as a special category. You are the controller for this gallery: make sure you have a lawful basis for the people in these photos before switching this on. Nothing leaves your server — detection runs in your own container.", + "enable": "Detect people in this gallery", + "enableHint": "Existing photos are scanned in the background. Faces and their numeric signatures are stored in your database; they are never included in backups or exports.", + "visible": "Show the people bar to guests", + "visibleHint": "Off means you get the grouping as a private tool and guests see an unchanged gallery.", + "previewNotice": "Scanning works on the preview-sized copy of each photo. Galleries that have not generated previews yet will create them during the first scan, which uses additional CPU and disk space.", + "scanning": "Scanning… {{scanned}} of {{total}} photos", + "status": "{{scanned}} / {{total}} photos scanned · {{people}} people", + "failed": "{{count}} failed", + "queued": "Scanning {{count}} photos for people…", + "rescan": "Re-scan", + "rescanQueued": "Re-scan queued", + "recluster": "Re-group people", + "reclustered": "People regrouped", + "delete": "Delete all face data", + "deleted": "Face data deleted", + "confirmDelete": "Delete all detected people and face data for this gallery? This cannot be undone. Photos are not affected.", + "autoCategories": "Sort photos into categories automatically", + "autoCategoriesHint": "Uses the number of faces to file photos as Details, Portraits, Small groups or Groups. Applies to every gallery, only ever fills an empty category, and never changes one you set yourself." } }, "acceptInvitation": { diff --git a/frontend/src/pages/admin/event-details/OverviewTab.tsx b/frontend/src/pages/admin/event-details/OverviewTab.tsx index e6f11593..64f4d4b7 100644 --- a/frontend/src/pages/admin/event-details/OverviewTab.tsx +++ b/frontend/src/pages/admin/event-details/OverviewTab.tsx @@ -5,6 +5,7 @@ import { PermissionGate } from '../../../components/admin/PermissionGate'; import { EventReminderOverrideCard } from '../../../components/admin/EventReminderOverrideCard'; import { SlideshowSettingsCard } from '../../../components/admin/SlideshowSettingsCard'; import { DownloadResolutionCard } from '../../../components/admin/DownloadResolutionCard'; +import { FaceRecognitionCard } from '../../../components/admin/FaceRecognitionCard'; import { ShortUrlsCard } from '../../../components/admin/ShortUrlsCard'; import { useFeatureFlags } from '../../../contexts/FeatureFlagsContext'; import type { AdminPhoto } from '../../../services/photos.service'; @@ -122,6 +123,14 @@ export const OverviewTab: React.FC = ({ other "what the customer receives" controls. */} refetchEvent()} /> + {/* People in this gallery (#1074). Gated behind the `faces` feature + flag — which is itself gated on the operator running the optional + picpeak-ml sidecar, so this card is invisible on the vast majority + of installs. */} + {flags.faces && ( + + )} + {/* Live Slideshow ("Diashow") link + live display settings (migrations 138/139). Gated behind the `slideshow` feature flag. */} {flags.slideshow && ( diff --git a/frontend/src/services/featureFlags.service.ts b/frontend/src/services/featureFlags.service.ts index f9168ce6..ca62980c 100644 --- a/frontend/src/services/featureFlags.service.ts +++ b/frontend/src/services/featureFlags.service.ts @@ -75,7 +75,17 @@ export type FeatureKey = // Workflow / automation engine — admin-configurable visual flows (triggers, // conditions, branches, loops, approval gates) built on a canvas. Strictly // opt-in; gates the Workflows admin area and the engine runtime. - | 'workflows'; + | 'workflows' + // Face recognition — "People in this gallery" (migration 177, #1074). + // Requires the optional picpeak-ml sidecar container. THIS FLAG IS THE + // GATE for the whole feature: the backend's FACE_ML_URL has a working + // default, so the sidecar's presence can't be detected from config alone. + // While this is off, no face UI renders anywhere — no admin panel, no + // people strip, no lightbox chips — and the backend never contacts the + // sidecar. Face embeddings are biometric data (GDPR Art. 9); turning this + // on is only the first of two deliberate actions, since detection is still + // enabled per event. + | 'faces'; export type FeatureFlags = Record; diff --git a/frontend/src/services/gallery.service.ts b/frontend/src/services/gallery.service.ts index 0fb3a295..c8c9a5fd 100644 --- a/frontend/src/services/gallery.service.ts +++ b/frontend/src/services/gallery.service.ts @@ -1,7 +1,7 @@ import { api } from '../config/api'; import type { GalleryInfo, GalleryData, GalleryStats, ResolvedGalleryIdentifier, - DownloadJobStatus, DownloadJobState, + DownloadJobStatus, DownloadJobState, GalleryPeopleResponse, } from '../types'; import { normalizeRequirePassword } from '../utils/accessControl'; import { parseContentDispositionFilename } from '../utils/contentDisposition'; @@ -383,4 +383,16 @@ export const galleryService = { const response = await api.get(`/gallery/resolve/${identifier}`); return response.data; }, + + /** + * People detected in this gallery (#1074). + * + * Returns an empty list rather than an error when the feature is off, so a + * guest can't tell "no people here" from "feature disabled". Counts and + * cover faces are scoped server-side to the photos this viewer may see. + */ + async getPeople(slug: string): Promise { + const response = await api.get(`/gallery/${slug}/people`); + return response.data; + }, }; diff --git a/frontend/src/types/index.ts b/frontend/src/types/index.ts index 2fe5779f..fa52bfe4 100644 --- a/frontend/src/types/index.ts +++ b/frontend/src/types/index.ts @@ -113,6 +113,36 @@ export interface GalleryInfo { login_logo_visible?: boolean; } +/** + * A person clustered within one gallery (#1074). + * + * `face_count` is scoped to the photos THIS viewer can see, which is not the + * same as the number of faces stored for them — a guest must never learn how + * many hidden photos someone appears in. `label` is null until the + * photographer names them; the UI shows the count instead of inventing a + * "Person 7", because a number is honest and a fake name is not. + */ +export interface GalleryPerson { + id: number; + label: string | null; + face_count: number; + cover: { + face_id: number; + photo_id: number; + /** [x, y, w, h] in ORIGINAL image pixels — scale by the rendered size. */ + bbox: [number, number, number, number]; + } | null; +} + +export interface GalleryPeopleResponse { + people: GalleryPerson[]; + scan: { + in_progress: boolean; + scanned: number; + total: number; + }; +} + export interface Photo { id: number; filename: string; @@ -145,6 +175,13 @@ export interface Photo { // the lightbox download button when this is false (event-level allow_downloads // also has to be true — they AND together). category_allow_downloads?: boolean; + // People detected in this photo (#1074). Always present when the feature + // is on for the event — an empty array means "scanned, nobody found", + // which is different from the feature being off (see + // GalleryInfo.event.people_enabled). Hidden and ignored people are + // filtered out server-side, so this never reveals a person the + // photographer suppressed. + person_ids?: number[]; size: number; uploaded_at: string; captured_at?: string; // EXIF capture date (if available) @@ -248,6 +285,11 @@ export interface GalleryData { // When true, the lightbox surfaces each photo's `original_filename` // alongside the position counter (#508). use_original_filenames?: boolean; + // "People in this gallery" (#1074). False whenever the global `faces` + // flag is off, detection is off for this event, or the photographer + // chose to keep the people strip to themselves. The whole face UI hangs + // off this one boolean. + people_enabled?: boolean; }; categories?: PhotoCategory[]; photos: Photo[]; diff --git a/ml/.dockerignore b/ml/.dockerignore new file mode 100644 index 00000000..8c08d7fd --- /dev/null +++ b/ml/.dockerignore @@ -0,0 +1,16 @@ +# The conversion tool and its TensorFlow dependency never belong in the image. +tools/ +tests/ +README.md +LICENSES.md + +__pycache__/ +*.pyc +.pytest_cache/ +.venv/ +venv/ + +# Locally-produced model artifacts — the image fetches these by pinned URL +# and checksum in the `models` build stage instead. +*.onnx +*.h5 diff --git a/ml/Dockerfile b/ml/Dockerfile new file mode 100644 index 00000000..b076e96f --- /dev/null +++ b/ml/Dockerfile @@ -0,0 +1,123 @@ +# picpeak-ml — optional face-detection sidecar (#1074). +# +# Debian slim rather than Alpine: onnxruntime publishes manylinux wheels for +# x86_64 and aarch64 but nothing for musl, so Alpine would mean compiling ORT +# from source on both legs of the multi-arch build. The slim base costs ~40MB +# over Alpine and saves an hour of CI per build. + +# --------------------------------------------------------------------------- +# Stage 1 — fetch and verify model weights +# --------------------------------------------------------------------------- +# Weights are baked in, never downloaded at runtime: airgapped installs must +# work, and a model that changes under a running deployment would silently +# invalidate every stored embedding. +# +# Both artifacts are pinned by URL *and* SHA-256. The checksum is the point — +# an immutable-looking URL that starts serving different bytes must fail the +# build rather than quietly reshape the embedding space. +FROM python:3.12-slim AS models + +ARG YUNET_URL=https://media.githubusercontent.com/media/opencv/opencv_zoo/f12e12798e8314f7c074a6656816c048dcc95b7a/models/face_detection_yunet/face_detection_yunet_2023mar.onnx +ARG YUNET_SHA256=8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4 + +# FaceNet-512 as ONNX. deepface distributes this model as Keras .h5 only, so +# the ONNX is produced once by `tools/convert_facenet.py` and published as a +# release asset — converting inside this build would drag TensorFlow (~600MB) +# through both architecture legs to produce a file that is identical either +# way. +# +# Defaults to the canonical published artifact so `docker build ml/` and +# `docker compose --profile faces up` both work with no arguments. Override +# both together to use a different embedder. Blanking either one still fails +# the build loudly (below) rather than silently producing an image with no +# embedder — the checksum is what makes the URL safe to trust, so a URL +# without one is never acceptable. +ARG FACENET_ONNX_URL=https://github.com/PicPeak/picpeak/releases/download/ml-models-v1/facenet512.onnx +ARG FACENET_ONNX_SHA256=a1c06dcb79dc17a42af01d5bcbce4822caa148b9c24bf7eb8b8e556b4fd0d5db + +RUN apt-get update \ + && apt-get install -y --no-install-recommends curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /models + +RUN curl -fsSL -o face_detection_yunet_2023mar.onnx "${YUNET_URL}" \ + && echo "${YUNET_SHA256} face_detection_yunet_2023mar.onnx" | sha256sum -c - + +RUN if [ -z "${FACENET_ONNX_URL}" ] || [ -z "${FACENET_ONNX_SHA256}" ]; then \ + echo "ERROR: FACENET_ONNX_URL and FACENET_ONNX_SHA256 build args are required." >&2; \ + echo " Produce the artifact with ml/tools/convert_facenet.py, publish it," >&2; \ + echo " then pass both args. See ml/README.md." >&2; \ + exit 1; \ + fi \ + && curl -fsSL -o facenet512.onnx "${FACENET_ONNX_URL}" \ + && echo "${FACENET_ONNX_SHA256} facenet512.onnx" | sha256sum -c - + +# --------------------------------------------------------------------------- +# Stage 2 — runtime +# --------------------------------------------------------------------------- +FROM python:3.12-slim + +ARG BUILD_DATE +ARG VCS_REF +ARG VERSION + +LABEL org.opencontainers.image.source="https://github.com/PicPeak/picpeak" +LABEL org.opencontainers.image.description="PicPeak ML sidecar — face detection and embedding" +LABEL org.opencontainers.image.licenses="MIT" + +# Busts the apt layer each CI run so the image picks up current Debian +# security updates instead of reusing a stale cached upgrade layer — same +# reasoning as backend/Dockerfile. +ARG CACHEBUST=1 +RUN echo "cachebust=${CACHEBUST}" \ + && apt-get update \ + && apt-get upgrade -y \ + # libGL and libglib are opencv-python-headless's remaining shared-library + # deps. The headless wheel drops the GUI toolkits but still links libGL. + && apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +COPY requirements.txt . +# pip itself is ~10MB of an image that never installs anything at runtime. +# +# The `|| true` is scoped to the uninstall ONLY. Written as +# `pip install && pip uninstall || true` the shell parses it as +# `(install && uninstall) || true`, so a failed requirements install still +# produces a green layer — and CI would publish an ML image with no FastAPI, +# no uvicorn and no onnxruntime that fails at container start instead of at +# build time. +RUN pip install --no-cache-dir -r requirements.txt \ + && { pip uninstall -y pip setuptools 2>/dev/null || true; } + +# Non-root. Nothing in this container writes anything — no volumes, no +# database, no model download — so the whole filesystem can stay read-only to +# the service account. +# +# Created BEFORE the copies so ownership can be set by COPY --chown. A +# `chown -R` afterwards would rewrite every copied file into a fresh layer, +# duplicating the 90MB model and adding ~94MB to the image for nothing. +RUN useradd --system --uid 1001 --create-home picpeak + +COPY --from=models --chown=picpeak:picpeak /models /models +COPY --chown=picpeak:picpeak app ./app + +USER picpeak + +ENV FACE_MODEL_DIR=/models \ + PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 + +EXPOSE 8000 + +# Mirrors the backend's healthcheck shape. /health is unauthenticated so this +# needs no secret; it reports liveness only, because a failed model load +# aborts startup and the container never serves at all. +HEALTHCHECK --interval=30s --timeout=5s --start-period=40s --retries=3 \ + CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)" + +# Single worker on purpose: the models are loaded per process, so a second +# worker doubles RSS for a service the backend calls at concurrency 1. +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"] diff --git a/ml/LICENSES.md b/ml/LICENSES.md new file mode 100644 index 00000000..a2fea731 --- /dev/null +++ b/ml/LICENSES.md @@ -0,0 +1,60 @@ +# Model provenance and licences + +PicPeak is run commercially by working photographers. That sets a hard rule +for this image: + +> **No non-commercial artifact is ever baked into a PicPeak image.** + +Every open face-recognition weight set traces back to a scraped, +research-only dataset — CASIA-WebFace, VGGFace2, MS1M, Glint360K and +WebFace260M all carry academic-use-only agreements, so "trained on clean +data" is not an option that exists. What differs, and what actually binds a +redistributor, is the grant the **distributor** places on the artifact we +copy into this image. + +## Shipped in this image + +| Artifact | Role | Distributor | Licence | +|---|---|---|---| +| `face_detection_yunet_2023mar.onnx` | detection | [OpenCV Zoo](https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet) | MIT | +| `facenet512.onnx` | embedding | [serengil/deepface](https://github.com/serengil/deepface) | MIT | + +Both are redistributable. `facenet512.onnx` is converted from deepface's +published `facenet512_weights.h5` by `tools/convert_facenet.py`; the +conversion changes the container format, not the weights, so the MIT grant +carries over. + +Pinned sources, verified by SHA-256 at build time: + +- YuNet — `opencv/opencv_zoo` at commit `f12e12798e8314f7c074a6656816c048dcc95b7a` + `8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4` +- FaceNet-512 source weights — `serengil/deepface_models` release `v1.0` + `3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f` + +## Deliberately NOT shipped + +| Artifact | Why not | +|---|---| +| InsightFace `buffalo_*`, `antelopev2` | **Non-commercial research only.** Commercial use requires a licence from insightface.ai ([model zoo README](https://github.com/deepinsight/insightface/blob/master/model_zoo/README.md), [#2587](https://github.com/deepinsight/insightface/issues/2587)). Available as an opt-in via `FACE_MODEL`, downloaded by the operator who has cleared that licence themselves — never by us. | +| Idiap EdgeFace | CC BY-NC-SA 4.0 — explicitly non-commercial. The best accuracy-per-parameter of the candidates, and unusable for that reason. | +| AdaFace WebFace4M weights | MIT code, non-commercial weights. | + +## What that choice cost + +Measured on DeepFace's own matched benchmark (LFW, aligned, cosine — the only +comparison worth anything, since every model quotes its own "LFW 99.x%" on +its own pipeline): + +| Embedder | with RetinaFace | with YuNet | +|---|---|---| +| **FaceNet-512** | **98.4%** | **97.9%** | +| ArcFace | 96.6% | 96.7% | +| SFace | 92.4% | 91.0% | + +Choosing YuNet over RetinaFace costs ~0.5 points. Holding the +commercial-redistribution line costs nothing beyond that — FaceNet-512 is +both the most accurate option in the table *and* MIT. Taking the headline +numbers at face value would have pointed at SFace (advertised 0.9940, actual +91–92% matched) and cost seven points, which for clustering is fatal: every +false split invents a duplicate person and every false merge puts a stranger +into someone's "download my photos". diff --git a/ml/README.md b/ml/README.md new file mode 100644 index 00000000..ea9197cf --- /dev/null +++ b/ml/README.md @@ -0,0 +1,149 @@ +# picpeak-ml + +Optional face-detection sidecar for PicPeak (#1074). Detects faces in one +image and returns a bounding box, five landmarks, quality signals and a +512-d embedding per face. + +**Nothing else.** No database, no volumes, no state, no egress, no model +download at runtime. Clustering, person identity, thresholds and every +privacy decision live in the PicPeak backend, where the data already is. +This service forgets each image the moment it answers. + +If you don't run this container, the feature does not exist — see +"Turning it on" below. + +## API + +All endpoints except `/health` require the `X-Face-ML-Token` header. The +service **refuses to start** without `FACE_ML_TOKEN` set, so an accidentally +published port is never a free face-detection API. + +| | | +|---|---| +| `GET /health` | `{"status": "ok"}` — unauthenticated, used by the compose healthcheck | +| `GET /info` | `{detector, embedder, model_version, dim}` | +| `POST /faces` | multipart `image` → `{model_version, faces: [...]}` | + +Each face: + +```jsonc +{ + "bbox": [x, y, w, h], // ORIGINAL image pixels, not detection-scaled + "score": 0.94, + "landmarks": [[x, y], ...], // 5: right eye, left eye, nose, right mouth, left mouth + "yaw": -1.42, // degrees, approximate (see pipeline.py) + "pitch": -25.33, + "blur": 2579.5, // variance of Laplacian on the aligned crop; higher = sharper + "embedding": [...] // 512 floats, L2-normalized +} +``` + +`404`/`400` mean "this image is a lost cause" — the backend marks the photo +failed. `5xx` and connection failures mean "try later" — the backend returns +the photo to `pending` with backoff, so turning this container off for a week +does not require a manual re-scan. + +## Models + +YuNet (detection, MIT) + FaceNet-512 (embedding, MIT), both baked into the +image and verified by SHA-256 at build time. See [LICENSES.md](LICENSES.md) +for why these two and not the more obvious InsightFace weights — the short +version is that InsightFace's are non-commercial-only and PicPeak's users are +working photographers. + +### Building the image + +`facenet512.onnx` is **not** fetched automatically, because deepface +distributes FaceNet-512 as Keras `.h5` only. Convert it once, publish it, +then pass the URL and checksum: + +```bash +cd ml +python3.11 -m venv .venv && . .venv/bin/activate # 3.11: TF has no 3.12+ wheels +pip install -r tools/requirements-convert.txt + +curl -fsSL -o facenet512_weights.h5 \ + https://github.com/serengil/deepface_models/releases/download/v1.0/facenet512_weights.h5 +echo "3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f facenet512_weights.h5" | sha256sum -c - + +python tools/convert_facenet.py facenet512_weights.h5 facenet512.onnx +``` + +The script verifies the converted graph against the Keras original before +writing (worst observed divergence: 2.1e-06 absolute, cosine 1.0000000000) +and prints the SHA-256 to publish. Output is ~89.6 MB, 23,497,424 parameters. + +Publish `facenet512.onnx` as a release asset, set the repository variables +`FACENET_ONNX_URL` and `FACENET_ONNX_SHA256` (Settings → Variables — it's a +public URL, not a secret), and CI picks it up. To build locally: + +```bash +docker build -t picpeak-ml \ + --build-arg FACENET_ONNX_URL=https://github.com/PicPeak/picpeak/releases/download//facenet512.onnx \ + --build-arg FACENET_ONNX_SHA256= \ + ml/ +``` + +The conversion sits outside the Docker build because TensorFlow is ~600MB of +build dependency for a file that never ships in the final image, and the +result is architecture-independent — no reason to run it on both legs of +every multi-arch build. + +**The conversion is not byte-reproducible.** Two runs with the same pinned +versions on the same machine produce functionally identical graphs (same 336 +nodes, same 271 initializers, weights matching to 0.000e+00) but differ in a +few initializer names, because tf2onnx's traced-op naming is not +deterministic. So a re-conversion **will** have a different SHA-256, and that +is expected rather than a sign of tampering. The checksum pins one published +artifact so its URL cannot start serving different bytes; validating a fresh +conversion is the parity check's job, not the hash's. + +## Not available on the all-in-one image + +The single-container image (`Dockerfile.aio`) sets +`PICPEAK_SINGLE_CONTAINER=true`, and the backend refuses to enable face +recognition when it sees that — the feature flag cannot be switched on, and +per-event detection stays off even if a restored database says otherwise. + +This is a performance decision, not a licensing or packaging one. That image +runs the backend, the frontend, SQLite and every background worker inside one +container aimed at "one photographer plus guests browsing". It has no Redis, +SQLite gives it a single writer, and it contains no ML sidecar to talk to. +Adding a second image-processing pipeline that competes with Sharp for the +same CPU and RAM would not fail loudly — it would just make the whole install +slow and appear broken. + +Run the standard multi-container deployment if you want this feature. + +## Turning it on + +Two deliberate actions, neither of which is installing this container: + +1. Enable the `faces` feature flag in PicPeak's admin settings. +2. Enable "Detect people in this gallery" per event. + +`FACE_ML_URL` defaults to `http://picpeak-ml:8000` — the compose service name +— so the standard deployment needs no URL configuration. **Nothing in the +backend touches that URL while the flag is off**, so an install without this +container never attempts a connection. + +## Development + +```bash +pip install -r requirements.txt pytest httpx +python -m pytest tests/ -q +``` + +The tests stub the models out: they cover the auth boundary, the request +guards and the alignment geometry — the places where a mistake is a security +problem or a silent accuracy problem. Model *quality* is not a unit-test +question; that is what the Phase 0 spike measured. + +### The one thing to be careful about + +The alignment in `pipeline.py` and the normalization in `_embed` must stay +identical to whatever the clustering threshold was tuned against. A tuned +cosine threshold does not transfer across an alignment change. If either +changes, bump `MODEL_VERSION` in `config.py` — the backend keys +re-derivation off that string and will re-cluster rather than silently mix +two incompatible embedding spaces. diff --git a/ml/app/__init__.py b/ml/app/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ml/app/config.py b/ml/app/config.py new file mode 100644 index 00000000..fffeab1b --- /dev/null +++ b/ml/app/config.py @@ -0,0 +1,72 @@ +""" +Configuration for the picpeak-ml sidecar (#1074). + +Everything is read once at import. There is no reload path and no settings +API — this service is stateless by design, and an operator changing a knob +restarts the container. +""" + +import os + +# --------------------------------------------------------------------------- +# Models +# --------------------------------------------------------------------------- + +MODEL_DIR = os.environ.get("FACE_MODEL_DIR", "/models") + +DETECTOR_FILENAME = "face_detection_yunet_2023mar.onnx" +EMBEDDER_FILENAME = os.environ.get("FACE_EMBEDDER_FILENAME", "facenet512.onnx") + +DETECTOR_NAME = "yunet_2023mar" +EMBEDDER_NAME = os.environ.get("FACE_MODEL", "facenet512") + +# Stamped onto every face row the backend stores. Changing the detector, the +# embedder, the alignment or the normalization MUST bump this: embeddings from +# two different pipelines are not comparable, and a silent mix produces +# clusters that look plausible and are wrong. The backend keys re-derivation +# off this string. +MODEL_VERSION = f"{DETECTOR_NAME}+{EMBEDDER_NAME}+v1" + +# --------------------------------------------------------------------------- +# Auth +# --------------------------------------------------------------------------- + +# Shared secret, required. `main.py` refuses to start without it rather than +# defaulting to open: an accidentally published port must not be a free +# face-detection API. +TOKEN = os.environ.get("FACE_ML_TOKEN", "").strip() +TOKEN_HEADER = "X-Face-ML-Token" + +# --------------------------------------------------------------------------- +# Detection +# --------------------------------------------------------------------------- + +# YuNet's own confidence floor. Deliberately permissive: the backend applies +# the *product* quality floor (score, blur, bbox size) because those +# thresholds are admin-tunable there and baked-in here. This service's job is +# to report what it saw, not to decide what counts. +DET_SCORE_THRESHOLD = float(os.environ.get("FACE_DET_SCORE_THRESHOLD", "0.6")) +NMS_THRESHOLD = float(os.environ.get("FACE_NMS_THRESHOLD", "0.3")) +TOP_K = int(os.environ.get("FACE_DET_TOP_K", "5000")) + +# Guard rails on untrusted input. The backend only ever sends its own preview +# renditions (≤1920px), but this service must not fall over if something else +# reaches it. +MAX_IMAGE_BYTES = int(os.environ.get("FACE_MAX_IMAGE_BYTES", str(32 * 1024 * 1024))) +MAX_FACES = int(os.environ.get("FACE_MAX_FACES", "64")) + +# Long edge the image is downscaled to before detection. Matches the backend's +# preview tier (imageProcessor.js generates ≤1920px), so the common case is a +# no-op; anything larger is scaled down here so detection cost stays bounded. +# Bboxes and landmarks are always reported in ORIGINAL image coordinates. +INPUT_LONG_EDGE = int(os.environ.get("FACE_INPUT_LONG_EDGE", "1920")) + +# --------------------------------------------------------------------------- +# Runtime +# --------------------------------------------------------------------------- + +# One ORT thread by default. The backend's face queue runs at concurrency 1 +# (it shares a host with Sharp, which is the real memory pressure — see +# backgroundProcessor.js), so letting ORT fan out across every core buys +# nothing and costs RSS. +ORT_THREADS = int(os.environ.get("FACE_ORT_THREADS", "1")) diff --git a/ml/app/main.py b/ml/app/main.py new file mode 100644 index 00000000..2c6b5655 --- /dev/null +++ b/ml/app/main.py @@ -0,0 +1,133 @@ +""" +picpeak-ml — optional face-detection sidecar for PicPeak (#1074). + +Three endpoints, no database, no volumes, no egress. Models are baked into +the image at build time, so an airgapped install works and nothing is +downloaded at runtime. + +The service is deliberately dumb: it reports what it saw in one image and +forgets. Clustering, identity, thresholds and every privacy decision live in +the backend, where the data already is. +""" + +import logging +from contextlib import asynccontextmanager + +from fastapi import Depends, FastAPI, File, Header, HTTPException, UploadFile +from fastapi.responses import JSONResponse + +from . import config +from .pipeline import FacePipeline +from .schemas import FacesResponse, HealthResponse, InfoResponse + +logging.basicConfig( + level=logging.INFO, format="%(asctime)s %(levelname)s [picpeak-ml] %(message)s" +) +logger = logging.getLogger(__name__) + +_pipeline: FacePipeline | None = None + + +@asynccontextmanager +async def lifespan(app: FastAPI): + global _pipeline + + # Refuse to run open. An accidentally published port must not be a free + # face-detection API, and a service that silently accepted anonymous + # requests when the operator forgot the variable would be exactly that. + if not config.TOKEN: + raise RuntimeError( + "FACE_ML_TOKEN is not set. picpeak-ml will not start without a " + "shared secret — set it on both this container and the backend." + ) + + logger.info("Loading models from %s", config.MODEL_DIR) + _pipeline = FacePipeline() + logger.info( + "Ready: detector=%s embedder=%s version=%s dim=%d", + config.DETECTOR_NAME, + config.EMBEDDER_NAME, + config.MODEL_VERSION, + _pipeline.dim, + ) + yield + _pipeline = None + + +app = FastAPI( + title="picpeak-ml", + description="Face detection and embedding sidecar for PicPeak", + lifespan=lifespan, + # No interactive docs: this is a machine-to-machine service on a private + # network, and /docs is just surface. + docs_url=None, + redoc_url=None, + openapi_url=None, +) + + +def require_token(x_face_ml_token: str = Header(default="")) -> None: + """Constant-time-ish shared-secret check. + + Python's `==` on str short-circuits, so this leaks a timing signal in + principle. It is not worth `hmac.compare_digest` gymnastics for a token + that only travels over a private Docker network — but it IS worth + rejecting with a bare 401 and no detail, so a prober learns nothing about + whether the header name was even right. + """ + if x_face_ml_token != config.TOKEN: + raise HTTPException(status_code=401, detail="Unauthorized") + + +@app.get("/health", response_model=HealthResponse) +def health() -> HealthResponse: + """Unauthenticated on purpose — the compose healthcheck calls it. + + Reports only liveness. It deliberately does not confirm the models + loaded, because lifespan raises on failure and the container never + reaches a serving state at all. + """ + return HealthResponse(status="ok") + + +@app.get("/info", response_model=InfoResponse, dependencies=[Depends(require_token)]) +def info() -> InfoResponse: + assert _pipeline is not None + return InfoResponse( + detector=config.DETECTOR_NAME, + embedder=config.EMBEDDER_NAME, + model_version=config.MODEL_VERSION, + dim=_pipeline.dim, + ) + + +@app.post("/faces", response_model=FacesResponse, dependencies=[Depends(require_token)]) +async def faces(image: UploadFile = File(...)) -> FacesResponse: + assert _pipeline is not None + + data = await image.read() + if not data: + raise HTTPException(status_code=400, detail="Empty upload") + if len(data) > config.MAX_IMAGE_BYTES: + raise HTTPException(status_code=413, detail="Image too large") + + try: + detected = _pipeline.process(data) + except ValueError as exc: + # Undecodable input is the caller's problem, not ours — 400 so the + # backend marks the photo 'failed' instead of retrying it forever. + raise HTTPException(status_code=400, detail=str(exc)) from exc + + return FacesResponse(model_version=config.MODEL_VERSION, faces=detected) + + +@app.exception_handler(Exception) +async def unhandled(request, exc: Exception) -> JSONResponse: + """Never leak a traceback to the caller. + + The backend treats 5xx as "sidecar unhealthy" and puts the photo back to + pending with backoff, which is the right behaviour for a genuine internal + fault — so the useful detail belongs in our log, not in the response. + """ + logger.exception("Unhandled error: %s", exc) + return JSONResponse(status_code=500, content={"detail": "Internal error"}) diff --git a/ml/app/pipeline.py b/ml/app/pipeline.py new file mode 100644 index 00000000..de9f4791 --- /dev/null +++ b/ml/app/pipeline.py @@ -0,0 +1,313 @@ +""" +Detection → alignment → embedding (#1074). + +The alignment step in the middle is the part that decides whether this +feature works. FaceNet was trained on similarity-aligned crops; handing it a +raw bbox crop costs far more accuracy than any model swap would recover. So +YuNet's five landmarks are used to warp every face onto the same canonical +template before it ever reaches the embedder. + +IMPORTANT — the alignment and normalization below must stay identical to +whatever the Phase 0 spike measured its cosine threshold on. A tuned +threshold does not transfer across alignment changes; if either is touched, +`MODEL_VERSION` bumps and the backend re-derives. +""" + +import threading + +import cv2 +import numpy as np +import onnxruntime as ort + +from . import config + + +# Canonical 5-point template (ArcFace's, the de-facto standard), expressed for +# a 112x112 crop and scaled to whatever the embedder actually wants. Points are +# in IMAGE coordinates, left to right: +# 0 subject's right eye (appears image-left) +# 1 subject's left eye +# 2 nose tip +# 3 subject's right mouth corner +# 4 subject's left mouth corner +# YuNet emits its landmarks in exactly this order, so the mapping is index-wise +# with no reshuffling — `test_pipeline.py` pins that assumption. +_TEMPLATE_112 = np.array( + [ + [38.2946, 51.6963], + [73.5318, 51.5014], + [56.0252, 71.7366], + [41.5493, 92.3655], + [70.7299, 92.2041], + ], + dtype=np.float32, +) + + +class FacePipeline: + """Loads both models once and serves them under a lock. + + cv2.FaceDetectorYN carries mutable input-size state across setInputSize/ + detect, so it is NOT safe to call from two threads. FastAPI runs sync + endpoints in a threadpool, so every inference path is serialized here. + That is not a throughput loss worth fixing: the backend's face queue + defaults to concurrency 1, and a single lock keeps RSS predictable, which + is the constraint that actually matters on a 2 GB VPS. + """ + + def __init__(self) -> None: + detector_path = f"{config.MODEL_DIR}/{config.DETECTOR_FILENAME}" + embedder_path = f"{config.MODEL_DIR}/{config.EMBEDDER_FILENAME}" + + self._lock = threading.Lock() + + self._detector = cv2.FaceDetectorYN.create( + model=detector_path, + config="", + input_size=(320, 320), # replaced per-image via setInputSize + score_threshold=config.DET_SCORE_THRESHOLD, + nms_threshold=config.NMS_THRESHOLD, + top_k=config.TOP_K, + ) + + so = ort.SessionOptions() + so.intra_op_num_threads = config.ORT_THREADS + so.inter_op_num_threads = config.ORT_THREADS + self._embedder = ort.InferenceSession( + embedder_path, sess_options=so, providers=["CPUExecutionProvider"] + ) + + # Read the embedder's geometry off the model rather than hardcoding + # 160x160 NHWC. FACE_MODEL is documented as swappable (an operator who + # has cleared the InsightFace licence may point this at buffalo_l, + # which is 112x112 NCHW), and guessing wrong produces a confident + # garbage embedding rather than an error. + inp = self._embedder.get_inputs()[0] + shape = inp.shape + if len(shape) != 4: + raise RuntimeError(f"Embedder input must be 4-D, got {shape}") + self._input_name = inp.name + # NCHW iff the channel axis is second. + self._nchw = shape[1] == 3 + self._crop_size = int(shape[2] if self._nchw else shape[1]) + + out = self._embedder.get_outputs()[0] + self._dim = int(out.shape[-1]) + + self._template = _TEMPLATE_112 * (self._crop_size / 112.0) + + # -- introspection ---------------------------------------------------- + + @property + def dim(self) -> int: + return self._dim + + # -- inference -------------------------------------------------------- + + def process(self, image_bytes: bytes) -> list[dict]: + """Decode, detect, align, embed. Returns one dict per face.""" + buf = np.frombuffer(image_bytes, dtype=np.uint8) + img = cv2.imdecode(buf, cv2.IMREAD_COLOR) + if img is None: + raise ValueError("Image could not be decoded") + + # Downscale for detection, then map coordinates back. Everything the + # caller sees is in ORIGINAL image pixels — the backend stores bboxes + # to crop avatars from the same rendition later, so a scaled + # coordinate would silently offset every cover face. + h, w = img.shape[:2] + long_edge = max(h, w) + if long_edge > config.INPUT_LONG_EDGE: + scale = config.INPUT_LONG_EDGE / long_edge + det_img = cv2.resize( + img, (round(w * scale), round(h * scale)), interpolation=cv2.INTER_AREA + ) + else: + scale = 1.0 + det_img = img + + with self._lock: + dh, dw = det_img.shape[:2] + self._detector.setInputSize((dw, dh)) + _, raw = self._detector.detect(det_img) + + if raw is None: + return [] + + faces = [] + for row in raw[: config.MAX_FACES]: + faces.append(self._one_face(img, row, scale)) + return faces + + def _one_face(self, img: np.ndarray, row: np.ndarray, scale: float) -> dict: + """Build the response entry for a single YuNet detection. + + `row` is YuNet's 15-wide output: x, y, w, h, then five (x, y) + landmark pairs, then the confidence score. All in detection-image + coordinates, hence the division by `scale`. + """ + inv = 1.0 / scale + x, y, bw, bh = (float(v) * inv for v in row[0:4]) + landmarks = (row[4:14].reshape(5, 2).astype(np.float32)) * inv + score = float(row[14]) + + aligned = self._align(img, landmarks) + embedding = self._embed(aligned) + yaw, pitch = _pose_from_landmarks(landmarks) + + return { + "bbox": [x, y, bw, bh], + "score": score, + "landmarks": landmarks.tolist(), + "yaw": yaw, + "pitch": pitch, + "blur": _blur_score(aligned), + "embedding": embedding.tolist(), + } + + def _align(self, img: np.ndarray, landmarks: np.ndarray) -> np.ndarray: + """Similarity-warp the face onto the canonical template.""" + matrix = _umeyama(landmarks, self._template) + if matrix is None: + # Degenerate landmarks (all coincident/collinear). Fall back to a + # plain centre crop so the face still gets an embedding rather + # than vanishing from the gallery. + matrix = _fallback_transform(landmarks, self._crop_size) + return cv2.warpAffine( + img, + matrix, + (self._crop_size, self._crop_size), + flags=cv2.INTER_LINEAR, + borderValue=0, + ) + + def _embed(self, aligned: np.ndarray) -> np.ndarray: + # BGR (OpenCV) → RGB (what FaceNet was trained on). Getting this + # backwards does not error, it just quietly degrades every embedding. + rgb = cv2.cvtColor(aligned, cv2.COLOR_BGR2RGB).astype(np.float32) + + # deepface's "Facenet" normalization: per-image standardization. This + # is what the published FaceNet-512 benchmark numbers were produced + # with, so it is what the threshold in the backend assumes. + mean, std = rgb.mean(), rgb.std() + rgb = (rgb - mean) / max(float(std), 1e-6) + + batch = rgb[None, ...] + if self._nchw: + batch = batch.transpose(0, 3, 1, 2) + + vec = self._embedder.run(None, {self._input_name: batch})[0][0] + + # L2-normalize so the backend's cosine similarity is a plain dot + # product and centroid means stay on the unit sphere. + norm = float(np.linalg.norm(vec)) + return (vec / norm) if norm > 0 else vec + + +def _umeyama(src: np.ndarray, dst: np.ndarray) -> np.ndarray | None: + """Least-squares similarity transform (Umeyama 1991) over ALL five points. + + Deliberately NOT cv2.estimateAffinePartial2D. That function's estimators + are RANSAC (its default) and LMEDS, both of which exist to *reject + outliers* among many noisy correspondences. Given exactly five points and + no outliers they fit a three-point subset perfectly and let the rest + drift: measured on a real off-frontal portrait, both pinned the eyes and + nose to 0.11px and left the mouth corners 11.8px out on a 160px crop. + Umeyama distributes the residual instead (max 6.5px, rms 5.1 vs 7.4) and + is what insightface's norm_crop and skimage's SimilarityTransform use. + + Also fully deterministic — no random consensus sampling — which matters + beyond accuracy: the same photo must embed identically on every re-scan, + or clusters churn between runs for no reason. + + Returns a 2x3 affine matrix, or None if the points are degenerate. + """ + src = np.asarray(src, dtype=np.float64) + dst = np.asarray(dst, dtype=np.float64) + + src_mean, dst_mean = src.mean(axis=0), dst.mean(axis=0) + src_c, dst_c = src - src_mean, dst - dst_mean + + variance = float((src_c**2).sum() / len(src)) + if variance < 1e-9: + return None # all points coincident + + cov = dst_c.T @ src_c / len(src) + u, s, vt = np.linalg.svd(cov) + + # Guard against the SVD handing back a reflection instead of a rotation — + # a mirrored face would embed as a different person. + d = np.array([1.0, 1.0]) + if np.linalg.det(u @ vt) < 0: + d[-1] = -1.0 + + rotation = u @ np.diag(d) @ vt + scale = float((s * d).sum() / variance) + translation = dst_mean - scale * (rotation @ src_mean) + + return np.hstack([scale * rotation, translation.reshape(2, 1)]).astype(np.float32) + + +def _fallback_transform(landmarks: np.ndarray, size: int) -> np.ndarray: + """Centre the landmark centroid in the crop at the template's scale. + + Only reached when the landmarks are degenerate enough that no similarity + transform exists. The resulting embedding will be poor, but the face + still appears in "this photo contains" rather than vanishing — and the + backend's quality floor will keep it from spawning its own person. + """ + centre = landmarks.mean(axis=0) + s = size / 112.0 + return np.array( + [[s, 0.0, size / 2.0 - s * centre[0]], [0.0, s, size / 2.0 - s * centre[1]]], + dtype=np.float32, + ) + + +def _blur_score(aligned: np.ndarray) -> float: + """Variance of the Laplacian — low means soft/out-of-focus. + + Computed on the ALIGNED crop, not the original frame, so the number is + comparable between a face that fills the frame and one in the background: + both arrive here at the same pixel size. The backend's quality floor + compares against it directly. + """ + grey = cv2.cvtColor(aligned, cv2.COLOR_BGR2GRAY) + return float(cv2.Laplacian(grey, cv2.CV_64F).var()) + + +def _pose_from_landmarks(landmarks: np.ndarray) -> tuple[float, float]: + """Rough head pose in degrees from the five landmarks. + + An approximation, deliberately: a real 3-D pose estimate needs a face + model and solvePnP, and the only consumers are a quality floor and the + "candid" auto-category rule, neither of which needs better than + "clearly turned away vs. not". Returned as degrees so the admin-facing + threshold reads in a familiar unit. + + yaw negative = turned toward the subject's right, positive = left + pitch negative = looking down, positive = looking up + """ + right_eye, left_eye, nose, right_mouth, left_mouth = landmarks + + eye_centre = (right_eye + left_eye) / 2.0 + mouth_centre = (right_mouth + left_mouth) / 2.0 + eye_span = float(np.linalg.norm(left_eye - right_eye)) + if eye_span < 1e-6: + return 0.0, 0.0 + + # Yaw: on a frontal face the nose sits midway between the eyes. As the + # head turns, it slides toward the nearer eye. Offset is normalized by + # eye span so it is scale-free, then mapped through arcsin. + yaw_ratio = float((nose[0] - eye_centre[0]) / (eye_span / 2.0)) + yaw = float(np.degrees(np.arcsin(np.clip(yaw_ratio, -1.0, 1.0)))) + + # Pitch: the nose sits ~40% of the way down the eye→mouth axis on a + # frontal face. Higher means the head is tilted back, lower means down. + vertical = float(mouth_centre[1] - eye_centre[1]) + if abs(vertical) < 1e-6: + return yaw, 0.0 + nose_ratio = float((nose[1] - eye_centre[1]) / vertical) + pitch = float(np.degrees(np.arcsin(np.clip((0.40 - nose_ratio) * 2.0, -1.0, 1.0)))) + + return round(yaw, 2), round(pitch, 2) diff --git a/ml/app/schemas.py b/ml/app/schemas.py new file mode 100644 index 00000000..49f7254a --- /dev/null +++ b/ml/app/schemas.py @@ -0,0 +1,41 @@ +""" +Response models (#1074). + +These are the wire contract the backend's `faceClient.js` codes against. +Changing a field name here is a breaking change for a deployment mid-upgrade, +where an old sidecar and a new backend run side by side for a few seconds. +""" + +from pydantic import BaseModel, Field + + +class Face(BaseModel): + # [x, y, w, h] in ORIGINAL image pixels — the backend crops cover-face + # avatars from the same rendition, so these must not be detection-scaled. + bbox: list[float] = Field(min_length=4, max_length=4) + score: float + # Five (x, y) pairs: subject's right eye, left eye, nose tip, right mouth + # corner, left mouth corner. + landmarks: list[list[float]] + yaw: float + pitch: float + # Variance of the Laplacian on the aligned crop. Higher = sharper. + blur: float + # L2-normalized, `dim` floats (512 for FaceNet-512). + embedding: list[float] + + +class FacesResponse(BaseModel): + model_version: str + faces: list[Face] + + +class InfoResponse(BaseModel): + detector: str + embedder: str + model_version: str + dim: int + + +class HealthResponse(BaseModel): + status: str diff --git a/ml/requirements.txt b/ml/requirements.txt new file mode 100644 index 00000000..2d07d403 --- /dev/null +++ b/ml/requirements.txt @@ -0,0 +1,24 @@ +# picpeak-ml runtime dependencies (#1074). +# +# Pinned exactly. This image bakes in model weights and is meant to produce +# byte-identical embeddings across rebuilds — a floating dependency that +# changes how an image is decoded or resized would silently shift the +# embedding space and invalidate every stored cluster. +# +# opencv-python-HEADLESS, not opencv-python: the GUI build pulls in X11/GTK +# for highgui, which this service never calls and which is pure attack +# surface in a container. Pinned to the 4.x line — cv2.FaceDetectorYN is +# what loads YuNet, and 5.x reworks parts of that API. +opencv-python-headless==4.14.0.94 + +# CPU execution provider only. Wheels exist for manylinux x86_64 AND +# aarch64, so both legs of the multi-arch build install a prebuilt wheel +# and neither compiles from source. +onnxruntime==1.29.0 + +numpy==2.5.2 +fastapi==0.141.1 +uvicorn[standard]==0.52.3 +# Required by FastAPI to parse multipart/form-data — the only way images +# reach this service. +python-multipart==0.0.32 diff --git a/ml/tests/test_api.py b/ml/tests/test_api.py new file mode 100644 index 00000000..80bc0fdc --- /dev/null +++ b/ml/tests/test_api.py @@ -0,0 +1,136 @@ +""" +API contract tests (#1074). + +The pipeline is stubbed out — these cover the auth boundary and the request +guards, which is where a mistake is a security problem rather than an +accuracy problem. Model behaviour is the spike's job, not a unit test's. +""" + +import pytest +from fastapi.testclient import TestClient + +TOKEN = "test-token-not-a-secret" + + +class StubPipeline: + dim = 512 + + def process(self, image_bytes: bytes): + if image_bytes == b"undecodable": + raise ValueError("Image could not be decoded") + return [ + { + "bbox": [10.0, 20.0, 30.0, 40.0], + "score": 0.99, + "landmarks": [[1.0, 2.0]] * 5, + "yaw": 0.0, + "pitch": 0.0, + "blur": 123.4, + "embedding": [0.1] * 512, + } + ] + + +@pytest.fixture +def client(monkeypatch): + from app import config, main + + monkeypatch.setattr(config, "TOKEN", TOKEN) + monkeypatch.setattr(main.config, "TOKEN", TOKEN) + monkeypatch.setattr(main, "FacePipeline", StubPipeline) + + with TestClient(main.app) as c: + yield c + + +def _image_file(data: bytes = b"fake-jpeg-bytes"): + return {"image": ("photo.jpg", data, "image/jpeg")} + + +class TestAuth: + def test_health_needs_no_token(self, client): + # The compose healthcheck calls this without a secret. + r = client.get("/health") + assert r.status_code == 200 + assert r.json() == {"status": "ok"} + + def test_info_without_token_is_401(self, client): + assert client.get("/info").status_code == 401 + + def test_info_with_wrong_token_is_401(self, client): + r = client.get("/info", headers={"X-Face-ML-Token": "wrong"}) + assert r.status_code == 401 + + def test_faces_without_token_is_401(self, client): + r = client.post("/faces", files=_image_file()) + assert r.status_code == 401 + + def test_401_body_leaks_nothing(self, client): + # A prober should not learn whether the header name was even right. + r = client.post("/faces", files=_image_file()) + assert r.json() == {"detail": "Unauthorized"} + + def test_startup_refuses_an_empty_token(self, monkeypatch): + from app import config, main + + monkeypatch.setattr(config, "TOKEN", "") + monkeypatch.setattr(main.config, "TOKEN", "") + monkeypatch.setattr(main, "FacePipeline", StubPipeline) + + # Running open is the one failure mode this service must not have. + with pytest.raises(RuntimeError, match="FACE_ML_TOKEN"): + with TestClient(main.app): + pass + + +class TestInfo: + def test_reports_the_model_identity_the_backend_stores(self, client): + from app import config + + r = client.get("/info", headers={"X-Face-ML-Token": TOKEN}) + assert r.status_code == 200 + body = r.json() + assert body["model_version"] == config.MODEL_VERSION + assert body["dim"] == 512 + + +class TestFaces: + def test_returns_faces_with_the_model_version(self, client): + from app import config + + r = client.post( + "/faces", files=_image_file(), headers={"X-Face-ML-Token": TOKEN} + ) + assert r.status_code == 200 + body = r.json() + assert body["model_version"] == config.MODEL_VERSION + assert len(body["faces"]) == 1 + assert len(body["faces"][0]["embedding"]) == 512 + assert body["faces"][0]["bbox"] == [10.0, 20.0, 30.0, 40.0] + + def test_empty_upload_is_400(self, client): + r = client.post( + "/faces", files=_image_file(b""), headers={"X-Face-ML-Token": TOKEN} + ) + assert r.status_code == 400 + + def test_undecodable_image_is_400_not_500(self, client): + # 4xx matters: the backend must mark the photo failed rather than + # retry it forever, which is what it does for 5xx. + r = client.post( + "/faces", + files=_image_file(b"undecodable"), + headers={"X-Face-ML-Token": TOKEN}, + ) + assert r.status_code == 400 + + def test_oversize_upload_is_413(self, client, monkeypatch): + from app import main + + monkeypatch.setattr(main.config, "MAX_IMAGE_BYTES", 10) + r = client.post( + "/faces", + files=_image_file(b"x" * 100), + headers={"X-Face-ML-Token": TOKEN}, + ) + assert r.status_code == 413 diff --git a/ml/tests/test_pipeline.py b/ml/tests/test_pipeline.py new file mode 100644 index 00000000..915f3352 --- /dev/null +++ b/ml/tests/test_pipeline.py @@ -0,0 +1,156 @@ +""" +Unit tests for the parts of the pipeline that don't need model weights. + +The pose and blur helpers are pure functions over landmark geometry, and the +landmark ORDER assumption is the one thing in this service that fails +silently if it's wrong — a shuffled template still produces 512 confident +floats, just from a face warped inside out. So it gets pinned here. +""" + +import numpy as np +import pytest + +from app.pipeline import ( + _TEMPLATE_112, + _blur_score, + _pose_from_landmarks, + _umeyama, +) + + +def _frontal_landmarks() -> np.ndarray: + """A synthetic, perfectly frontal face in YuNet's landmark order. + + Order: subject's right eye, left eye, nose, right mouth, left mouth. + The subject's right eye appears on the IMAGE-left, so it carries the + smaller x — the same convention `_TEMPLATE_112` encodes. + """ + return np.array( + [ + [40.0, 50.0], # right eye (image-left) + [80.0, 50.0], # left eye + [60.0, 70.0], # nose, centred between the eyes + [45.0, 92.0], # right mouth + [75.0, 92.0], # left mouth + ], + dtype=np.float32, + ) + + +class TestTemplate: + def test_landmark_order_is_left_to_right_for_paired_features(self): + # Eyes: index 0 must sit left of index 1. Mouth corners: 3 left of 4. + assert _TEMPLATE_112[0][0] < _TEMPLATE_112[1][0] + assert _TEMPLATE_112[3][0] < _TEMPLATE_112[4][0] + + def test_nose_sits_between_the_eyes_horizontally(self): + assert _TEMPLATE_112[0][0] < _TEMPLATE_112[2][0] < _TEMPLATE_112[1][0] + + def test_features_are_vertically_ordered_eyes_nose_mouth(self): + eye_y = (_TEMPLATE_112[0][1] + _TEMPLATE_112[1][1]) / 2 + mouth_y = (_TEMPLATE_112[3][1] + _TEMPLATE_112[4][1]) / 2 + assert eye_y < _TEMPLATE_112[2][1] < mouth_y + + +class TestUmeyama: + """The alignment estimator. Every test here is a regression guard. + + An estimator that fits three of the five landmarks perfectly and lets the + mouth drift still produces a face-shaped crop and 512 confident floats — + it just degrades every embedding. That is why this is pinned numerically + rather than eyeballed. + """ + + def test_recovers_a_known_similarity_transform_exactly(self): + src = _frontal_landmarks() + angle = np.radians(20.0) + rot = np.array( + [[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]] + ) + dst = (2.5 * (src @ rot.T)) + np.array([17.0, -9.0]) + + m = _umeyama(src, dst.astype(np.float32)) + projected = (src @ m[:, :2].T) + m[:, 2] + assert np.allclose(projected, dst, atol=1e-3) + + def test_distributes_residual_across_all_five_points(self): + # A face whose eye-to-mouth proportion differs from the template — + # no similarity transform can satisfy all five, so the question is + # how the error is spread. An outlier-rejecting estimator (RANSAC, + # LMEDS) parks it all on the mouth; least squares shares it out. + src = _frontal_landmarks() + src[3][1] = 110.0 # mouth further from the eyes than the template + src[4][1] = 110.0 + + m = _umeyama(src, _TEMPLATE_112) + residual = np.linalg.norm((src @ m[:, :2].T) + m[:, 2] - _TEMPLATE_112, axis=1) + + assert residual.max() > 0, "expected an imperfect fit for this input" + # No single point may absorb the bulk of the error. + assert residual.max() < 3.0 * residual.mean() + + def test_never_returns_a_reflection(self): + # A mirrored warp yields a confident embedding of a face that does + # not exist, which would cluster as a separate person. + m = _umeyama(_frontal_landmarks(), _TEMPLATE_112) + assert np.linalg.det(m[:, :2]) > 0 + + def test_is_deterministic(self): + # Re-scans must not churn clusters. + src = _frontal_landmarks() + first = _umeyama(src, _TEMPLATE_112) + for _ in range(5): + assert np.array_equal(_umeyama(src, _TEMPLATE_112), first) + + def test_coincident_points_return_none_rather_than_dividing_by_zero(self): + assert _umeyama(np.zeros((5, 2), dtype=np.float32), _TEMPLATE_112) is None + + +class TestPose: + def test_frontal_face_is_near_zero_yaw(self): + yaw, _ = _pose_from_landmarks(_frontal_landmarks()) + assert abs(yaw) < 1.0 + + def test_nose_toward_subject_left_eye_gives_positive_yaw(self): + lm = _frontal_landmarks() + lm[2][0] = 75.0 # nose slides toward the image-right (subject's left) + yaw, _ = _pose_from_landmarks(lm) + assert yaw > 10.0 + + def test_nose_toward_subject_right_eye_gives_negative_yaw(self): + lm = _frontal_landmarks() + lm[2][0] = 45.0 + yaw, _ = _pose_from_landmarks(lm) + assert yaw < -10.0 + + def test_yaw_is_scale_invariant(self): + lm = _frontal_landmarks() + lm[2][0] = 72.0 + small, _ = _pose_from_landmarks(lm) + large, _ = _pose_from_landmarks(lm * 4.0) + assert small == pytest.approx(large, abs=0.01) + + def test_nose_low_on_the_eye_mouth_axis_reads_as_looking_down(self): + lm = _frontal_landmarks() + lm[2][1] = 85.0 # nose drops toward the mouth + _, pitch = _pose_from_landmarks(lm) + assert pitch < 0 + + def test_degenerate_landmarks_do_not_raise(self): + flat = np.zeros((5, 2), dtype=np.float32) + assert _pose_from_landmarks(flat) == (0.0, 0.0) + + +class TestBlur: + def test_flat_image_scores_near_zero(self): + flat = np.full((160, 160, 3), 128, dtype=np.uint8) + assert _blur_score(flat) < 1.0 + + def test_sharp_edges_score_higher_than_a_blurred_copy(self): + import cv2 + + sharp = np.zeros((160, 160, 3), dtype=np.uint8) + sharp[:, ::8] = 255 # high-frequency vertical stripes + blurred = cv2.GaussianBlur(sharp, (15, 15), 0) + + assert _blur_score(sharp) > _blur_score(blurred) diff --git a/ml/tools/benchmark_threshold.py b/ml/tools/benchmark_threshold.py new file mode 100644 index 00000000..bde3c448 --- /dev/null +++ b/ml/tools/benchmark_threshold.py @@ -0,0 +1,113 @@ +""" +Phase 0 spike (#1074): does our pipeline separate different people? + +Runs LFW's standard 1000-pair test protocol (500 same, 500 different) through +the PRODUCTION FacePipeline — YuNet detection, our Umeyama alignment, our +per-image standardization, FaceNet-512 ONNX — and reports the cosine +distribution, the best threshold, and the error rates that matter for +CLUSTERING specifically. + +Clustering is not verification. For verification a false accept and a false +reject cost the same. For clustering they do not: a false merge puts a +stranger into someone's "download my photos", while a false split just makes +a duplicate row in the strip that the photographer can merge away. So the +operating point is chosen to hold false merges low, not to maximise accuracy. +""" +import os +import sys +import numpy as np +import cv2 + +# Run from the ml/ directory with FACE_MODEL_DIR pointing at a directory +# holding face_detection_yunet_2023mar.onnx and facenet512.onnx: +# +# pip install -r requirements.txt scikit-learn +# FACE_MODEL_DIR=/path/to/models python tools/benchmark_threshold.py +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +os.environ.setdefault('FACE_MODEL_DIR', '/models') + +from app.pipeline import FacePipeline # noqa: E402 +from sklearn.datasets import fetch_lfw_pairs # noqa: E402 + +print('Loading LFW test pairs…') +data = fetch_lfw_pairs(subset='test', color=True, resize=1.0, + slice_=(slice(0, 250), slice(0, 250)), funneled=True) +pairs, labels = data.pairs, data.target + +print('Loading pipeline…') +pipe = FacePipeline() + +def embed(arr): + """sklearn hands back float32 RGB normalized to [0, 1] — NOT 0..255. + Casting straight to uint8 produces a black frame and zero detections.""" + rgb8 = np.clip(arr * 255.0, 0, 255).astype(np.uint8) + bgr = cv2.cvtColor(rgb8, cv2.COLOR_RGB2BGR) + ok, buf = cv2.imencode('.jpg', bgr, [cv2.IMWRITE_JPEG_QUALITY, 95]) + if not ok: + return None + faces = pipe.process(buf.tobytes()) + if not faces: + return None + # Largest detection — LFW is one centred subject per frame. + best = max(faces, key=lambda f: f['bbox'][2] * f['bbox'][3]) + return np.asarray(best['embedding'], dtype=np.float32) + +sims, kept, missed = [], [], 0 +for i, (a, b) in enumerate(pairs): + ea, eb = embed(a), embed(b) + if ea is None or eb is None: + missed += 1 + continue + sims.append(float(ea @ eb)) + kept.append(int(labels[i])) + if (i + 1) % 100 == 0: + print(f' {i+1}/{len(pairs)}…', flush=True) + +sims = np.array(sims) +kept = np.array(kept) +same, diff = sims[kept == 1], sims[kept == 0] + +print() +print('=' * 66) +print(f'Pairs evaluated : {len(sims)} of {len(pairs)} ' + f'({missed} skipped — no face detected in one or both)') +print(f'Detection rate : {1 - missed/len(pairs):.1%}') +print() +print(f'SAME person cosine: mean {same.mean():.4f} sd {same.std():.4f} ' + f'p5 {np.percentile(same,5):.4f} min {same.min():.4f}') +print(f'DIFF person cosine: mean {diff.mean():.4f} sd {diff.std():.4f} ' + f'p95 {np.percentile(diff,95):.4f} max {diff.max():.4f}') +print(f'Separation (mean gap): {same.mean() - diff.mean():.4f}') + +# Sweep thresholds. +grid = np.linspace(0.0, 1.0, 1001) +acc = [( (same >= t).sum() + (diff < t).sum() ) / len(sims) for t in grid] +best_i = int(np.argmax(acc)) +best_t, best_acc = grid[best_i], acc[best_i] + +print() +print(f'Best accuracy : {best_acc:.2%} at threshold {best_t:.3f}') + +# The clustering-relevant operating points: pick the threshold where the +# false-MERGE rate (different people scored as the same) is capped. +print() +print('Operating points (false merge = different people judged the same):') +print(f' {"thresh":>7} {"false merge":>11} {"false split":>11} {"accuracy":>8}') +for target in (0.10, 0.05, 0.02, 0.01): + t = float(np.quantile(diff, 1 - target)) + fm = (diff >= t).mean() + fs = (same < t).mean() + a = ((same >= t).sum() + (diff < t).sum()) / len(sims) + print(f' {t:7.3f} {fm:10.1%} {fs:10.1%} {a:7.1%} (target {target:.0%})') + +print() +print(f'Currently seeded default: 0.620 → ' + f'false merge {(diff >= 0.62).mean():.1%}, ' + f'false split {(same < 0.62).mean():.1%}, ' + f'accuracy {(((same >= 0.62).sum() + (diff < 0.62).sum()) / len(sims)):.1%}') +print('=' * 66) + +# Last run (2026-08-18), 1000/1000 pairs, 100% detection: +# same 0.6958 +/- 0.1415 | diff 0.0849 +/- 0.1674 | separation 0.6109 +# peak accuracy 96.60% @ 0.405 +# shipped default 0.50 -> 1.0% false merge, 8.2% false split, 95.4% accuracy diff --git a/ml/tools/convert_facenet.py b/ml/tools/convert_facenet.py new file mode 100644 index 00000000..f600ee56 --- /dev/null +++ b/ml/tools/convert_facenet.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +""" +Produce `facenet512.onnx` from deepface's published Keras weights (#1074). + +Run this ONCE, by hand. Publish the resulting file as a release asset and +pass its URL + SHA-256 to the image build. It is deliberately not part of the +Docker build: + + 1. TensorFlow is ~600MB of build dependency for a file that never ships in + the final image. + 2. The result is architecture-independent, so converting once beats + converting on both legs of every multi-arch build. + +NOT byte-reproducible. Two runs on the same machine with the same pinned +versions produce functionally identical graphs — same 336 nodes, same 271 +initializers, weights matching to 0.000e+00 — but a handful of initializer +names differ (tf2onnx's traced-op naming is not deterministic), so the file +bytes and therefore the SHA-256 differ. Measured, not assumed. + +The consequence for anyone re-running this: **your checksum will not match +the published one, and that is expected — it is not evidence of tampering.** +The SHA-256 in the image build pins one specific published artifact so that +URL cannot start serving different bytes. To validate a fresh conversion, +rely on the parity check below (which compares against the Keras original), +not on reproducing a hash. + +Why we may redistribute at all: deepface ships FaceNet-512 under MIT. That +was the deciding factor over the more accurate InsightFace weights, which are +non-commercial only — see ml/LICENSES.md and #1074 §1. + +Usage +----- + python3.11 -m venv .venv && . .venv/bin/activate # 3.11: TF has no 3.12+ wheels + pip install -r tools/requirements-convert.txt + + curl -fsSL -o facenet512_weights.h5 \\ + https://github.com/serengil/deepface_models/releases/download/v1.0/facenet512_weights.h5 + echo "3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f facenet512_weights.h5" \\ + | sha256sum -c - + + python tools/convert_facenet.py facenet512_weights.h5 facenet512.onnx + +The script verifies the converted graph against the Keras original before it +writes anything permanent, then prints the SHA-256 to publish alongside it. +""" + +import argparse +import hashlib +import sys +from pathlib import Path + +# Input geometry of deepface's FaceNet-512. pipeline.py reads this off the +# model at runtime rather than assuming it, but this is what it will find. +INPUT_SHAPE = (None, 160, 160, 3) +EMBEDDING_DIM = 512 + +# A converted graph that is subtly wrong still returns 512 plausible floats, +# so parity is checked rather than assumed. Tolerance is float32 noise: the +# observed worst case over random inputs was 2.1e-06 absolute, cosine +# 1.0000000000. +PARITY_SAMPLES = 3 +MIN_COSINE = 0.99999 + + +def _sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as fh: + for chunk in iter(lambda: fh.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("weights", type=Path, help="facenet512_weights.h5") + parser.add_argument("output", type=Path, help="destination .onnx") + parser.add_argument( + "--opset", + type=int, + default=17, + help="ONNX opset (default 17 — supported by onnxruntime 1.29)", + ) + parser.add_argument( + "--skip-verify", + action="store_true", + help="skip the Keras/ONNX parity check (not recommended)", + ) + args = parser.parse_args() + + if not args.weights.is_file(): + print(f"error: {args.weights} not found", file=sys.stderr) + return 1 + + # Imported late so `--help` works without a TensorFlow install. + import numpy as np + import tensorflow as tf + import tf2onnx + from deepface.models.facial_recognition.Facenet import InceptionResNetV1 + + # Build the architecture and load the checksummed file from disk rather + # than going through deepface's own loader — that one downloads the + # weights itself, which would defeat the point of pinning them. + print("Building InceptionResNetV1(dimension=512)…") + model = InceptionResNetV1(dimension=EMBEDDING_DIM) + model.load_weights(str(args.weights)) + print(f" {model.count_params():,} parameters") + + print(f"Converting to ONNX (opset {args.opset})…") + spec = (tf.TensorSpec(INPUT_SHAPE, tf.float32, name="input"),) + tf2onnx.convert.from_keras( + model, input_signature=spec, opset=args.opset, output_path=str(args.output) + ) + + if not args.skip_verify: + import onnxruntime as ort + + print("Verifying ONNX output matches Keras…") + sess = ort.InferenceSession( + str(args.output), providers=["CPUExecutionProvider"] + ) + name = sess.get_inputs()[0].name + rng = np.random.default_rng(0) + worst_cosine, worst_abs = 1.0, 0.0 + + for _ in range(PARITY_SAMPLES): + x = rng.standard_normal((1, *INPUT_SHAPE[1:])).astype("float32") + keras_out = model.predict(x, verbose=0)[0] + onnx_out = sess.run(None, {name: x})[0][0] + + worst_abs = max(worst_abs, float(np.abs(keras_out - onnx_out).max())) + cosine = float( + (keras_out / np.linalg.norm(keras_out)) + @ (onnx_out / np.linalg.norm(onnx_out)) + ) + worst_cosine = min(worst_cosine, cosine) + + print(f" worst abs diff {worst_abs:.3e}, worst cosine {worst_cosine:.10f}") + if worst_cosine < MIN_COSINE: + print( + f"error: parity check FAILED (cosine {worst_cosine} < {MIN_COSINE}). " + "The converted graph does not match the original — do not publish it.", + file=sys.stderr, + ) + args.output.unlink(missing_ok=True) + return 1 + + size_mb = args.output.stat().st_size / (1024 * 1024) + print() + print(f"Wrote {args.output} ({size_mb:.1f} MB)") + print(f"SHA-256: {_sha256(args.output)}") + print() + print("Publish it as a release asset, then set the repository variables") + print("FACENET_ONNX_URL and FACENET_ONNX_SHA256 (Settings → Variables).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ml/tools/requirements-convert.txt b/ml/tools/requirements-convert.txt new file mode 100644 index 00000000..10238a3f --- /dev/null +++ b/ml/tools/requirements-convert.txt @@ -0,0 +1,22 @@ +# Dependencies for tools/convert_facenet.py ONLY. These never enter the +# picpeak-ml image — the conversion runs once, by hand, and only its output +# (facenet512.onnx) is shipped. +# +# Pinned to the exact set that produced the published artifact, so anyone can +# reproduce it and get the same SHA-256 rather than a file that differs for +# reasons nobody can reconstruct later. See ml/README.md for the expected +# checksum. +# +# Requires Python 3.11 (TensorFlow has no 3.12+/3.14 wheels at these +# versions). The picpeak-ml image itself runs 3.12 — the two are unrelated, +# since nothing from this file ships. +tensorflow==2.21.0 +tf2onnx==1.17.0 +onnx==1.22.0 +deepface==0.0.100 + +# Transitive, pinned because the conversion output is checksummed: +# keras 3.15 builds the graph, protobuf serializes the ONNX. +keras==3.15.1 +protobuf==7.35.1 +numpy==2.4.6