* feat(ml): optional face-detection sidecar, opt-in and inert by default (#1074) First of four PRs for "People in this gallery". This one ships only the sidecar, its wiring and its CI — no schema, no backend code, no UI. Nothing in PicPeak calls it yet. picpeak-ml is a single FastAPI + onnxruntime container: three endpoints (/health, /info, /faces), no database, no volumes, no egress, no model download at runtime. Clustering, person identity and every privacy decision stay in the backend where the data already lives. Models are YuNet (detection) + FaceNet-512 (embedding), both MIT, both pinned by URL and SHA-256 and verified at build time. The licence analysis is in ml/LICENSES.md: the more accurate InsightFace weights are non-commercial-only and PicPeak's users are working photographers, so they are never baked into an image we publish. Two things worth review attention: - Alignment uses a least-squares similarity transform (Umeyama), NOT cv2.estimateAffinePartial2D. RANSAC and LMEDS exist to reject outliers among many correspondences; given five landmarks and no outliers they fit a three-point subset exactly and let the rest drift. Measured on a real off-frontal portrait: eyes and nose pinned to 0.11px, mouth corners 11.8px out on a 160px crop. Umeyama distributes it (max 6.5px, rms 5.1 vs 7.4). The failure mode is silent — a bad warp still yields 512 confident floats — so tests/test_pipeline.py pins it numerically. - FACENET_ONNX_URL has no default and the build fails loudly without it. deepface distributes FaceNet-512 as Keras .h5 only, so the ONNX is produced once by tools/convert_facenet.py and published as a release asset. Converting inside the build would drag TensorFlow through both architecture legs of every build to produce a byte-identical file. The CI jobs are gated on the FACENET_ONNX_URL repository variable and skip cleanly until it is set. Off by default, twice over: the sidecar is behind the `faces` compose profile, and the backend will gate on a `faces` feature flag that defaults to false. FACE_ML_URL defaults to http://picpeak-ml:8000 so the standard deployment needs no configuration — nothing dials that host while the flag is off, which is why a non-resolving default is harmless. Verified: 27 pytest tests green; YuNet loads and detects against a real portrait with its landmark order matching the alignment template index-for-index; both compose files validate and the faces profile is correctly excluded from a default `up`; workflow YAML parses and the job graph resolves. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(ml): pin the converter toolchain, verify parity, drop a false reproducibility claim (#1074) Ran the FaceNet-512 conversion for real and corrected what the previous commit assumed about it. The conversion works: 23,497,424 parameters, 89.6 MB ONNX, and the converted graph matches the Keras original to 2.086e-06 absolute / cosine 1.0000000000. That check is now part of the script rather than something I did once by hand — a subtly wrong graph still returns 512 plausible floats, so it refuses to leave the file on disk if parity fails. Also ran the full pipeline against both real models end to end. The embedding is L2-normalized to 1.000000, and the same face survives being re-rendered: half scale 0.973, double scale 0.984, JPEG q40 0.987, rotated 8 degrees 0.984, brightness +40 0.988. Scale invariance in particular is evidence the alignment warp is doing its job. Corrected claim: the conversion is NOT byte-reproducible. Two runs with the same pinned versions on the same machine gave different SHA-256s. The graphs are functionally identical — same 336 nodes, same 271 initializers, every weight matching to 0.000e+00 — but a few initializer names differ because tf2onnx's traced-op naming is not deterministic (Keras layer naming is deterministic; I checked). The previous commit message and README both claimed byte-identical output. They were wrong, and it matters: anyone re-running the conversion gets a different hash, and without this note that reads like tampering. The build-time SHA-256 pins one published artifact so its URL cannot start serving different bytes; validating a fresh conversion is the parity check's job. requirements-convert.txt now pins the exact set that produced the artifact, including transitive keras/protobuf/numpy, and documents that the converter needs Python 3.11 while the image runs 3.12. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): schema, queue, clustering and API for People in this gallery (#1074) Backend half of the feature. Migration 177, a face-detection queue, the clustering engine, the gallery and admin APIs, and the privacy wiring. No UI yet; nothing is reachable until the `faces` feature flag is on, which defaults to false. The flag is the gate, not FACE_ML_URL. That variable now has a working default (the compose service name), so its presence proves nothing about intent — if it were the gate, every install would poll a hostname that does not resolve. faceQueue re-checks the flag every tick, so turning it off stops the workers without a restart. Visibility scoping is the part worth reviewing closely. Face rows have no concept of photo visibility, but guests are restricted to photos.visibility='visible'. A raw count leaks how many hidden photos someone appears in, and an unscoped cover face renders a crop of a photo the guest may not open — with the best-scoring face being the likeliest pick, so it would happen often rather than rarely. facePeopleService recomputes both per request against the caller's own scope, and event_people.face_count_total is named to be conspicuous in a guest path. Six tests cover it, including the case where a person's photos are ALL hidden and they must vanish entirely. Face data is excluded from backups and .picpeak exports, per the decision in the thread: it is derived, so a restore re-scans rather than carrying biometrics between operators. Three separate mechanisms, because the engines cannot be filtered alike — EXCLUDED_TABLES for export, --exclude-table-data (not --exclude-table; the CREATE TABLE must survive or restore breaks on the first query) for Postgres, and DELETE + VACUUM on the temp copy for SQLite, which has no way to exclude a table from a whole-file .backup. The VACUUM is not cosmetic: without it the pages stay in the file and the claim is false on disk. Archiving now purges face data explicitly. photo_faces cascades off photos, but archive deletes neither the photo rows nor the event, so without this an archived gallery kept its biometrics indefinitely. Other decisions: clustering keeps names across a re-cluster by majority inheritance (without it, one button click silently discards every name the photographer typed); consolidation refuses to merge two people who were named differently; assignment never compares across model_version, since embeddings from two pipelines are not comparable; low-quality faces are stored but left unassigned so they show in "this photo contains" without spawning junk people. Migration is 177, not 174 — 174/175/176 landed on main while this branch was open. 29 tests green: 7 migration (idempotency, down(), cascade, and that installing it enqueues NOTHING), 11 clustering, 11 privacy/visibility. Lint clean; the pre-existing error counts in databaseBackup.js and server.js are unchanged. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): People strip, face filter and admin controls (#1074) Frontend half. Renders nothing anywhere unless the `faces` feature flag is on AND the photographer enabled detection for the gallery — the whole guest surface hangs off one boolean, event.people_enabled, which the server computes from the flag, the per-event toggle and the show-to-guests toggle together. Guest side: a People strip between the filter bar and the grid, circular crops from each person's cover face, an active-filter chip row, and a "Show all" bottom sheet. The face filter composes with category, search, media type and the liked/saved/rated filters in the same useMemo rather than replacing them, so "photos of Anna that I liked" works. Two people selected means AND by default — that is what picking a second face almost always asks for — with a toggle to OR that appears only once a second person is picked. Unnamed people show a photo count and never "Person 7". A number is honest about what the system knows; an invented name is not. There is a test asserting we don't do it. The strip renders nothing below two people, collapses to one line when dismissed (persisted per slug, so dismissing one gallery says nothing about the next), and appears mid-backfill with a progress line rather than blocking the gallery behind a spinner. Avatar crops are computed in ratios of the source dimensions so they survive whatever rendition the browser gets; without width/height they fall back to an uncropped thumbnail, since a wrongly-offset crop is worse than no crop. No new download endpoint: "download these N" rides the existing photoIds path, which already enforces access level and per-category permissions server-side. Adding a person_id selector would have been a second thing to authorize for no gain. Guest-facing copy never says "biometric" or "recognition" — those words describe our implementation, not the guest's experience. The sheet's footnote answers the first question every guest has (where does this go?) inline. The admin card, by contrast, is explicit: it states the controller obligation next to the toggle, and warns that scanning materializes the preview tier on galleries that never generated one, which is real CPU and disk an admin should know about before a 2,000-photo backfill. EN + DE translations. 140 frontend tests green (8 new), tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): measured match threshold, working build defaults, 89MB smaller image (#1074) Ran the Phase 0 spike that had been outstanding, published the model, and fixed what both turned up. THRESHOLD IS NOW MEASURED, NOT GUESSED. LFW's standard 1000-pair protocol run through this exact pipeline (YuNet -> Umeyama alignment -> FaceNet-512 ONNX), 100% detection on 2000 images: same person cosine 0.6958 +/- 0.1415 diff person cosine 0.0849 +/- 0.1674 separation 0.6109 peak accuracy 96.60% @ 0.405 So the pipeline separates people well — the thing I could not previously claim, since every earlier number was the same face re-rendered. Default moves 0.62 -> 0.50. The old value was a placeholder and a bad one: it gave 0% false merges but 22.4% false splits, i.e. roughly one in four same-person pairs failing to join, which fragments a gallery badly. 0.50 gives 1.0% false merge / 8.2% false split. Peak accuracy (0.405) is deliberately NOT chosen: for clustering the two errors do not cost the same. A false split is a duplicate row the photographer can merge away; a false merge puts a stranger into someone's "download my photos" — and until the Phase 2 merge/split UI ships, there is no way to undo one. So this sits on the conservative side of the optimum. The spike is committed as ml/tools/benchmark_threshold.py rather than thrown away, so "why 0.50?" has an answer in six months and a re-tune is one command. BUILD DEFAULTS. FACENET_ONNX_URL/_SHA256 now default to the published ml-models-v1 release asset, so `docker build ml/` and `docker compose --profile faces up` work with no arguments. Blanking either still fails loudly — a URL without a checksum is never acceptable, since the checksum is what makes the URL safe to trust. Found by running compose for real: it failed exactly as designed, which was correct behaviour and a bad out-of-box experience now that a canonical artifact exists. IMAGE SIZE. 389MB -> 300MB single-arch. `chown -R` after COPY rewrote every copied file into a fresh layer, duplicating the 90MB model for nothing; the user is now created before the copies and ownership set via COPY --chown. Also drops pip/setuptools from the runtime image. Measured RSS is 186MiB idle, and the container answers /faces end-to-end in well under the 80-150ms/photo the issue budgeted. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): threshold 0.50 -> 0.60 from real clustering, theme-aware People strip (#1074) Both fixes come from running the feature on an actual gallery — 61 photos, 5 real identities — rather than reasoning about it. THRESHOLD. The LFW pairwise sweep in the previous commit said 0.50, and it was wrong. On a real gallery at 0.50, three of six visible clusters were contaminated: two different people merged into one strip entry, which is the exact failure that puts a stranger into someone's "download my photos". Pairwise error rates do not predict cluster purity. Greedy assignment compounds — one wrong face drags the centroid toward the midpoint between two identities, making the next wrong face likelier. A 1% pairwise false-merge rate is not a 1% chance of a clean gallery, and no amount of staring at an ROC curve would have shown that. Sweep against ground truth (5 identities): 0.50 -> 6 clusters, 3 contaminated 0.56 -> 6 clusters, 0 contaminated 0.60 -> 5 clusters, 0 contaminated <- exactly right 0.64 -> 5 clusters, 0 contaminated, fewer faces assigned 0.60 recovers the right number of people with no contamination; higher only loses coverage. Migration 177 carries the full reasoning so the next person to touch this knows why the obvious pairwise answer is the wrong one. THEME. The People strip hardcoded `text-neutral-800` for named people. On a dark gallery — which the screenshot immediately showed — that renders a named person's label almost invisibly, while UNNAMED people stayed legible. Exactly backwards. Labels, headings, the collapsed summary, the scan line and the filter chip row now read the gallery's own theme tokens (--color-text / --color-muted-text / --color-accent / --color-surface-border) like the rest of the gallery surface. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): keep the mobile filter row inside the viewport (#1074) At 390px the photo count and Clear link were pushed against the right edge by ml-auto and clipped. Only apply it from the sm breakpoint up, where there is room; below that they flow after the chips. Found by screenshotting the real thing on an iPhone-sized viewport. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): complete Phase 1, add People management and auto-categories (#1074) Closes the two Phase 1 gaps, then builds Phase 2 and Phase 3. PHASE 1 GAPS. "Download these N" was specified, described as done in an earlier summary, and never actually built — I had verified the backend needed no new endpoint and let that stand as if the button existed. It now hands the filtered photo ids to the same path as a manual selection, so the server re-applies access level and per-category permissions on the way through. Photos in a downloads-disabled category are excluded client-side too, so the number on the button is the number the guest receives. Hidden entirely when downloads are off for the gallery. Lightbox person chips ("In this photo: Anna") are the second way into the face filter — a guest looking at a photo of themselves can act on it without scrolling back to the strip. Tapping one closes the lightbox and filters the grid behind it. PHASE 2. A People management modal over the endpoints that already existed and were already tested: rename inline, merge (multi-select, first pick is the target so the name a photographer typed survives), split via a face picker, hide, ignore. This matters more than it sounds — clustering deliberately errs toward splitting because a wrong merge puts a stranger into someone's download, and that trade only works if merging is easy. PHASE 3. Rule engine over face_count plus face-area ratio: 0 -> Details, 1 large -> Portraits, 2-5 -> Small groups, >5 -> Groups. The area ratio is what separates "a portrait of someone" from "someone is in this landscape". Three guarantees, all tested: it only ever fills an EMPTY category (enforced in the query AND re-checked in the UPDATE, so a photographer setting one mid-run still wins), everything it touches is marked auto_categorized so undo is exact, and it is a no-op unless separately enabled. Migration 178 adds the column — separate from 177, which has already run wherever this branch is deployed. Verified on the real gallery: 61 photos -> 48 portraits + 13 small groups, undo cleared exactly 61 and left the manual ones alone. Merge moved faces and removed the source. Both confirmed against the database, not just the UI. TWO BUGS THE BROWSER CAUGHT, both invisible to tsc: - The lightbox destructure never landed — my patch targeted a line that has a default value, matched nothing, and failed silently. `people` resolved to something else entirely and the chips would never have rendered. eslint's "outer scope value" warning is what surfaced it. - Admin face thumbnails 403'd because <AuthenticatedImage> attaches whatever gallery token is in session storage; an admin who has also opened one of their own galleries sends a type:"gallery" bearer to an admin route. Admin routes authenticate from the httpOnly cookie, which a plain same-origin <img> sends by itself. Worth noting AdminPhotoGrid has the same latent shape; not touched here. Also: the admin card now reports "N people (M shown to guests)" when those differ, so the settings page and the gallery stop disagreeing without explanation. 45 backend tests (8 new) and 140 frontend tests green; tsc and eslint clean. EN + DE for every new string. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * perf(faces): batch migration DDL and drop the face stack from server.js import (#1074) CI's backend job timed out at 10 minutes on the first run of this branch. Nothing failed — 132 of 182 suites passed and the wall clock ran out. Main does the same 182 in 124s, and where main has 12 suites slow enough for jest to print a duration, this branch had 77. Two changes, both worth making regardless of how much of the gap they close: - Migration 177 added its columns one ALTER TABLE at a time (four on photos, three on events, plus a separate index statement) and seeded settings with a SELECT and an INSERT per key. It now uses one alterTable per table and one SELECT plus one bulk INSERT. 178 folds its index into the same statement as its column. That chain replays in ~90 suites, so statement count there is multiplied by 90. - server.js required faceQueue at module scope, which pulls in axios and — through imageProcessor — sharp. Every supertest suite that imports server.js was paying for a module graph it never uses. Now required inside the startup block, next to the call that needs it. Honest about the evidence: locally the migration delta measures at zero (1.15s vs 1.13s for the same suite, three runs each), so batching alone does not explain an eight-minute regression. A fast local disk and many cores mask per-statement and per-import costs that a two-core runner with a shared disk does not. These are the two real costs this branch added to a path that runs in almost every suite; whether they are sufficient is a question for CI, not for another round of local speculation. 37 face tests still green after the change. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * i18n(faces): complete EN and DE coverage for the face feature (#1074) The admin card and the Features toggle were rendering entirely from inline English `defaultValue` fallbacks — 22 keys existed in no locale file at all, so a German admin saw an English consent notice, English toggles and English buttons. The gallery side was already translated; the admin side was not, and nothing in the toolchain flags this because a `defaultValue` always renders something. Adds the missing `admin.faces.*` (19), `settings.features.faces.*` (2) and shared `common.clear/saved/saveFailed` in both languages. Existing keys are left alone (setdefault, not overwrite), so the shared `common` strings other features rely on are untouched. Committed the audit as frontend/scripts/i18n-faces-audit.py rather than throwing it away: it extracts every t() key the face components actually use and diffs it against each locale, and it also reports German values that are byte-identical to English, which is the usual shape of an untranslated copy-paste. Currently: 69 keys in use, EN complete, DE complete, no identical pairs. Verified in the browser, not just in the JSON — the German card reads "61 / 61 Fotos durchsucht · 16 Personen (5 für Gäste sichtbar)" end to end. Also checked the components for hardcoded user-facing text (JSX nodes, title/aria-label/placeholder attributes) outside t(); there is none. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 13 defects from external review — coordinates, counts, erasure, races (#1074) Codex reviewed the branch against main. Thirteen findings, nine P1. I checked every one against the code and could not dismiss a single one as a false positive, so all thirteen are fixed here. THE WORST ONE: bounding boxes were stored in the wrong coordinate system. The sidecar reports coordinates in the space of the image it was HANDED — which is the ≤1920px preview, not the original — while every consumer compares them against photos.width/height, the original dimensions. A 6000px photo therefore produced boxes ~3x too small and areas ~9x too small: avatar crops landed in the wrong place and the Portraits rule could never fire. It is invisible on any photo already under 1920px, which is exactly why the demo gallery and every screenshot looked correct. Now scaled once in faceProcessor so everything downstream can assume original-image coordinates. ERASURE. The FK cascade on photo_faces is decorative on SQLite: PicPeak never enables `PRAGMA foreign_keys`, so deleting a photo left its embeddings behind. I first enabled the pragma globally and reverted it — six unrelated suites immediately failed on pre-existing dangling references, and switching it on would start rejecting inserts on every existing install. That is a real change worth making, but it is its own PR, not a rider on this one. Instead deletion purges explicitly: purgePhotoFaces in the photo paths (single, bulk, service) and photo_faces/event_people in deleteEventCascade. Tests assert this with the pragma explicitly OFF, so they can only pass if the code does the work. COUNTS. A re-scan deleted the old face rows without undoing their contribution to event_people, so counts inflated on every re-scan and ghost people survived. Now the affected people are recomputed before the replacements are assigned. My own "must not double its faces" test only checked photo_faces rows, which is why it passed throughout. RACES. A worker that finished after an admin purged the event committed its rows anyway — erasure reported success and the data reappeared. The commit is now conditional on the row still being 'processing'. And assignFaces is read-modify-write over an event's people, so two workers lost each other's updates; it is now serialised per event with an in-process mutex plus a Postgres advisory lock for the multi-pod case the queue advertises. METADATA LOSS. Merging discarded the source's name and suppression flags, so a merge could erase a typed name or un-hide someone. Reclustering remembered only people with a label, so an unnamed-but-hidden bystander came back guest-visible after one "Re-group people" — and suppression now propagates to every descendant cluster, not just the majority one. Also: export reset face_status so a restored gallery re-scans instead of claiming to be scanned forever; manual category edits clear auto_categorized so "undo automatic" cannot delete a photographer's own choice; external photos are skipped rather than failed (resolvePhotoStorageKey returns null for them by design); the gallery refetches photo memberships as a scan progresses so filtering is not stale; a failed VACUUM now fails the backup rather than publishing one that may retain biometric pages; and the ML Dockerfile's `|| true` is scoped to the uninstall — as written it was `(install && uninstall) || true`, so a failed dependency install produced a green layer and an image with no onnxruntime. Four new regression tests. Full backend suite failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 12 more defects from review round 2 — cross-event purge, leaks, lifecycle (#1074) Second Codex round on the same diff, now including round 1's fixes. Twelve findings, seven P1. Again none were false positives. SECURITY, AND MINE FROM ROUND 1: the bulk-delete face purge iterated the raw `photoIds` from the request instead of the event-scoped `photos` rows the handler had already validated. purgePhotoFaces has no event scope of its own, so an editor could pass another gallery's photo id and delete its face data — even though the photo deletion right below it was correctly scoped. Fixing one thing and introducing another is exactly why the second round was worth running. ANOTHER VISIBILITY LEAK, same class as the one round 1 fixed: /people returns scan progress, and getScanStatus counted every photo with a face_status — including hidden ones. Guests could read the hidden-photo count off the progress bar while the people list and covers beside it were properly scoped. Now scoped by the same predicate, with the caller passing its audience. RECLUSTER, ROUND 1'S FIX WAS INCOMPLETE. I made suppression follow every descendant but still copied the flags from the majority ANCESTOR. When reclustering merges a visible named person with a hidden one, the majority ancestor is often the visible one — republishing the hidden person's photos. Suppression is now OR-ed across every ancestor contributing faces. The name also now goes to the genuine largest descendant; the previous code took whichever cluster came first in map order, which the comment already claimed it did not. LIFECYCLE. Face data is excluded from backups and exports, but photos. face_status came across intact, so a restored install claimed every photo was scanned while holding no faces — and the worker only claims 'pending', so it stayed that way forever. Now: the SQLite backup requeues in the dump, restore requeues after the pool reinit (the Postgres path cannot rewrite rows inside pg_dump), the portable importer purges LOCAL face tables (they were excluded from the replace list, so another instance's embeddings survived an import with FK checks suspended) and requeues, and archiving disables detection so a restored archive is honestly off rather than enabled-and-empty. WRITE PATHS. Only processPhoto enqueued. The synchronous upload path (chunked-upload completion, watch-folder) left photos unscanned, and replacePhoto kept the OLD image's faces on a row now pointing at a different picture — stale identities shown on the new photo. FRONTEND. PeopleSheet and the admin manager rendered centred thumbnails and ignored the bbox, so on group photos the avatar showed whoever stood in the middle and two people from one photo were indistinguishable — in the manager whose entire job is telling faces apart. The crop maths is now one shared helper (faceCrop.ts) so the three surfaces cannot drift again. Full-page layouts (gallery-premium, gallery-story) render their own lightbox and never received the people props. Backend failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): round 3 — five of round 2's fixes were wrong or no-ops (#1074) Third and final Codex round. Eight findings, four P1 — and the important part is that FIVE of them are defects in round 2's fixes, not in the original code. - The sync-upload enqueue I added was a silent no-op. It queried through `trx` after the transaction had already been committed, which throws "Transaction query already complete" straight into the catch I had wrapped it in. Chunked uploads and watch-folder imports were still never scanned, and the code read as though they were. Uses `db` now. - The post-restore requeue ran BEFORE the files were restored, in both the portable importer and the native restore. The face worker is live during a restore, so it could claim those rows and scan the previous instance's files, or fail them for originals not yet on disk — with nothing to requeue them afterwards. Both now run after file restoration; the native one is extracted into requeueFaceScans() and called from the full and database-only paths. - The admin face crop mixed coordinate spaces: an original-pixel bbox scaled against the THUMBNAIL's natural size. The API now returns the source dimensions alongside the box, so there is one space to reason about. - Forwarding people props through layoutProps did not make them work — the full-page layouts never destructured them. GalleryStoryLayout now threads them to its own lightbox. Genuinely new findings, all in the same class as ones already fixed: - releaseToPending updated unconditionally, so a photo purged while its sidecar request was in flight came back as 'pending' and was rescanned — biometric rows reappearing after the purge reported success. Round 2 fixed exactly this on the COMMIT path and I did not carry it to the retry path. Now guarded on 'processing'. - purgePhotoFaces left face_status alone, so a worker mid-scan still satisfied its commit guard and could write fresh faces into a photo being deleted — orphans, since the FK cascade is inert on SQLite. It now clears the claim as part of the purge. - Phase 3 was unreachable: the migration seeds face_auto_categorize_enabled false and nothing could ever write it, so the rule engine and its undo endpoint returned "disabled" in every real flow. Added GET/PUT and a toggle on the admin card, EN + DE. NOT fixed, deliberately: GalleryPremiumLayout uses yet-another-react-lightbox rather than the shared PhotoLightbox, so person chips there are a real port rather than a prop forward. Recorded as open rather than bodged. Backend failure set identical to origin/main; 41 face tests and 140 frontend tests green; i18n audit reports EN and DE complete at 71 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): block face recognition on the all-in-one image (#1074, #1042) The single-container image cannot run this feature, so it is refused there rather than left to degrade. WHY, since the reason is not obvious from the code: the AIO image runs the backend, the frontend, SQLite and every background worker inside one container aimed at "one photographer plus guests browsing". It has no Redis, SQLite gives it a single writer, and it contains no ML sidecar to talk to. Face detection would add a second image-processing pipeline competing with Sharp for the same CPU and memory. That failure is not loud — the install just becomes slow and looks broken, which is the worst possible shape for a deployment whose whole promise is one container and no decisions. Gated on an explicit PICPEAK_SINGLE_CONTAINER marker, NOT inferred from SERVE_FRONTEND or a SQLite path: plenty of legitimate multi-container setups serve the frontend from the backend or run SQLite, and none of them should lose the feature by accident. Three layers, because the first is the only one that enforces: - faceSettings.isFeatureEnabled() returns false before consulting the flag, so a database restored from a full deployment with `faces` enabled still cannot switch it on here. - The feature-flag API forces `faces: false` in both directions, so the admin UI reflects reality instead of offering a switch that refuses to stay on. - The Features tab renders the card disabled with a plain-language reason, read from a new `single_container` field on /admin/system/version (an endpoint the admin UI already calls). Documented in ml/README.md and .env.example. Three tests pin the behaviour, including that the marker only accepts explicit truthy values. NOTE FOR PR #1068: this expects `Dockerfile.aio` to set `ENV PICPEAK_SINGLE_CONTAINER=true`. That one line lives on that branch and is not in this commit — until it lands, an AIO build would still offer the feature. Worth adding alongside the `Limits` section of docs/single-container.md. 44 face tests green; EN + DE complete at 72 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(faces): pin the bbox coordinate space with a real scale factor (#1074) The coordinate-space bug — boxes stored in preview space while every consumer reads them as original-image pixels — had no test, and could not have been caught by the ones that existed: every photo in the demo gallery is 750px, so the scale factor was always exactly 1.0 and the correction never executed. Verified by hand first, on a real 4000x3000 upload with the face placed off-centre so a wrong crop would be unmistakable. Before the fix the stored box was 1493,204 (preview space, face actually at x≈2850-3618); after, 3110,426 — a factor of 2.083, exactly 4000/1920, landing inside the face. The admin crop then resolved to left=-395px/top=-46px on a 64px window, which is the face centred. That verification is now a test rather than a memory. Three cases: a 4000px photo must scale by 4000/1920, a 1920px photo must NOT change (the case that hid the bug), and a row with no width must fall back to unscaled rather than storing NaN. Note for anyone extending these: jest hoists mock factories above the file, so anything they close over has to be `mock`-prefixed. Getting that wrong fails at transform time with a message that does not name the variable. 47 face tests green. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
co-authored by
Paul Nothaft
Paul Nothaft
parent
6ebdd13dde
commit
b69dd134d0
@@ -0,0 +1,16 @@
|
||||
# The conversion tool and its TensorFlow dependency never belong in the image.
|
||||
tools/
|
||||
tests/
|
||||
README.md
|
||||
LICENSES.md
|
||||
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Locally-produced model artifacts — the image fetches these by pinned URL
|
||||
# and checksum in the `models` build stage instead.
|
||||
*.onnx
|
||||
*.h5
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
# picpeak-ml — optional face-detection sidecar (#1074).
|
||||
#
|
||||
# Debian slim rather than Alpine: onnxruntime publishes manylinux wheels for
|
||||
# x86_64 and aarch64 but nothing for musl, so Alpine would mean compiling ORT
|
||||
# from source on both legs of the multi-arch build. The slim base costs ~40MB
|
||||
# over Alpine and saves an hour of CI per build.
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stage 1 — fetch and verify model weights
|
||||
# ---------------------------------------------------------------------------
|
||||
# Weights are baked in, never downloaded at runtime: airgapped installs must
|
||||
# work, and a model that changes under a running deployment would silently
|
||||
# invalidate every stored embedding.
|
||||
#
|
||||
# Both artifacts are pinned by URL *and* SHA-256. The checksum is the point —
|
||||
# an immutable-looking URL that starts serving different bytes must fail the
|
||||
# build rather than quietly reshape the embedding space.
|
||||
FROM python:3.12-slim AS models
|
||||
|
||||
ARG YUNET_URL=https://media.githubusercontent.com/media/opencv/opencv_zoo/f12e12798e8314f7c074a6656816c048dcc95b7a/models/face_detection_yunet/face_detection_yunet_2023mar.onnx
|
||||
ARG YUNET_SHA256=8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4
|
||||
|
||||
# FaceNet-512 as ONNX. deepface distributes this model as Keras .h5 only, so
|
||||
# the ONNX is produced once by `tools/convert_facenet.py` and published as a
|
||||
# release asset — converting inside this build would drag TensorFlow (~600MB)
|
||||
# through both architecture legs to produce a file that is identical either
|
||||
# way.
|
||||
#
|
||||
# Defaults to the canonical published artifact so `docker build ml/` and
|
||||
# `docker compose --profile faces up` both work with no arguments. Override
|
||||
# both together to use a different embedder. Blanking either one still fails
|
||||
# the build loudly (below) rather than silently producing an image with no
|
||||
# embedder — the checksum is what makes the URL safe to trust, so a URL
|
||||
# without one is never acceptable.
|
||||
ARG FACENET_ONNX_URL=https://github.com/PicPeak/picpeak/releases/download/ml-models-v1/facenet512.onnx
|
||||
ARG FACENET_ONNX_SHA256=a1c06dcb79dc17a42af01d5bcbce4822caa148b9c24bf7eb8b8e556b4fd0d5db
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends curl ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /models
|
||||
|
||||
RUN curl -fsSL -o face_detection_yunet_2023mar.onnx "${YUNET_URL}" \
|
||||
&& echo "${YUNET_SHA256} face_detection_yunet_2023mar.onnx" | sha256sum -c -
|
||||
|
||||
RUN if [ -z "${FACENET_ONNX_URL}" ] || [ -z "${FACENET_ONNX_SHA256}" ]; then \
|
||||
echo "ERROR: FACENET_ONNX_URL and FACENET_ONNX_SHA256 build args are required." >&2; \
|
||||
echo " Produce the artifact with ml/tools/convert_facenet.py, publish it," >&2; \
|
||||
echo " then pass both args. See ml/README.md." >&2; \
|
||||
exit 1; \
|
||||
fi \
|
||||
&& curl -fsSL -o facenet512.onnx "${FACENET_ONNX_URL}" \
|
||||
&& echo "${FACENET_ONNX_SHA256} facenet512.onnx" | sha256sum -c -
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stage 2 — runtime
|
||||
# ---------------------------------------------------------------------------
|
||||
FROM python:3.12-slim
|
||||
|
||||
ARG BUILD_DATE
|
||||
ARG VCS_REF
|
||||
ARG VERSION
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/PicPeak/picpeak"
|
||||
LABEL org.opencontainers.image.description="PicPeak ML sidecar — face detection and embedding"
|
||||
LABEL org.opencontainers.image.licenses="MIT"
|
||||
|
||||
# Busts the apt layer each CI run so the image picks up current Debian
|
||||
# security updates instead of reusing a stale cached upgrade layer — same
|
||||
# reasoning as backend/Dockerfile.
|
||||
ARG CACHEBUST=1
|
||||
RUN echo "cachebust=${CACHEBUST}" \
|
||||
&& apt-get update \
|
||||
&& apt-get upgrade -y \
|
||||
# libGL and libglib are opencv-python-headless's remaining shared-library
|
||||
# deps. The headless wheel drops the GUI toolkits but still links libGL.
|
||||
&& apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
# pip itself is ~10MB of an image that never installs anything at runtime.
|
||||
#
|
||||
# The `|| true` is scoped to the uninstall ONLY. Written as
|
||||
# `pip install && pip uninstall || true` the shell parses it as
|
||||
# `(install && uninstall) || true`, so a failed requirements install still
|
||||
# produces a green layer — and CI would publish an ML image with no FastAPI,
|
||||
# no uvicorn and no onnxruntime that fails at container start instead of at
|
||||
# build time.
|
||||
RUN pip install --no-cache-dir -r requirements.txt \
|
||||
&& { pip uninstall -y pip setuptools 2>/dev/null || true; }
|
||||
|
||||
# Non-root. Nothing in this container writes anything — no volumes, no
|
||||
# database, no model download — so the whole filesystem can stay read-only to
|
||||
# the service account.
|
||||
#
|
||||
# Created BEFORE the copies so ownership can be set by COPY --chown. A
|
||||
# `chown -R` afterwards would rewrite every copied file into a fresh layer,
|
||||
# duplicating the 90MB model and adding ~94MB to the image for nothing.
|
||||
RUN useradd --system --uid 1001 --create-home picpeak
|
||||
|
||||
COPY --from=models --chown=picpeak:picpeak /models /models
|
||||
COPY --chown=picpeak:picpeak app ./app
|
||||
|
||||
USER picpeak
|
||||
|
||||
ENV FACE_MODEL_DIR=/models \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
# Mirrors the backend's healthcheck shape. /health is unauthenticated so this
|
||||
# needs no secret; it reports liveness only, because a failed model load
|
||||
# aborts startup and the container never serves at all.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=40s --retries=3 \
|
||||
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status == 200 else 1)"
|
||||
|
||||
# Single worker on purpose: the models are loaded per process, so a second
|
||||
# worker doubles RSS for a service the backend calls at concurrency 1.
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]
|
||||
@@ -0,0 +1,60 @@
|
||||
# Model provenance and licences
|
||||
|
||||
PicPeak is run commercially by working photographers. That sets a hard rule
|
||||
for this image:
|
||||
|
||||
> **No non-commercial artifact is ever baked into a PicPeak image.**
|
||||
|
||||
Every open face-recognition weight set traces back to a scraped,
|
||||
research-only dataset — CASIA-WebFace, VGGFace2, MS1M, Glint360K and
|
||||
WebFace260M all carry academic-use-only agreements, so "trained on clean
|
||||
data" is not an option that exists. What differs, and what actually binds a
|
||||
redistributor, is the grant the **distributor** places on the artifact we
|
||||
copy into this image.
|
||||
|
||||
## Shipped in this image
|
||||
|
||||
| Artifact | Role | Distributor | Licence |
|
||||
|---|---|---|---|
|
||||
| `face_detection_yunet_2023mar.onnx` | detection | [OpenCV Zoo](https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet) | MIT |
|
||||
| `facenet512.onnx` | embedding | [serengil/deepface](https://github.com/serengil/deepface) | MIT |
|
||||
|
||||
Both are redistributable. `facenet512.onnx` is converted from deepface's
|
||||
published `facenet512_weights.h5` by `tools/convert_facenet.py`; the
|
||||
conversion changes the container format, not the weights, so the MIT grant
|
||||
carries over.
|
||||
|
||||
Pinned sources, verified by SHA-256 at build time:
|
||||
|
||||
- YuNet — `opencv/opencv_zoo` at commit `f12e12798e8314f7c074a6656816c048dcc95b7a`
|
||||
`8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4`
|
||||
- FaceNet-512 source weights — `serengil/deepface_models` release `v1.0`
|
||||
`3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f`
|
||||
|
||||
## Deliberately NOT shipped
|
||||
|
||||
| Artifact | Why not |
|
||||
|---|---|
|
||||
| InsightFace `buffalo_*`, `antelopev2` | **Non-commercial research only.** Commercial use requires a licence from insightface.ai ([model zoo README](https://github.com/deepinsight/insightface/blob/master/model_zoo/README.md), [#2587](https://github.com/deepinsight/insightface/issues/2587)). Available as an opt-in via `FACE_MODEL`, downloaded by the operator who has cleared that licence themselves — never by us. |
|
||||
| Idiap EdgeFace | CC BY-NC-SA 4.0 — explicitly non-commercial. The best accuracy-per-parameter of the candidates, and unusable for that reason. |
|
||||
| AdaFace WebFace4M weights | MIT code, non-commercial weights. |
|
||||
|
||||
## What that choice cost
|
||||
|
||||
Measured on DeepFace's own matched benchmark (LFW, aligned, cosine — the only
|
||||
comparison worth anything, since every model quotes its own "LFW 99.x%" on
|
||||
its own pipeline):
|
||||
|
||||
| Embedder | with RetinaFace | with YuNet |
|
||||
|---|---|---|
|
||||
| **FaceNet-512** | **98.4%** | **97.9%** |
|
||||
| ArcFace | 96.6% | 96.7% |
|
||||
| SFace | 92.4% | 91.0% |
|
||||
|
||||
Choosing YuNet over RetinaFace costs ~0.5 points. Holding the
|
||||
commercial-redistribution line costs nothing beyond that — FaceNet-512 is
|
||||
both the most accurate option in the table *and* MIT. Taking the headline
|
||||
numbers at face value would have pointed at SFace (advertised 0.9940, actual
|
||||
91–92% matched) and cost seven points, which for clustering is fatal: every
|
||||
false split invents a duplicate person and every false merge puts a stranger
|
||||
into someone's "download my photos".
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
# picpeak-ml
|
||||
|
||||
Optional face-detection sidecar for PicPeak (#1074). Detects faces in one
|
||||
image and returns a bounding box, five landmarks, quality signals and a
|
||||
512-d embedding per face.
|
||||
|
||||
**Nothing else.** No database, no volumes, no state, no egress, no model
|
||||
download at runtime. Clustering, person identity, thresholds and every
|
||||
privacy decision live in the PicPeak backend, where the data already is.
|
||||
This service forgets each image the moment it answers.
|
||||
|
||||
If you don't run this container, the feature does not exist — see
|
||||
"Turning it on" below.
|
||||
|
||||
## API
|
||||
|
||||
All endpoints except `/health` require the `X-Face-ML-Token` header. The
|
||||
service **refuses to start** without `FACE_ML_TOKEN` set, so an accidentally
|
||||
published port is never a free face-detection API.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GET /health` | `{"status": "ok"}` — unauthenticated, used by the compose healthcheck |
|
||||
| `GET /info` | `{detector, embedder, model_version, dim}` |
|
||||
| `POST /faces` | multipart `image` → `{model_version, faces: [...]}` |
|
||||
|
||||
Each face:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"bbox": [x, y, w, h], // ORIGINAL image pixels, not detection-scaled
|
||||
"score": 0.94,
|
||||
"landmarks": [[x, y], ...], // 5: right eye, left eye, nose, right mouth, left mouth
|
||||
"yaw": -1.42, // degrees, approximate (see pipeline.py)
|
||||
"pitch": -25.33,
|
||||
"blur": 2579.5, // variance of Laplacian on the aligned crop; higher = sharper
|
||||
"embedding": [...] // 512 floats, L2-normalized
|
||||
}
|
||||
```
|
||||
|
||||
`404`/`400` mean "this image is a lost cause" — the backend marks the photo
|
||||
failed. `5xx` and connection failures mean "try later" — the backend returns
|
||||
the photo to `pending` with backoff, so turning this container off for a week
|
||||
does not require a manual re-scan.
|
||||
|
||||
## Models
|
||||
|
||||
YuNet (detection, MIT) + FaceNet-512 (embedding, MIT), both baked into the
|
||||
image and verified by SHA-256 at build time. See [LICENSES.md](LICENSES.md)
|
||||
for why these two and not the more obvious InsightFace weights — the short
|
||||
version is that InsightFace's are non-commercial-only and PicPeak's users are
|
||||
working photographers.
|
||||
|
||||
### Building the image
|
||||
|
||||
`facenet512.onnx` is **not** fetched automatically, because deepface
|
||||
distributes FaceNet-512 as Keras `.h5` only. Convert it once, publish it,
|
||||
then pass the URL and checksum:
|
||||
|
||||
```bash
|
||||
cd ml
|
||||
python3.11 -m venv .venv && . .venv/bin/activate # 3.11: TF has no 3.12+ wheels
|
||||
pip install -r tools/requirements-convert.txt
|
||||
|
||||
curl -fsSL -o facenet512_weights.h5 \
|
||||
https://github.com/serengil/deepface_models/releases/download/v1.0/facenet512_weights.h5
|
||||
echo "3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f facenet512_weights.h5" | sha256sum -c -
|
||||
|
||||
python tools/convert_facenet.py facenet512_weights.h5 facenet512.onnx
|
||||
```
|
||||
|
||||
The script verifies the converted graph against the Keras original before
|
||||
writing (worst observed divergence: 2.1e-06 absolute, cosine 1.0000000000)
|
||||
and prints the SHA-256 to publish. Output is ~89.6 MB, 23,497,424 parameters.
|
||||
|
||||
Publish `facenet512.onnx` as a release asset, set the repository variables
|
||||
`FACENET_ONNX_URL` and `FACENET_ONNX_SHA256` (Settings → Variables — it's a
|
||||
public URL, not a secret), and CI picks it up. To build locally:
|
||||
|
||||
```bash
|
||||
docker build -t picpeak-ml \
|
||||
--build-arg FACENET_ONNX_URL=https://github.com/PicPeak/picpeak/releases/download/<tag>/facenet512.onnx \
|
||||
--build-arg FACENET_ONNX_SHA256=<sha256> \
|
||||
ml/
|
||||
```
|
||||
|
||||
The conversion sits outside the Docker build because TensorFlow is ~600MB of
|
||||
build dependency for a file that never ships in the final image, and the
|
||||
result is architecture-independent — no reason to run it on both legs of
|
||||
every multi-arch build.
|
||||
|
||||
**The conversion is not byte-reproducible.** Two runs with the same pinned
|
||||
versions on the same machine produce functionally identical graphs (same 336
|
||||
nodes, same 271 initializers, weights matching to 0.000e+00) but differ in a
|
||||
few initializer names, because tf2onnx's traced-op naming is not
|
||||
deterministic. So a re-conversion **will** have a different SHA-256, and that
|
||||
is expected rather than a sign of tampering. The checksum pins one published
|
||||
artifact so its URL cannot start serving different bytes; validating a fresh
|
||||
conversion is the parity check's job, not the hash's.
|
||||
|
||||
## Not available on the all-in-one image
|
||||
|
||||
The single-container image (`Dockerfile.aio`) sets
|
||||
`PICPEAK_SINGLE_CONTAINER=true`, and the backend refuses to enable face
|
||||
recognition when it sees that — the feature flag cannot be switched on, and
|
||||
per-event detection stays off even if a restored database says otherwise.
|
||||
|
||||
This is a performance decision, not a licensing or packaging one. That image
|
||||
runs the backend, the frontend, SQLite and every background worker inside one
|
||||
container aimed at "one photographer plus guests browsing". It has no Redis,
|
||||
SQLite gives it a single writer, and it contains no ML sidecar to talk to.
|
||||
Adding a second image-processing pipeline that competes with Sharp for the
|
||||
same CPU and RAM would not fail loudly — it would just make the whole install
|
||||
slow and appear broken.
|
||||
|
||||
Run the standard multi-container deployment if you want this feature.
|
||||
|
||||
## Turning it on
|
||||
|
||||
Two deliberate actions, neither of which is installing this container:
|
||||
|
||||
1. Enable the `faces` feature flag in PicPeak's admin settings.
|
||||
2. Enable "Detect people in this gallery" per event.
|
||||
|
||||
`FACE_ML_URL` defaults to `http://picpeak-ml:8000` — the compose service name
|
||||
— so the standard deployment needs no URL configuration. **Nothing in the
|
||||
backend touches that URL while the flag is off**, so an install without this
|
||||
container never attempts a connection.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt pytest httpx
|
||||
python -m pytest tests/ -q
|
||||
```
|
||||
|
||||
The tests stub the models out: they cover the auth boundary, the request
|
||||
guards and the alignment geometry — the places where a mistake is a security
|
||||
problem or a silent accuracy problem. Model *quality* is not a unit-test
|
||||
question; that is what the Phase 0 spike measured.
|
||||
|
||||
### The one thing to be careful about
|
||||
|
||||
The alignment in `pipeline.py` and the normalization in `_embed` must stay
|
||||
identical to whatever the clustering threshold was tuned against. A tuned
|
||||
cosine threshold does not transfer across an alignment change. If either
|
||||
changes, bump `MODEL_VERSION` in `config.py` — the backend keys
|
||||
re-derivation off that string and will re-cluster rather than silently mix
|
||||
two incompatible embedding spaces.
|
||||
@@ -0,0 +1,72 @@
|
||||
"""
|
||||
Configuration for the picpeak-ml sidecar (#1074).
|
||||
|
||||
Everything is read once at import. There is no reload path and no settings
|
||||
API — this service is stateless by design, and an operator changing a knob
|
||||
restarts the container.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MODEL_DIR = os.environ.get("FACE_MODEL_DIR", "/models")
|
||||
|
||||
DETECTOR_FILENAME = "face_detection_yunet_2023mar.onnx"
|
||||
EMBEDDER_FILENAME = os.environ.get("FACE_EMBEDDER_FILENAME", "facenet512.onnx")
|
||||
|
||||
DETECTOR_NAME = "yunet_2023mar"
|
||||
EMBEDDER_NAME = os.environ.get("FACE_MODEL", "facenet512")
|
||||
|
||||
# Stamped onto every face row the backend stores. Changing the detector, the
|
||||
# embedder, the alignment or the normalization MUST bump this: embeddings from
|
||||
# two different pipelines are not comparable, and a silent mix produces
|
||||
# clusters that look plausible and are wrong. The backend keys re-derivation
|
||||
# off this string.
|
||||
MODEL_VERSION = f"{DETECTOR_NAME}+{EMBEDDER_NAME}+v1"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Shared secret, required. `main.py` refuses to start without it rather than
|
||||
# defaulting to open: an accidentally published port must not be a free
|
||||
# face-detection API.
|
||||
TOKEN = os.environ.get("FACE_ML_TOKEN", "").strip()
|
||||
TOKEN_HEADER = "X-Face-ML-Token"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# YuNet's own confidence floor. Deliberately permissive: the backend applies
|
||||
# the *product* quality floor (score, blur, bbox size) because those
|
||||
# thresholds are admin-tunable there and baked-in here. This service's job is
|
||||
# to report what it saw, not to decide what counts.
|
||||
DET_SCORE_THRESHOLD = float(os.environ.get("FACE_DET_SCORE_THRESHOLD", "0.6"))
|
||||
NMS_THRESHOLD = float(os.environ.get("FACE_NMS_THRESHOLD", "0.3"))
|
||||
TOP_K = int(os.environ.get("FACE_DET_TOP_K", "5000"))
|
||||
|
||||
# Guard rails on untrusted input. The backend only ever sends its own preview
|
||||
# renditions (≤1920px), but this service must not fall over if something else
|
||||
# reaches it.
|
||||
MAX_IMAGE_BYTES = int(os.environ.get("FACE_MAX_IMAGE_BYTES", str(32 * 1024 * 1024)))
|
||||
MAX_FACES = int(os.environ.get("FACE_MAX_FACES", "64"))
|
||||
|
||||
# Long edge the image is downscaled to before detection. Matches the backend's
|
||||
# preview tier (imageProcessor.js generates ≤1920px), so the common case is a
|
||||
# no-op; anything larger is scaled down here so detection cost stays bounded.
|
||||
# Bboxes and landmarks are always reported in ORIGINAL image coordinates.
|
||||
INPUT_LONG_EDGE = int(os.environ.get("FACE_INPUT_LONG_EDGE", "1920"))
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Runtime
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# One ORT thread by default. The backend's face queue runs at concurrency 1
|
||||
# (it shares a host with Sharp, which is the real memory pressure — see
|
||||
# backgroundProcessor.js), so letting ORT fan out across every core buys
|
||||
# nothing and costs RSS.
|
||||
ORT_THREADS = int(os.environ.get("FACE_ORT_THREADS", "1"))
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
"""
|
||||
picpeak-ml — optional face-detection sidecar for PicPeak (#1074).
|
||||
|
||||
Three endpoints, no database, no volumes, no egress. Models are baked into
|
||||
the image at build time, so an airgapped install works and nothing is
|
||||
downloaded at runtime.
|
||||
|
||||
The service is deliberately dumb: it reports what it saw in one image and
|
||||
forgets. Clustering, identity, thresholds and every privacy decision live in
|
||||
the backend, where the data already is.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import Depends, FastAPI, File, Header, HTTPException, UploadFile
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from . import config
|
||||
from .pipeline import FacePipeline
|
||||
from .schemas import FacesResponse, HealthResponse, InfoResponse
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO, format="%(asctime)s %(levelname)s [picpeak-ml] %(message)s"
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_pipeline: FacePipeline | None = None
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
global _pipeline
|
||||
|
||||
# Refuse to run open. An accidentally published port must not be a free
|
||||
# face-detection API, and a service that silently accepted anonymous
|
||||
# requests when the operator forgot the variable would be exactly that.
|
||||
if not config.TOKEN:
|
||||
raise RuntimeError(
|
||||
"FACE_ML_TOKEN is not set. picpeak-ml will not start without a "
|
||||
"shared secret — set it on both this container and the backend."
|
||||
)
|
||||
|
||||
logger.info("Loading models from %s", config.MODEL_DIR)
|
||||
_pipeline = FacePipeline()
|
||||
logger.info(
|
||||
"Ready: detector=%s embedder=%s version=%s dim=%d",
|
||||
config.DETECTOR_NAME,
|
||||
config.EMBEDDER_NAME,
|
||||
config.MODEL_VERSION,
|
||||
_pipeline.dim,
|
||||
)
|
||||
yield
|
||||
_pipeline = None
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="picpeak-ml",
|
||||
description="Face detection and embedding sidecar for PicPeak",
|
||||
lifespan=lifespan,
|
||||
# No interactive docs: this is a machine-to-machine service on a private
|
||||
# network, and /docs is just surface.
|
||||
docs_url=None,
|
||||
redoc_url=None,
|
||||
openapi_url=None,
|
||||
)
|
||||
|
||||
|
||||
def require_token(x_face_ml_token: str = Header(default="")) -> None:
|
||||
"""Constant-time-ish shared-secret check.
|
||||
|
||||
Python's `==` on str short-circuits, so this leaks a timing signal in
|
||||
principle. It is not worth `hmac.compare_digest` gymnastics for a token
|
||||
that only travels over a private Docker network — but it IS worth
|
||||
rejecting with a bare 401 and no detail, so a prober learns nothing about
|
||||
whether the header name was even right.
|
||||
"""
|
||||
if x_face_ml_token != config.TOKEN:
|
||||
raise HTTPException(status_code=401, detail="Unauthorized")
|
||||
|
||||
|
||||
@app.get("/health", response_model=HealthResponse)
|
||||
def health() -> HealthResponse:
|
||||
"""Unauthenticated on purpose — the compose healthcheck calls it.
|
||||
|
||||
Reports only liveness. It deliberately does not confirm the models
|
||||
loaded, because lifespan raises on failure and the container never
|
||||
reaches a serving state at all.
|
||||
"""
|
||||
return HealthResponse(status="ok")
|
||||
|
||||
|
||||
@app.get("/info", response_model=InfoResponse, dependencies=[Depends(require_token)])
|
||||
def info() -> InfoResponse:
|
||||
assert _pipeline is not None
|
||||
return InfoResponse(
|
||||
detector=config.DETECTOR_NAME,
|
||||
embedder=config.EMBEDDER_NAME,
|
||||
model_version=config.MODEL_VERSION,
|
||||
dim=_pipeline.dim,
|
||||
)
|
||||
|
||||
|
||||
@app.post("/faces", response_model=FacesResponse, dependencies=[Depends(require_token)])
|
||||
async def faces(image: UploadFile = File(...)) -> FacesResponse:
|
||||
assert _pipeline is not None
|
||||
|
||||
data = await image.read()
|
||||
if not data:
|
||||
raise HTTPException(status_code=400, detail="Empty upload")
|
||||
if len(data) > config.MAX_IMAGE_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Image too large")
|
||||
|
||||
try:
|
||||
detected = _pipeline.process(data)
|
||||
except ValueError as exc:
|
||||
# Undecodable input is the caller's problem, not ours — 400 so the
|
||||
# backend marks the photo 'failed' instead of retrying it forever.
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
return FacesResponse(model_version=config.MODEL_VERSION, faces=detected)
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def unhandled(request, exc: Exception) -> JSONResponse:
|
||||
"""Never leak a traceback to the caller.
|
||||
|
||||
The backend treats 5xx as "sidecar unhealthy" and puts the photo back to
|
||||
pending with backoff, which is the right behaviour for a genuine internal
|
||||
fault — so the useful detail belongs in our log, not in the response.
|
||||
"""
|
||||
logger.exception("Unhandled error: %s", exc)
|
||||
return JSONResponse(status_code=500, content={"detail": "Internal error"})
|
||||
@@ -0,0 +1,313 @@
|
||||
"""
|
||||
Detection → alignment → embedding (#1074).
|
||||
|
||||
The alignment step in the middle is the part that decides whether this
|
||||
feature works. FaceNet was trained on similarity-aligned crops; handing it a
|
||||
raw bbox crop costs far more accuracy than any model swap would recover. So
|
||||
YuNet's five landmarks are used to warp every face onto the same canonical
|
||||
template before it ever reaches the embedder.
|
||||
|
||||
IMPORTANT — the alignment and normalization below must stay identical to
|
||||
whatever the Phase 0 spike measured its cosine threshold on. A tuned
|
||||
threshold does not transfer across alignment changes; if either is touched,
|
||||
`MODEL_VERSION` bumps and the backend re-derives.
|
||||
"""
|
||||
|
||||
import threading
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
|
||||
from . import config
|
||||
|
||||
|
||||
# Canonical 5-point template (ArcFace's, the de-facto standard), expressed for
|
||||
# a 112x112 crop and scaled to whatever the embedder actually wants. Points are
|
||||
# in IMAGE coordinates, left to right:
|
||||
# 0 subject's right eye (appears image-left)
|
||||
# 1 subject's left eye
|
||||
# 2 nose tip
|
||||
# 3 subject's right mouth corner
|
||||
# 4 subject's left mouth corner
|
||||
# YuNet emits its landmarks in exactly this order, so the mapping is index-wise
|
||||
# with no reshuffling — `test_pipeline.py` pins that assumption.
|
||||
_TEMPLATE_112 = np.array(
|
||||
[
|
||||
[38.2946, 51.6963],
|
||||
[73.5318, 51.5014],
|
||||
[56.0252, 71.7366],
|
||||
[41.5493, 92.3655],
|
||||
[70.7299, 92.2041],
|
||||
],
|
||||
dtype=np.float32,
|
||||
)
|
||||
|
||||
|
||||
class FacePipeline:
|
||||
"""Loads both models once and serves them under a lock.
|
||||
|
||||
cv2.FaceDetectorYN carries mutable input-size state across setInputSize/
|
||||
detect, so it is NOT safe to call from two threads. FastAPI runs sync
|
||||
endpoints in a threadpool, so every inference path is serialized here.
|
||||
That is not a throughput loss worth fixing: the backend's face queue
|
||||
defaults to concurrency 1, and a single lock keeps RSS predictable, which
|
||||
is the constraint that actually matters on a 2 GB VPS.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
detector_path = f"{config.MODEL_DIR}/{config.DETECTOR_FILENAME}"
|
||||
embedder_path = f"{config.MODEL_DIR}/{config.EMBEDDER_FILENAME}"
|
||||
|
||||
self._lock = threading.Lock()
|
||||
|
||||
self._detector = cv2.FaceDetectorYN.create(
|
||||
model=detector_path,
|
||||
config="",
|
||||
input_size=(320, 320), # replaced per-image via setInputSize
|
||||
score_threshold=config.DET_SCORE_THRESHOLD,
|
||||
nms_threshold=config.NMS_THRESHOLD,
|
||||
top_k=config.TOP_K,
|
||||
)
|
||||
|
||||
so = ort.SessionOptions()
|
||||
so.intra_op_num_threads = config.ORT_THREADS
|
||||
so.inter_op_num_threads = config.ORT_THREADS
|
||||
self._embedder = ort.InferenceSession(
|
||||
embedder_path, sess_options=so, providers=["CPUExecutionProvider"]
|
||||
)
|
||||
|
||||
# Read the embedder's geometry off the model rather than hardcoding
|
||||
# 160x160 NHWC. FACE_MODEL is documented as swappable (an operator who
|
||||
# has cleared the InsightFace licence may point this at buffalo_l,
|
||||
# which is 112x112 NCHW), and guessing wrong produces a confident
|
||||
# garbage embedding rather than an error.
|
||||
inp = self._embedder.get_inputs()[0]
|
||||
shape = inp.shape
|
||||
if len(shape) != 4:
|
||||
raise RuntimeError(f"Embedder input must be 4-D, got {shape}")
|
||||
self._input_name = inp.name
|
||||
# NCHW iff the channel axis is second.
|
||||
self._nchw = shape[1] == 3
|
||||
self._crop_size = int(shape[2] if self._nchw else shape[1])
|
||||
|
||||
out = self._embedder.get_outputs()[0]
|
||||
self._dim = int(out.shape[-1])
|
||||
|
||||
self._template = _TEMPLATE_112 * (self._crop_size / 112.0)
|
||||
|
||||
# -- introspection ----------------------------------------------------
|
||||
|
||||
@property
|
||||
def dim(self) -> int:
|
||||
return self._dim
|
||||
|
||||
# -- inference --------------------------------------------------------
|
||||
|
||||
def process(self, image_bytes: bytes) -> list[dict]:
|
||||
"""Decode, detect, align, embed. Returns one dict per face."""
|
||||
buf = np.frombuffer(image_bytes, dtype=np.uint8)
|
||||
img = cv2.imdecode(buf, cv2.IMREAD_COLOR)
|
||||
if img is None:
|
||||
raise ValueError("Image could not be decoded")
|
||||
|
||||
# Downscale for detection, then map coordinates back. Everything the
|
||||
# caller sees is in ORIGINAL image pixels — the backend stores bboxes
|
||||
# to crop avatars from the same rendition later, so a scaled
|
||||
# coordinate would silently offset every cover face.
|
||||
h, w = img.shape[:2]
|
||||
long_edge = max(h, w)
|
||||
if long_edge > config.INPUT_LONG_EDGE:
|
||||
scale = config.INPUT_LONG_EDGE / long_edge
|
||||
det_img = cv2.resize(
|
||||
img, (round(w * scale), round(h * scale)), interpolation=cv2.INTER_AREA
|
||||
)
|
||||
else:
|
||||
scale = 1.0
|
||||
det_img = img
|
||||
|
||||
with self._lock:
|
||||
dh, dw = det_img.shape[:2]
|
||||
self._detector.setInputSize((dw, dh))
|
||||
_, raw = self._detector.detect(det_img)
|
||||
|
||||
if raw is None:
|
||||
return []
|
||||
|
||||
faces = []
|
||||
for row in raw[: config.MAX_FACES]:
|
||||
faces.append(self._one_face(img, row, scale))
|
||||
return faces
|
||||
|
||||
def _one_face(self, img: np.ndarray, row: np.ndarray, scale: float) -> dict:
|
||||
"""Build the response entry for a single YuNet detection.
|
||||
|
||||
`row` is YuNet's 15-wide output: x, y, w, h, then five (x, y)
|
||||
landmark pairs, then the confidence score. All in detection-image
|
||||
coordinates, hence the division by `scale`.
|
||||
"""
|
||||
inv = 1.0 / scale
|
||||
x, y, bw, bh = (float(v) * inv for v in row[0:4])
|
||||
landmarks = (row[4:14].reshape(5, 2).astype(np.float32)) * inv
|
||||
score = float(row[14])
|
||||
|
||||
aligned = self._align(img, landmarks)
|
||||
embedding = self._embed(aligned)
|
||||
yaw, pitch = _pose_from_landmarks(landmarks)
|
||||
|
||||
return {
|
||||
"bbox": [x, y, bw, bh],
|
||||
"score": score,
|
||||
"landmarks": landmarks.tolist(),
|
||||
"yaw": yaw,
|
||||
"pitch": pitch,
|
||||
"blur": _blur_score(aligned),
|
||||
"embedding": embedding.tolist(),
|
||||
}
|
||||
|
||||
def _align(self, img: np.ndarray, landmarks: np.ndarray) -> np.ndarray:
|
||||
"""Similarity-warp the face onto the canonical template."""
|
||||
matrix = _umeyama(landmarks, self._template)
|
||||
if matrix is None:
|
||||
# Degenerate landmarks (all coincident/collinear). Fall back to a
|
||||
# plain centre crop so the face still gets an embedding rather
|
||||
# than vanishing from the gallery.
|
||||
matrix = _fallback_transform(landmarks, self._crop_size)
|
||||
return cv2.warpAffine(
|
||||
img,
|
||||
matrix,
|
||||
(self._crop_size, self._crop_size),
|
||||
flags=cv2.INTER_LINEAR,
|
||||
borderValue=0,
|
||||
)
|
||||
|
||||
def _embed(self, aligned: np.ndarray) -> np.ndarray:
|
||||
# BGR (OpenCV) → RGB (what FaceNet was trained on). Getting this
|
||||
# backwards does not error, it just quietly degrades every embedding.
|
||||
rgb = cv2.cvtColor(aligned, cv2.COLOR_BGR2RGB).astype(np.float32)
|
||||
|
||||
# deepface's "Facenet" normalization: per-image standardization. This
|
||||
# is what the published FaceNet-512 benchmark numbers were produced
|
||||
# with, so it is what the threshold in the backend assumes.
|
||||
mean, std = rgb.mean(), rgb.std()
|
||||
rgb = (rgb - mean) / max(float(std), 1e-6)
|
||||
|
||||
batch = rgb[None, ...]
|
||||
if self._nchw:
|
||||
batch = batch.transpose(0, 3, 1, 2)
|
||||
|
||||
vec = self._embedder.run(None, {self._input_name: batch})[0][0]
|
||||
|
||||
# L2-normalize so the backend's cosine similarity is a plain dot
|
||||
# product and centroid means stay on the unit sphere.
|
||||
norm = float(np.linalg.norm(vec))
|
||||
return (vec / norm) if norm > 0 else vec
|
||||
|
||||
|
||||
def _umeyama(src: np.ndarray, dst: np.ndarray) -> np.ndarray | None:
|
||||
"""Least-squares similarity transform (Umeyama 1991) over ALL five points.
|
||||
|
||||
Deliberately NOT cv2.estimateAffinePartial2D. That function's estimators
|
||||
are RANSAC (its default) and LMEDS, both of which exist to *reject
|
||||
outliers* among many noisy correspondences. Given exactly five points and
|
||||
no outliers they fit a three-point subset perfectly and let the rest
|
||||
drift: measured on a real off-frontal portrait, both pinned the eyes and
|
||||
nose to 0.11px and left the mouth corners 11.8px out on a 160px crop.
|
||||
Umeyama distributes the residual instead (max 6.5px, rms 5.1 vs 7.4) and
|
||||
is what insightface's norm_crop and skimage's SimilarityTransform use.
|
||||
|
||||
Also fully deterministic — no random consensus sampling — which matters
|
||||
beyond accuracy: the same photo must embed identically on every re-scan,
|
||||
or clusters churn between runs for no reason.
|
||||
|
||||
Returns a 2x3 affine matrix, or None if the points are degenerate.
|
||||
"""
|
||||
src = np.asarray(src, dtype=np.float64)
|
||||
dst = np.asarray(dst, dtype=np.float64)
|
||||
|
||||
src_mean, dst_mean = src.mean(axis=0), dst.mean(axis=0)
|
||||
src_c, dst_c = src - src_mean, dst - dst_mean
|
||||
|
||||
variance = float((src_c**2).sum() / len(src))
|
||||
if variance < 1e-9:
|
||||
return None # all points coincident
|
||||
|
||||
cov = dst_c.T @ src_c / len(src)
|
||||
u, s, vt = np.linalg.svd(cov)
|
||||
|
||||
# Guard against the SVD handing back a reflection instead of a rotation —
|
||||
# a mirrored face would embed as a different person.
|
||||
d = np.array([1.0, 1.0])
|
||||
if np.linalg.det(u @ vt) < 0:
|
||||
d[-1] = -1.0
|
||||
|
||||
rotation = u @ np.diag(d) @ vt
|
||||
scale = float((s * d).sum() / variance)
|
||||
translation = dst_mean - scale * (rotation @ src_mean)
|
||||
|
||||
return np.hstack([scale * rotation, translation.reshape(2, 1)]).astype(np.float32)
|
||||
|
||||
|
||||
def _fallback_transform(landmarks: np.ndarray, size: int) -> np.ndarray:
|
||||
"""Centre the landmark centroid in the crop at the template's scale.
|
||||
|
||||
Only reached when the landmarks are degenerate enough that no similarity
|
||||
transform exists. The resulting embedding will be poor, but the face
|
||||
still appears in "this photo contains" rather than vanishing — and the
|
||||
backend's quality floor will keep it from spawning its own person.
|
||||
"""
|
||||
centre = landmarks.mean(axis=0)
|
||||
s = size / 112.0
|
||||
return np.array(
|
||||
[[s, 0.0, size / 2.0 - s * centre[0]], [0.0, s, size / 2.0 - s * centre[1]]],
|
||||
dtype=np.float32,
|
||||
)
|
||||
|
||||
|
||||
def _blur_score(aligned: np.ndarray) -> float:
|
||||
"""Variance of the Laplacian — low means soft/out-of-focus.
|
||||
|
||||
Computed on the ALIGNED crop, not the original frame, so the number is
|
||||
comparable between a face that fills the frame and one in the background:
|
||||
both arrive here at the same pixel size. The backend's quality floor
|
||||
compares against it directly.
|
||||
"""
|
||||
grey = cv2.cvtColor(aligned, cv2.COLOR_BGR2GRAY)
|
||||
return float(cv2.Laplacian(grey, cv2.CV_64F).var())
|
||||
|
||||
|
||||
def _pose_from_landmarks(landmarks: np.ndarray) -> tuple[float, float]:
|
||||
"""Rough head pose in degrees from the five landmarks.
|
||||
|
||||
An approximation, deliberately: a real 3-D pose estimate needs a face
|
||||
model and solvePnP, and the only consumers are a quality floor and the
|
||||
"candid" auto-category rule, neither of which needs better than
|
||||
"clearly turned away vs. not". Returned as degrees so the admin-facing
|
||||
threshold reads in a familiar unit.
|
||||
|
||||
yaw negative = turned toward the subject's right, positive = left
|
||||
pitch negative = looking down, positive = looking up
|
||||
"""
|
||||
right_eye, left_eye, nose, right_mouth, left_mouth = landmarks
|
||||
|
||||
eye_centre = (right_eye + left_eye) / 2.0
|
||||
mouth_centre = (right_mouth + left_mouth) / 2.0
|
||||
eye_span = float(np.linalg.norm(left_eye - right_eye))
|
||||
if eye_span < 1e-6:
|
||||
return 0.0, 0.0
|
||||
|
||||
# Yaw: on a frontal face the nose sits midway between the eyes. As the
|
||||
# head turns, it slides toward the nearer eye. Offset is normalized by
|
||||
# eye span so it is scale-free, then mapped through arcsin.
|
||||
yaw_ratio = float((nose[0] - eye_centre[0]) / (eye_span / 2.0))
|
||||
yaw = float(np.degrees(np.arcsin(np.clip(yaw_ratio, -1.0, 1.0))))
|
||||
|
||||
# Pitch: the nose sits ~40% of the way down the eye→mouth axis on a
|
||||
# frontal face. Higher means the head is tilted back, lower means down.
|
||||
vertical = float(mouth_centre[1] - eye_centre[1])
|
||||
if abs(vertical) < 1e-6:
|
||||
return yaw, 0.0
|
||||
nose_ratio = float((nose[1] - eye_centre[1]) / vertical)
|
||||
pitch = float(np.degrees(np.arcsin(np.clip((0.40 - nose_ratio) * 2.0, -1.0, 1.0))))
|
||||
|
||||
return round(yaw, 2), round(pitch, 2)
|
||||
@@ -0,0 +1,41 @@
|
||||
"""
|
||||
Response models (#1074).
|
||||
|
||||
These are the wire contract the backend's `faceClient.js` codes against.
|
||||
Changing a field name here is a breaking change for a deployment mid-upgrade,
|
||||
where an old sidecar and a new backend run side by side for a few seconds.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class Face(BaseModel):
|
||||
# [x, y, w, h] in ORIGINAL image pixels — the backend crops cover-face
|
||||
# avatars from the same rendition, so these must not be detection-scaled.
|
||||
bbox: list[float] = Field(min_length=4, max_length=4)
|
||||
score: float
|
||||
# Five (x, y) pairs: subject's right eye, left eye, nose tip, right mouth
|
||||
# corner, left mouth corner.
|
||||
landmarks: list[list[float]]
|
||||
yaw: float
|
||||
pitch: float
|
||||
# Variance of the Laplacian on the aligned crop. Higher = sharper.
|
||||
blur: float
|
||||
# L2-normalized, `dim` floats (512 for FaceNet-512).
|
||||
embedding: list[float]
|
||||
|
||||
|
||||
class FacesResponse(BaseModel):
|
||||
model_version: str
|
||||
faces: list[Face]
|
||||
|
||||
|
||||
class InfoResponse(BaseModel):
|
||||
detector: str
|
||||
embedder: str
|
||||
model_version: str
|
||||
dim: int
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
@@ -0,0 +1,24 @@
|
||||
# picpeak-ml runtime dependencies (#1074).
|
||||
#
|
||||
# Pinned exactly. This image bakes in model weights and is meant to produce
|
||||
# byte-identical embeddings across rebuilds — a floating dependency that
|
||||
# changes how an image is decoded or resized would silently shift the
|
||||
# embedding space and invalidate every stored cluster.
|
||||
#
|
||||
# opencv-python-HEADLESS, not opencv-python: the GUI build pulls in X11/GTK
|
||||
# for highgui, which this service never calls and which is pure attack
|
||||
# surface in a container. Pinned to the 4.x line — cv2.FaceDetectorYN is
|
||||
# what loads YuNet, and 5.x reworks parts of that API.
|
||||
opencv-python-headless==4.14.0.94
|
||||
|
||||
# CPU execution provider only. Wheels exist for manylinux x86_64 AND
|
||||
# aarch64, so both legs of the multi-arch build install a prebuilt wheel
|
||||
# and neither compiles from source.
|
||||
onnxruntime==1.29.0
|
||||
|
||||
numpy==2.5.2
|
||||
fastapi==0.141.1
|
||||
uvicorn[standard]==0.52.3
|
||||
# Required by FastAPI to parse multipart/form-data — the only way images
|
||||
# reach this service.
|
||||
python-multipart==0.0.32
|
||||
@@ -0,0 +1,136 @@
|
||||
"""
|
||||
API contract tests (#1074).
|
||||
|
||||
The pipeline is stubbed out — these cover the auth boundary and the request
|
||||
guards, which is where a mistake is a security problem rather than an
|
||||
accuracy problem. Model behaviour is the spike's job, not a unit test's.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
TOKEN = "test-token-not-a-secret"
|
||||
|
||||
|
||||
class StubPipeline:
|
||||
dim = 512
|
||||
|
||||
def process(self, image_bytes: bytes):
|
||||
if image_bytes == b"undecodable":
|
||||
raise ValueError("Image could not be decoded")
|
||||
return [
|
||||
{
|
||||
"bbox": [10.0, 20.0, 30.0, 40.0],
|
||||
"score": 0.99,
|
||||
"landmarks": [[1.0, 2.0]] * 5,
|
||||
"yaw": 0.0,
|
||||
"pitch": 0.0,
|
||||
"blur": 123.4,
|
||||
"embedding": [0.1] * 512,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(monkeypatch):
|
||||
from app import config, main
|
||||
|
||||
monkeypatch.setattr(config, "TOKEN", TOKEN)
|
||||
monkeypatch.setattr(main.config, "TOKEN", TOKEN)
|
||||
monkeypatch.setattr(main, "FacePipeline", StubPipeline)
|
||||
|
||||
with TestClient(main.app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _image_file(data: bytes = b"fake-jpeg-bytes"):
|
||||
return {"image": ("photo.jpg", data, "image/jpeg")}
|
||||
|
||||
|
||||
class TestAuth:
|
||||
def test_health_needs_no_token(self, client):
|
||||
# The compose healthcheck calls this without a secret.
|
||||
r = client.get("/health")
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {"status": "ok"}
|
||||
|
||||
def test_info_without_token_is_401(self, client):
|
||||
assert client.get("/info").status_code == 401
|
||||
|
||||
def test_info_with_wrong_token_is_401(self, client):
|
||||
r = client.get("/info", headers={"X-Face-ML-Token": "wrong"})
|
||||
assert r.status_code == 401
|
||||
|
||||
def test_faces_without_token_is_401(self, client):
|
||||
r = client.post("/faces", files=_image_file())
|
||||
assert r.status_code == 401
|
||||
|
||||
def test_401_body_leaks_nothing(self, client):
|
||||
# A prober should not learn whether the header name was even right.
|
||||
r = client.post("/faces", files=_image_file())
|
||||
assert r.json() == {"detail": "Unauthorized"}
|
||||
|
||||
def test_startup_refuses_an_empty_token(self, monkeypatch):
|
||||
from app import config, main
|
||||
|
||||
monkeypatch.setattr(config, "TOKEN", "")
|
||||
monkeypatch.setattr(main.config, "TOKEN", "")
|
||||
monkeypatch.setattr(main, "FacePipeline", StubPipeline)
|
||||
|
||||
# Running open is the one failure mode this service must not have.
|
||||
with pytest.raises(RuntimeError, match="FACE_ML_TOKEN"):
|
||||
with TestClient(main.app):
|
||||
pass
|
||||
|
||||
|
||||
class TestInfo:
|
||||
def test_reports_the_model_identity_the_backend_stores(self, client):
|
||||
from app import config
|
||||
|
||||
r = client.get("/info", headers={"X-Face-ML-Token": TOKEN})
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["model_version"] == config.MODEL_VERSION
|
||||
assert body["dim"] == 512
|
||||
|
||||
|
||||
class TestFaces:
|
||||
def test_returns_faces_with_the_model_version(self, client):
|
||||
from app import config
|
||||
|
||||
r = client.post(
|
||||
"/faces", files=_image_file(), headers={"X-Face-ML-Token": TOKEN}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["model_version"] == config.MODEL_VERSION
|
||||
assert len(body["faces"]) == 1
|
||||
assert len(body["faces"][0]["embedding"]) == 512
|
||||
assert body["faces"][0]["bbox"] == [10.0, 20.0, 30.0, 40.0]
|
||||
|
||||
def test_empty_upload_is_400(self, client):
|
||||
r = client.post(
|
||||
"/faces", files=_image_file(b""), headers={"X-Face-ML-Token": TOKEN}
|
||||
)
|
||||
assert r.status_code == 400
|
||||
|
||||
def test_undecodable_image_is_400_not_500(self, client):
|
||||
# 4xx matters: the backend must mark the photo failed rather than
|
||||
# retry it forever, which is what it does for 5xx.
|
||||
r = client.post(
|
||||
"/faces",
|
||||
files=_image_file(b"undecodable"),
|
||||
headers={"X-Face-ML-Token": TOKEN},
|
||||
)
|
||||
assert r.status_code == 400
|
||||
|
||||
def test_oversize_upload_is_413(self, client, monkeypatch):
|
||||
from app import main
|
||||
|
||||
monkeypatch.setattr(main.config, "MAX_IMAGE_BYTES", 10)
|
||||
r = client.post(
|
||||
"/faces",
|
||||
files=_image_file(b"x" * 100),
|
||||
headers={"X-Face-ML-Token": TOKEN},
|
||||
)
|
||||
assert r.status_code == 413
|
||||
@@ -0,0 +1,156 @@
|
||||
"""
|
||||
Unit tests for the parts of the pipeline that don't need model weights.
|
||||
|
||||
The pose and blur helpers are pure functions over landmark geometry, and the
|
||||
landmark ORDER assumption is the one thing in this service that fails
|
||||
silently if it's wrong — a shuffled template still produces 512 confident
|
||||
floats, just from a face warped inside out. So it gets pinned here.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from app.pipeline import (
|
||||
_TEMPLATE_112,
|
||||
_blur_score,
|
||||
_pose_from_landmarks,
|
||||
_umeyama,
|
||||
)
|
||||
|
||||
|
||||
def _frontal_landmarks() -> np.ndarray:
|
||||
"""A synthetic, perfectly frontal face in YuNet's landmark order.
|
||||
|
||||
Order: subject's right eye, left eye, nose, right mouth, left mouth.
|
||||
The subject's right eye appears on the IMAGE-left, so it carries the
|
||||
smaller x — the same convention `_TEMPLATE_112` encodes.
|
||||
"""
|
||||
return np.array(
|
||||
[
|
||||
[40.0, 50.0], # right eye (image-left)
|
||||
[80.0, 50.0], # left eye
|
||||
[60.0, 70.0], # nose, centred between the eyes
|
||||
[45.0, 92.0], # right mouth
|
||||
[75.0, 92.0], # left mouth
|
||||
],
|
||||
dtype=np.float32,
|
||||
)
|
||||
|
||||
|
||||
class TestTemplate:
|
||||
def test_landmark_order_is_left_to_right_for_paired_features(self):
|
||||
# Eyes: index 0 must sit left of index 1. Mouth corners: 3 left of 4.
|
||||
assert _TEMPLATE_112[0][0] < _TEMPLATE_112[1][0]
|
||||
assert _TEMPLATE_112[3][0] < _TEMPLATE_112[4][0]
|
||||
|
||||
def test_nose_sits_between_the_eyes_horizontally(self):
|
||||
assert _TEMPLATE_112[0][0] < _TEMPLATE_112[2][0] < _TEMPLATE_112[1][0]
|
||||
|
||||
def test_features_are_vertically_ordered_eyes_nose_mouth(self):
|
||||
eye_y = (_TEMPLATE_112[0][1] + _TEMPLATE_112[1][1]) / 2
|
||||
mouth_y = (_TEMPLATE_112[3][1] + _TEMPLATE_112[4][1]) / 2
|
||||
assert eye_y < _TEMPLATE_112[2][1] < mouth_y
|
||||
|
||||
|
||||
class TestUmeyama:
|
||||
"""The alignment estimator. Every test here is a regression guard.
|
||||
|
||||
An estimator that fits three of the five landmarks perfectly and lets the
|
||||
mouth drift still produces a face-shaped crop and 512 confident floats —
|
||||
it just degrades every embedding. That is why this is pinned numerically
|
||||
rather than eyeballed.
|
||||
"""
|
||||
|
||||
def test_recovers_a_known_similarity_transform_exactly(self):
|
||||
src = _frontal_landmarks()
|
||||
angle = np.radians(20.0)
|
||||
rot = np.array(
|
||||
[[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]]
|
||||
)
|
||||
dst = (2.5 * (src @ rot.T)) + np.array([17.0, -9.0])
|
||||
|
||||
m = _umeyama(src, dst.astype(np.float32))
|
||||
projected = (src @ m[:, :2].T) + m[:, 2]
|
||||
assert np.allclose(projected, dst, atol=1e-3)
|
||||
|
||||
def test_distributes_residual_across_all_five_points(self):
|
||||
# A face whose eye-to-mouth proportion differs from the template —
|
||||
# no similarity transform can satisfy all five, so the question is
|
||||
# how the error is spread. An outlier-rejecting estimator (RANSAC,
|
||||
# LMEDS) parks it all on the mouth; least squares shares it out.
|
||||
src = _frontal_landmarks()
|
||||
src[3][1] = 110.0 # mouth further from the eyes than the template
|
||||
src[4][1] = 110.0
|
||||
|
||||
m = _umeyama(src, _TEMPLATE_112)
|
||||
residual = np.linalg.norm((src @ m[:, :2].T) + m[:, 2] - _TEMPLATE_112, axis=1)
|
||||
|
||||
assert residual.max() > 0, "expected an imperfect fit for this input"
|
||||
# No single point may absorb the bulk of the error.
|
||||
assert residual.max() < 3.0 * residual.mean()
|
||||
|
||||
def test_never_returns_a_reflection(self):
|
||||
# A mirrored warp yields a confident embedding of a face that does
|
||||
# not exist, which would cluster as a separate person.
|
||||
m = _umeyama(_frontal_landmarks(), _TEMPLATE_112)
|
||||
assert np.linalg.det(m[:, :2]) > 0
|
||||
|
||||
def test_is_deterministic(self):
|
||||
# Re-scans must not churn clusters.
|
||||
src = _frontal_landmarks()
|
||||
first = _umeyama(src, _TEMPLATE_112)
|
||||
for _ in range(5):
|
||||
assert np.array_equal(_umeyama(src, _TEMPLATE_112), first)
|
||||
|
||||
def test_coincident_points_return_none_rather_than_dividing_by_zero(self):
|
||||
assert _umeyama(np.zeros((5, 2), dtype=np.float32), _TEMPLATE_112) is None
|
||||
|
||||
|
||||
class TestPose:
|
||||
def test_frontal_face_is_near_zero_yaw(self):
|
||||
yaw, _ = _pose_from_landmarks(_frontal_landmarks())
|
||||
assert abs(yaw) < 1.0
|
||||
|
||||
def test_nose_toward_subject_left_eye_gives_positive_yaw(self):
|
||||
lm = _frontal_landmarks()
|
||||
lm[2][0] = 75.0 # nose slides toward the image-right (subject's left)
|
||||
yaw, _ = _pose_from_landmarks(lm)
|
||||
assert yaw > 10.0
|
||||
|
||||
def test_nose_toward_subject_right_eye_gives_negative_yaw(self):
|
||||
lm = _frontal_landmarks()
|
||||
lm[2][0] = 45.0
|
||||
yaw, _ = _pose_from_landmarks(lm)
|
||||
assert yaw < -10.0
|
||||
|
||||
def test_yaw_is_scale_invariant(self):
|
||||
lm = _frontal_landmarks()
|
||||
lm[2][0] = 72.0
|
||||
small, _ = _pose_from_landmarks(lm)
|
||||
large, _ = _pose_from_landmarks(lm * 4.0)
|
||||
assert small == pytest.approx(large, abs=0.01)
|
||||
|
||||
def test_nose_low_on_the_eye_mouth_axis_reads_as_looking_down(self):
|
||||
lm = _frontal_landmarks()
|
||||
lm[2][1] = 85.0 # nose drops toward the mouth
|
||||
_, pitch = _pose_from_landmarks(lm)
|
||||
assert pitch < 0
|
||||
|
||||
def test_degenerate_landmarks_do_not_raise(self):
|
||||
flat = np.zeros((5, 2), dtype=np.float32)
|
||||
assert _pose_from_landmarks(flat) == (0.0, 0.0)
|
||||
|
||||
|
||||
class TestBlur:
|
||||
def test_flat_image_scores_near_zero(self):
|
||||
flat = np.full((160, 160, 3), 128, dtype=np.uint8)
|
||||
assert _blur_score(flat) < 1.0
|
||||
|
||||
def test_sharp_edges_score_higher_than_a_blurred_copy(self):
|
||||
import cv2
|
||||
|
||||
sharp = np.zeros((160, 160, 3), dtype=np.uint8)
|
||||
sharp[:, ::8] = 255 # high-frequency vertical stripes
|
||||
blurred = cv2.GaussianBlur(sharp, (15, 15), 0)
|
||||
|
||||
assert _blur_score(sharp) > _blur_score(blurred)
|
||||
@@ -0,0 +1,113 @@
|
||||
"""
|
||||
Phase 0 spike (#1074): does our pipeline separate different people?
|
||||
|
||||
Runs LFW's standard 1000-pair test protocol (500 same, 500 different) through
|
||||
the PRODUCTION FacePipeline — YuNet detection, our Umeyama alignment, our
|
||||
per-image standardization, FaceNet-512 ONNX — and reports the cosine
|
||||
distribution, the best threshold, and the error rates that matter for
|
||||
CLUSTERING specifically.
|
||||
|
||||
Clustering is not verification. For verification a false accept and a false
|
||||
reject cost the same. For clustering they do not: a false merge puts a
|
||||
stranger into someone's "download my photos", while a false split just makes
|
||||
a duplicate row in the strip that the photographer can merge away. So the
|
||||
operating point is chosen to hold false merges low, not to maximise accuracy.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import numpy as np
|
||||
import cv2
|
||||
|
||||
# Run from the ml/ directory with FACE_MODEL_DIR pointing at a directory
|
||||
# holding face_detection_yunet_2023mar.onnx and facenet512.onnx:
|
||||
#
|
||||
# pip install -r requirements.txt scikit-learn
|
||||
# FACE_MODEL_DIR=/path/to/models python tools/benchmark_threshold.py
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ.setdefault('FACE_MODEL_DIR', '/models')
|
||||
|
||||
from app.pipeline import FacePipeline # noqa: E402
|
||||
from sklearn.datasets import fetch_lfw_pairs # noqa: E402
|
||||
|
||||
print('Loading LFW test pairs…')
|
||||
data = fetch_lfw_pairs(subset='test', color=True, resize=1.0,
|
||||
slice_=(slice(0, 250), slice(0, 250)), funneled=True)
|
||||
pairs, labels = data.pairs, data.target
|
||||
|
||||
print('Loading pipeline…')
|
||||
pipe = FacePipeline()
|
||||
|
||||
def embed(arr):
|
||||
"""sklearn hands back float32 RGB normalized to [0, 1] — NOT 0..255.
|
||||
Casting straight to uint8 produces a black frame and zero detections."""
|
||||
rgb8 = np.clip(arr * 255.0, 0, 255).astype(np.uint8)
|
||||
bgr = cv2.cvtColor(rgb8, cv2.COLOR_RGB2BGR)
|
||||
ok, buf = cv2.imencode('.jpg', bgr, [cv2.IMWRITE_JPEG_QUALITY, 95])
|
||||
if not ok:
|
||||
return None
|
||||
faces = pipe.process(buf.tobytes())
|
||||
if not faces:
|
||||
return None
|
||||
# Largest detection — LFW is one centred subject per frame.
|
||||
best = max(faces, key=lambda f: f['bbox'][2] * f['bbox'][3])
|
||||
return np.asarray(best['embedding'], dtype=np.float32)
|
||||
|
||||
sims, kept, missed = [], [], 0
|
||||
for i, (a, b) in enumerate(pairs):
|
||||
ea, eb = embed(a), embed(b)
|
||||
if ea is None or eb is None:
|
||||
missed += 1
|
||||
continue
|
||||
sims.append(float(ea @ eb))
|
||||
kept.append(int(labels[i]))
|
||||
if (i + 1) % 100 == 0:
|
||||
print(f' {i+1}/{len(pairs)}…', flush=True)
|
||||
|
||||
sims = np.array(sims)
|
||||
kept = np.array(kept)
|
||||
same, diff = sims[kept == 1], sims[kept == 0]
|
||||
|
||||
print()
|
||||
print('=' * 66)
|
||||
print(f'Pairs evaluated : {len(sims)} of {len(pairs)} '
|
||||
f'({missed} skipped — no face detected in one or both)')
|
||||
print(f'Detection rate : {1 - missed/len(pairs):.1%}')
|
||||
print()
|
||||
print(f'SAME person cosine: mean {same.mean():.4f} sd {same.std():.4f} '
|
||||
f'p5 {np.percentile(same,5):.4f} min {same.min():.4f}')
|
||||
print(f'DIFF person cosine: mean {diff.mean():.4f} sd {diff.std():.4f} '
|
||||
f'p95 {np.percentile(diff,95):.4f} max {diff.max():.4f}')
|
||||
print(f'Separation (mean gap): {same.mean() - diff.mean():.4f}')
|
||||
|
||||
# Sweep thresholds.
|
||||
grid = np.linspace(0.0, 1.0, 1001)
|
||||
acc = [( (same >= t).sum() + (diff < t).sum() ) / len(sims) for t in grid]
|
||||
best_i = int(np.argmax(acc))
|
||||
best_t, best_acc = grid[best_i], acc[best_i]
|
||||
|
||||
print()
|
||||
print(f'Best accuracy : {best_acc:.2%} at threshold {best_t:.3f}')
|
||||
|
||||
# The clustering-relevant operating points: pick the threshold where the
|
||||
# false-MERGE rate (different people scored as the same) is capped.
|
||||
print()
|
||||
print('Operating points (false merge = different people judged the same):')
|
||||
print(f' {"thresh":>7} {"false merge":>11} {"false split":>11} {"accuracy":>8}')
|
||||
for target in (0.10, 0.05, 0.02, 0.01):
|
||||
t = float(np.quantile(diff, 1 - target))
|
||||
fm = (diff >= t).mean()
|
||||
fs = (same < t).mean()
|
||||
a = ((same >= t).sum() + (diff < t).sum()) / len(sims)
|
||||
print(f' {t:7.3f} {fm:10.1%} {fs:10.1%} {a:7.1%} (target {target:.0%})')
|
||||
|
||||
print()
|
||||
print(f'Currently seeded default: 0.620 → '
|
||||
f'false merge {(diff >= 0.62).mean():.1%}, '
|
||||
f'false split {(same < 0.62).mean():.1%}, '
|
||||
f'accuracy {(((same >= 0.62).sum() + (diff < 0.62).sum()) / len(sims)):.1%}')
|
||||
print('=' * 66)
|
||||
|
||||
# Last run (2026-08-18), 1000/1000 pairs, 100% detection:
|
||||
# same 0.6958 +/- 0.1415 | diff 0.0849 +/- 0.1674 | separation 0.6109
|
||||
# peak accuracy 96.60% @ 0.405
|
||||
# shipped default 0.50 -> 1.0% false merge, 8.2% false split, 95.4% accuracy
|
||||
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Produce `facenet512.onnx` from deepface's published Keras weights (#1074).
|
||||
|
||||
Run this ONCE, by hand. Publish the resulting file as a release asset and
|
||||
pass its URL + SHA-256 to the image build. It is deliberately not part of the
|
||||
Docker build:
|
||||
|
||||
1. TensorFlow is ~600MB of build dependency for a file that never ships in
|
||||
the final image.
|
||||
2. The result is architecture-independent, so converting once beats
|
||||
converting on both legs of every multi-arch build.
|
||||
|
||||
NOT byte-reproducible. Two runs on the same machine with the same pinned
|
||||
versions produce functionally identical graphs — same 336 nodes, same 271
|
||||
initializers, weights matching to 0.000e+00 — but a handful of initializer
|
||||
names differ (tf2onnx's traced-op naming is not deterministic), so the file
|
||||
bytes and therefore the SHA-256 differ. Measured, not assumed.
|
||||
|
||||
The consequence for anyone re-running this: **your checksum will not match
|
||||
the published one, and that is expected — it is not evidence of tampering.**
|
||||
The SHA-256 in the image build pins one specific published artifact so that
|
||||
URL cannot start serving different bytes. To validate a fresh conversion,
|
||||
rely on the parity check below (which compares against the Keras original),
|
||||
not on reproducing a hash.
|
||||
|
||||
Why we may redistribute at all: deepface ships FaceNet-512 under MIT. That
|
||||
was the deciding factor over the more accurate InsightFace weights, which are
|
||||
non-commercial only — see ml/LICENSES.md and #1074 §1.
|
||||
|
||||
Usage
|
||||
-----
|
||||
python3.11 -m venv .venv && . .venv/bin/activate # 3.11: TF has no 3.12+ wheels
|
||||
pip install -r tools/requirements-convert.txt
|
||||
|
||||
curl -fsSL -o facenet512_weights.h5 \\
|
||||
https://github.com/serengil/deepface_models/releases/download/v1.0/facenet512_weights.h5
|
||||
echo "3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f facenet512_weights.h5" \\
|
||||
| sha256sum -c -
|
||||
|
||||
python tools/convert_facenet.py facenet512_weights.h5 facenet512.onnx
|
||||
|
||||
The script verifies the converted graph against the Keras original before it
|
||||
writes anything permanent, then prints the SHA-256 to publish alongside it.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Input geometry of deepface's FaceNet-512. pipeline.py reads this off the
|
||||
# model at runtime rather than assuming it, but this is what it will find.
|
||||
INPUT_SHAPE = (None, 160, 160, 3)
|
||||
EMBEDDING_DIM = 512
|
||||
|
||||
# A converted graph that is subtly wrong still returns 512 plausible floats,
|
||||
# so parity is checked rather than assumed. Tolerance is float32 noise: the
|
||||
# observed worst case over random inputs was 2.1e-06 absolute, cosine
|
||||
# 1.0000000000.
|
||||
PARITY_SAMPLES = 3
|
||||
MIN_COSINE = 0.99999
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with path.open("rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(1024 * 1024), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("weights", type=Path, help="facenet512_weights.h5")
|
||||
parser.add_argument("output", type=Path, help="destination .onnx")
|
||||
parser.add_argument(
|
||||
"--opset",
|
||||
type=int,
|
||||
default=17,
|
||||
help="ONNX opset (default 17 — supported by onnxruntime 1.29)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-verify",
|
||||
action="store_true",
|
||||
help="skip the Keras/ONNX parity check (not recommended)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.weights.is_file():
|
||||
print(f"error: {args.weights} not found", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# Imported late so `--help` works without a TensorFlow install.
|
||||
import numpy as np
|
||||
import tensorflow as tf
|
||||
import tf2onnx
|
||||
from deepface.models.facial_recognition.Facenet import InceptionResNetV1
|
||||
|
||||
# Build the architecture and load the checksummed file from disk rather
|
||||
# than going through deepface's own loader — that one downloads the
|
||||
# weights itself, which would defeat the point of pinning them.
|
||||
print("Building InceptionResNetV1(dimension=512)…")
|
||||
model = InceptionResNetV1(dimension=EMBEDDING_DIM)
|
||||
model.load_weights(str(args.weights))
|
||||
print(f" {model.count_params():,} parameters")
|
||||
|
||||
print(f"Converting to ONNX (opset {args.opset})…")
|
||||
spec = (tf.TensorSpec(INPUT_SHAPE, tf.float32, name="input"),)
|
||||
tf2onnx.convert.from_keras(
|
||||
model, input_signature=spec, opset=args.opset, output_path=str(args.output)
|
||||
)
|
||||
|
||||
if not args.skip_verify:
|
||||
import onnxruntime as ort
|
||||
|
||||
print("Verifying ONNX output matches Keras…")
|
||||
sess = ort.InferenceSession(
|
||||
str(args.output), providers=["CPUExecutionProvider"]
|
||||
)
|
||||
name = sess.get_inputs()[0].name
|
||||
rng = np.random.default_rng(0)
|
||||
worst_cosine, worst_abs = 1.0, 0.0
|
||||
|
||||
for _ in range(PARITY_SAMPLES):
|
||||
x = rng.standard_normal((1, *INPUT_SHAPE[1:])).astype("float32")
|
||||
keras_out = model.predict(x, verbose=0)[0]
|
||||
onnx_out = sess.run(None, {name: x})[0][0]
|
||||
|
||||
worst_abs = max(worst_abs, float(np.abs(keras_out - onnx_out).max()))
|
||||
cosine = float(
|
||||
(keras_out / np.linalg.norm(keras_out))
|
||||
@ (onnx_out / np.linalg.norm(onnx_out))
|
||||
)
|
||||
worst_cosine = min(worst_cosine, cosine)
|
||||
|
||||
print(f" worst abs diff {worst_abs:.3e}, worst cosine {worst_cosine:.10f}")
|
||||
if worst_cosine < MIN_COSINE:
|
||||
print(
|
||||
f"error: parity check FAILED (cosine {worst_cosine} < {MIN_COSINE}). "
|
||||
"The converted graph does not match the original — do not publish it.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
args.output.unlink(missing_ok=True)
|
||||
return 1
|
||||
|
||||
size_mb = args.output.stat().st_size / (1024 * 1024)
|
||||
print()
|
||||
print(f"Wrote {args.output} ({size_mb:.1f} MB)")
|
||||
print(f"SHA-256: {_sha256(args.output)}")
|
||||
print()
|
||||
print("Publish it as a release asset, then set the repository variables")
|
||||
print("FACENET_ONNX_URL and FACENET_ONNX_SHA256 (Settings → Variables).")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,22 @@
|
||||
# Dependencies for tools/convert_facenet.py ONLY. These never enter the
|
||||
# picpeak-ml image — the conversion runs once, by hand, and only its output
|
||||
# (facenet512.onnx) is shipped.
|
||||
#
|
||||
# Pinned to the exact set that produced the published artifact, so anyone can
|
||||
# reproduce it and get the same SHA-256 rather than a file that differs for
|
||||
# reasons nobody can reconstruct later. See ml/README.md for the expected
|
||||
# checksum.
|
||||
#
|
||||
# Requires Python 3.11 (TensorFlow has no 3.12+/3.14 wheels at these
|
||||
# versions). The picpeak-ml image itself runs 3.12 — the two are unrelated,
|
||||
# since nothing from this file ships.
|
||||
tensorflow==2.21.0
|
||||
tf2onnx==1.17.0
|
||||
onnx==1.22.0
|
||||
deepface==0.0.100
|
||||
|
||||
# Transitive, pinned because the conversion output is checksummed:
|
||||
# keras 3.15 builds the graph, protobuf serializes the ONNX.
|
||||
keras==3.15.1
|
||||
protobuf==7.35.1
|
||||
numpy==2.4.6
|
||||
Reference in New Issue
Block a user