Paul Nothaft
198df02d2a
* test(ml): add an embedding-space fingerprint tool (#1084) requirements.txt is pinned exactly so rebuilds produce byte-identical embeddings, but nothing verified that. The API tests stub FacePipeline, so a decode/resize/kernel change could move every stored cluster without failing anything. This prints a hash per stage — decode, resize, cvtColor, YuNet detect, FaceNet forward pass — so the old and new image can be diffed on the same host before any base-image or dependency bump lands. Used it to answer the open question in #1084: Debian/Python 3.12 and Wolfi/Python 3.14 produce identical hashes at every stage, so a base swap would not invalidate stored clusters. The input is generated rather than a fixture, and the hashes are deliberately not compared across architectures — OpenCV and onnxruntime dispatch different SIMD kernels on x86 and aarch64, so this answers "did this change move the numbers", not "is every platform identical". * test(ml): fingerprint the production path, not a parallel one External review found the first cut was largely theatre. The documented command could not run: tools/ is in .dockerignore and the Dockerfile copies only app/, so the script is never inside the image. It has to be mounted — which is what I actually did when producing the numbers, while documenting something else. Three stages were fingerprinting the wrong thing: - The detector recorded "none" plus a return status, because synthetic input has no face to find. It would have stayed green through any change to YuNet or its kernels. Now the ONNX graph is driven directly, so all twelve output heads always produce numbers, and the reported thresholds are the service's (0.6/0.3) rather than FaceDetectorYN's 0.9 default. - The embedding used a hand-rolled tensor, bypassing everything that actually places a face in the embedding space: umeyama + warpAffine, BGR->RGB, per-image standardization, layout, and the L2 normalization the backend's cosine similarity depends on. It now calls _align and _embed directly. Private, deliberately — reimplementing the maths here would drift from pipeline.py and fingerprint a path nothing runs. - Decode exercised PNG, but the worker only ever receives the preview rendition, which imageProcessor.js writes as JPEG. Now a fixed JPEG, embedded as bytes so the input cannot depend on the encoder version being held still. Verified SOI/EOI-clean; the first attempt at this produced "Corrupt JPEG data: 22 extraneous bytes". Sensitivity checked rather than assumed: a one-pixel landmark nudge moves align_warp and embed and leaves decode and the detector heads alone, which is exactly the dependency structure expected. Debian/Python 3.12 vs Wolfi/Python 3.14 remain identical across all sixteen stages, so the #1084 parity conclusion still holds under the stronger check. * test(ml): measure the image's own pipeline, and the detector OpenCV runs Two more from external review, both of which let matching hashes mean less than they claimed. The documented bind mount put the checkout's app/ ahead of the image's /app/app, so comparing two images built from different revisions would have executed the same pipeline source twice and reported a match no matter how the images differed. /app now wins whenever it exists, so the tool measures the image under test however it is invoked, and the loaded path is printed as _app_source so that is auditable rather than assumed. The detector was fingerprinted through onnxruntime, but production runs cv2.FaceDetectorYN — OpenCV's own preprocessing, DNN engine and NMS/landmark decode, none of which ORT touches. An OpenCV upgrade could therefore move real landmarks, and with them alignment and embeddings, while every detector hash held still. It now runs the OpenCV path too, with the score threshold at the floor so synthetic input still yields candidates (594 here) instead of the empty result the production 0.6 gives on an image with no face. The ORT pass is kept alongside it to separate a model change from an OpenCV change. Parity across debian/3.12 and wolfi/3.14 still holds across all 19 stages, and a one-pixel landmark nudge still moves align_warp and embed and nothing else. * test(ml): cover the orchestration and progressive decode too Round three of external review found two more ways the hashes could match while production moved. The isolated stages never fed the detector's output into alignment — _align got fixed landmarks — so INPUT_LONG_EDGE resizing and the row -> landmark scaling in _one_face were invisible. process() now runs end to end on the fixture, with the pipeline's own detector threshold dropped so a faceless frame still yields rows to carry through (26 faces here). A first attempt at that still missed the resize: the embedded fixture is 48px, so `long_edge > INPUT_LONG_EDGE` never fired and changing 1920 to 960 moved nothing. It now runs a second pass with the threshold lowered under the fixture, which executes the same downscale and inverse landmark scaling without carrying a 1920px image in the source. Verified sensitive: moving that bound 32 -> 24 changes both the face count and the embedding. The fixture was also a baseline JPEG, while generatePreview writes progressive (imageProcessor.js:236/480/617) — a different path through libjpeg. Swapped for a progressive fixture, SOF2 confirmed present and SOF0 absent. 24 stages now. Debian/3.12 and Wolfi/3.14 remain identical across all of them. * test(ml): close three more false-negative paths in the fingerprint Round four of external review. All three let hashes match while production moved. INPUT_LONG_EDGE was used but never printed. The fixture is too small to trip the resize in either image, and the forced pass overrides the value in both, so a production change from 1920 to 960 moved no hash at all. It is now emitted alongside the other thresholds, where a reviewer sees it in the diff. The fixture was square, so a width/height swap in setInputSize or the resize produced identical dimensions and identical hashes. It is now 64x48. The forced-downscale pass hashed only an embedding, which is derived from separately scaled landmarks — a regression in the inverse scaling of row[0:4] would have shown up nowhere, because the normal pass runs at scale 1. That bbox is now hashed too; a wrong one is what breaks avatar crops and area calculations. Changing the fixture to 64x48 also broke the forced pass: at the old bound of 32 the downscaled frame is 32x24 and YuNet returns nothing, so the stage pinned nothing. The NO-DETECTIONS-STAGE-VACUOUS marker added last round caught it immediately rather than printing a reassuring hash of an empty result. Bound moved to 48, which still triggers the resize and still yields rows. 25 stages, no vacuous markers. Debian/3.12 and Wolfi/3.14 identical across all of them. * test(ml): hash every detection, not just the first Round five of external review. Both end-to-end passes hashed only candidate 0, so a change that moved candidates 1..n — or merely reordered them — matched as long as the count and the first candidate held. With the threshold at the floor those passes return 24 and 27 candidates, so that was most of the evidence being thrown away. Both now stack every returned face, in order, via a shared _hash_all. Stacking preserves order, so a reshuffle is caught too. Verified against the exact case: reversing candidates 1..n while leaving the count and candidate 0 untouched now moves process_embedding and process_bbox. Before this it moved nothing. * test(ml): hash every persisted field, and emit the model version Round six of external review, plus the adjacent gaps it implied. Two findings: MODEL_VERSION was never emitted, and _hash_all discarded score. Both matter to the backend rather than to the numbers — a model_version change makes faceClustering.js:190 refuse to compare new faces against existing people, forcing a rescan, and det_score decides via meetsQualityFloor (faceClustering.js:96-100) whether a face joins clustering at all. Either could change while every hash held still. Rather than fix only the two named, I checked what faceProcessor.js actually stores per face (:157-167) and covered all of it: bbox, score, yaw, pitch, blur, embedding. yaw/pitch/blur were heading for the same finding next round. One hash per field, so a diff says which thing moved rather than only that something did. model_version is emitted as a compatibility key alongside the thresholds, not hashed — it is a string, and its job is to be read. Verified: scaling score alone by 0.999 now moves process_score and nothing else. 33 stages, no vacuous markers, debian/3.12 and wolfi/3.14 still identical. * test(ml): split verdict from diagnostic, and stop masking the threshold Round seven of external review. The ORT detector hashes were being read as part of the compatibility verdict, but production never runs YuNet through onnxruntime. An ORT change touching a YuNet operator would have moved them while real behaviour was untouched, and the docstring said any difference means re-scan — so the tool could have ordered a full-gallery rescan for nothing. They are now diag_-prefixed, and the docstring states which keys carry a verdict, which are diagnostic, and which are metadata a reviewer has to read rather than diff. setScoreThreshold(1e-6) also overwrote the detector's real threshold before anything recorded it, and _thresholds.det_score only echoes config. If FacePipeline ever stopped applying DET_SCORE_THRESHOLD — falling back to OpenCV's 0.9 default — production would detect a different face set while every hash matched. The constructed value is now read first and emitted as _effective_det_score; simulating the regression makes it read 0.9 instead of 0.6. MAX_FACES is emitted for the same reason INPUT_LONG_EDGE is: the fixture never reaches the pipeline.py:138 slice, so 64 -> 128 would move no hash while real group photos persisted a different face set. 21 verdict keys, 12 diagnostic, no vacuous markers, debian/3.12 and wolfi/3.14 still identical across both sets. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
picpeak-ml
Optional face-detection sidecar for PicPeak (#1074). Detects faces in one image and returns a bounding box, five landmarks, quality signals and a 512-d embedding per face.
Nothing else. No database, no volumes, no state, no egress, no model download at runtime. Clustering, person identity, thresholds and every privacy decision live in the PicPeak backend, where the data already is. This service forgets each image the moment it answers.
If you don't run this container, the feature does not exist — see "Turning it on" below.
API
All endpoints except /health require the X-Face-ML-Token header. The
service refuses to start without FACE_ML_TOKEN set, so an accidentally
published port is never a free face-detection API.
GET /health |
{"status": "ok"} — unauthenticated, used by the compose healthcheck |
GET /info |
{detector, embedder, model_version, dim} |
POST /faces |
multipart image → {model_version, faces: [...]} |
Each face:
{
"bbox": [x, y, w, h], // ORIGINAL image pixels, not detection-scaled
"score": 0.94,
"landmarks": [[x, y], ...], // 5: right eye, left eye, nose, right mouth, left mouth
"yaw": -1.42, // degrees, approximate (see pipeline.py)
"pitch": -25.33,
"blur": 2579.5, // variance of Laplacian on the aligned crop; higher = sharper
"embedding": [...] // 512 floats, L2-normalized
}
404/400 mean "this image is a lost cause" — the backend marks the photo
failed. 5xx and connection failures mean "try later" — the backend returns
the photo to pending with backoff, so turning this container off for a week
does not require a manual re-scan.
Models
YuNet (detection, MIT) + FaceNet-512 (embedding, MIT), both baked into the image and verified by SHA-256 at build time. See LICENSES.md for why these two and not the more obvious InsightFace weights — the short version is that InsightFace's are non-commercial-only and PicPeak's users are working photographers.
Building the image
facenet512.onnx is not fetched automatically, because deepface
distributes FaceNet-512 as Keras .h5 only. Convert it once, publish it,
then pass the URL and checksum:
cd ml
python3.11 -m venv .venv && . .venv/bin/activate # 3.11: TF has no 3.12+ wheels
pip install -r tools/requirements-convert.txt
curl -fsSL -o facenet512_weights.h5 \
https://github.com/serengil/deepface_models/releases/download/v1.0/facenet512_weights.h5
echo "3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f facenet512_weights.h5" | sha256sum -c -
python tools/convert_facenet.py facenet512_weights.h5 facenet512.onnx
The script verifies the converted graph against the Keras original before writing (worst observed divergence: 2.1e-06 absolute, cosine 1.0000000000) and prints the SHA-256 to publish. Output is ~89.6 MB, 23,497,424 parameters.
Publish facenet512.onnx as a release asset, set the repository variables
FACENET_ONNX_URL and FACENET_ONNX_SHA256 (Settings → Variables — it's a
public URL, not a secret), and CI picks it up. To build locally:
docker build -t picpeak-ml \
--build-arg FACENET_ONNX_URL=https://github.com/PicPeak/picpeak/releases/download/<tag>/facenet512.onnx \
--build-arg FACENET_ONNX_SHA256=<sha256> \
ml/
The conversion sits outside the Docker build because TensorFlow is ~600MB of build dependency for a file that never ships in the final image, and the result is architecture-independent — no reason to run it on both legs of every multi-arch build.
The conversion is not byte-reproducible. Two runs with the same pinned versions on the same machine produce functionally identical graphs (same 336 nodes, same 271 initializers, weights matching to 0.000e+00) but differ in a few initializer names, because tf2onnx's traced-op naming is not deterministic. So a re-conversion will have a different SHA-256, and that is expected rather than a sign of tampering. The checksum pins one published artifact so its URL cannot start serving different bytes; validating a fresh conversion is the parity check's job, not the hash's.
Not available on the all-in-one image
The single-container image (Dockerfile.aio) sets
PICPEAK_SINGLE_CONTAINER=true, and the backend refuses to enable face
recognition when it sees that — the feature flag cannot be switched on, and
per-event detection stays off even if a restored database says otherwise.
This is a performance decision, not a licensing or packaging one. That image runs the backend, the frontend, SQLite and every background worker inside one container aimed at "one photographer plus guests browsing". It has no Redis, SQLite gives it a single writer, and it contains no ML sidecar to talk to. Adding a second image-processing pipeline that competes with Sharp for the same CPU and RAM would not fail loudly — it would just make the whole install slow and appear broken.
Run the standard multi-container deployment if you want this feature.
Turning it on
Two deliberate actions, neither of which is installing this container:
- Enable the
facesfeature flag in PicPeak's admin settings. - Enable "Detect people in this gallery" per event.
FACE_ML_URL defaults to http://picpeak-ml:8000 — the compose service name
— so the standard deployment needs no URL configuration. Nothing in the
backend touches that URL while the flag is off, so an install without this
container never attempts a connection.
Development
pip install -r requirements.txt pytest httpx
python -m pytest tests/ -q
The tests stub the models out: they cover the auth boundary, the request guards and the alignment geometry — the places where a mistake is a security problem or a silent accuracy problem. Model quality is not a unit-test question; that is what the Phase 0 spike measured.
The one thing to be careful about
The alignment in pipeline.py and the normalization in _embed must stay
identical to whatever the clustering threshold was tuned against. A tuned
cosine threshold does not transfer across an alignment change. If either
changes, bump MODEL_VERSION in config.py — the backend keys
re-derivation off that string and will re-cluster rather than silently mix
two incompatible embedding spaces.