* test(ml): add an embedding-space fingerprint tool (#1084)
requirements.txt is pinned exactly so rebuilds produce byte-identical
embeddings, but nothing verified that. The API tests stub FacePipeline, so
a decode/resize/kernel change could move every stored cluster without
failing anything.
This prints a hash per stage — decode, resize, cvtColor, YuNet detect,
FaceNet forward pass — so the old and new image can be diffed on the same
host before any base-image or dependency bump lands.
Used it to answer the open question in #1084: Debian/Python 3.12 and
Wolfi/Python 3.14 produce identical hashes at every stage, so a base swap
would not invalidate stored clusters. The input is generated rather than
a fixture, and the hashes are deliberately not compared across
architectures — OpenCV and onnxruntime dispatch different SIMD kernels on
x86 and aarch64, so this answers "did this change move the numbers", not
"is every platform identical".
* test(ml): fingerprint the production path, not a parallel one
External review found the first cut was largely theatre.
The documented command could not run: tools/ is in .dockerignore and the
Dockerfile copies only app/, so the script is never inside the image. It
has to be mounted — which is what I actually did when producing the
numbers, while documenting something else.
Three stages were fingerprinting the wrong thing:
- The detector recorded "none" plus a return status, because synthetic
input has no face to find. It would have stayed green through any
change to YuNet or its kernels. Now the ONNX graph is driven directly,
so all twelve output heads always produce numbers, and the reported
thresholds are the service's (0.6/0.3) rather than FaceDetectorYN's
0.9 default.
- The embedding used a hand-rolled tensor, bypassing everything that
actually places a face in the embedding space: umeyama + warpAffine,
BGR->RGB, per-image standardization, layout, and the L2 normalization
the backend's cosine similarity depends on. It now calls _align and
_embed directly. Private, deliberately — reimplementing the maths here
would drift from pipeline.py and fingerprint a path nothing runs.
- Decode exercised PNG, but the worker only ever receives the preview
rendition, which imageProcessor.js writes as JPEG. Now a fixed JPEG,
embedded as bytes so the input cannot depend on the encoder version
being held still. Verified SOI/EOI-clean; the first attempt at this
produced "Corrupt JPEG data: 22 extraneous bytes".
Sensitivity checked rather than assumed: a one-pixel landmark nudge
moves align_warp and embed and leaves decode and the detector heads
alone, which is exactly the dependency structure expected.
Debian/Python 3.12 vs Wolfi/Python 3.14 remain identical across all
sixteen stages, so the #1084 parity conclusion still holds under the
stronger check.
* test(ml): measure the image's own pipeline, and the detector OpenCV runs
Two more from external review, both of which let matching hashes mean
less than they claimed.
The documented bind mount put the checkout's app/ ahead of the image's
/app/app, so comparing two images built from different revisions would
have executed the same pipeline source twice and reported a match no
matter how the images differed. /app now wins whenever it exists, so the
tool measures the image under test however it is invoked, and the loaded
path is printed as _app_source so that is auditable rather than assumed.
The detector was fingerprinted through onnxruntime, but production runs
cv2.FaceDetectorYN — OpenCV's own preprocessing, DNN engine and
NMS/landmark decode, none of which ORT touches. An OpenCV upgrade could
therefore move real landmarks, and with them alignment and embeddings,
while every detector hash held still. It now runs the OpenCV path too,
with the score threshold at the floor so synthetic input still yields
candidates (594 here) instead of the empty result the production 0.6
gives on an image with no face. The ORT pass is kept alongside it to
separate a model change from an OpenCV change.
Parity across debian/3.12 and wolfi/3.14 still holds across all 19
stages, and a one-pixel landmark nudge still moves align_warp and embed
and nothing else.
* test(ml): cover the orchestration and progressive decode too
Round three of external review found two more ways the hashes could
match while production moved.
The isolated stages never fed the detector's output into alignment —
_align got fixed landmarks — so INPUT_LONG_EDGE resizing and the row ->
landmark scaling in _one_face were invisible. process() now runs end to
end on the fixture, with the pipeline's own detector threshold dropped
so a faceless frame still yields rows to carry through (26 faces here).
A first attempt at that still missed the resize: the embedded fixture is
48px, so `long_edge > INPUT_LONG_EDGE` never fired and changing 1920 to
960 moved nothing. It now runs a second pass with the threshold lowered
under the fixture, which executes the same downscale and inverse
landmark scaling without carrying a 1920px image in the source. Verified
sensitive: moving that bound 32 -> 24 changes both the face count and
the embedding.
The fixture was also a baseline JPEG, while generatePreview writes
progressive (imageProcessor.js:236/480/617) — a different path through
libjpeg. Swapped for a progressive fixture, SOF2 confirmed present and
SOF0 absent.
24 stages now. Debian/3.12 and Wolfi/3.14 remain identical across all of
them.
* test(ml): close three more false-negative paths in the fingerprint
Round four of external review. All three let hashes match while
production moved.
INPUT_LONG_EDGE was used but never printed. The fixture is too small to
trip the resize in either image, and the forced pass overrides the value
in both, so a production change from 1920 to 960 moved no hash at all.
It is now emitted alongside the other thresholds, where a reviewer sees
it in the diff.
The fixture was square, so a width/height swap in setInputSize or the
resize produced identical dimensions and identical hashes. It is now
64x48.
The forced-downscale pass hashed only an embedding, which is derived
from separately scaled landmarks — a regression in the inverse scaling
of row[0:4] would have shown up nowhere, because the normal pass runs at
scale 1. That bbox is now hashed too; a wrong one is what breaks avatar
crops and area calculations.
Changing the fixture to 64x48 also broke the forced pass: at the old
bound of 32 the downscaled frame is 32x24 and YuNet returns nothing, so
the stage pinned nothing. The NO-DETECTIONS-STAGE-VACUOUS marker added
last round caught it immediately rather than printing a reassuring hash
of an empty result. Bound moved to 48, which still triggers the resize
and still yields rows.
25 stages, no vacuous markers. Debian/3.12 and Wolfi/3.14 identical
across all of them.
* test(ml): hash every detection, not just the first
Round five of external review. Both end-to-end passes hashed only
candidate 0, so a change that moved candidates 1..n — or merely
reordered them — matched as long as the count and the first candidate
held. With the threshold at the floor those passes return 24 and 27
candidates, so that was most of the evidence being thrown away.
Both now stack every returned face, in order, via a shared _hash_all.
Stacking preserves order, so a reshuffle is caught too.
Verified against the exact case: reversing candidates 1..n while leaving
the count and candidate 0 untouched now moves process_embedding and
process_bbox. Before this it moved nothing.
* test(ml): hash every persisted field, and emit the model version
Round six of external review, plus the adjacent gaps it implied.
Two findings: MODEL_VERSION was never emitted, and _hash_all discarded
score. Both matter to the backend rather than to the numbers — a
model_version change makes faceClustering.js:190 refuse to compare new
faces against existing people, forcing a rescan, and det_score decides
via meetsQualityFloor (faceClustering.js:96-100) whether a face joins
clustering at all. Either could change while every hash held still.
Rather than fix only the two named, I checked what faceProcessor.js
actually stores per face (:157-167) and covered all of it: bbox, score,
yaw, pitch, blur, embedding. yaw/pitch/blur were heading for the same
finding next round. One hash per field, so a diff says which thing moved
rather than only that something did.
model_version is emitted as a compatibility key alongside the
thresholds, not hashed — it is a string, and its job is to be read.
Verified: scaling score alone by 0.999 now moves process_score and
nothing else. 33 stages, no vacuous markers, debian/3.12 and wolfi/3.14
still identical.
* test(ml): split verdict from diagnostic, and stop masking the threshold
Round seven of external review.
The ORT detector hashes were being read as part of the compatibility
verdict, but production never runs YuNet through onnxruntime. An ORT
change touching a YuNet operator would have moved them while real
behaviour was untouched, and the docstring said any difference means
re-scan — so the tool could have ordered a full-gallery rescan for
nothing. They are now diag_-prefixed, and the docstring states which
keys carry a verdict, which are diagnostic, and which are metadata a
reviewer has to read rather than diff.
setScoreThreshold(1e-6) also overwrote the detector's real threshold
before anything recorded it, and _thresholds.det_score only echoes
config. If FacePipeline ever stopped applying DET_SCORE_THRESHOLD —
falling back to OpenCV's 0.9 default — production would detect a
different face set while every hash matched. The constructed value is
now read first and emitted as _effective_det_score; simulating the
regression makes it read 0.9 instead of 0.6.
MAX_FACES is emitted for the same reason INPUT_LONG_EDGE is: the fixture
never reaches the pipeline.py:138 slice, so 64 -> 128 would move no hash
while real group photos persisted a different face set.
21 verdict keys, 12 diagnostic, no vacuous markers, debian/3.12 and
wolfi/3.14 still identical across both sets.
---------
Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
Paul Nothaft
198df02d2a