Paul Nothaft
25fbefc703
Every other feature routes readers to docs.picpeak.app. Face recognition was the one that either pointed somewhere else or pointed at nothing — poor placement for the feature with the highest read-before-you-enable burden anything here ships. .env.example referenced docs/feature-face-recognition.md, which does not exist — and creating it is not the fix, because .gitignore:89 ignores docs/feature-*.md outright, so the file would be invisible to anyone who cloned. That was the only pointer to legal guidance an operator got while editing the variables that turn Art. 9 processing on. Also: the README linked the sidecar's developer README for the feature name and had no row in the documentation table, docs/single-container.md left readers who wanted the feature nowhere to go, ml/README.md had no backlink, and the admin consent callout had no link at all. It does now, inline at the end of the obligation. Reported by @Luca-Timo.
155 lines
6.5 KiB
Markdown
155 lines
6.5 KiB
Markdown
# picpeak-ml
|
|
|
|
> **Looking for how to use the feature?** →
|
|
> [docs.picpeak.app/features/face-recognition](https://docs.picpeak.app/features/face-recognition)
|
|
> covers enabling it, the per-gallery toggle, consent and what guests see.
|
|
> This document is the sidecar's HTTP contract and deployment notes.
|
|
|
|
Optional face-detection sidecar for PicPeak (#1074). Detects faces in one
|
|
image and returns a bounding box, five landmarks, quality signals and a
|
|
512-d embedding per face.
|
|
|
|
**Nothing else.** No database, no volumes, no state, no egress, no model
|
|
download at runtime. Clustering, person identity, thresholds and every
|
|
privacy decision live in the PicPeak backend, where the data already is.
|
|
This service forgets each image the moment it answers.
|
|
|
|
If you don't run this container, the feature does not exist — see
|
|
"Turning it on" below.
|
|
|
|
## API
|
|
|
|
All endpoints except `/health` require the `X-Face-ML-Token` header. The
|
|
service **refuses to start** without `FACE_ML_TOKEN` set, so an accidentally
|
|
published port is never a free face-detection API.
|
|
|
|
| | |
|
|
|---|---|
|
|
| `GET /health` | `{"status": "ok"}` — unauthenticated, used by the compose healthcheck |
|
|
| `GET /info` | `{detector, embedder, model_version, dim}` |
|
|
| `POST /faces` | multipart `image` → `{model_version, faces: [...]}` |
|
|
|
|
Each face:
|
|
|
|
```jsonc
|
|
{
|
|
"bbox": [x, y, w, h], // ORIGINAL image pixels, not detection-scaled
|
|
"score": 0.94,
|
|
"landmarks": [[x, y], ...], // 5: right eye, left eye, nose, right mouth, left mouth
|
|
"yaw": -1.42, // degrees, approximate (see pipeline.py)
|
|
"pitch": -25.33,
|
|
"blur": 2579.5, // variance of Laplacian on the aligned crop; higher = sharper
|
|
"embedding": [...] // 512 floats, L2-normalized
|
|
}
|
|
```
|
|
|
|
`404`/`400` mean "this image is a lost cause" — the backend marks the photo
|
|
failed. `5xx` and connection failures mean "try later" — the backend returns
|
|
the photo to `pending` with backoff, so turning this container off for a week
|
|
does not require a manual re-scan.
|
|
|
|
## Models
|
|
|
|
YuNet (detection, MIT) + FaceNet-512 (embedding, MIT), both baked into the
|
|
image and verified by SHA-256 at build time. See [LICENSES.md](LICENSES.md)
|
|
for why these two and not the more obvious InsightFace weights — the short
|
|
version is that InsightFace's are non-commercial-only and PicPeak's users are
|
|
working photographers.
|
|
|
|
### Building the image
|
|
|
|
`facenet512.onnx` is **not** fetched automatically, because deepface
|
|
distributes FaceNet-512 as Keras `.h5` only. Convert it once, publish it,
|
|
then pass the URL and checksum:
|
|
|
|
```bash
|
|
cd ml
|
|
python3.11 -m venv .venv && . .venv/bin/activate # 3.11: TF has no 3.12+ wheels
|
|
pip install -r tools/requirements-convert.txt
|
|
|
|
curl -fsSL -o facenet512_weights.h5 \
|
|
https://github.com/serengil/deepface_models/releases/download/v1.0/facenet512_weights.h5
|
|
echo "3f76b5117a9ca574d536af8199e6720089eb4ad3dc7e93534496d88265de864f facenet512_weights.h5" | sha256sum -c -
|
|
|
|
python tools/convert_facenet.py facenet512_weights.h5 facenet512.onnx
|
|
```
|
|
|
|
The script verifies the converted graph against the Keras original before
|
|
writing (worst observed divergence: 2.1e-06 absolute, cosine 1.0000000000)
|
|
and prints the SHA-256 to publish. Output is ~89.6 MB, 23,497,424 parameters.
|
|
|
|
Publish `facenet512.onnx` as a release asset, set the repository variables
|
|
`FACENET_ONNX_URL` and `FACENET_ONNX_SHA256` (Settings → Variables — it's a
|
|
public URL, not a secret), and CI picks it up. To build locally:
|
|
|
|
```bash
|
|
docker build -t picpeak-ml \
|
|
--build-arg FACENET_ONNX_URL=https://github.com/PicPeak/picpeak/releases/download/<tag>/facenet512.onnx \
|
|
--build-arg FACENET_ONNX_SHA256=<sha256> \
|
|
ml/
|
|
```
|
|
|
|
The conversion sits outside the Docker build because TensorFlow is ~600MB of
|
|
build dependency for a file that never ships in the final image, and the
|
|
result is architecture-independent — no reason to run it on both legs of
|
|
every multi-arch build.
|
|
|
|
**The conversion is not byte-reproducible.** Two runs with the same pinned
|
|
versions on the same machine produce functionally identical graphs (same 336
|
|
nodes, same 271 initializers, weights matching to 0.000e+00) but differ in a
|
|
few initializer names, because tf2onnx's traced-op naming is not
|
|
deterministic. So a re-conversion **will** have a different SHA-256, and that
|
|
is expected rather than a sign of tampering. The checksum pins one published
|
|
artifact so its URL cannot start serving different bytes; validating a fresh
|
|
conversion is the parity check's job, not the hash's.
|
|
|
|
## Not available on the all-in-one image
|
|
|
|
The single-container image (`Dockerfile.aio`) sets
|
|
`PICPEAK_SINGLE_CONTAINER=true`, and the backend refuses to enable face
|
|
recognition when it sees that — the feature flag cannot be switched on, and
|
|
per-event detection stays off even if a restored database says otherwise.
|
|
|
|
This is a performance decision, not a licensing or packaging one. That image
|
|
runs the backend, the frontend, SQLite and every background worker inside one
|
|
container aimed at "one photographer plus guests browsing". It has no Redis,
|
|
SQLite gives it a single writer, and it contains no ML sidecar to talk to.
|
|
Adding a second image-processing pipeline that competes with Sharp for the
|
|
same CPU and RAM would not fail loudly — it would just make the whole install
|
|
slow and appear broken.
|
|
|
|
Run the standard multi-container deployment if you want this feature.
|
|
|
|
## Turning it on
|
|
|
|
Two deliberate actions, neither of which is installing this container:
|
|
|
|
1. Enable the `faces` feature flag in PicPeak's admin settings.
|
|
2. Enable "Detect people in this gallery" per event.
|
|
|
|
`FACE_ML_URL` defaults to `http://picpeak-ml:8000` — the compose service name
|
|
— so the standard deployment needs no URL configuration. **Nothing in the
|
|
backend touches that URL while the flag is off**, so an install without this
|
|
container never attempts a connection.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
pip install -r requirements.txt pytest httpx
|
|
python -m pytest tests/ -q
|
|
```
|
|
|
|
The tests stub the models out: they cover the auth boundary, the request
|
|
guards and the alignment geometry — the places where a mistake is a security
|
|
problem or a silent accuracy problem. Model *quality* is not a unit-test
|
|
question; that is what the Phase 0 spike measured.
|
|
|
|
### The one thing to be careful about
|
|
|
|
The alignment in `pipeline.py` and the normalization in `_embed` must stay
|
|
identical to whatever the clustering threshold was tuned against. A tuned
|
|
cosine threshold does not transfer across an alignment change. If either
|
|
changes, bump `MODEL_VERSION` in `config.py` — the backend keys
|
|
re-derivation off that string and will re-cluster rather than silently mix
|
|
two incompatible embedding spaces.
|