Files
picpeak/backend/src/services/faceClient.js
T
Paul NothaftandPaul Nothaft 37a15e3d49 fix(faces): restore the :beta image tag and surface sidecar health (#1087)
* fix(faces): restore the :beta image tag and surface sidecar health

Both halves of what a user hit on discussions/1069: the People card sat
at "Scanning… 0 of 227" for 30 minutes with no explanation, because the
sidecar container could never have started.

docker-build.yml — republish `:beta`. It used to come for free via
`type=ref,event=branch` when the active development branch was literally
named `beta`; the rename to `main` silently retired it. backend:beta has
been frozen at 2026-06-29 (448da950) ever since while :main moved on, so
PICPEAK_CHANNEL=beta has been serving a seven-week-old build across every
image. The ml sidecar was added after the rename and so never had a
`:beta` at all, which left docker-compose.production.yml:158 unable to
resolve ghcr.io/picpeak/picpeak/ml for any documented channel — the
image simply does not exist as :beta or :stable, only as :main and
pinned versions. Tag added to all four merge jobs, gated on main.

`:stable` stays absent for ml on purpose: it is gated on refs/heads/stable
and the sidecar does not exist there. stable's docker-compose.production.yml
carries no picpeak-ml service, so nothing can reference the missing tag.

FaceRecognitionCard — show when the sidecar is unreachable. An
unreachable sidecar is not an error by design: faceQueue.js:132-136
releases the photo back to `pending` and retries forever so a restart
does not burn the queue. The cost was that a stopped container looked
exactly like a slow scan, indefinitely, and the only signal was a
backend log line rate-limited to once per five minutes.
/admin/events/faces/health already existed and nothing in the frontend
called it. It is now polled while a scan is in progress, and a failing
check replaces the spinner with the sidecar URL, the underlying error
(which distinguishes a stopped container from a token mismatch) and the
command to start it.

Health is only polled while a scan is running — an idle card has no
reason to care whether the sidecar is up.

* fix(faces): tell the three sidecar failure modes apart

Follow-up to the health surface in this branch, from an external review
pass. The original warning was right about "the sidecar is not working"
and wrong about almost everything after that.

faceClient.checkHealth now returns a `reason` rather than only a message,
because the caller has to know whether photos survive:
  - 'unauthorized' (401) and 'rejected' (any other 4xx) both become
    SidecarRejectedError in classify(), which workerLoop does NOT retry —
    every claimed photo is marked 'failed'. Telling the admin the scan
    resumes on its own was simply untrue there; both now say to fix the
    cause and Re-scan.
  - 'unreachable' (refused/DNS/timeout/5xx) is the retryable one.

The card also no longer cries wolf. /faces runs inference synchronously
inside an `async def`, so one slow photo blocks the event loop and stalls
/info past its 5s timeout — a healthy sidecar can fail a probe. Verified
with an isolated uvicorn repro: a blocking call in an async handler
stalled the sync /info endpoint to 5.01s. The warning now needs three
consecutive failures AND no drop in `pending`. Three because a single
/faces call may legitimately run to FACE_ML_TIMEOUT_MS (30s) and two
probes 15s apart both fit inside that window; `pending` rather than
`scanned` because scanned counts only 'done', so a run producing
skipped/failed photos is progress that counter misses.

A 4xx burns the queue with no backoff, so it can empty before anyone
opens the card — in_progress goes false and only "227 failed" is left.
The probe therefore also runs when a finished scan has failures, and the
notice renders under the counts instead of replacing them. It is worded
as present-tense service state, not as a claim about those specific
failures: a live probe cannot know whether they came from this
misconfiguration or from corrupt images earlier. Attributing them exactly
needs stored face_error rows, which is a bigger change than this.

Also adds the missing-token case to the unreachable text: FACE_ML_TOKEN
has no default and the container refuses to start without it, so the most
likely first run fails as a plain connection refusal that "just start it"
does not fix.

---------

Co-authored-by: Paul Nothaft <[email protected]>
2026-08-19 16:49:47 +02:00

161 lines
5.4 KiB
JavaScript

/**
* HTTP client for the picpeak-ml sidecar (#1074).
*
* The important behaviour here is the error taxonomy, because the queue
* treats the two classes completely differently:
*
* SidecarUnavailableError — the sidecar is down, unreachable, timing out,
* or returned 5xx. The photo goes BACK to 'pending' and is retried later.
* Turning the container off for a week must not require a manual re-scan.
*
* Everything else (4xx) — this image is a lost cause. The photo is marked
* 'failed' and never retried, because retrying an undecodable file
* forever is just a busy loop.
*
* Log volume matters too: an hour of downtime at a 1s poll is 3,600 identical
* warnings. Unavailability is logged at most once per LOG_INTERVAL_MS.
*/
const axios = require('axios');
const logger = require('../utils/logger');
const { getSidecarUrl, getSidecarToken } = require('./faceSettings');
const REQUEST_TIMEOUT_MS = parseInt(process.env.FACE_ML_TIMEOUT_MS || '30000', 10);
const LOG_INTERVAL_MS = 5 * 60 * 1000;
let lastUnavailableLogAt = 0;
class SidecarUnavailableError extends Error {
constructor(message) {
super(message);
this.name = 'SidecarUnavailableError';
}
}
class SidecarRejectedError extends Error {
constructor(message, status) {
super(message);
this.name = 'SidecarRejectedError';
this.status = status;
}
}
function logUnavailable(message) {
const now = Date.now();
if (now - lastUnavailableLogAt < LOG_INTERVAL_MS) return;
lastUnavailableLogAt = now;
logger.warn(
`faceClient: ML sidecar unavailable (${message}). Photos stay queued and will ` +
'be retried; no action needed unless this persists. Further identical warnings ' +
'are suppressed for 5 minutes.'
);
}
function authHeaders() {
const token = getSidecarToken();
return token ? { 'X-Face-ML-Token': token } : {};
}
/**
* Translate an axios failure into our two-class taxonomy.
*/
function classify(err) {
const status = err.response?.status;
if (status && status >= 400 && status < 500) {
// 401 is a configuration error, not a bad image — but it is also not
// something retrying fixes, so it surfaces loudly and stops the photo.
if (status === 401) {
logger.error(
'faceClient: sidecar rejected our token (401). FACE_ML_TOKEN must match ' +
'on both the backend and the picpeak-ml container.'
);
}
return new SidecarRejectedError(
err.response?.data?.detail || `Sidecar rejected the request (${status})`,
status
);
}
logUnavailable(err.code || err.message || `HTTP ${status}`);
return new SidecarUnavailableError(err.message || 'Sidecar unreachable');
}
/**
* Detect faces in an image. `buffer` is the preview rendition's bytes.
* Returns the sidecar's `{ model_version, faces: [...] }`.
*/
async function detectFaces(buffer, filename = 'photo.jpg') {
const form = new FormData();
// Node 18+ ships FormData/Blob globally, so no multipart dependency is
// needed for the one endpoint that uploads anything.
form.append('image', new Blob([buffer]), filename);
try {
const response = await axios.post(`${getSidecarUrl()}/faces`, form, {
headers: authHeaders(),
timeout: REQUEST_TIMEOUT_MS,
// A 45MP preview is ~2MB; the cap is generous but not unbounded.
maxBodyLength: 64 * 1024 * 1024,
maxContentLength: 64 * 1024 * 1024,
});
return response.data;
} catch (err) {
throw classify(err);
}
}
/**
* Sidecar identity + liveness, for the admin connection test.
* Returns { ok: true, info } or { ok: false, reason, error } — never throws,
* because the caller is a UI button and a stack trace helps nobody there.
*
* `reason` exists because the failure modes need opposite advice, and the
* message string is the wrong thing for a caller to match on. The split
* mirrors classify() above, because that is what decides whether a photo is
* retried or burnt:
* - 'unauthorized' — 401. classify() returns SidecarRejectedError, which
* workerLoop does NOT retry, so every claimed photo is marked 'failed'.
* Fixing the token does not resume anything; the admin has to re-scan.
* - 'rejected' — any other 4xx (a wrong FACE_ML_URL answering 404, a proxy
* returning 403). classify() treats the whole 4xx range the same way, so
* these burn photos exactly like a 401 does and must not be reported as
* temporary.
* - 'unreachable' — everything else: connection refused, DNS, timeouts,
* 5xx. SidecarUnavailableError, photos go back to 'pending' and the scan
* picks up on its own.
*/
async function checkHealth() {
try {
const { data } = await axios.get(`${getSidecarUrl()}/info`, {
headers: authHeaders(),
timeout: 5000,
});
return { ok: true, info: data };
} catch (err) {
const status = err.response?.status;
if (status === 401) {
return {
ok: false,
reason: 'unauthorized',
error: 'Sidecar rejected the token (check FACE_ML_TOKEN on both containers)',
};
}
if (status && status >= 400 && status < 500) {
return {
ok: false,
reason: 'rejected',
error: `Sidecar answered ${status} — check FACE_ML_URL points at picpeak-ml and no proxy sits in front of it`,
};
}
return { ok: false, reason: 'unreachable', error: err.message || 'Sidecar unreachable' };
}
}
module.exports = {
detectFaces,
checkHealth,
SidecarUnavailableError,
SidecarRejectedError,
};