fix(faces): restore the :beta image tag and surface sidecar health (#1087)
* fix(faces): restore the :beta image tag and surface sidecar health
Both halves of what a user hit on discussions/1069: the People card sat
at "Scanning… 0 of 227" for 30 minutes with no explanation, because the
sidecar container could never have started.
docker-build.yml — republish `:beta`. It used to come for free via
`type=ref,event=branch` when the active development branch was literally
named `beta`; the rename to `main` silently retired it. backend:beta has
been frozen at 2026-06-29 (448da950) ever since while :main moved on, so
PICPEAK_CHANNEL=beta has been serving a seven-week-old build across every
image. The ml sidecar was added after the rename and so never had a
`:beta` at all, which left docker-compose.production.yml:158 unable to
resolve ghcr.io/picpeak/picpeak/ml for any documented channel — the
image simply does not exist as :beta or :stable, only as :main and
pinned versions. Tag added to all four merge jobs, gated on main.
`:stable` stays absent for ml on purpose: it is gated on refs/heads/stable
and the sidecar does not exist there. stable's docker-compose.production.yml
carries no picpeak-ml service, so nothing can reference the missing tag.
FaceRecognitionCard — show when the sidecar is unreachable. An
unreachable sidecar is not an error by design: faceQueue.js:132-136
releases the photo back to `pending` and retries forever so a restart
does not burn the queue. The cost was that a stopped container looked
exactly like a slow scan, indefinitely, and the only signal was a
backend log line rate-limited to once per five minutes.
/admin/events/faces/health already existed and nothing in the frontend
called it. It is now polled while a scan is in progress, and a failing
check replaces the spinner with the sidecar URL, the underlying error
(which distinguishes a stopped container from a token mismatch) and the
command to start it.
Health is only polled while a scan is running — an idle card has no
reason to care whether the sidecar is up.
* fix(faces): tell the three sidecar failure modes apart
Follow-up to the health surface in this branch, from an external review
pass. The original warning was right about "the sidecar is not working"
and wrong about almost everything after that.
faceClient.checkHealth now returns a `reason` rather than only a message,
because the caller has to know whether photos survive:
- 'unauthorized' (401) and 'rejected' (any other 4xx) both become
SidecarRejectedError in classify(), which workerLoop does NOT retry —
every claimed photo is marked 'failed'. Telling the admin the scan
resumes on its own was simply untrue there; both now say to fix the
cause and Re-scan.
- 'unreachable' (refused/DNS/timeout/5xx) is the retryable one.
The card also no longer cries wolf. /faces runs inference synchronously
inside an `async def`, so one slow photo blocks the event loop and stalls
/info past its 5s timeout — a healthy sidecar can fail a probe. Verified
with an isolated uvicorn repro: a blocking call in an async handler
stalled the sync /info endpoint to 5.01s. The warning now needs three
consecutive failures AND no drop in `pending`. Three because a single
/faces call may legitimately run to FACE_ML_TIMEOUT_MS (30s) and two
probes 15s apart both fit inside that window; `pending` rather than
`scanned` because scanned counts only 'done', so a run producing
skipped/failed photos is progress that counter misses.
A 4xx burns the queue with no backoff, so it can empty before anyone
opens the card — in_progress goes false and only "227 failed" is left.
The probe therefore also runs when a finished scan has failures, and the
notice renders under the counts instead of replacing them. It is worded
as present-tense service state, not as a claim about those specific
failures: a live probe cannot know whether they came from this
misconfiguration or from corrupt images earlier. Attributing them exactly
needs stored face_error rows, which is a bigger change than this.
Also adds the missing-token case to the unreachable text: FACE_ML_TOKEN
has no default and the container refuses to start without it, so the most
likely first run fails as a plain connection refusal that "just start it"
does not fix.
---------
Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
co-authored by
Paul Nothaft
Paul Nothaft
parent
74a8f9bf24
commit
37a15e3d49
@@ -331,6 +331,15 @@ jobs:
|
||||
# so `is_default_branch` no longer maps to "stable" — be explicit.
|
||||
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
type=raw,value=stable,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
# `:beta` follows the active development branch. This used to happen
|
||||
# for free via `type=ref,event=branch` back when that branch was
|
||||
# literally named `beta`; the rename to `main` silently retired the
|
||||
# tag, so `PICPEAK_CHANNEL=beta` has been pinned to the last pre-rename
|
||||
# build (backend:beta sat at 2026-06-29 / 448da950) while :main moved
|
||||
# on. The ml sidecar was added after the rename and so never had a
|
||||
# `:beta` at all, which left docker-compose.production.yml unable to
|
||||
# resolve the image for any documented channel.
|
||||
type=raw,value=beta,enable=${{ github.ref == 'refs/heads/main' }}
|
||||
# `:beta` is RETIRED post-rename (Option B / #669). Active-dev pulls
|
||||
# are `:main` (auto via type=ref,event=branch). The pre-rename `:beta`
|
||||
# tag remains frozen at its last build — operators should update.
|
||||
@@ -596,6 +605,15 @@ jobs:
|
||||
# so `is_default_branch` no longer maps to "stable" — be explicit.
|
||||
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
type=raw,value=stable,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
# `:beta` follows the active development branch. This used to happen
|
||||
# for free via `type=ref,event=branch` back when that branch was
|
||||
# literally named `beta`; the rename to `main` silently retired the
|
||||
# tag, so `PICPEAK_CHANNEL=beta` has been pinned to the last pre-rename
|
||||
# build (backend:beta sat at 2026-06-29 / 448da950) while :main moved
|
||||
# on. The ml sidecar was added after the rename and so never had a
|
||||
# `:beta` at all, which left docker-compose.production.yml unable to
|
||||
# resolve the image for any documented channel.
|
||||
type=raw,value=beta,enable=${{ github.ref == 'refs/heads/main' }}
|
||||
# `:beta` is RETIRED post-rename (Option B / #669). Active-dev pulls
|
||||
# are `:main` (auto via type=ref,event=branch). The pre-rename `:beta`
|
||||
# tag remains frozen at its last build — operators should update.
|
||||
@@ -819,6 +837,15 @@ jobs:
|
||||
type=sha,format=short
|
||||
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
type=raw,value=stable,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
# `:beta` follows the active development branch. This used to happen
|
||||
# for free via `type=ref,event=branch` back when that branch was
|
||||
# literally named `beta`; the rename to `main` silently retired the
|
||||
# tag, so `PICPEAK_CHANNEL=beta` has been pinned to the last pre-rename
|
||||
# build (backend:beta sat at 2026-06-29 / 448da950) while :main moved
|
||||
# on. The ml sidecar was added after the rename and so never had a
|
||||
# `:beta` at all, which left docker-compose.production.yml unable to
|
||||
# resolve the image for any documented channel.
|
||||
type=raw,value=beta,enable=${{ github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Create and push multi-arch manifest
|
||||
working-directory: /tmp/digests
|
||||
@@ -1232,6 +1259,15 @@ jobs:
|
||||
type=sha,format=short
|
||||
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
type=raw,value=stable,enable=${{ github.ref == 'refs/heads/stable' || (startsWith(github.ref, 'refs/tags/v') && steps.context.outputs.is_prerelease == 'false') }}
|
||||
# `:beta` follows the active development branch. This used to happen
|
||||
# for free via `type=ref,event=branch` back when that branch was
|
||||
# literally named `beta`; the rename to `main` silently retired the
|
||||
# tag, so `PICPEAK_CHANNEL=beta` has been pinned to the last pre-rename
|
||||
# build (backend:beta sat at 2026-06-29 / 448da950) while :main moved
|
||||
# on. The ml sidecar was added after the rename and so never had a
|
||||
# `:beta` at all, which left docker-compose.production.yml unable to
|
||||
# resolve the image for any documented channel.
|
||||
type=raw,value=beta,enable=${{ github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Create and push multi-arch manifest
|
||||
working-directory: /tmp/digests
|
||||
|
||||
@@ -107,8 +107,23 @@ async function detectFaces(buffer, filename = 'photo.jpg') {
|
||||
|
||||
/**
|
||||
* Sidecar identity + liveness, for the admin connection test.
|
||||
* Returns { ok: true, info } or { ok: false, error } — never throws, because
|
||||
* the caller is a UI button and a stack trace helps nobody there.
|
||||
* Returns { ok: true, info } or { ok: false, reason, error } — never throws,
|
||||
* because the caller is a UI button and a stack trace helps nobody there.
|
||||
*
|
||||
* `reason` exists because the failure modes need opposite advice, and the
|
||||
* message string is the wrong thing for a caller to match on. The split
|
||||
* mirrors classify() above, because that is what decides whether a photo is
|
||||
* retried or burnt:
|
||||
* - 'unauthorized' — 401. classify() returns SidecarRejectedError, which
|
||||
* workerLoop does NOT retry, so every claimed photo is marked 'failed'.
|
||||
* Fixing the token does not resume anything; the admin has to re-scan.
|
||||
* - 'rejected' — any other 4xx (a wrong FACE_ML_URL answering 404, a proxy
|
||||
* returning 403). classify() treats the whole 4xx range the same way, so
|
||||
* these burn photos exactly like a 401 does and must not be reported as
|
||||
* temporary.
|
||||
* - 'unreachable' — everything else: connection refused, DNS, timeouts,
|
||||
* 5xx. SidecarUnavailableError, photos go back to 'pending' and the scan
|
||||
* picks up on its own.
|
||||
*/
|
||||
async function checkHealth() {
|
||||
try {
|
||||
@@ -120,9 +135,20 @@ async function checkHealth() {
|
||||
} catch (err) {
|
||||
const status = err.response?.status;
|
||||
if (status === 401) {
|
||||
return { ok: false, error: 'Sidecar rejected the token (check FACE_ML_TOKEN on both containers)' };
|
||||
return {
|
||||
ok: false,
|
||||
reason: 'unauthorized',
|
||||
error: 'Sidecar rejected the token (check FACE_ML_TOKEN on both containers)',
|
||||
};
|
||||
}
|
||||
return { ok: false, error: err.message || 'Sidecar unreachable' };
|
||||
if (status && status >= 400 && status < 500) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: 'rejected',
|
||||
error: `Sidecar answered ${status} — check FACE_ML_URL points at picpeak-ml and no proxy sits in front of it`,
|
||||
};
|
||||
}
|
||||
return { ok: false, reason: 'unreachable', error: err.message || 'Sidecar unreachable' };
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
* provide the switch, the photographer provides the lawful basis — so it
|
||||
* renders next to the toggle rather than behind a "learn more".
|
||||
*/
|
||||
import React, { useEffect, useState } from 'react';
|
||||
import React, { useEffect, useRef, useState } from 'react';
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { toast } from 'react-toastify';
|
||||
@@ -42,6 +42,18 @@ interface FacesPayload {
|
||||
};
|
||||
}
|
||||
|
||||
interface SidecarHealth {
|
||||
url: string;
|
||||
ok: boolean;
|
||||
reason?: 'unauthorized' | 'rejected' | 'unreachable';
|
||||
error?: string;
|
||||
}
|
||||
|
||||
// 4xx of any kind burns photos: faceClient.classify() turns the whole range
|
||||
// into SidecarRejectedError, which faceQueue does not retry. Those cases need
|
||||
// "fix it, then re-scan", never "it resumes on its own".
|
||||
const BURNS_PHOTOS = new Set(['unauthorized', 'rejected']);
|
||||
|
||||
interface FaceRecognitionCardProps {
|
||||
eventId: number;
|
||||
isArchived?: boolean;
|
||||
@@ -61,6 +73,89 @@ export const FaceRecognitionCard: React.FC<FaceRecognitionCardProps> = ({ eventI
|
||||
refetchInterval: (query) => (query.state.data?.status?.in_progress ? 5000 : false),
|
||||
});
|
||||
|
||||
// A scan that cannot reach the sidecar does not fail — faceQueue releases the
|
||||
// photo back to `pending` and retries forever, so a restart doesn't burn the
|
||||
// queue. The cost is that "Scanning… 0 of 227" is also what a missing
|
||||
// picpeak-ml container looks like, indefinitely. Poll the connection test
|
||||
// while a scan is running so that case says so instead of spinning.
|
||||
const scanRunning = !!data?.enabled && !!data.status.in_progress;
|
||||
// A 4xx burns photos with no backoff (faceQueue.js:139-152 marks failed and
|
||||
// loops straight on), so a misconfigured token or URL can empty the queue
|
||||
// faster than this card polls. By the time anyone looks, in_progress is
|
||||
// false and all that is left is "N failed" — the diagnosis has to outlive
|
||||
// the scan, so keep probing while there are failures to explain.
|
||||
const hasFailures = !!data?.enabled && (data.status.failed ?? 0) > 0;
|
||||
const shouldProbe = scanRunning || hasFailures;
|
||||
const { data: health, dataUpdatedAt: healthAt } = useQuery<SidecarHealth>({
|
||||
queryKey: ['admin-faces-health'],
|
||||
queryFn: async () => (await api.get('/admin/events/faces/health')).data,
|
||||
enabled: shouldProbe,
|
||||
// Only worth re-polling while work is actually moving; one probe is enough
|
||||
// to explain a finished run.
|
||||
refetchInterval: scanRunning ? 15000 : false,
|
||||
// A failing connection test is the signal itself, not an error state.
|
||||
retry: false,
|
||||
});
|
||||
|
||||
// One failed probe is not enough to cry wolf. The sidecar serves /faces from
|
||||
// an `async def` that runs inference synchronously, so a single slow photo
|
||||
// blocks the event loop and stalls /info past its 5s timeout — a healthy,
|
||||
// actively-working sidecar can fail a probe. Require consecutive failures
|
||||
// AND no queue movement: `pending` draining is direct proof the sidecar is
|
||||
// processing, whatever the probe says. (`pending`, not `scanned` — scanned
|
||||
// counts only `done`, so a run producing skipped/failed photos is still
|
||||
// progress the counter would miss.)
|
||||
//
|
||||
// Three, not two, because the streak has to outlast one whole inference: a
|
||||
// single /faces call may legitimately run to FACE_ML_TIMEOUT_MS (30s by
|
||||
// default) on a slow host, blocking /info the entire time, and two probes
|
||||
// 15s apart both fit inside that window. Three spans >30s, by which point
|
||||
// the backend's own request has timed out and freed the event loop — so a
|
||||
// still-failing probe means the sidecar really is gone, not merely busy.
|
||||
//
|
||||
// The photo-burning reasons skip the liveness guard on purpose: there the
|
||||
// queue drains too, but every photo drains into `failed`.
|
||||
const failStreak = useRef(0);
|
||||
const lastPending = useRef<number | null>(null);
|
||||
const [sidecarWarning, setSidecarWarning] =
|
||||
useState<'unauthorized' | 'rejected' | 'unreachable' | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (!shouldProbe || !health) {
|
||||
failStreak.current = 0;
|
||||
lastPending.current = null;
|
||||
setSidecarWarning(null);
|
||||
return;
|
||||
}
|
||||
if (health.reason && BURNS_PHOTOS.has(health.reason)) {
|
||||
setSidecarWarning(health.reason);
|
||||
return;
|
||||
}
|
||||
if (!scanRunning) {
|
||||
// Probing only to explain existing failures. A reachable sidecar means
|
||||
// those failures were per-photo (undecodable images), not config — and
|
||||
// an unreachable one leaves photos pending, not failed, so it cannot be
|
||||
// the explanation either.
|
||||
setSidecarWarning(null);
|
||||
return;
|
||||
}
|
||||
|
||||
const pending = data?.status.pending ?? 0;
|
||||
const draining = lastPending.current !== null && pending < lastPending.current;
|
||||
lastPending.current = pending;
|
||||
|
||||
if (health.ok || draining) {
|
||||
failStreak.current = 0;
|
||||
setSidecarWarning(null);
|
||||
return;
|
||||
}
|
||||
failStreak.current += 1;
|
||||
setSidecarWarning(failStreak.current >= 3 ? 'unreachable' : null);
|
||||
// Keyed on the probe, not the status poll: the two queries tick at
|
||||
// different rates and the streak counts probes.
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [healthAt, shouldProbe, scanRunning, health]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!data?.enabled) return;
|
||||
api.get('/admin/events/faces/auto-categories')
|
||||
@@ -128,6 +223,34 @@ export const FaceRecognitionCard: React.FC<FaceRecognitionCardProps> = ({ eventI
|
||||
|
||||
const { status } = data;
|
||||
|
||||
// Rendered in two places — replacing the spinner mid-scan, or under the
|
||||
// final counts once a scan has ended with failures it can explain.
|
||||
const sidecarNotice = sidecarWarning && health ? (
|
||||
<div className={`flex items-start gap-2 ${BURNS_PHOTOS.has(sidecarWarning) ? 'text-red-700' : 'text-amber-700'}`}>
|
||||
<AlertTriangle size={14} className="mt-0.5 shrink-0" />
|
||||
<p>
|
||||
{sidecarWarning === 'unauthorized' && t('admin.faces.sidecarUnauthorized', {
|
||||
url: health.url,
|
||||
defaultValue: `The face-detection service at ${health.url} is rejecting our token, and photos are being marked failed rather than retried. Make FACE_ML_TOKEN identical on the backend and the picpeak-ml container, restart both, then use Re-scan — fixing the token alone will not reprocess the photos that already failed.`,
|
||||
})}
|
||||
{sidecarWarning === 'rejected' && t('admin.faces.sidecarRejected', {
|
||||
url: health.url,
|
||||
defaultValue: `${health.url} answered, but not like the face-detection service — photos are being marked failed rather than retried. Check FACE_ML_URL points at the picpeak-ml container and that nothing is proxying that address, then use Re-scan for the photos that already failed.`,
|
||||
})}
|
||||
{sidecarWarning === 'unreachable' && t('admin.faces.sidecarUnreachable', {
|
||||
url: health.url,
|
||||
pending: status.pending,
|
||||
defaultValue: `Can't reach the face-detection service at ${health.url}, so the ${status.pending} queued photos aren't being processed. Nothing is lost — the scan resumes on its own once the service is up. Start it with \`docker compose --profile faces up -d\`, and note it exits immediately unless FACE_ML_TOKEN is set to the same value as the backend — there is no default.`,
|
||||
})}
|
||||
{health.error && (
|
||||
<span className={`block mt-1 text-xs font-mono ${BURNS_PHOTOS.has(sidecarWarning) ? 'text-red-600' : 'text-amber-600'}`}>
|
||||
{health.error}
|
||||
</span>
|
||||
)}
|
||||
</p>
|
||||
</div>
|
||||
) : null;
|
||||
|
||||
return (
|
||||
<Card>
|
||||
<div className="flex items-start gap-3 mb-4">
|
||||
@@ -253,7 +376,13 @@ export const FaceRecognitionCard: React.FC<FaceRecognitionCardProps> = ({ eventI
|
||||
{data.enabled && (
|
||||
<>
|
||||
<div className="mt-4 pt-4 border-t border-neutral-100 text-sm text-neutral-600">
|
||||
{status.in_progress ? (
|
||||
{/* While scanning, the warning replaces the spinner — a progress
|
||||
indicator that cannot progress is the misleading part. Once the
|
||||
scan has ended the counts are what the admin came for, so the
|
||||
warning renders underneath them instead (below). */}
|
||||
{sidecarWarning && health && status.in_progress ? (
|
||||
sidecarNotice
|
||||
) : status.in_progress ? (
|
||||
<p className="flex items-center gap-2">
|
||||
<RefreshCw size={14} className="animate-spin text-primary-500" />
|
||||
{t('admin.faces.scanning', {
|
||||
@@ -294,6 +423,26 @@ export const FaceRecognitionCard: React.FC<FaceRecognitionCardProps> = ({ eventI
|
||||
)}
|
||||
</p>
|
||||
)}
|
||||
{/* Scan already over: keep the counts, add the reason those photos
|
||||
failed. Without this a misconfigured token shows only
|
||||
"227 failed" and no way to act on it.
|
||||
|
||||
Deliberately worded as present-tense state rather than a claim
|
||||
about these specific failures: this is a live probe, so it
|
||||
cannot know whether the recorded failures came from the current
|
||||
misconfiguration or from corrupt images at some earlier point.
|
||||
Attributing them properly would mean reading stored face_error
|
||||
rows — worth doing, but a bigger change than this. */}
|
||||
{!status.in_progress && sidecarNotice && (
|
||||
<div className="mt-2">
|
||||
<p className="text-xs text-neutral-500 mb-1">
|
||||
{t('admin.faces.sidecarStateNow', {
|
||||
defaultValue: 'Service state right now — some of the failures above may have a different cause, but a re-scan will not succeed until this is fixed:',
|
||||
})}
|
||||
</p>
|
||||
{sidecarNotice}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap gap-2 mt-4">
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
/**
|
||||
* A scan whose sidecar is unreachable does not fail — faceQueue releases the
|
||||
* photo back to `pending` and retries forever. That is deliberate, but it made
|
||||
* "Scanning… 0 of N" indistinguishable from a missing picpeak-ml container,
|
||||
* which is exactly what a user hit on discussions/1069: 0/227 for 30 minutes
|
||||
* with no explanation anywhere in the UI.
|
||||
*
|
||||
* These pin the branch: while a scan is in progress, a failing connection test
|
||||
* replaces the spinner with something actionable, and a passing one leaves the
|
||||
* spinner alone.
|
||||
*/
|
||||
import React from 'react';
|
||||
import { render, screen, waitFor } from '@testing-library/react';
|
||||
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
|
||||
import { FaceRecognitionCard } from '../FaceRecognitionCard';
|
||||
|
||||
vi.mock('react-i18next', () => ({
|
||||
useTranslation: () => ({
|
||||
t: (_key: string, opts?: any) => opts?.defaultValue ?? _key,
|
||||
}),
|
||||
// src/i18n/config.ts is pulled in transitively via ErrorBoundary and calls
|
||||
// .use(initReactI18next) at import time.
|
||||
initReactI18next: { type: '3rdParty', init: () => {} },
|
||||
}));
|
||||
|
||||
vi.mock('../PeopleManagerModal', () => ({
|
||||
PeopleManagerModal: () => null,
|
||||
}));
|
||||
|
||||
vi.mock('react-toastify', () => ({
|
||||
toast: { success: vi.fn(), error: vi.fn() },
|
||||
}));
|
||||
|
||||
const get = vi.fn();
|
||||
vi.mock('../../../config/api', () => ({
|
||||
api: {
|
||||
get: (...args: any[]) => get(...args),
|
||||
put: vi.fn().mockResolvedValue({ data: {} }),
|
||||
post: vi.fn().mockResolvedValue({ data: {} }),
|
||||
},
|
||||
}));
|
||||
|
||||
function facesPayload(over: Record<string, any> = {}) {
|
||||
return {
|
||||
enabled: true,
|
||||
visible_to_guests: true,
|
||||
last_scan_at: null,
|
||||
status: {
|
||||
scanned: 0,
|
||||
total: 227,
|
||||
pending: 227,
|
||||
failed: 0,
|
||||
people: 0,
|
||||
in_progress: true,
|
||||
...over,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function mockApi(faces: any, health: any) {
|
||||
get.mockImplementation((rawUrl: unknown) => {
|
||||
const url = String(rawUrl ?? '');
|
||||
if (url.includes('/faces/health')) {
|
||||
if (health === 'reject') return Promise.reject(new Error('boom'));
|
||||
return Promise.resolve({ data: health });
|
||||
}
|
||||
if (url.includes('/auto-categories')) return Promise.resolve({ data: { enabled: false } });
|
||||
return Promise.resolve({ data: faces });
|
||||
});
|
||||
}
|
||||
|
||||
function renderCard() {
|
||||
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
||||
const utils = render(
|
||||
<QueryClientProvider client={qc}>
|
||||
<FaceRecognitionCard eventId={1} />
|
||||
</QueryClientProvider>
|
||||
);
|
||||
return {
|
||||
...utils,
|
||||
qc,
|
||||
rerender: () =>
|
||||
utils.rerender(
|
||||
<QueryClientProvider client={qc}>
|
||||
<FaceRecognitionCard eventId={1} />
|
||||
</QueryClientProvider>
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
// Back-to-back refetchQueries calls coalesce, so drive probes one at a time.
|
||||
async function probe(qc: QueryClient, times: number) {
|
||||
for (let i = 0; i < times; i++) {
|
||||
await qc.refetchQueries({ queryKey: ['admin-faces-health'] });
|
||||
await new Promise((r) => setTimeout(r, 0));
|
||||
}
|
||||
}
|
||||
|
||||
describe('FaceRecognitionCard — sidecar health during a scan', () => {
|
||||
beforeEach(() => get.mockReset());
|
||||
|
||||
it('does not cry wolf on a single failed probe', async () => {
|
||||
// The sidecar serves /faces from an async handler that runs inference
|
||||
// synchronously, so one slow photo stalls /info past its 5s timeout. A
|
||||
// healthy sidecar can fail a probe; one failure must not raise the alarm.
|
||||
mockApi(facesPayload(), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
reason: 'unreachable',
|
||||
error: 'timeout of 5000ms exceeded',
|
||||
});
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() => expect(screen.getByText(/Scanning… 0 of 227/)).toBeInTheDocument());
|
||||
expect(screen.queryByText(/Can't reach the face-detection service/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('stays quiet at two failures — still inside one inference window', async () => {
|
||||
// A single /faces call may legitimately run to FACE_ML_TIMEOUT_MS (30s),
|
||||
// blocking /info throughout, and two probes 15s apart both fit inside that
|
||||
// window. Warning at two would falsely accuse a healthy, working sidecar.
|
||||
mockApi(facesPayload(), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
reason: 'unreachable',
|
||||
error: 'timeout of 5000ms exceeded',
|
||||
});
|
||||
|
||||
const { rerender, qc } = renderCard();
|
||||
await waitFor(() => expect(screen.getByText(/Scanning… 0 of 227/)).toBeInTheDocument());
|
||||
|
||||
await probe(qc, 1); // now two consecutive failures
|
||||
rerender();
|
||||
|
||||
expect(screen.queryByText(/Can't reach the face-detection service/)).not.toBeInTheDocument();
|
||||
expect(screen.getByText(/Scanning… 0 of 227/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('warns only after the streak outlasts one full inference window', async () => {
|
||||
mockApi(facesPayload(), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
reason: 'unreachable',
|
||||
error: 'connect ECONNREFUSED 172.18.0.5:8000',
|
||||
});
|
||||
|
||||
const { rerender, qc } = renderCard();
|
||||
await waitFor(() => expect(screen.getByText(/Scanning… 0 of 227/)).toBeInTheDocument());
|
||||
|
||||
// Two more probes, pending unchanged -> the streak now spans >30s, past a
|
||||
// full inference window, so the sidecar really is gone rather than busy.
|
||||
await probe(qc, 2);
|
||||
rerender();
|
||||
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText(/Can't reach the face-detection service/)).toBeInTheDocument()
|
||||
);
|
||||
|
||||
// URL and underlying error are both shown: without them the admin cannot
|
||||
// tell a stopped container from a token mismatch.
|
||||
expect(screen.getByText(/http:\/\/picpeak-ml:8000/)).toBeInTheDocument();
|
||||
expect(screen.getByText(/ECONNREFUSED/)).toBeInTheDocument();
|
||||
expect(screen.queryByText(/Scanning… 0 of 227/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('treats a non-401 4xx as burning photos too, not as downtime', async () => {
|
||||
// classify() turns the whole 4xx range into SidecarRejectedError, so a
|
||||
// wrong FACE_ML_URL answering 404 marks photos failed exactly like a 401.
|
||||
// Reporting it as temporary downtime would promise recovery that never
|
||||
// comes.
|
||||
mockApi(facesPayload({ pending: 210, failed: 17 }), {
|
||||
url: 'http://wrong-host:8000',
|
||||
ok: false,
|
||||
reason: 'rejected',
|
||||
error: 'Sidecar answered 404 — check FACE_ML_URL points at picpeak-ml',
|
||||
});
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText(/answered, but not like the face-detection service/)).toBeInTheDocument()
|
||||
);
|
||||
expect(screen.queryByText(/resumes on its own/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('tells the admin the sidecar needs FACE_ML_TOKEN to start at all', async () => {
|
||||
// The most likely first-run failure: FACE_ML_TOKEN has no default, so the
|
||||
// container raises at startup and never listens. That surfaces as a plain
|
||||
// connection refusal, and "just start it" is not enough to fix it.
|
||||
mockApi(facesPayload(), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
reason: 'unreachable',
|
||||
error: 'connect ECONNREFUSED 172.18.0.5:8000',
|
||||
});
|
||||
|
||||
const { rerender, qc } = renderCard();
|
||||
// let the first probe land before forcing the rest — the warning is
|
||||
// deliberately gated on three consecutive failures
|
||||
await waitFor(() => expect(screen.getByText(/Scanning… 0 of 227/)).toBeInTheDocument());
|
||||
await probe(qc, 2);
|
||||
rerender();
|
||||
|
||||
await waitFor(() => expect(screen.getByText(/FACE_ML_TOKEN/)).toBeInTheDocument());
|
||||
expect(screen.getByText(/there is no default/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('warns immediately on a rejected token, and says a re-scan is needed', async () => {
|
||||
// 401 becomes SidecarRejectedError, which faceQueue does NOT retry — every
|
||||
// claimed photo is marked `failed`. Promising the scan auto-resumes here
|
||||
// would be wrong, and the liveness guard must not mask it: the queue is
|
||||
// draining, just into failures.
|
||||
mockApi(facesPayload({ pending: 200, scanned: 0, failed: 27 }), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
reason: 'unauthorized',
|
||||
error: 'Sidecar rejected the token (check FACE_ML_TOKEN on both containers)',
|
||||
});
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText(/rejecting our token/)).toBeInTheDocument()
|
||||
);
|
||||
// scoped to the message, not the Re-scan button that also matches /Re-scan/
|
||||
expect(screen.getByText(/then use Re-scan/)).toBeInTheDocument();
|
||||
expect(screen.getByText(/marked failed rather than retried/)).toBeInTheDocument();
|
||||
// must NOT claim it resumes on its own
|
||||
expect(screen.queryByText(/resumes on its own/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps the normal progress line when the sidecar is healthy', async () => {
|
||||
mockApi(facesPayload(), { url: 'http://picpeak-ml:8000', ok: true });
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText(/Scanning… 0 of 227/)).toBeInTheDocument()
|
||||
);
|
||||
expect(screen.queryByText(/Can't reach the face-detection service/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not probe health on a clean idle card', async () => {
|
||||
mockApi(facesPayload({ in_progress: false, scanned: 227, pending: 0, failed: 0, people: 12 }), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
});
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() => expect(screen.getByText(/227 photos scanned/)).toBeInTheDocument());
|
||||
|
||||
// Nothing queued and nothing failed — there is no question to answer, so
|
||||
// the card must not call the sidecar at all.
|
||||
expect(screen.queryByText(/Can't reach the face-detection service/)).not.toBeInTheDocument();
|
||||
expect(get.mock.calls.some(([url]: any[]) => String(url).includes('/faces/health'))).toBe(false);
|
||||
});
|
||||
|
||||
it('still explains a 4xx after the scan has already burnt through the queue', async () => {
|
||||
// A 4xx marks photos failed with no backoff (faceQueue.js:139-152), so the
|
||||
// queue can empty before anyone opens the card. in_progress is false and
|
||||
// only "227 failed" remains — the diagnosis has to outlive the scan.
|
||||
mockApi(facesPayload({ in_progress: false, pending: 0, scanned: 0, failed: 227 }), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: false,
|
||||
reason: 'unauthorized',
|
||||
error: 'Sidecar rejected the token (check FACE_ML_TOKEN on both containers)',
|
||||
});
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() => expect(screen.getByText(/rejecting our token/)).toBeInTheDocument());
|
||||
// the counts the admin came for are still there, not replaced by the warning
|
||||
expect(screen.getByText(/227 failed/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not blame the sidecar for per-photo failures when it is healthy', async () => {
|
||||
// Failures with a reachable sidecar mean undecodable images, not config.
|
||||
mockApi(facesPayload({ in_progress: false, pending: 0, scanned: 220, failed: 7 }), {
|
||||
url: 'http://picpeak-ml:8000',
|
||||
ok: true,
|
||||
});
|
||||
|
||||
renderCard();
|
||||
|
||||
await waitFor(() => expect(screen.getByText(/7 failed/)).toBeInTheDocument());
|
||||
expect(screen.queryByText(/rejecting our token/)).not.toBeInTheDocument();
|
||||
expect(screen.queryByText(/Can't reach the face-detection service/)).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -3161,7 +3161,11 @@
|
||||
"deleted": "Gesichtsdaten gelöscht",
|
||||
"confirmDelete": "Alle erkannten Personen und Gesichtsdaten dieser Galerie löschen? Dies kann nicht rückgängig gemacht werden. Die Fotos selbst bleiben unverändert.",
|
||||
"autoCategories": "Fotos automatisch in Kategorien einsortieren",
|
||||
"autoCategoriesHint": "Nutzt die Anzahl der Gesichter, um Fotos als Details, Porträts, Kleine Gruppen oder Gruppen abzulegen. Gilt für alle Galerien, füllt ausschließlich leere Kategorien und ändert niemals eine von Ihnen gesetzte."
|
||||
"autoCategoriesHint": "Nutzt die Anzahl der Gesichter, um Fotos als Details, Porträts, Kleine Gruppen oder Gruppen abzulegen. Gilt für alle Galerien, füllt ausschließlich leere Kategorien und ändert niemals eine von Ihnen gesetzte.",
|
||||
"sidecarUnreachable": "Der Gesichtserkennungs-Dienst unter {{url}} ist nicht erreichbar, deshalb werden die {{pending}} eingereihten Fotos nicht verarbeitet. Es geht nichts verloren — der Scan läuft von selbst weiter, sobald der Dienst wieder da ist. Starte ihn mit `docker compose --profile faces up -d`; er beendet sich sofort, wenn FACE_ML_TOKEN nicht auf denselben Wert wie im Backend gesetzt ist — einen Standardwert gibt es nicht.",
|
||||
"sidecarUnauthorized": "Der Gesichtserkennungs-Dienst unter {{url}} weist unser Token zurück; Fotos werden als fehlgeschlagen markiert statt erneut versucht. Setze FACE_ML_TOKEN im Backend und im picpeak-ml-Container identisch, starte beide neu und nutze dann Neu scannen — das Token allein zu korrigieren verarbeitet die bereits fehlgeschlagenen Fotos nicht erneut.",
|
||||
"sidecarRejected": "{{url}} antwortet, aber nicht wie der Gesichtserkennungs-Dienst — Fotos werden als fehlgeschlagen markiert statt erneut versucht. Prüfe, ob FACE_ML_URL auf den picpeak-ml-Container zeigt und kein Proxy dazwischenliegt, und nutze dann Neu scannen für die bereits fehlgeschlagenen Fotos.",
|
||||
"sidecarStateNow": "Aktueller Zustand des Dienstes — einige der Fehler oben können eine andere Ursache haben, aber ein erneuter Scan wird erst nach der Behebung erfolgreich sein:"
|
||||
}
|
||||
},
|
||||
"acceptInvitation": {
|
||||
|
||||
@@ -2733,7 +2733,11 @@
|
||||
"deleted": "Face data deleted",
|
||||
"confirmDelete": "Delete all detected people and face data for this gallery? This cannot be undone. Photos are not affected.",
|
||||
"autoCategories": "Sort photos into categories automatically",
|
||||
"autoCategoriesHint": "Uses the number of faces to file photos as Details, Portraits, Small groups or Groups. Applies to every gallery, only ever fills an empty category, and never changes one you set yourself."
|
||||
"autoCategoriesHint": "Uses the number of faces to file photos as Details, Portraits, Small groups or Groups. Applies to every gallery, only ever fills an empty category, and never changes one you set yourself.",
|
||||
"sidecarUnreachable": "Can't reach the face-detection service at {{url}}, so the {{pending}} queued photos aren't being processed. Nothing is lost — the scan resumes on its own once the service is up. Start it with `docker compose --profile faces up -d`, and note it exits immediately unless FACE_ML_TOKEN is set to the same value as the backend — there is no default.",
|
||||
"sidecarUnauthorized": "The face-detection service at {{url}} is rejecting our token, and photos are being marked failed rather than retried. Make FACE_ML_TOKEN identical on the backend and the picpeak-ml container, restart both, then use Re-scan — fixing the token alone will not reprocess the photos that already failed.",
|
||||
"sidecarRejected": "{{url}} answered, but not like the face-detection service — photos are being marked failed rather than retried. Check FACE_ML_URL points at the picpeak-ml container and that nothing is proxying that address, then use Re-scan for the photos that already failed.",
|
||||
"sidecarStateNow": "Service state right now — some of the failures above may have a different cause, but a re-scan will not succeed until this is fixed:"
|
||||
}
|
||||
},
|
||||
"acceptInvitation": {
|
||||
|
||||
Reference in New Issue
Block a user