2df455784c813e44409d87420ebea9ae73bdeeef
50 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Luca
|
2df455784c |
ci(docker): mirror the all-in-one image to Docker Hub
The aio image (#1042) shipped GHCR-only with a TODO to wire the Docker Hub mirror once the Hub repo existed. backend, frontend and the ml sidecar all publish to docker.io/picpeak/*; aio was the only image a Docker Hub user could not pull. merge-aio now follows merge-backend/merge-ml verbatim: DOCKERHUB_ENABLED computed from the repository slug (so forks stay GHCR-only), a gated Docker Hub login, docker.io/picpeak/aio added to the metadata images list, and a Docker Hub manifest inspect. Tag scheme is untouched — the same beta/main/stable/latest/semver tags land in both registries. The build summary drops the "Docker Hub mirror pending" note and lists the aio (and ml) Hub images when the mirror is active. |
||
|
Paul Nothaft
|
37a15e3d49 |
fix(faces): restore the :beta image tag and surface sidecar health (#1087)
* fix(faces): restore the :beta image tag and surface sidecar health
Both halves of what a user hit on discussions/1069: the People card sat
at "Scanning… 0 of 227" for 30 minutes with no explanation, because the
sidecar container could never have started.
docker-build.yml — republish `:beta`. It used to come for free via
`type=ref,event=branch` when the active development branch was literally
named `beta`; the rename to `main` silently retired it. backend:beta has
been frozen at 2026-06-29 (
|
||
|
Paul Nothaft
|
74a8f9bf24 |
chore(security): shrink the ML image's CVE surface, override deepmerge-ts (#1083)
Three Trivy cleanups off the code-scanning tab. ml/Dockerfile — install no runtime apt packages at all. Neither libgl1 nor libglib2.0-0 is needed: opencv-python-headless 4.14 bundles what it needs and `ldd .../cv2/cv2*.so` resolves fully on a bare slim base. The old comment claimed the headless wheel still links libGL, which was true of much older wheels. libgl1 was dragging in 36 transitive packages (mesa, LLVM, X11) for a service that never opens a display. Measured with `trivy image` on locally built variants: before: 165 findings — 88 low / 49 med / 19 high / 6 crit without libgl1: 133 findings — 58 low / 48 med / 19 high / 5 crit without either: 123 findings — 57 low / 46 med / 13 high / 4 crit 42 findings gone, image 1.05GB -> 774MB. Not one of the 165 had an upstream fix available, so not installing the packages is the only lever there is. docker-build.yml — set ignore-unfixed on all four Trivy steps. All 123 remaining ML findings are unfixed base-OS CVEs; Debian has them resolved in sid and pending backport to trixie, and apt-get upgrade -y behind CACHEBUST picks each one up automatically. Reporting them buries anything actionable, and suppressing them is the precondition for ever setting exit-code: 1. backend — deepmerge-ts <8.0.0 has a stack-exhaustion advisory (CVE-2026-40345, high) reached via mailparser -> html-to-text, which pins ^7.1.5 so npm cannot get there alone. Not reachable in our code: html-to-text only feeds deepmerge-ts its options object (html-to-text.mjs:1468, :1442), never parsed email content. npm audit goes 3 high -> 0. The lockfile also picks up the version field release-please had left at 3.103.1-beta.0, plus some "peer": true metadata npm 11.6 recomputes. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
b69dd134d0 |
feat(faces): People in this gallery — face recognition via an optional ML sidecar (#1074) (#1075)
* feat(ml): optional face-detection sidecar, opt-in and inert by default (#1074) First of four PRs for "People in this gallery". This one ships only the sidecar, its wiring and its CI — no schema, no backend code, no UI. Nothing in PicPeak calls it yet. picpeak-ml is a single FastAPI + onnxruntime container: three endpoints (/health, /info, /faces), no database, no volumes, no egress, no model download at runtime. Clustering, person identity and every privacy decision stay in the backend where the data already lives. Models are YuNet (detection) + FaceNet-512 (embedding), both MIT, both pinned by URL and SHA-256 and verified at build time. The licence analysis is in ml/LICENSES.md: the more accurate InsightFace weights are non-commercial-only and PicPeak's users are working photographers, so they are never baked into an image we publish. Two things worth review attention: - Alignment uses a least-squares similarity transform (Umeyama), NOT cv2.estimateAffinePartial2D. RANSAC and LMEDS exist to reject outliers among many correspondences; given five landmarks and no outliers they fit a three-point subset exactly and let the rest drift. Measured on a real off-frontal portrait: eyes and nose pinned to 0.11px, mouth corners 11.8px out on a 160px crop. Umeyama distributes it (max 6.5px, rms 5.1 vs 7.4). The failure mode is silent — a bad warp still yields 512 confident floats — so tests/test_pipeline.py pins it numerically. - FACENET_ONNX_URL has no default and the build fails loudly without it. deepface distributes FaceNet-512 as Keras .h5 only, so the ONNX is produced once by tools/convert_facenet.py and published as a release asset. Converting inside the build would drag TensorFlow through both architecture legs of every build to produce a byte-identical file. The CI jobs are gated on the FACENET_ONNX_URL repository variable and skip cleanly until it is set. Off by default, twice over: the sidecar is behind the `faces` compose profile, and the backend will gate on a `faces` feature flag that defaults to false. FACE_ML_URL defaults to http://picpeak-ml:8000 so the standard deployment needs no configuration — nothing dials that host while the flag is off, which is why a non-resolving default is harmless. Verified: 27 pytest tests green; YuNet loads and detects against a real portrait with its landmark order matching the alignment template index-for-index; both compose files validate and the faces profile is correctly excluded from a default `up`; workflow YAML parses and the job graph resolves. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(ml): pin the converter toolchain, verify parity, drop a false reproducibility claim (#1074) Ran the FaceNet-512 conversion for real and corrected what the previous commit assumed about it. The conversion works: 23,497,424 parameters, 89.6 MB ONNX, and the converted graph matches the Keras original to 2.086e-06 absolute / cosine 1.0000000000. That check is now part of the script rather than something I did once by hand — a subtly wrong graph still returns 512 plausible floats, so it refuses to leave the file on disk if parity fails. Also ran the full pipeline against both real models end to end. The embedding is L2-normalized to 1.000000, and the same face survives being re-rendered: half scale 0.973, double scale 0.984, JPEG q40 0.987, rotated 8 degrees 0.984, brightness +40 0.988. Scale invariance in particular is evidence the alignment warp is doing its job. Corrected claim: the conversion is NOT byte-reproducible. Two runs with the same pinned versions on the same machine gave different SHA-256s. The graphs are functionally identical — same 336 nodes, same 271 initializers, every weight matching to 0.000e+00 — but a few initializer names differ because tf2onnx's traced-op naming is not deterministic (Keras layer naming is deterministic; I checked). The previous commit message and README both claimed byte-identical output. They were wrong, and it matters: anyone re-running the conversion gets a different hash, and without this note that reads like tampering. The build-time SHA-256 pins one published artifact so its URL cannot start serving different bytes; validating a fresh conversion is the parity check's job. requirements-convert.txt now pins the exact set that produced the artifact, including transitive keras/protobuf/numpy, and documents that the converter needs Python 3.11 while the image runs 3.12. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): schema, queue, clustering and API for People in this gallery (#1074) Backend half of the feature. Migration 177, a face-detection queue, the clustering engine, the gallery and admin APIs, and the privacy wiring. No UI yet; nothing is reachable until the `faces` feature flag is on, which defaults to false. The flag is the gate, not FACE_ML_URL. That variable now has a working default (the compose service name), so its presence proves nothing about intent — if it were the gate, every install would poll a hostname that does not resolve. faceQueue re-checks the flag every tick, so turning it off stops the workers without a restart. Visibility scoping is the part worth reviewing closely. Face rows have no concept of photo visibility, but guests are restricted to photos.visibility='visible'. A raw count leaks how many hidden photos someone appears in, and an unscoped cover face renders a crop of a photo the guest may not open — with the best-scoring face being the likeliest pick, so it would happen often rather than rarely. facePeopleService recomputes both per request against the caller's own scope, and event_people.face_count_total is named to be conspicuous in a guest path. Six tests cover it, including the case where a person's photos are ALL hidden and they must vanish entirely. Face data is excluded from backups and .picpeak exports, per the decision in the thread: it is derived, so a restore re-scans rather than carrying biometrics between operators. Three separate mechanisms, because the engines cannot be filtered alike — EXCLUDED_TABLES for export, --exclude-table-data (not --exclude-table; the CREATE TABLE must survive or restore breaks on the first query) for Postgres, and DELETE + VACUUM on the temp copy for SQLite, which has no way to exclude a table from a whole-file .backup. The VACUUM is not cosmetic: without it the pages stay in the file and the claim is false on disk. Archiving now purges face data explicitly. photo_faces cascades off photos, but archive deletes neither the photo rows nor the event, so without this an archived gallery kept its biometrics indefinitely. Other decisions: clustering keeps names across a re-cluster by majority inheritance (without it, one button click silently discards every name the photographer typed); consolidation refuses to merge two people who were named differently; assignment never compares across model_version, since embeddings from two pipelines are not comparable; low-quality faces are stored but left unassigned so they show in "this photo contains" without spawning junk people. Migration is 177, not 174 — 174/175/176 landed on main while this branch was open. 29 tests green: 7 migration (idempotency, down(), cascade, and that installing it enqueues NOTHING), 11 clustering, 11 privacy/visibility. Lint clean; the pre-existing error counts in databaseBackup.js and server.js are unchanged. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): People strip, face filter and admin controls (#1074) Frontend half. Renders nothing anywhere unless the `faces` feature flag is on AND the photographer enabled detection for the gallery — the whole guest surface hangs off one boolean, event.people_enabled, which the server computes from the flag, the per-event toggle and the show-to-guests toggle together. Guest side: a People strip between the filter bar and the grid, circular crops from each person's cover face, an active-filter chip row, and a "Show all" bottom sheet. The face filter composes with category, search, media type and the liked/saved/rated filters in the same useMemo rather than replacing them, so "photos of Anna that I liked" works. Two people selected means AND by default — that is what picking a second face almost always asks for — with a toggle to OR that appears only once a second person is picked. Unnamed people show a photo count and never "Person 7". A number is honest about what the system knows; an invented name is not. There is a test asserting we don't do it. The strip renders nothing below two people, collapses to one line when dismissed (persisted per slug, so dismissing one gallery says nothing about the next), and appears mid-backfill with a progress line rather than blocking the gallery behind a spinner. Avatar crops are computed in ratios of the source dimensions so they survive whatever rendition the browser gets; without width/height they fall back to an uncropped thumbnail, since a wrongly-offset crop is worse than no crop. No new download endpoint: "download these N" rides the existing photoIds path, which already enforces access level and per-category permissions server-side. Adding a person_id selector would have been a second thing to authorize for no gain. Guest-facing copy never says "biometric" or "recognition" — those words describe our implementation, not the guest's experience. The sheet's footnote answers the first question every guest has (where does this go?) inline. The admin card, by contrast, is explicit: it states the controller obligation next to the toggle, and warns that scanning materializes the preview tier on galleries that never generated one, which is real CPU and disk an admin should know about before a 2,000-photo backfill. EN + DE translations. 140 frontend tests green (8 new), tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): measured match threshold, working build defaults, 89MB smaller image (#1074) Ran the Phase 0 spike that had been outstanding, published the model, and fixed what both turned up. THRESHOLD IS NOW MEASURED, NOT GUESSED. LFW's standard 1000-pair protocol run through this exact pipeline (YuNet -> Umeyama alignment -> FaceNet-512 ONNX), 100% detection on 2000 images: same person cosine 0.6958 +/- 0.1415 diff person cosine 0.0849 +/- 0.1674 separation 0.6109 peak accuracy 96.60% @ 0.405 So the pipeline separates people well — the thing I could not previously claim, since every earlier number was the same face re-rendered. Default moves 0.62 -> 0.50. The old value was a placeholder and a bad one: it gave 0% false merges but 22.4% false splits, i.e. roughly one in four same-person pairs failing to join, which fragments a gallery badly. 0.50 gives 1.0% false merge / 8.2% false split. Peak accuracy (0.405) is deliberately NOT chosen: for clustering the two errors do not cost the same. A false split is a duplicate row the photographer can merge away; a false merge puts a stranger into someone's "download my photos" — and until the Phase 2 merge/split UI ships, there is no way to undo one. So this sits on the conservative side of the optimum. The spike is committed as ml/tools/benchmark_threshold.py rather than thrown away, so "why 0.50?" has an answer in six months and a re-tune is one command. BUILD DEFAULTS. FACENET_ONNX_URL/_SHA256 now default to the published ml-models-v1 release asset, so `docker build ml/` and `docker compose --profile faces up` work with no arguments. Blanking either still fails loudly — a URL without a checksum is never acceptable, since the checksum is what makes the URL safe to trust. Found by running compose for real: it failed exactly as designed, which was correct behaviour and a bad out-of-box experience now that a canonical artifact exists. IMAGE SIZE. 389MB -> 300MB single-arch. `chown -R` after COPY rewrote every copied file into a fresh layer, duplicating the 90MB model for nothing; the user is now created before the copies and ownership set via COPY --chown. Also drops pip/setuptools from the runtime image. Measured RSS is 186MiB idle, and the container answers /faces end-to-end in well under the 80-150ms/photo the issue budgeted. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): threshold 0.50 -> 0.60 from real clustering, theme-aware People strip (#1074) Both fixes come from running the feature on an actual gallery — 61 photos, 5 real identities — rather than reasoning about it. THRESHOLD. The LFW pairwise sweep in the previous commit said 0.50, and it was wrong. On a real gallery at 0.50, three of six visible clusters were contaminated: two different people merged into one strip entry, which is the exact failure that puts a stranger into someone's "download my photos". Pairwise error rates do not predict cluster purity. Greedy assignment compounds — one wrong face drags the centroid toward the midpoint between two identities, making the next wrong face likelier. A 1% pairwise false-merge rate is not a 1% chance of a clean gallery, and no amount of staring at an ROC curve would have shown that. Sweep against ground truth (5 identities): 0.50 -> 6 clusters, 3 contaminated 0.56 -> 6 clusters, 0 contaminated 0.60 -> 5 clusters, 0 contaminated <- exactly right 0.64 -> 5 clusters, 0 contaminated, fewer faces assigned 0.60 recovers the right number of people with no contamination; higher only loses coverage. Migration 177 carries the full reasoning so the next person to touch this knows why the obvious pairwise answer is the wrong one. THEME. The People strip hardcoded `text-neutral-800` for named people. On a dark gallery — which the screenshot immediately showed — that renders a named person's label almost invisibly, while UNNAMED people stayed legible. Exactly backwards. Labels, headings, the collapsed summary, the scan line and the filter chip row now read the gallery's own theme tokens (--color-text / --color-muted-text / --color-accent / --color-surface-border) like the rest of the gallery surface. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): keep the mobile filter row inside the viewport (#1074) At 390px the photo count and Clear link were pushed against the right edge by ml-auto and clipped. Only apply it from the sm breakpoint up, where there is room; below that they flow after the chips. Found by screenshotting the real thing on an iPhone-sized viewport. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): complete Phase 1, add People management and auto-categories (#1074) Closes the two Phase 1 gaps, then builds Phase 2 and Phase 3. PHASE 1 GAPS. "Download these N" was specified, described as done in an earlier summary, and never actually built — I had verified the backend needed no new endpoint and let that stand as if the button existed. It now hands the filtered photo ids to the same path as a manual selection, so the server re-applies access level and per-category permissions on the way through. Photos in a downloads-disabled category are excluded client-side too, so the number on the button is the number the guest receives. Hidden entirely when downloads are off for the gallery. Lightbox person chips ("In this photo: Anna") are the second way into the face filter — a guest looking at a photo of themselves can act on it without scrolling back to the strip. Tapping one closes the lightbox and filters the grid behind it. PHASE 2. A People management modal over the endpoints that already existed and were already tested: rename inline, merge (multi-select, first pick is the target so the name a photographer typed survives), split via a face picker, hide, ignore. This matters more than it sounds — clustering deliberately errs toward splitting because a wrong merge puts a stranger into someone's download, and that trade only works if merging is easy. PHASE 3. Rule engine over face_count plus face-area ratio: 0 -> Details, 1 large -> Portraits, 2-5 -> Small groups, >5 -> Groups. The area ratio is what separates "a portrait of someone" from "someone is in this landscape". Three guarantees, all tested: it only ever fills an EMPTY category (enforced in the query AND re-checked in the UPDATE, so a photographer setting one mid-run still wins), everything it touches is marked auto_categorized so undo is exact, and it is a no-op unless separately enabled. Migration 178 adds the column — separate from 177, which has already run wherever this branch is deployed. Verified on the real gallery: 61 photos -> 48 portraits + 13 small groups, undo cleared exactly 61 and left the manual ones alone. Merge moved faces and removed the source. Both confirmed against the database, not just the UI. TWO BUGS THE BROWSER CAUGHT, both invisible to tsc: - The lightbox destructure never landed — my patch targeted a line that has a default value, matched nothing, and failed silently. `people` resolved to something else entirely and the chips would never have rendered. eslint's "outer scope value" warning is what surfaced it. - Admin face thumbnails 403'd because <AuthenticatedImage> attaches whatever gallery token is in session storage; an admin who has also opened one of their own galleries sends a type:"gallery" bearer to an admin route. Admin routes authenticate from the httpOnly cookie, which a plain same-origin <img> sends by itself. Worth noting AdminPhotoGrid has the same latent shape; not touched here. Also: the admin card now reports "N people (M shown to guests)" when those differ, so the settings page and the gallery stop disagreeing without explanation. 45 backend tests (8 new) and 140 frontend tests green; tsc and eslint clean. EN + DE for every new string. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * perf(faces): batch migration DDL and drop the face stack from server.js import (#1074) CI's backend job timed out at 10 minutes on the first run of this branch. Nothing failed — 132 of 182 suites passed and the wall clock ran out. Main does the same 182 in 124s, and where main has 12 suites slow enough for jest to print a duration, this branch had 77. Two changes, both worth making regardless of how much of the gap they close: - Migration 177 added its columns one ALTER TABLE at a time (four on photos, three on events, plus a separate index statement) and seeded settings with a SELECT and an INSERT per key. It now uses one alterTable per table and one SELECT plus one bulk INSERT. 178 folds its index into the same statement as its column. That chain replays in ~90 suites, so statement count there is multiplied by 90. - server.js required faceQueue at module scope, which pulls in axios and — through imageProcessor — sharp. Every supertest suite that imports server.js was paying for a module graph it never uses. Now required inside the startup block, next to the call that needs it. Honest about the evidence: locally the migration delta measures at zero (1.15s vs 1.13s for the same suite, three runs each), so batching alone does not explain an eight-minute regression. A fast local disk and many cores mask per-statement and per-import costs that a two-core runner with a shared disk does not. These are the two real costs this branch added to a path that runs in almost every suite; whether they are sufficient is a question for CI, not for another round of local speculation. 37 face tests still green after the change. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * i18n(faces): complete EN and DE coverage for the face feature (#1074) The admin card and the Features toggle were rendering entirely from inline English `defaultValue` fallbacks — 22 keys existed in no locale file at all, so a German admin saw an English consent notice, English toggles and English buttons. The gallery side was already translated; the admin side was not, and nothing in the toolchain flags this because a `defaultValue` always renders something. Adds the missing `admin.faces.*` (19), `settings.features.faces.*` (2) and shared `common.clear/saved/saveFailed` in both languages. Existing keys are left alone (setdefault, not overwrite), so the shared `common` strings other features rely on are untouched. Committed the audit as frontend/scripts/i18n-faces-audit.py rather than throwing it away: it extracts every t() key the face components actually use and diffs it against each locale, and it also reports German values that are byte-identical to English, which is the usual shape of an untranslated copy-paste. Currently: 69 keys in use, EN complete, DE complete, no identical pairs. Verified in the browser, not just in the JSON — the German card reads "61 / 61 Fotos durchsucht · 16 Personen (5 für Gäste sichtbar)" end to end. Also checked the components for hardcoded user-facing text (JSX nodes, title/aria-label/placeholder attributes) outside t(); there is none. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 13 defects from external review — coordinates, counts, erasure, races (#1074) Codex reviewed the branch against main. Thirteen findings, nine P1. I checked every one against the code and could not dismiss a single one as a false positive, so all thirteen are fixed here. THE WORST ONE: bounding boxes were stored in the wrong coordinate system. The sidecar reports coordinates in the space of the image it was HANDED — which is the ≤1920px preview, not the original — while every consumer compares them against photos.width/height, the original dimensions. A 6000px photo therefore produced boxes ~3x too small and areas ~9x too small: avatar crops landed in the wrong place and the Portraits rule could never fire. It is invisible on any photo already under 1920px, which is exactly why the demo gallery and every screenshot looked correct. Now scaled once in faceProcessor so everything downstream can assume original-image coordinates. ERASURE. The FK cascade on photo_faces is decorative on SQLite: PicPeak never enables `PRAGMA foreign_keys`, so deleting a photo left its embeddings behind. I first enabled the pragma globally and reverted it — six unrelated suites immediately failed on pre-existing dangling references, and switching it on would start rejecting inserts on every existing install. That is a real change worth making, but it is its own PR, not a rider on this one. Instead deletion purges explicitly: purgePhotoFaces in the photo paths (single, bulk, service) and photo_faces/event_people in deleteEventCascade. Tests assert this with the pragma explicitly OFF, so they can only pass if the code does the work. COUNTS. A re-scan deleted the old face rows without undoing their contribution to event_people, so counts inflated on every re-scan and ghost people survived. Now the affected people are recomputed before the replacements are assigned. My own "must not double its faces" test only checked photo_faces rows, which is why it passed throughout. RACES. A worker that finished after an admin purged the event committed its rows anyway — erasure reported success and the data reappeared. The commit is now conditional on the row still being 'processing'. And assignFaces is read-modify-write over an event's people, so two workers lost each other's updates; it is now serialised per event with an in-process mutex plus a Postgres advisory lock for the multi-pod case the queue advertises. METADATA LOSS. Merging discarded the source's name and suppression flags, so a merge could erase a typed name or un-hide someone. Reclustering remembered only people with a label, so an unnamed-but-hidden bystander came back guest-visible after one "Re-group people" — and suppression now propagates to every descendant cluster, not just the majority one. Also: export reset face_status so a restored gallery re-scans instead of claiming to be scanned forever; manual category edits clear auto_categorized so "undo automatic" cannot delete a photographer's own choice; external photos are skipped rather than failed (resolvePhotoStorageKey returns null for them by design); the gallery refetches photo memberships as a scan progresses so filtering is not stale; a failed VACUUM now fails the backup rather than publishing one that may retain biometric pages; and the ML Dockerfile's `|| true` is scoped to the uninstall — as written it was `(install && uninstall) || true`, so a failed dependency install produced a green layer and an image with no onnxruntime. Four new regression tests. Full backend suite failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): 12 more defects from review round 2 — cross-event purge, leaks, lifecycle (#1074) Second Codex round on the same diff, now including round 1's fixes. Twelve findings, seven P1. Again none were false positives. SECURITY, AND MINE FROM ROUND 1: the bulk-delete face purge iterated the raw `photoIds` from the request instead of the event-scoped `photos` rows the handler had already validated. purgePhotoFaces has no event scope of its own, so an editor could pass another gallery's photo id and delete its face data — even though the photo deletion right below it was correctly scoped. Fixing one thing and introducing another is exactly why the second round was worth running. ANOTHER VISIBILITY LEAK, same class as the one round 1 fixed: /people returns scan progress, and getScanStatus counted every photo with a face_status — including hidden ones. Guests could read the hidden-photo count off the progress bar while the people list and covers beside it were properly scoped. Now scoped by the same predicate, with the caller passing its audience. RECLUSTER, ROUND 1'S FIX WAS INCOMPLETE. I made suppression follow every descendant but still copied the flags from the majority ANCESTOR. When reclustering merges a visible named person with a hidden one, the majority ancestor is often the visible one — republishing the hidden person's photos. Suppression is now OR-ed across every ancestor contributing faces. The name also now goes to the genuine largest descendant; the previous code took whichever cluster came first in map order, which the comment already claimed it did not. LIFECYCLE. Face data is excluded from backups and exports, but photos. face_status came across intact, so a restored install claimed every photo was scanned while holding no faces — and the worker only claims 'pending', so it stayed that way forever. Now: the SQLite backup requeues in the dump, restore requeues after the pool reinit (the Postgres path cannot rewrite rows inside pg_dump), the portable importer purges LOCAL face tables (they were excluded from the replace list, so another instance's embeddings survived an import with FK checks suspended) and requeues, and archiving disables detection so a restored archive is honestly off rather than enabled-and-empty. WRITE PATHS. Only processPhoto enqueued. The synchronous upload path (chunked-upload completion, watch-folder) left photos unscanned, and replacePhoto kept the OLD image's faces on a row now pointing at a different picture — stale identities shown on the new photo. FRONTEND. PeopleSheet and the admin manager rendered centred thumbnails and ignored the bbox, so on group photos the avatar showed whoever stood in the middle and two people from one photo were indistinguishable — in the manager whose entire job is telling faces apart. The crop maths is now one shared helper (faceCrop.ts) so the three surfaces cannot drift again. Full-page layouts (gallery-premium, gallery-story) render their own lightbox and never received the people props. Backend failure set verified identical to origin/main; frontend 140 green; tsc and eslint clean. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(faces): round 3 — five of round 2's fixes were wrong or no-ops (#1074) Third and final Codex round. Eight findings, four P1 — and the important part is that FIVE of them are defects in round 2's fixes, not in the original code. - The sync-upload enqueue I added was a silent no-op. It queried through `trx` after the transaction had already been committed, which throws "Transaction query already complete" straight into the catch I had wrapped it in. Chunked uploads and watch-folder imports were still never scanned, and the code read as though they were. Uses `db` now. - The post-restore requeue ran BEFORE the files were restored, in both the portable importer and the native restore. The face worker is live during a restore, so it could claim those rows and scan the previous instance's files, or fail them for originals not yet on disk — with nothing to requeue them afterwards. Both now run after file restoration; the native one is extracted into requeueFaceScans() and called from the full and database-only paths. - The admin face crop mixed coordinate spaces: an original-pixel bbox scaled against the THUMBNAIL's natural size. The API now returns the source dimensions alongside the box, so there is one space to reason about. - Forwarding people props through layoutProps did not make them work — the full-page layouts never destructured them. GalleryStoryLayout now threads them to its own lightbox. Genuinely new findings, all in the same class as ones already fixed: - releaseToPending updated unconditionally, so a photo purged while its sidecar request was in flight came back as 'pending' and was rescanned — biometric rows reappearing after the purge reported success. Round 2 fixed exactly this on the COMMIT path and I did not carry it to the retry path. Now guarded on 'processing'. - purgePhotoFaces left face_status alone, so a worker mid-scan still satisfied its commit guard and could write fresh faces into a photo being deleted — orphans, since the FK cascade is inert on SQLite. It now clears the claim as part of the purge. - Phase 3 was unreachable: the migration seeds face_auto_categorize_enabled false and nothing could ever write it, so the rule engine and its undo endpoint returned "disabled" in every real flow. Added GET/PUT and a toggle on the admin card, EN + DE. NOT fixed, deliberately: GalleryPremiumLayout uses yet-another-react-lightbox rather than the shared PhotoLightbox, so person chips there are a real port rather than a prop forward. Recorded as open rather than bodged. Backend failure set identical to origin/main; 41 face tests and 140 frontend tests green; i18n audit reports EN and DE complete at 71 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(faces): block face recognition on the all-in-one image (#1074, #1042) The single-container image cannot run this feature, so it is refused there rather than left to degrade. WHY, since the reason is not obvious from the code: the AIO image runs the backend, the frontend, SQLite and every background worker inside one container aimed at "one photographer plus guests browsing". It has no Redis, SQLite gives it a single writer, and it contains no ML sidecar to talk to. Face detection would add a second image-processing pipeline competing with Sharp for the same CPU and memory. That failure is not loud — the install just becomes slow and looks broken, which is the worst possible shape for a deployment whose whole promise is one container and no decisions. Gated on an explicit PICPEAK_SINGLE_CONTAINER marker, NOT inferred from SERVE_FRONTEND or a SQLite path: plenty of legitimate multi-container setups serve the frontend from the backend or run SQLite, and none of them should lose the feature by accident. Three layers, because the first is the only one that enforces: - faceSettings.isFeatureEnabled() returns false before consulting the flag, so a database restored from a full deployment with `faces` enabled still cannot switch it on here. - The feature-flag API forces `faces: false` in both directions, so the admin UI reflects reality instead of offering a switch that refuses to stay on. - The Features tab renders the card disabled with a plain-language reason, read from a new `single_container` field on /admin/system/version (an endpoint the admin UI already calls). Documented in ml/README.md and .env.example. Three tests pin the behaviour, including that the marker only accepts explicit truthy values. NOTE FOR PR #1068: this expects `Dockerfile.aio` to set `ENV PICPEAK_SINGLE_CONTAINER=true`. That one line lives on that branch and is not in this commit — until it lands, an AIO build would still offer the feature. Worth adding alongside the `Limits` section of docs/single-container.md. 44 face tests green; EN + DE complete at 72 keys. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(faces): pin the bbox coordinate space with a real scale factor (#1074) The coordinate-space bug — boxes stored in preview space while every consumer reads them as original-image pixels — had no test, and could not have been caught by the ones that existed: every photo in the demo gallery is 750px, so the scale factor was always exactly 1.0 and the correction never executed. Verified by hand first, on a real 4000x3000 upload with the face placed off-centre so a wrong crop would be unmistakable. Before the fix the stored box was 1493,204 (preview space, face actually at x≈2850-3618); after, 3110,426 — a factor of 2.083, exactly 4000/1920, landing inside the face. The admin crop then resolved to left=-395px/top=-46px on a 64px window, which is the face centred. That verification is now a test rather than a memory. Three cases: a 4000px photo must scale by 4000/1920, a 1920px photo must NOT change (the case that hid the bug), and a row with no width must fall back to unscaled rather than storing NaN. Note for anyone extending these: jest hoists mock factories above the file, so anything they close over has to be `mock`-prefixed. Getting that wrong fails at transform time with a message that does not name the variable. 47 face tests green. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Luca
|
0874a30ac9 |
feat(docker): all-in-one image (#1042) — my version of #1067 (#1068)
* feat(docker): add all-in-one image — backend + frontend in one container (#1042) One container, one Node process, SQLite by default: `docker run` with no compose file, no nginx, no supervisor, no bundled Postgres/Redis. - Dockerfile.aio (repo-root context): frontend build stage + backend deps stage + a runtime stage mirroring backend/Dockerfile's production stage, with the built SPA copied to /app/frontend/dist and SERVE_FRONTEND=true. DATABASE_CLIENT=sqlite3 and STORAGE_PATH=/app/storage are pinned explicitly — the storage fallback resolves to container-root /storage, which EACCESes after the su-exec drop. - server.js: the SERVE_FRONTEND block now does what the nginx image did — renders ${BRAND_TITLE}/${BRAND_DESCRIPTION} into index.html once at boot, serves that rendered shell on /index.html and every SPA route, caches hashed /assets/* immutably while the shell revalidates, and gzips the bundle via compression() mounted after all /api routers. express.static now runs with index:false so `/` keeps flowing to handlePublicSiteRequest — its default index option was shadowing the landing page on native installs. - wait-for-db.sh: skip the Postgres readiness wait when DATABASE_CLIENT is sqlite3. The engine resolver still runs, still logs, and still refuses the populated-both conflict (#1038). - .dockerignore: **/node_modules, so the root-context build can't pick up host deps from backend/ or frontend/. - docker-build.yml: build-aio / merge-aio follow the same per-arch build → digest-merge → per-version tag scheme as backend/frontend (GHCR only for now; the Docker Hub mirror is wired once the Hub repo exists), plus a smoke-aio job that boots the image on every PR and asserts /health, the SPA shell, the rendered brand title, immutable asset caching and the SQLite engine resolution. Pointing DB_HOST/DB_USER/DB_PASSWORD + DATABASE_CLIENT=pg at an external Postgres works exactly like the backend image. * fix(ci): correct three smoke-aio assertions that would fail a green image (#1042) Found by running the smoke job locally against a real build — the image passed every behavioral check, but three assertions were wrong: - `/` asserts 200, but handlePublicSiteRequest 302s to /admin/login while the public landing site is disabled, which is the state of the fresh install the smoke container always is. Assert the redirect target instead — that still proves express.static's index option is not shadowing the handler, which is the thing the check exists for. - The placeholder-leak grep matched index.html's explanatory comment, which mentions BRAND_TITLE in prose and survives into the built shell. Match the literal ${BRAND_TITLE}/${BRAND_DESCRIPTION} tokens with -F, and cover the description token too. - Add a gzip assertion, probing with GET: the compression middleware skips bodyless responses, so a HEAD probe reports no Content-Encoding even when compression is active. Verified locally on linux/arm64: image builds clean, boots to healthy in ~8s on the SQLite default, and 25/25 checks pass (SPA shell, rendered brand title, immutable+gzipped assets, no-store shell, SPA fallbacks, npm removed, su-exec drop to nodejs, no errors in the boot log). The DATABASE_CLIENT=pg override was exercised against a real Postgres too — the readiness wait still runs and the engine resolves to postgres. * fix(server): serve the SPA for every client route, not just /admin and /gallery (#1042) nginx did `try_files $uri $uri/ /index.html`, so behind compose every client-side route survived a direct hit or a refresh and the short `['/admin', '/admin/*', '/gallery/*']` list was never exercised. Without nginx that list is the whole contract, and everything outside it 404'd: /setup /customer /impressum /datenschutz /payment-check /quote/:token /contract/:token /invite/:token /transfer/:token /transfer-upload/:token /setup is the first URL a new install visits, so the all-in-one image was unusable from a cold start. The catch-all is registered after `app.use('/api', notFoundHandler)`, so an unknown /api route still answers JSON instead of being handed the HTML shell, and after the /s/:shortSlug resolver, so a typo'd short URL still 404s (#699). It is GET-only — a stray POST keeps 404ing rather than getting a 200 page back. The handler is hoisted out of the SERVE_FRONTEND block via `spaCatchAll` because that block runs before the API 404 handler is registered. Verified on the built image: all ten routes above now 200, /api/nope still returns JSON 404, /s/nonexistent still returns 404, / still 302s to /admin/login, and the smoke suite is 25/25. Both boundaries are now asserted in the smoke-aio job. * docs(readme): document the single-container install (#1042) The README had no mention of the all-in-one image, so the only way to discover it was reading the workflow file. Adds a Quick Start subsection with the one-line `docker run` and the `docker exec … cat SETUP_TOKEN` step, plus a row in the documentation table. Deliberately does not sell it as the default: the note says the compose stack is still the right choice for anything busier, gives the reason (SQLite takes one writer at a time), and points at the `.picpeak` restore as the way out, so nobody picks it and then finds themselves stuck. Full details live at docs.picpeak.app/deployment/single-container (PicPeak/docs#8). * feat(docker): fold #1067's items into the all-in-one image (#1042) Consolidating the two parallel AIO branches into this one. This PR's approach is kept wherever the two differed on design — in particular the in-process brand render, `index: false` (which fixes express.static shadowing handlePublicSiteRequest, a bug #1067 had), the compression middleware, and the smoke-aio job. What follows is what #1067 had that this branch did not. Layout — the issue asks for a single mountable root, and this moves to one: /data/db picpeak.db (+ -wal/-shm) and SETUP_TOKEN /data/storage originals, thumbnails, archives /data/logs application logs /data/backup built-in backup output; /backup symlinks here `-v picpeak:/data` and nothing else to remember. README and the smoke job's database-path assertion follow the new layout. Correctness items: - sqlite CLI. DatabaseBackupService SPAWNS `sqlite3` for `.backup` and PRAGMA integrity_check; the npm module does not ship that binary. backend/Dockerfile omits it because compose always runs Postgres — this image defaults to SQLite, so every database backup failed with ENOENT. - /backup wired in. Migrations 029 + 030 seed /backup/picpeak and /backup/database as the backup destinations; nothing created or mounted them, so backups had nowhere to write and anything written would die with the container. Symlinked into the volume, subdirectories created at startup (a bind mount hides the tree baked into the image), and adopted only when BACKUP_DIR is set so it never gates boot for compose deployments that do not mount it. - logger.js honours LOG_DIR. It hard-coded <backend>/logs, so logs could not leave the container. Unset keeps the old path for every existing install. - wait-for-db.sh derives its writable roots from STORAGE_PATH / DATA_DIR / LOG_DIR instead of hard-coded /app paths, and mkdir -p's them before chown — a bind-mounted /data hides the image's tree, and chown against a missing path reports "the filesystem rejects chown", which is both wrong and a dead end. - .dockerignore excludes backend/-prefixed runtime data. Docker reads only the root file, so the unprefixed data/*.db, logs/* and storage/* rules missed backend/data, backend/logs and backend/storage entirely; a checkout used to run PicPeak would bake its database, photos, logs and SETUP_TOKEN into a published layer. - HEALTHCHECK follows $PORT rather than a hard-coded 3000. - --max-http-header-size=32768 matches nginx's large_client_header_buffers 4 32k; Node's 16 KiB default would reject a guest carrying several per-gallery JWT cookies. docs/single-container.md is added as the in-repo reference the README links to. The smoke job gains four assertions for the above: the one-volume layout and writable backup destinations, the sqlite3 CLI, logs landing on the volume, and the image carrying no runtime data from the build context. Verified on a built image — named volume, bind mount and PORT=8080 all healthy; every existing smoke assertion still passes, including / -> 302 /admin/login, the rendered BRAND_TITLE, immutable assets, gzip and /s/<unknown> -> 404. Co-authored-by: Luca-Timo <102960244+Luca-Timo@users.noreply.github.com> * fix(docker): restore the SPA-fallback exclusions and close the build-context leak (#1042) Both found by external review of the consolidated branch. - The SPA catch-all had no backend-owned exclusions. This was a regression I introduced while merging: #1067 carried a BACKEND_OWNED prefix list, and taking this branch's server.js wholesale (correctly — its index:false and in-process brand render are the better design) dropped it. /photos, /thumbnails, /uploads and /fonts are static mounts whose middleware calls next() on a miss, so the catch-all was answering 200 text/html under image and font URLs instead of 404. nginx gave each of those its own location block, so try_files never applied to them. - backend/data is now excluded wholesale rather than by suffix. The suffix list (*.db, *.db-wal, *.db-shm, SETUP_TOKEN) let real secrets through: a used checkout carries ADMIN_CREDENTIALS.txt next to the database, plus -journal files and any DATABASE_PATH not ending in .db. Since Dockerfile.aio builds from the repository root and COPYs backend/ wholesale, any of those would be baked into a published layer. The directory holds only runtime state and is already gitignored in full. smoke-aio gains an assertion that the backend static routes still 404, so the exclusion cannot be dropped again silently. Verified on a built image: /photos, /thumbnails, /fonts and /uploads misses all 404; /setup, /impressum, /gallery/x, /admin/login still 200; / still 302s to /admin/login; /api/nope still answers JSON; /s/<unknown> still 404s; and the image carries no *.db, ADMIN_CREDENTIALS.txt, logs or storage from the context. * fix(aio): three failures that only surface outside a dev laptop (#1042) Backups aborted on SQLite. getTableChecksums() built its digest with `CAST(t.* AS TEXT)`, which is Postgres row-to-text syntax; SQLite parses `*` there as a syntax error, so every backup threw before reaching the .backup call. Since the all-in-one image ships SQLite by default, that is every AIO install. Enumerate the columns via columnInfo() and sum their lengths instead. The shared /data mount root was never adopted. wait-for-db.sh chowned the children it creates but not the mount point itself, so a host directory arriving as 0700 with a foreign owner stayed untraversable by UID 1001 after the su-exec drop. Docker Desktop's permissive bind mounts hide this completely, which is why local testing passed; a NAS share does not. DATA_ROOT is now adopted first. Maintenance mode locked the admin out of the box. The middleware runs at server.js:493, long before the static block at 891, and exempted the auth endpoints but not the page that calls them. With the backend serving the frontend, /admin/login and /assets/* returned 503 JSON, so an admin who enabled maintenance mode could never load the UI to turn it off. nginx serves those paths in the compose stack, which is why it never surfaced there. Guest and API surfaces stay gated. Verified on a built image: checksums compute across all 95 tables; a bind mount created 0700/4000:4000 boots healthy and ends up 1001:1001; with general_maintenance_mode=true, /admin/login, /admin and /assets/* return 200 while /gallery/* and /api/gallery/* return 503 — and 503 across all three once the exemption is removed again. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): stop leaking .env into the image, fix the broken checksum test (#1042) The Jest suite was red: mocking db.raw is no longer enough now that the SQLite checksum branch asks the query builder for its column list, so db(table) came back undefined and getTableChecksums failed on every PR. The production code is right; the fixture needed to know about the call. backend/.env was landing in the published layer. The root ignore file's `.env`, `.env.*` and `data/*.db` rules read as unanchored but Docker matches them from the context root, so they catch ./.env and never backend/.env — and `COPY backend/ .` then puts a real JWT_SECRET at /app/.env. Matched at any depth instead, the way **/node_modules in the same file already is. Confirmed by building from a checkout carrying a planted secret: before, `cat /app/.env` printed it back. Business documents wrote outside the volume. quoteService, invoice sending/reminders and contract signatures build paths from process.cwd()/storage and never read STORAGE_PATH; compose hides it by setting STORAGE_PATH=/app/storage with WORKDIR /app so the two are the same directory. Here they are not, and /app is root-owned, so a quote or invoice PDF failed to write as UID 1001 — and would not survive the container if it had. Symlinked /app/storage into the volume, matching the /backup symlink beside it. Teaching those services STORAGE_PATH is the real fix and wants its own change. Two smaller ones: the mount root is now chowned shallow rather than recursively, since every child below it is already walked recursively and a NAS-sized photo library should not be traversed twice on each restart; and /assets/ joins the backend-owned prefixes, so a stale hashed chunk requested by a tab left open across an upgrade gets a 404 instead of index.html served with 200 under a .js URL. Verified on a built image: planted backend/.env and backend/probe.db are absent; /app/storage resolves to /data/storage and a business-doc write as UID 1001 appears on the host; a 0700 bind mount owned by 4000:4000 boots healthy; a missing /assets chunk 404s while the real bundle still serves 200 as application/javascript. The databaseBackup suite is green again, and the branch adds no failing suite that origin/main does not already fail on the same machine. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * test(aio): teach the leak assertion about the storage symlink (#1042) The previous check listed /app/storage/events and treated a hit as a leak. That was true while /app/storage was either absent or a copied directory; now it is a symlink into the volume, so the check followed it and found the empty tree the image itself creates — a false positive on its own design. Check the shape instead: /app/storage must be a symlink pointing at /data/storage, and the volume's photo tree must contain no files on a fresh install. A real directory there now fails loudly, which is the condition the assertion was always trying to catch. Also extended the path list to /app/.env and loose database files, matching the .dockerignore rules added alongside. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): show the maintenance screen instead of raw JSON to guests (#1042) The previous commit exempted the admin shell so an admin could still reach the switch they had just flipped. Guests had the same problem for the same reason: with no nginx in front, /gallery/<slug> reaches this middleware long before the static block, so a visitor during maintenance got a 503 JSON body where every other deployment shows the branded maintenance screen the frontend already ships. Replaced the two path-specific exemptions with the rule they were both special cases of: a GET that is not an API call and not a backend-owned content mount is the SPA shell, and the shell is inert HTML — it boots, reads /api/public/settings (already exempt) and renders MaintenanceMode on its own. Everything that carries real data stays gated: /api/*, /photos/, /thumbnails/, /fonts/, and any non-GET. Compose is untouched by construction, since nginx answers those paths and they never arrive here. Verified on a built image with the flag on: /gallery/x, /customer/x, /admin and /admin/login return 200 text/html while /api/gallery/x/verify, /photos/x.jpg and /thumbnails/x.jpg return 503 and a POST to a public API still returns 503; with the flag off the same paths go back to 404. Added a middleware test over that exemption matrix — over-exemption is the real risk in this change, so it asserts the gated half too. It fails on five cases without the fix. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): stop the shell exemption from un-gating /og and the public CMS (#1042) The previous commit exempted "any GET that is not an API call". That negative rule reads as safe and is not: /og/gallery/<slug> and its /cover render the event name and the hero thumbnail, /s/<code> renders short-link previews, and `/` is handed to the public CMS. All four are proxy_passed to the backend by nginx, so they were gated before this PR in every deployment — the rule un-gated them, and for compose too, not just the new image. A site switched to maintenance would have kept publishing gallery metadata. Replaced the guess with the split nginx already defines: exempt what the frontend container answers itself, gate what it proxies. That is the same rule the all-in-one image needs by definition, since its whole job is to be both halves of that stack, and it now matches compose in both directions rather than only in the direction the last commit tested. Verified on a built image with the flag on: /admin/login, /gallery/<slug> and /customer/* return 200, while /, /og/gallery/x, /og/gallery/x/cover, /s/abc, /robots.txt, /api/* and /photos/* return 503; with the flag off all of them behave normally again. The middleware test grew the gated cases — it now covers 21, most of them asserting what must NOT be exempt. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * fix(aio): give the image a FRONTEND_URL default so share links are absolute (#1042) getFrontendBaseUrl() reads FRONTEND_URL, falls back to the general_site_url setting, and otherwise returns an empty string — which makes share_url come back as a bare "/gallery/<slug>/<token>". Compose defaults the variable to http://localhost:3000, but the documented one-liner for this image passes only JWT_SECRET, so every fresh single-container install handed out relative links in API responses, QR codes and emails. Defaulted to the same value compose uses; -e FRONTEND_URL=https://... overrides it, as does the site URL field in Settings. Found by pointing tests/e2e/local at a running AIO container: auth/06-api-tokens asserts share_url matches /^https?:\/\//, and it was the one spec that failed for a product reason rather than a harness one. It passes now, and the suite is 19/20 against the image — the remaining failure is smoke/02-auth-flow, whose seed helper shells out to a hard-coded `docker exec picpeak-backend`, so it cannot arrange its precondition against any other container. Claude-Session: https://claude.ai/code/session_01Ra4hcsYiKuQLbbRsg6EjAc * feat(aio): mark the image so face recognition stays off (#1042, #1074) Face recognition needs a separate ML container this image does not contain, and enabling it here would add a second image-processing pipeline competing with Sharp for the CPU and memory of a container sized for one photographer plus guests browsing. The failure mode would not be a clear error — just a slow install that looks broken. The backend gate for this lands in #1075 and keys on PICPEAK_SINGLE_CONTAINER. Without this line the guard never triggers on an actual all-in-one build, so the two changes have to arrive together: whichever merges second completes the pair. Verified against this file's exact value — isFeatureEnabled() returns false with it set. An explicit marker rather than inferring from SERVE_FRONTEND or the SQLite path, because legitimate multi-container deployments do both of those and should keep the feature. Also adds it to the Limits section of docs/single-container.md, next to the SQLite and Redis constraints, since that is where someone will look before choosing this image. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> Co-authored-by: the-luap <paul-nothaft@hotmail.de> |
||
|
Paul Nothaft
|
18b1e0f66e |
ci(tests): run the gated real-Postgres .picpeak cases in the backend job (#1056)
The .picpeak restore suites gate their Postgres cases behind PICPEAK_PG_TEST_URL and describe.skip themselves out when it is unset. That variable was set in no workflow, so those cases have never run in CI — the suites reported green while silently skipping the half that needs a real database: sequence resync, operator/role preservation across a cross-instance restore, and whether a coerced row lands with the right STORED VALUES rather than merely not throwing. Add a postgres:15-alpine service to the backend job (same shape schema-drift already uses) and point the variable at it. Everything else in the suite still runs on SQLite; this only un-gates the cases that were skipping. Verified against a real Postgres 15 before wiring: picpeakRestorePg 4/4 and picpeakCrossEngine 11/11 (8 of which were previously skipped across both). Matters now because #1043 opens sqlite -> pg restore to the upload UI, so the coercion layer's correctness stops being a CLI-only concern. Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
fbc18a386b |
ci: batch stable releases into one daily version (#919)
* ci: batch stable releases into one daily version The stable release PR was auto-merged the instant it went green, so a day with N bugfixes produced N patch releases (3.45.8 AND 3.45.9 on 2026-07-29 alone) — N upgrade notifications for stable users and N full Docker build cycles. Fixes now accumulate in release-please's rolling release PR and are cut as ONE version per day by release-stable-daily.yml (18:00 UTC). Approval/merge mechanics are unchanged from the inline step (#719): approve as github-actions[bot], auto-merge as the PAT so the merge triggers the tag-cutting run. - Urgent fix? workflow_dispatch the daily job or merge the release PR by hand — the schedule is a default, not a gate. - Beta is untouched: instant beta releases are load-bearing for same-day reporter verification. - schedule only fires from the default branch; the stable copy of the new workflow is inert and exists to keep branches in sync. * ci: harden the daily stable-release cut (review round) - P1: the daily job runs on a schedule, so a fork PR can spoof the head branch name 'release-please--branches--stable' — gh --head matches the name only. Pin --base stable AND require isCrossRepository == false so a fork PR can never be approved+auto-merged with the release PAT. - P2: this scheduled job is now the ONLY automatic stable cut, so the auto-merge-enable step no longer swallows failures (|| true); it fails loudly and verifies autoMergeRequest is actually set. A silently expired PAT would otherwise stop releases while the workflow stays green. Approve stays tolerant (re-approval can return non-zero). * ci: accept an immediately-merged release PR as success (review round 2) gh pr merge --auto merges immediately when required checks are already green — the normal 18:00 case, since fixes land hours earlier and CI passes. The autoMergeRequest verify then saw null on a MERGED PR and failed the job on the happy path. Now: MERGED = success, pending auto-merge = success, still-open-with-no-auto-merge = real failure. * ci: read release-PR state + auto-merge in one snapshot (review round 3) Two separate gh pr view calls raced: a pending auto-merge completing between them made the first read OPEN and the second read null on the now-merged PR, failing the job on a successful release. Fetch state and autoMergeRequest together. --------- Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local> |
||
|
Paul Nothaft
|
e3d597b89a |
Merge pull request #787 from PicPeak/ci/push-images-to-dockerhub
ci(docker): also publish images to Docker Hub (picpeak/backend, picpeak/frontend) |
||
|
Paul Nothaft
|
39db7bf6cb |
fix(ci): publish v-prefixed image tags via type=ref,event=tag (#668)
#783 added `type=semver,pattern=v{{version}}` to the merge-job metadata, but metadata-action silently dropped it on prereleases — the 3.84.0-beta.0 build published only :3.84.0-beta.0 + :sha, not :v3.84.0-beta.0 (verified in the merge-backend push log + GHCR: :v3.84.0-beta.0 → 404). Replace the v{{version}}/v{{major}} semver patterns with type=ref,event=tag, which emits the git-tag name verbatim (v3.45.0 / v3.84.0-beta.0) for both stable and beta tags — exactly the string users pin (matches the GitHub release). Applies to both backend + frontend merge metadata steps. Takes effect on the next release build. The bare :3.84.0-beta.0 tags stay (the {{version}} patterns are unchanged), so both forms resolve. |
||
|
Luca
|
1f19fbb1b2 |
ci(docker): mirror published images to Docker Hub
Add picpeak/backend + picpeak/frontend on Docker Hub alongside GHCR. The merge jobs already assemble the multi-arch manifest from the per-arch GHCR digests via 'imagetools create'; adding Docker Hub to metadata-action's images list + a Docker Hub login makes the same command push the manifest to both registries (blobs copied from GHCR). No change to the build-by-digest jobs. Full tag parity (main, stable, latest, semver, sha). Gated on DOCKERHUB_ENABLED (github.repository == PicPeak/picpeak) so forks stay GHCR-only and keep building. Requires repo secrets DOCKERHUB_USERNAME and DOCKERHUB_TOKEN. |
||
|
Paul Nothaft
|
784d059c3d |
fix(ci): publish v-prefixed image tags so :vX.Y.Z resolves (#668)
docker/metadata-action's type=semver strips the leading 'v', so releases published only :3.45.0 / :3.83.1-beta.0. But git tags + GitHub releases are named v3.45.0, so anyone pinning ghcr.io/.../backend:v3.45.0 (the obvious choice) hit 'manifest unknown' — exactly #664. Add v-prefixed semver patterns (v{{version}}, v{{major}}.{{minor}}, v{{major}}) alongside the existing bare ones, for both backend and frontend. Now both :v3.45.0 and :3.45.0 resolve. Applies to future releases; the already-published v3.45.0 only has the bare :3.45.0 tag (retagging past releases is out of scope). |
||
|
Paul Nothaft
|
65ac6eddac |
fix(release): target stable in release-please.yml + undo the bogus 2.7.0 bump
The stable release-please workflow (release-please.yml, triggered on push to stable) had no `target-branch`, so it defaulted to the repo default branch (main) and computed the next version from main's stale `.release-please-manifest.json` (2.6.1) — cutting a spurious **v2.7.0** stable release (a version regression from 3.44.0) when #771 landed on stable, and bumping main's package.json + manifest to 2.7.0. - release-please.yml: add `target-branch: stable` so it releases from the stable branch (3.44.0 → 3.45.0), like release-please-beta.yml already pins `target-branch: main`. - Restore main's version to 3.83.0-beta.0 (backend + frontend package.json), set `.release-please-manifest.json` to 3.44.0, and drop the bogus 2.7.0 CHANGELOG section. The v2.7.0 tag/release is deleted separately; the real v3.45.0 stable is cut by re-running release-please on the stable branch after this lands. |
||
|
Paul Nothaft
|
80503c52b9 |
ci: run the Tests workflow on stable-targeted PRs
tests.yml (the backend/frontend Jest+Vitest jobs) only triggered on main/beta, but those two jobs are required status checks on the stable branch. A beta→stable promote PR therefore hung forever on 'Expected — Waiting for status to be reported' for backend/frontend, while docker-build / install-smoke / schema-drift (already listing stable) ran fine. Add stable to the push + pull_request filters so the Tests suite runs on promote PRs too. |
||
|
Paul Nothaft
|
e08a33d9ea |
fix(ci): enable release-PR auto-merge with the PAT, not GITHUB_TOKEN
Auto-merge enabled via GITHUB_TOKEN attributes the eventual merge commit to github-actions[bot], so recursion prevention suppresses the resulting push to main — the follow-up release-please run that cuts the tag/release never fires. Net: the version PR merges but no release/tag/images are ever produced (#719). Enable auto-merge with RELEASE_PLEASE_TOKEN instead (a real identity) so the merge triggers the tag-cutting run. Approval stays on GITHUB_TOKEN because it must be a different identity than the PR author (the PAT) to count as a review. Observed on #723: merged 3.77.3-beta.0 but no run followed and no tag was cut. |
||
|
Paul Nothaft
|
0cab43ed89 |
fix(ci): set GH_REPO in release-please auto-merge step
The auto-merge step runs in a job with no actions/checkout, so gh could not
infer the repository from a git remote and failed with 'not a git repository'
(#719 follow-up). Set GH_REPO=${{ github.repository }} so gh pr list/review/
merge work without a checkout — same fix as the whatsnew workflow (
|
||
|
Paul Nothaft
|
fb64ec0910 |
Merge pull request #720 from PicPeak/fix/release-please-automation
fix: auto-publish release-please PRs without manual approval |
||
|
Paul Nothaft
|
a3e7232b8e |
fix(ci): auto-publish release-please PRs without manual approval (#719)
The release PR (authored by github-actions[bot] via GITHUB_TOKEN) sat open
forever: its workflows were held behind 'awaiting approval' and the required
review could not be satisfied by the bot. Both release-please workflows now:
- Use a dedicated ${{ secrets.RELEASE_PLEASE_TOKEN }} (fine-grained PAT) with a
GITHUB_TOKEN fallback. A PAT-authored PR runs CI automatically (no 'awaiting
approval') and can be merged without a human.
- Auto-approve (as github-actions[bot], a different identity than the PR
author) and enable auto-merge on the open release PR, so it publishes once
checks pass. Skipped when no PAT is configured — falls back to today's manual
flow, nothing breaks.
A PAT also un-suppresses the tag-push and release-published triggers on
docker-build (GITHUB_TOKEN suppressed them), so add a concurrency group there
to collapse the duplicate same-version builds into one.
Requires (repo/org settings, one-time):
- Create fine-grained PAT RELEASE_PLEASE_TOKEN (contents:write, pull-requests:write).
- Enable 'Allow auto-merge' on the repo (currently off).
- 'Allow GitHub Actions to approve pull requests' — already enabled.
|
||
|
Paul Nothaft
|
4881d2040a |
ci: validate PR titles against Conventional Commits
Release Please only recognizes Conventional Commit prefixes (feat:, fix:, ...). PRs merged with other conventions (gitmoji, free-form) are silently skipped, shipping changes with no version bump or changelog entry (see #707/#708). Fail such PRs early via amannn/action-semantic-pull-request. |
||
|
Paul Nothaft
|
2a5f0a8601 |
fix(ci): whatsnew highlights — set GH_REPO so gh runs without a checkout
Release Please Beta on v3.76.1-beta.0 hard-failed at the very first `gh release view "$TAG"` call: failed to run git: fatal: not a git repository (or any of the parent directories): .git The reusable `whatsnew-highlights.yml` (PR #703) doesn't run actions/checkout — so when `gh` tried to infer the target repo from the runner's empty workspace it errored out. The first time it ran against an actual release (#709 → 3.76.1-beta.0), the whole job died before the deterministic-fallback path could save it. Two changes, both single-line: 1. `env.GH_REPO: ${{ github.repository }}` at job scope. `gh` honours this and won't fall back to parsing `.git/config`, so no checkout is needed (the workflow only calls the GitHub API, never reads repo files). 2. `continue-on-error: true` on the "Extract Features" step. The file's comments say "never let highlights break a release", but the original wiring only soft-failed the AI + inject steps. A transient API hiccup at extract still hard-failed the whole job — defeating the design intent. Match the comment. Why not just add actions/checkout? It would work, but pulls the whole repo over the wire on every release just for `gh` to read its own config. GH_REPO is the lighter idiom. Net impact today: v3.76.1-beta.0 shipped without the `<!-- whatsnew -->` block; the app's parseWhatsNew() already falls back to the raw Features list so the admin "What's New" banner still works. The next beta release will pick up the polished version. |
||
|
Luca
|
d25178d2e5 |
ci(whatsnew): let the Models step fail soft so the fallback runs without Models
If GitHub Models is disabled for the org the ai-inference step errors; without continue-on-error the job would go red and skip the inject+fallback. Mark it continue-on-error so an unavailable Models cleanly degrades to the deterministic bullets — the feature now works with Models off, not just on. |
||
|
Luca
|
5aeb6905ac |
ci(whatsnew): generate release highlights via GitHub Models
Activate the What's New highlights step that condenses each release's Features into <=8 short bullets and injects a <!-- whatsnew --> block the app reads (utils/whatsNew.parseWhatsNew), with a deterministic fallback. Runs as a needs: job inside the release-please workflows rather than on a standalone release: published trigger, because release-please creates the release with GITHUB_TOKEN and GitHub never starts new workflow runs from token-generated events -- a standalone trigger would never fire. Shared as a reusable workflow_call so the stable and beta channels stay in sync. Best-effort: continue-on-error + fallback mean it can never break a release. Requires GitHub Models enabled for the org; until then the fallback is used. |
||
|
Paul Nothaft
|
a40ab6a9b1 |
ci: required-check workflows now fire on every PR (no paths filter)
Branch protection on `main` + `stable` lists `upgrade-from-bootstrap` and `fresh-install` as REQUIRED checks. The producing workflows had `paths:` filters in their `pull_request` triggers, so they correctly skipped on PRs that didn't touch migrations / package.json. But a skipped workflow doesn't satisfy a required check — it leaves the status "missing", which blocks merge on every unrelated PR. Concretely surfaced on PR #692 (security bumps): all 12 visible checks were green, but the merge button was blocked because the two path-filtered workflows skipped and their required-check names never reported. This PR drops the `paths:` filter from both workflows so they always fire on PRs against `main` + `stable`. Costs: - `schema-drift` (`upgrade-from-bootstrap`): ~75 s per PR (Postgres service boot + migrate:safe run + schema assertion). - `install-smoke` (`fresh-install`): ~2 min per PR (full Docker Compose boot + login). Both are buying unconditional safety nets on the install + migration paths, which is what the required-check gate is supposed to model. Also fixes the trigger branch list while in the file: `[main, beta]` → `[main, stable]`, completing the post-#669 rename for these two workflows that were missed in PR #686. ## What this does NOT fix `GitGuardian Security Checks` is the third required check that's currently missing on PRs — but that's a separate problem. The GitGuardian GitHub App was installed at the user-account level (`the-luap`) before the org transfer and didn't move with the repo. Re-installing it on the org via the GitHub Marketplace is a UI step the maintainer needs to do; can't be done via API. |
||
|
Paul Nothaft
|
806b1ac921 |
ci: bypass size gate — cap self-merge PR size for review-bypass users
@Luca-Timo is on main's review-bypass list so he can self-merge small
bugfixes without waiting for a maintainer review. The bypass list alone
is binary (he can merge anything), so this adds a complementary required
status check that fails when a bypass user's PR exceeds a configured
line-count threshold — blocking merge for genuine features while leaving
small bugfixes flowing.
How it works:
- Trigger: pull_request_target (so the workflow runs in the base repo's
context with permissions to write a check status — script never
executes PR code, so fork-PR-attack-safe).
- For PRs authored by a bypass user (default: @Luca-Timo):
- linesChanged = additions + deletions
- If ≤ LINE_LIMIT (300): check = success → bypass works → self-merge OK
- If > LINE_LIMIT: check = failure → required-check gate blocks merge
regardless of bypass; needs a maintainer review.
- For everyone else: check = success ("not applicable"). They go through
the normal review path and are unaffected.
Both constants (LINE_LIMIT, BYPASS_USERS) are at the top of the workflow
for easy tuning.
After this lands on main, a separate API step adds 'bypass-size-gate' to
the main branch's required_status_checks list so the gate is actually
enforced. Until that's in place the check runs but doesn't block.
|
||
|
Paul Nothaft
|
3ca6378bd7 |
chore: workflows + RELEASING.md for the post-rename branch model
After the org move + branch rename (#669): beta → main (active development) main → stable (curated release channel) This PR rewires the workflows that referenced the old branch names so release-please and the Docker build target the right channels. ## Workflow changes ### `.github/workflows/docker-build.yml` - **Push triggers**: `[main, beta]` → `[main, stable]` (both `push.branches` and `pull_request.branches`). `beta` no longer exists; `stable` is the curated channel that should also produce builds. - **`is_prerelease` detection**: pre-release context was decided by `refs/heads/beta`; now decided by `refs/heads/main` (active dev → prerelease, `-beta.N` version suffix unchanged). - **`:latest` + `:stable` tagging**: were gated on `{{is_default_branch}}` (which used to be `main` = stable channel). Default branch is now `main` = active dev, so the implicit gate would have aliased `:latest` to dev. Both tags now explicitly gate on `refs/heads/stable` OR a non-prerelease release tag. - **`:beta` tag**: REMOVED. Active-dev pulls are `:main` (auto-generated by `type=ref,event=branch`). The pre-rename `:beta` tag remains frozen at its last build under Option B / #669 — operators are expected to update to `:main` or pin to a versioned tag. ### `.github/workflows/release-please.yml` - `branches: [main]` → `branches: [stable]`. This is the **stable** release-please workflow (uses `release-please-config.json`); after the rename, the stable channel lives on the `stable` branch. ### `.github/workflows/release-please-beta.yml` - `branches: [beta]` → `branches: [main]`. - `target-branch: beta` → `target-branch: main`. - This is the **pre-release** release-please workflow (uses `release-please-config-beta.json`, `prerelease: true`); after the rename, pre-releases are cut from the new `main` (active dev). The version-suffix scheme stays `-beta.N` so existing operator pins keep working. ## RELEASING.md Rewrote the TL;DR, "How a stable release is cut", and hotfix path to reference the new branch names. Added a one-line "branch model background" note pointing at #669 so future maintainers know why `main` means active dev (the opposite of what some projects use). Filename conventions: `release/X.Y.Z-merge-from-main` (was `…-from-beta`); promotion PR title `promote main → stable as vX.Y.Z` (was `promote beta → main`). ## Why combined with PR A's content as a single PR Originally planned as two PRs (B = workflow triggers, C = release-please reconfigure). Splitting wasn't worth it: the configs are branch-agnostic (`release-please-config.json` and `release-please-config-beta.json` don't mention branch names internally), and not bundling them meant a window where the stable release-please workflow would fire on pushes to the new `main` (active dev) — exactly the wrong place. Single PR closes that gap. ## Versioning scheme — kept No version-scheme decision needed. The `-beta.N` suffix on pre-release versions is preserved (existing operator pins like `v3.71.3-beta.0` keep working). If a `v4.0.0-pre.N`-style reset is desired later, that's a separate PR with explicit operator-comms attached. |
||
|
Paul Nothaft
|
0205c7dcce |
chore: migrate Docker registry + GitHub URLs to PicPeak org
Repo transferred from the-luap/picpeak → PicPeak/picpeak. Docker images
publish to ghcr.io/picpeak/picpeak/{backend,frontend} (lowercase, per the
GHCR canonical form computed by docker-build.yml's `${GITHUB_REPOSITORY,,}`).
Sweep covers:
- docker-compose.production.yml + Dockerfiles → new image registry path
- README, CONTRIBUTING, SECURITY, SIMPLE_SETUP, scripts/picpeak-setup.sh
→ new GitHub URLs
- Update-check / release-notes services (updateCheckService,
environmentService, updateNotificationService, adminSystem,
UpdateNotification, githubReleaseUrl) → GitHub API + tag URLs use the
canonical PicPeak/picpeak path
- Issue templates + README-DOCKER + workflow README → updated package URLs
- One commit-context comment in migrations/090 + customerAccountsService
CHANGELOG.md is intentionally untouched (historical release entries are
immutable; GitHub auto-redirects the old URLs indefinitely).
CLAUDE.md keeps the bare `(the-luap)` reference — that's the maintainer's
personal handle, not a repo URL.
22 files, 48/48 line swaps (every change is a 1:1 URL replacement).
|
||
|
Luca
|
bbceac6cb0 |
ci: make GHA cache export non-fatal (ignore-error=true)
The frontend/backend image builds + pushes succeed, then the final 'exporting to GitHub Actions Cache' step intermittently fails with 'error writing layer blob: not_found' (a known flaky type=gha cache backend issue), failing the whole job. Add ignore-error=true to every cache-to so a cache-write hiccup can't break an otherwise-successful, already-pushed build. |
||
|
Paul Nothaft
|
29e63e5ce5 |
fix(notifications): restore /clear-all route the frontend already calls (#597)
The AdminHeader "Clear All" notifications button has been 404'ing for
a while: frontend `notifications.service.ts` calls
`DELETE /admin/notifications/clear-all`, backend only defined
`DELETE /admin/notifications/clear-old`.
The /clear-old route was misleadingly named anyway — it tried to
delete read OR >30-days-old rows, then had a fallback that nuked
EVERY row when nothing matched. Both the frontend and the existing
test expect a simple Clear All shape, so just rename to /clear-all,
drop the tiered logic, and return the plain
`{ message, deletedCount }` payload the test asserts on.
The test (adminNotifications.test.js) was hiding the breakage —
it was on CI's --testPathIgnorePatterns ignore list and so never
ran. Two reasons it failed locally before this fix:
1. Route path mismatch (the actual #597 bug).
2. The mock only stubbed adminAuth — requirePermission lives in
its own middleware module and ran for real, 403'ing before
the handler. Add a passthrough mock for that too.
With both fixed, the test passes. Drop adminNotifications from the
CI ignore list so future regressions in this route fail loudly
instead of going to ground.
|
||
|
Luca
|
482043f786 |
ci: run backend Jest + frontend Vitest on every PR
The suites already existed (538 backend tests, 40 frontend tests, with solid CRM coverage on quoteService/contractService/invoiceService/ customerHoursService/eventService.calendar) but no CI workflow invoked them. Wire both into a single Tests workflow that triggers on any push or PR to main/beta. Six backend suites are excluded — they fail on upstream/beta too (supertest fixture + knex mock chain issues unrelated to CRM). The explicit ignore pattern keeps the workflow green on day 1; each excluded suite is listed inline as test-infra debt to fix individually. Backend job pins SKIP_S3_TESTS=true (the same default the test setup file applies) so the backup-service integration doesn't try a real S3 round-trip when no MinIO is provisioned. |
||
|
Paul Nothaft
|
4d3f2470bc |
ci(schema-drift): handle absent migrations table in precondition (#530)
First CI run failed at the precondition check because the SQL `CASE WHEN to_regclass(...) IS NULL THEN 0 ELSE (SELECT count(*) FROM migrations)` expression doesn't short-circuit at parse time — Postgres parses the subquery against `migrations` even when the outer guard would skip it, fails the run with "relation 'migrations' does not exist". initializeDatabase() doesn't create the `migrations` tracking table — that's the migrate:safe runner's responsibility — so in the recovery scenario the table genuinely doesn't exist yet. Both "absent table" and "present but empty table" are valid recovery states. Split the check into two shell steps: to_regclass first, then count only if the table exists. Avoids the parse-time subquery error and accepts either state. |
||
|
Paul Nothaft
|
8f0108ce23 |
feat(install): skip legacy chain when modern bootstrap fingerprint detected (#530)
Refined from the original #530 framing after a dry-run uncovered that the "bootstrap vs migration chain" diff produces mostly noise — most of the ~200 lines of difference are expected (migrations add new tables and columns over time). initializeDatabase() isn't a parallel path that diverges from migrations; it's invoked by migration 001 itself, so every normal install/upgrade runs both. The genuine drift hazard surfaced during the dry-run: a DB with the modern bootstrap tables but an empty `migrations` table (which happens when a backup was restored that lost the migrations table, or someone invoked initializeDatabase() outside the runner, or the DB was moved between systems without copying the migrations row) fails to upgrade. Failure mode: 1. detectExistingSchema sees the bootstrap tables + empty migrations, treats it as an "existing deployment". 2. Runs the legacy chain first. 3. legacy/008 renames email_templates.subject → subject_en. 4. core/029 (later in the chain) inserts email templates referencing the pre-rename `subject` column. 5. Postgres rejects: column "subject" doesn't exist; subject_en is NOT NULL with no default. Fresh installs avoid this because they only run core/* (and core/059 handles the rename AFTER core/029 has inserted). Real legacy upgrades avoid it because their migrations table already records legacy/008–028 as applied historically. Fix in detectExistingSchema: - Detect the modern bootstrap fingerprint (photo_categories + cms_pages both present, which initializeDatabase produces as part of the consolidated post-004-era bootstrap). - When matched, enumerate every file in migrations/legacy/ and mark each as applied. This puts the recovery state on the same code path fresh installs use — only core migrations run, in core order. - Real legacy upgrades that already have entries in the migrations table hit no-op markings (markMigrationAsApplied skips duplicates), so their behaviour is unchanged. New CI workflow (`.github/workflows/schema-drift.yml`): - Boots fresh postgres. - Seeds via `node -e \"require('./src/database/db').initializeDatabase()\"` — reproduces the recovery state in one line. - Runs `npm run migrate:safe`. - Asserts: precondition (bootstrap fingerprint + empty migrations table), migrate:safe exits 0, final schema has ≥40 tables (soft floor, not exact pin so future migrations don't force workflow edits), legacy migrations marked applied (confirms the fingerprint check actually fired vs. the chain silently bailing). - Triggers only on PRs that touch backend/migrations/**, src/database/db.js, knexfile.js, or this workflow. Manually verified end-to-end before this commit: Before fix: migrate:safe dies at core/029 with NOT NULL violation on email_templates.subject_en (17/48 tables present). After fix: 82 migrations applied + 27 marked applied = 109 total, final state has all 48 tables matching fresh-install. Issue body in #530 has been updated to match this refined scope. Refs: #530, #484, #519 |
||
|
Paul Nothaft
|
763fd4593f |
ci(install-smoke): use BusyBox-compatible ps in node-user check
Alpine ships BusyBox ps (no -p PID, no pgrep), which failed CI on the first run of this workflow with "ps: unrecognized option: p". Replace the pgrep-then-ps chain with `ps -o user,comm | awk '$2=="node"'` which works on both BusyBox (Alpine, in the container) and procps (the GitHub runner host, though we don't use it here). |
||
|
Paul Nothaft
|
1505775678 |
fix(install): self-chowning entrypoint kills fresh-install restart loop (#484)
The fresh-install restart loop reported by @MrGabri (and confirmed by
@AloePacci with the user:0:0 workaround) had a clear root cause:
- Dockerfile pinned USER nodejs (UID 1001) before the entrypoint
ran, so the existing chown branch in init-production.sh:13 was
dead code.
- wait-for-db.sh (the actual entrypoint, not init-production.sh)
silently swallowed mkdir/EACCES on bind mounts with || true,
then a downstream migration error surfaced as the visible failure.
- Net effect on a typical Linux host where the bind-mount dir is
owned by UID 1000: container can't write, exits non-zero,
restarts forever with no clear error.
Switch to the standard Docker drop-privileges pattern:
1. Install su-exec, drop `USER nodejs` from the Dockerfile —
container now starts as root.
2. wait-for-db.sh: if running as root, chown /app/storage,
/app/data, /app/logs to nodejs and re-exec self via
su-exec nodejs:nodejs. App still ends up running as UID 1001.
3. Preflight check for non-root invocations (compose `user:`
overrides): verify the bind mounts are actually writable
before continuing. If not, exit 1 immediately with an
actionable error pointing at the docs — no more silent
restart loops.
Also:
- Delete backend/init-production.sh. It was an orphan — no caller
in the Dockerfile, compose, or anywhere else. Its chown logic
looked authoritative enough that @MrGabri ran it manually trying
to debug, which is what finally surfaced the EACCES.
- docker-compose.yml: drop user: + PUID/PGID env. The pattern-B
UID-matching workaround they implemented is obsolete now that
pattern A (root-then-drop) is in place.
- .env.example + README: drop PUID/PGID documentation.
- Add fresh-install smoke test workflow. Boots backend + postgres
against bind mounts owned by UID 1000 (the GitHub runner UID,
and the common-mismatch case on Linux hosts) and verifies:
+ container reaches healthy without restart-looping
+ chown happened (dirs now owned by 1001 inside the container)
+ node runs as nodejs, not root (su-exec drop worked)
+ /health returns status:ok
+ with --user 5005:5005 + unwritable mounts, preflight exits
loud with the expected error string
Verified locally end-to-end against a fresh Postgres + UID-501-owned
bind mount: backend reaches healthy in ~20s, chown applied, node
runs as nodejs, no restart loop. Docs in picpeak-docs cover the new
behavior + a Troubleshooting section for the install-path bugs
fixed in #484/#494/#511/#488.
Refs: #484
|
||
|
Paul Nothaft
|
c3256dc6bf |
fix(ci): pin TRIVY_PLATFORM per matrix arch (post-#477 follow-up)
PR #477 moved Trivy from the merge-* job into the per-arch build-* matrix scanning by digest. The amd64 leg works; the arm64 leg crashes with: remote error: no child with platform linux/amd64 in index ghcr.io/.../<image>@sha256:<digest> Root cause: docker/build-push-action wraps every push in an OCI index — the actual image manifest sits next to a SLSA provenance attestation manifest as siblings under the digest. Trivy's remote backend defaults to linux/amd64 when resolving an index, so: - amd64 leg → looks for amd64 child → finds the amd64 image → ok. - arm64 leg → looks for amd64 child → finds NO amd64 child (the only platform child is arm64) → fails. Fix: set TRIVY_PLATFORM = ${{ matrix.platform }} on each leg's Trivy step. Each scanner then asks for its own arch and finds it. SLSA provenance attestation stays attached to the per-arch images — a real win for supply-chain visibility we'd lose if we'd disabled provenance instead. amd64 was the only thing keeping CI partly green; this restores full green across both legs without touching the build artifact shape. |
||
|
Paul Nothaft
|
40e176cb46 |
fix(ci): trivy-action tag is v0.36.0 (was 0.28.0 — does not exist)
Initial pinning shipped a tag that doesn't exist in the aquasecurity/trivy-action repo. Workflow run failed with: Unable to resolve action 'aquasecurity/trivy-action@0.28.0', unable to find version '0.28.0' The repo's tags use a v prefix (v0.36.0, v0.35.0, …). Bumping both occurrences (build-backend and build-frontend matrix jobs) to v0.36.0, which is the latest stable as of 2026-04-22. |
||
|
Paul Nothaft
|
caf0d61857 |
fix(ci): scan multi-arch images per-arch by digest, pin trivy-action (#476)
Resolves the intermittent "no child with platform linux/amd64 in
index" failure on the merge-backend job — and fixes the same latent
bug on merge-frontend before it surfaces.
Two compounding root causes per Luca's diagnosis:
1. aquasecurity/trivy-action@master was unpinned, so the action and
its bundled Trivy binary float on every CI run. A green build
could flip red overnight without a single repo change.
2. Trivy was asked to scan a multi-platform OCI index by tag (the
merge-* jobs ran AFTER manifest creation). Its remote resolver
cannot reliably pick the right per-arch child out of an index
reference — it needs a single-platform reference (digest, or a
--platform flag).
Fix:
- Move the Trivy + upload-sarif steps OUT of merge-backend /
merge-frontend and INTO the per-arch build-backend / build-frontend
matrix jobs. Each leg scans the image it just pushed by its
sha256 digest (`...@${{ steps.build.outputs.digest }}`), which is
always single-platform by construction.
- Pin aquasecurity/trivy-action@0.28.0 (was @master).
- Distinct SARIF category per arch
(`backend-vulnerabilities-linux-amd64`, …-arm64) so an
amd64-only finding in a base layer doesn't get masked by the
arm64 scan in the Security tab.
- Move security-events: write down to the build-* jobs (where the
scan now runs) and remove it from the merge-* jobs (which only
publish the manifest now).
Out of scope: flipping `exit-code: '1'` to actually gate CI on
findings. Worth doing as a separate follow-up after an audit pass —
landing it here would surprise beta with a red build for any
pre-existing CRITICAL/HIGH in current images. Inline TODO in the
workflow notes the deferral.
|
||
|
Paul Nothaft
|
6b6191a426 |
fix(security): scan triage cleanup — drop dead deps, harden Docker/nginx/postMessage
Triage of an external SAST/SCA scan run on 2026-05-06. Most loud findings were already resolved by PR #412 (the 18-CVE backport); this PR addresses the residual real items: * Drop unused `handlebars` from backend deps. The runtime require was removed in PR #367 (#367) but the package.json line stayed. handlebars was the source of two flagged criticals (CVE-2026-33937 RCE, GHSA-2w6w-674q-4c4q AST injection) plus 8 highs — all now gone. * `npm audit fix` on backend + frontend. Bumps transitive picomatch, flatted, postcss, brace-expansion via lockfile, and direct dompurify, lodash, vite, i18next-http-backend within their existing semver ranges. Both audits now report 0 vulnerabilities. * Add `event.origin === window.location.origin` check to the THEME_PREVIEW message listener in PreviewPage. The branding page posts from the same origin, so nothing legitimate is rejected; without the check, any third party that window.open()'d the preview could push arbitrary branding/theme payloads (semgrep insufficient-postmessage-origin-validation). * nginx: `proxy_hide_header` for X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Content-Security-Policy, Permissions-Policy, Strict-Transport-Security at server level. nginx adds these itself, but helmet on the backend was also emitting them — clients were seeing duplicates (testssl flagged "Multiple X-Frame-Options / CSP / Permissions-Policy / Referrer-Policy headers" on the live origin). Single source of truth now. * Dockerfile hardening (checkov): - HEALTHCHECK on backend/Dockerfile, backend/Dockerfile.dev, frontend/Dockerfile.dev. Frontend production Dockerfile already had one. - USER node in frontend/Dockerfile.dev (was running as root). * GitHub Actions docker-build.yml: explicit top-level `permissions: contents: read`. Per-job blocks already declare `packages: write` where needed; this stops future steps from inheriting unintended privileges (CKV2_GHA_1). Backend npm audit: 4 vulns -> 0. Frontend npm audit: 6 vulns -> 0. Backend unit tests: 13 suites, 131/132 passing (1 pre-existing skip). Frontend type-check + lint: clean. The pre-existing integration-test failures (live DB / S3 required) and the ThemeCustomizerEnhanced QueryClientProvider failures are unrelated and reproduce on origin/beta without these changes. |
||
|
Luca
|
3440ecc999 | ci: lowercase image names for GHCR compatibility on forks | ||
|
Luca
|
ede5193e58 | Update docker-build.yml | ||
|
Luca
|
ec2eaf76ea | ci: build multi-arch images on every channel via native arm64 runners | ||
|
Paul Nothaft
|
fe7d45dd12 |
fix: use Release Please extra-files instead of sync-versions job
Remove sync-versions job that fails on protected branches. Instead, use Release Please's extra-files feature to update package.json versions as part of the release PR. |
||
|
Paul Nothaft
|
c05ae5b0b9 |
chore: upgrade CodeQL Action from v3 to v4
Address deprecation warning - CodeQL Action v3 will be deprecated in December 2026. |
||
|
Paul Nothaft
|
617e778a48 |
feat: implement beta/stable release channels with update notifications
Add dual-channel release strategy for stable and beta releases: Release Channels: - Stable channel: production-ready releases (stable, latest, v2.3.0) - Beta channel: early access features (beta, v2.3.0-beta.1) - Configurable via PICPEAK_CHANNEL environment variable Update Notifications: - Admin dashboard shows available updates for configured channel - Checks GitHub Releases API with 1-hour cache - Can be disabled with UPDATE_CHECK_ENABLED=false CI/CD Changes: - New release-please-beta.yml workflow for beta prereleases - Docker build workflow produces stable/beta tags based on branch - Beta versions use v2.3.0-beta.1 format New Files: - .github/workflows/release-please-beta.yml - release-please-config-beta.json - .release-please-manifest-beta.json - backend/src/services/updateCheckService.js - frontend/src/components/admin/UpdateNotification.tsx Modified Files: - docker-compose.production.yml (channel selection) - .env.example (PICPEAK_CHANNEL, UPDATE_CHECK_ENABLED) - backend/src/routes/adminSystem.js (/updates endpoint) - frontend components (VersionInfo, AdminDashboard) - i18n locales (en.json, de.json) - README.md and DEPLOYMENT_GUIDE.md (documentation) |
||
|
Paul Nothaft
|
4872ef71f8 |
ci: only build ARM64 images for tagged releases
QEMU emulation of ARM64 on x86 GitHub runners is too slow and unreliable for npm operations, causing builds to hang or crash with "Illegal instruction" errors. Changed platform detection logic to: - Tagged releases (v*.*.*): Build both amd64 and arm64 - All other builds (branches, PRs): Build amd64 only This ensures fast CI feedback during development while still providing multi-arch images for production releases. |
||
|
Paul Nothaft
|
0d36a273bb |
fix(ci): add QEMU setup for multi-arch builds and skip for PRs
- Add docker/setup-qemu-action for proper ARM64 emulation - Skip QEMU setup for PR builds (amd64 only) - Fix QEMU "Illegal instruction" errors during npm ci |
||
|
Paul Nothaft
|
6033461be1 |
feat: add Apple Liquid Glass templates, image security settings, and automated releases
## New Features - Apple Liquid Glass CSS template with iOS 26-inspired design - Liquid Glass Dark theme with neon accents - Image Security settings tab with per-event protection levels - Release Please automation for versioning and changelog ## Improvements - Update CSS template migration with final working templates - Add search placeholder visibility fix for glass themes - Update README roadmap (Download Protection, Gallery Templates, Filtering & Export now implemented) ## Infrastructure - Add release-please.yml workflow for automated releases - Add release-please-config.json and manifest - Update docker-build.yml with Release Please integration comments - Add comprehensive CHANGELOG.md ## Cleanup - Add working/planning docs to .gitignore (CLAUDE.md, test-*.md, feature-*.md, etc.) - Remove internal planning documents from git tracking (kept locally) ## Files Added - .github/workflows/release-please.yml - .release-please-manifest.json - release-please-config.json - CHANGELOG.md - frontend/src/features/settings/tabs/ImageSecurityTab.tsx |
||
|
Claude
|
f446335e81 |
Fix CI/CD: Build amd64 only for PRs to avoid QEMU ARM64 emulation issues
Sharp library native binaries cause QEMU 'Illegal instruction' errors during ARM64 emulation. This change builds only amd64 for PR checks (faster, reliable) while maintaining multi-arch (amd64+arm64) builds for main/develop/tags. |
||
|
Claude
|
ae93755dbb |
Fix GitHub Actions Docker tag generation
The workflow was generating invalid Docker tags with format ':-3b251d7'
due to empty branch names in PR contexts.
Problem:
- Tag config: type=sha,prefix={{branch}}-,format=short
- For PRs: {{branch}} is empty → results in ':-3b251d7' (invalid)
- Docker doesn't allow tags starting with hyphen
Solution:
- Changed to: type=sha,format=short
- Now generates: '3b251d7' (valid) without branch prefix
- Works correctly for PRs, branches, and tags
Valid tag examples now:
- PRs: pr-44, 3b251d7
- Branches: main, 3b251d7
- Tags: v1.0.0, 1.0, 1, 3b251d7
|
||
 paul
|
2f1a137342 |
ci: make ghcr login non-fatal and gate pushes/scans on login success; build images regardless (supports transient GHCR outages)
Mirror to GitHub / mirror (push) Successful in 45s
Test and Lint / backend-test (push) Successful in 1m31s
Test and Lint / frontend-test (push) Successful in 2m8s
Version and Release / version-bump (push) Successful in 44s
Version and Release / trigger-drone (push) Has been skipped
|
||
|
paul
|
986b101040 |
fix: remove unnecessary publish-manifest job from Docker workflow
Mirror to GitHub / mirror (push) Successful in 44s
Test and Lint / backend-test (push) Successful in 1m40s
Test and Lint / frontend-test (push) Successful in 2m19s
Version and Release / version-bump (push) Successful in 58s
Version and Release / trigger-drone (push) Has been skipped
The publish-manifest job was failing because it tried to create manifests from non-existent architecture-specific tags (latest-amd64, latest-arm64). docker/build-push-action@v5 already creates multi-arch manifests automatically when building for multiple platforms, making this job redundant. The workflow now correctly builds and pushes multi-arch images in a single step with proper manifest lists included. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
paul
|
4029559954 |
feat: add GitHub Actions workflow for Docker image builds
Mirror to GitHub / mirror (push) Successful in 42s
Test and Lint / backend-test (push) Successful in 1m43s
Test and Lint / frontend-test (push) Successful in 2m16s
Version and Release / version-bump (push) Successful in 1m11s
Version and Release / trigger-drone (push) Successful in 3s
- Created docker-build.yml workflow for automated Docker builds - Configured GitHub Container Registry (ghcr.io) with GITHUB_TOKEN auth - Added multi-architecture support (linux/amd64, linux/arm64) - Integrated Trivy security scanning for vulnerability detection - Implemented smart tagging based on branches, PRs, and releases - Added build caching for improved performance - Updated Dockerfiles with OCI labels for proper ghcr.io linking - Created comprehensive README-DOCKER.md documentation The workflow automatically builds and pushes images on: - Push to main/develop branches - Pull requests (build only, no push) - Release publications - Manual workflow dispatch 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |