Files
picpeak/backend/src/routes/adminSystem.js
T
Paul Nothaft 9003b34c8a fix(docker): default NODE_ENV=production so non-compose deploys don't fall back to SQLite (#1038) (#1040)
* fix(docker): default NODE_ENV=production so non-compose deploys don't fall back to SQLite (#1038)

knexfile.js selects its config block by NODE_ENV and the `development` block
defaults to sqlite3. The image never set NODE_ENV, so every deployment that
doesn't go through our compose files — Kubernetes, Helm, plain `docker run` —
silently ran on SQLite and ignored DB_HOST/DB_USER/DB_PASSWORD.

It stayed invisible because wait-for-db.sh is shell: it reads DB_HOST directly,
connects to Postgres, creates the database and logs "PostgreSQL is up" in the
same container where the Node process then writes to a SQLite file. Migrations
go through src/database/db.js → the same knexfile, so they also ran against
SQLite, leaving the provisioned Postgres database empty.

Setting the default alone would be unsafe: an affected install would flip to
Postgres on its next image pull and come up against an EMPTY database, which
reads as total data loss. So this adds a guard that runs before migrations
touch anything:

  - logs the resolved engine + target at boot
  - refuses to start when pointed at a virgin Postgres while a populated
    SQLite file exists, naming the file and the .picpeak export path for
    moving the data, with PICPEAK_ALLOW_EMPTY_PG=true as the escape hatch
  - warns but boots when Postgres settings are present yet SQLite is in use

Compose files already set NODE_ENV explicitly, so compose users are unaffected.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): stay on SQLite instead of blocking, and add a migration path (#1038)

Reworks the guard after walking through what an existing install actually
experiences on its next image pull.

Blocking was the wrong trade. An operator who had unknowingly been running on
SQLite (because the image left NODE_ENV unset) would have pulled the fix and
got a CrashLoopBackOff: data safe, galleries offline, for something they did
not do. Now the boot RESOLVES the engine before migrations run and stays on
whichever one holds the data:

  - Postgres configured but holding no galleries, while a populated SQLite file
    exists → keep serving from SQLite, print what happened and how to migrate.
  - once Postgres holds the data, the next restart switches over on its own.
  - an explicit DATABASE_CLIENT is always honoured.

Keyed on Postgres holding DATA, not on it having tables: a stray
`run-migrations` against the empty database creates every table, which would
otherwise blind the check.

Adds scripts/migrate-sqlite-to-postgres.js, which reuses the .picpeak
export/import services rather than hand-rolling a cross-engine copy. Two
additions were needed for the SQLite → Postgres direction, both opt-in and
CLI-only so the upload/restore UI is untouched:

  - `allowEngineSwitch` relaxes the importer's same-engine guard
  - cross-engine row coercion: SQLite has no real date or boolean types, so its
    rows carry epoch numbers where Postgres wants a timestamp and 0/1 where it
    wants a boolean, both of which Postgres rejects outright. Driven by the
    TARGET schema, never guessed from the value.

DELTA FROM THE BETA PR: this branch's import service has no resyncSequences()
— that landed on main only. Without it a cross-engine load leaves Postgres
identity sequences at 1 and the next insert collides on the primary key, so
the function is backported here and called ONLY on the cross-engine path.
Same-engine restores through the UI keep their current behaviour exactly.

Verified end to end on this branch against a real PostgreSQL 15: a seeded
SQLite install migrated across with booleans, timestamps and foreign keys
intact, and the next INSERT got id 2 rather than colliding.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close four review findings on the SQLite fallback + migration (#1038)

External review (codex) found four issues, all confirmed against the code and
fixed here. Two of them could have cost data.

1. The engine resolver was reachable only through wait-for-db.sh. A Kubernetes
   manifest that sets `command`/`args`, or a plain `docker run … node
   server.js`, bypasses the entrypoint — exactly the deployment styles this fix
   targets. With NODE_ENV now baked into the image, such an install would have
   resolved to Postgres and come up against an empty database while its SQLite
   data sat there unseen. server.js now resolves the engine itself, before
   anything requires knexfile, via the same script the entrypoint uses.
   Verified by running `node server.js` directly against an install with
   stranded SQLite data: it logs the banner and serves SQLite.

2. Cross-engine loads double-encoded JSON. SQLite has no json type, so its json
   columns are TEXT holding JSON; the export dumps that as a string and
   serialiseJsonColumns stringified it again, storing `true` as the scalar
   string "true". app_settings.setting_value is json on every install, so this
   reshaped every migrated setting. The text is decoded before serialisation
   now — verified against a real Postgres: json_typeof(setting_value) is
   `boolean`, matching a native install exactly.

3. The migration could silently miss concurrent writes. If the backend keeps
   serving, rows written after the export never reach Postgres and vanish from
   view once the engine switches. The script now fingerprints the SQLite tables
   whose loss would be noticed, checks for drift BEFORE loading Postgres (so a
   detected race leaves the target untouched) and again after, and refuses with
   the exact rows that moved. It also says plainly to stop the backend first.

4. The child phases shared stdout with winston. Outside production, and
   whenever LOG_TO_CONSOLE=true, createPicpeak's own log line was concatenated
   with the archive path and the migration failed on a bogus filename. Payloads
   travel through a result file now; verified with LOG_TO_CONSOLE=true.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 2 — six more data-safety findings (#1038)

1. The engine choice is now PINNED once the data is in Postgres. Previously the
   boot decided from "does Postgres hold galleries", so an operator who later
   deleted every gallery would be sent back to the stale pre-migration SQLite
   file while their settings, admins and CRM data stayed in Postgres. The
   migration writes a marker next to the database file (and retires the file
   itself by renaming it); the marker wins over any probe.

2. The migration refused to overwrite Postgres only when it held GALLERIES. A
   target with admins, customers, invoices or projects but no galleries was
   wiped without --force. Both the source and target checks now look for user
   data across the tables that are empty on a fresh install.

3. Same bug in the other direction: an install with no galleries but real
   admins/settings/customers was refused a migration it was entitled to.

4. Drift detection covered four tables and only count/max(id), so an in-place
   UPDATE (event edit, password change) or a write to any other table passed
   unnoticed. It now fingerprints every table the export carries, including
   max(updated_at). It still is not a substitute for stopping the backend, and
   the script says so rather than implying a guarantee.

5. probeSqliteData() treated an unreadable or corrupt file as "no data", which
   would have switched the install to an empty Postgres — the very failure this
   module exists to prevent. It fails closed now and stays on SQLite so the real
   error surfaces.

6. The "you are leaving SQLite data behind" warning was unreachable: setting
   DATABASE_CLIENT skipped the probes, so the branch that produces it never had
   the inputs. Postgres and SQLite are both probed whenever Postgres is the
   engine in play.

Also: the final verification compares row counts for EVERY table rather than
just galleries, and flags only a shortfall — the import legitimately adds an
app_settings row (setSessionsValidAfter) that made the strict equality fail on
a first real run.

Verified against a real PostgreSQL 15 end to end, including: the marker keeps
an install on Postgres after every gallery is deleted; removing the marker and
restoring the file rolls back to SQLite as documented.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 3 — occupancy, bootstrap admin, secrets in /tmp (#1038)

1. Both engine probes judged occupancy by GALLERIES alone. An install whose
   galleries were all deleted, but which still has admins, customers or
   accounting records, was treated as empty: on the SQLite side that meant
   booting the empty Postgres and appearing to lose everything; on the Postgres
   side it meant diverting a live install to a stale SQLite file. Both now look
   across the tables that are empty on a fresh install, matching the migration
   script.

2. The migration ran migrate-schema BEFORE checking the target, and migration
   001 seeds a bootstrap admin when ADMIN_PASSWORD is set (common on legacy
   installs). The occupancy check then saw that admin and refused, pushing the
   operator towards --force against a genuinely empty database. The target is
   read first now.

3. probeSqliteData()'s warning went through the app logger, which writes to
   STDOUT when LOG_TO_CONSOLE=true — and the resolver's stdout is the protocol
   channel wait-for-db.sh captures, so DATABASE_CLIENT could have been set to a
   JSON log line. Diagnostics take an injected sink (stderr in the resolver),
   and the shell now validates the value it captured instead of trusting it.

4. The .picpeak archive holds password hashes, SMTP credentials and API keys in
   plaintext, and was only removed on the fully-successful path — any drift or
   import failure left it in /tmp. Every exit path removes it now.

5. A database-only migration still hauled every business-doc and upload through
   /tmp and back into the same volume. createPicpeak takes includeFiles:false
   for this path; rows move, files stay where they already are.

Verified against a real PostgreSQL 15: a gallery-less install with only an admin
account now stays on SQLite and migrates successfully with ADMIN_PASSWORD set;
the resolver emits exactly one token on stdout with LOG_TO_CONSOLE=true and a
corrupt database; a drift failure leaves Postgres untouched and no archive
behind.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): pin the boot to SQLite while a migration is unfinished (#1038)

Review round 4. A migration that dies after touching Postgres leaves rows
behind — schema creation alone seeds a bootstrap admin when ADMIN_PASSWORD is
set, and a drift or row-count failure can leave a partial load. Since the
occupancy probes were widened in round 3, those rows read as "Postgres is
occupied", so the next restart would switch engines and hide the SQLite data
that is still the database of record.

The script now writes a pin file next to the database BEFORE its first Postgres
write and clears it only on success (after the success marker exists, so no
restart in between can pick the wrong engine). While the pin is present the
resolver stays on SQLite and explains why.

Verified against a real PostgreSQL 15 by reproducing the exact scenario: a
migration failed mid-run with ADMIN_PASSWORD set, leaving one bootstrap admin
in Postgres. With the pin the next boot resolves to sqlite3; with the pin
removed it resolves to pg — the failure this closes. The subsequent successful
re-run clears the pin and the boot moves to Postgres.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 5 — occupancy, path drift, retry, host default (#1038)

1. A seeded bootstrap admin counted as "Postgres is occupied". core/001_init.js
   inserts one whenever ADMIN_PASSWORD is set, so a Postgres that was
   initialised once and never used would have beaten a SQLite file full of real
   galleries — the exact failure the guard exists to prevent, reintroduced by
   widening the probe in round 3. The two sides are deliberately asymmetric now:
   the SQLite probe counts any user data (err towards keeping data visible),
   the Postgres probe ignores rows that schema creation seeds (err towards
   requiring proof of real use).

2. The guard resolved DATABASE_PATH with its own logic while knexfile trimmed
   whitespace and collapsed the legacy duplicated-backend form. A path either
   engine normalised differently meant probing a file nobody uses, concluding
   there was no SQLite data, and booting an empty Postgres. The resolution now
   lives in one module both require.

3. Re-running after a partial migration — the documented recovery — was refused
   unless the operator passed the destructive-sounding --force, because the
   half-written rows read as target data. An unfinished run of this same script
   is now recognised as a safe retry.

4. wait-for-db.sh verified readiness against its own default host (`postgres`)
   while knexfile's production block defaults to `db`. With NODE_ENV now baked
   in, a bare `docker run` without DB_HOST would have passed the readiness check
   against one host and then dialled another. The entrypoint exports the exact
   connection it verified. Compose sets DB_HOST explicitly and is unaffected.

Verified: a Postgres holding only a seeded admin now loses to real SQLite data;
a DATABASE_PATH with surrounding whitespace resolves to the identical file in
both knexfile and the guard.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 6 — explicit-client bypasses, retry scope, cleanup (#1038)

1. An explicit DATABASE_CLIENT bypassed the unfinished-migration pin, because
   decideBootEngine honoured it first. docker-compose sets DATABASE_CLIENT=pg,
   so a failed migration would have restarted on a half-written Postgres on
   exactly the deployments that pin it. Worse in the other direction: with
   DATABASE_CLIENT=sqlite3, a SUCCESSFUL migration renames the source file, so
   the next start created a NEW, empty SQLite database and served that. The pin
   now outranks explicit pg (clearing the marker is the override), explicit
   sqlite3 is left alone since it already points at the data, and the migration
   refuses up front when the deployment pins anything other than pg.

2. The retry allowance was bound to the SQLite file, not to the target. An
   operator who repointed DB_HOST/DB_NAME between attempts could have replaced
   an unrelated populated database without --force. The pin records the target
   and the allowance only applies when it matches.

3. The printed rollback did not roll back: with data on both sides and no
   marker, the resolver still selects Postgres. It now spells out all three
   steps, including DATABASE_CLIENT=sqlite3.

4. A failure inside createPicpeak left a partial archive — plaintext hashes and
   credentials — in the caller-supplied temp dir, which that service
   deliberately does not clean. The export phase removes it on error.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 7 — pin bypass on direct start, real admins (#1038)

1. server.js only ran the engine resolver when DATABASE_CLIENT was unset, so a
   deployment that both bypasses the entrypoint (Kubernetes `command:`) AND
   pins DATABASE_CLIENT=pg never consulted the migration pin — the round-6 fix
   was unreachable on exactly that path, and a failed migration would have
   served a half-populated Postgres. The resolver now also runs whenever a pin
   file exists.

2. Round 5 excluded admin_users from Postgres occupancy to stop a seeded
   bootstrap admin counting as real data. That over-corrected: an install that
   has completed first-run setup but has no galleries yet has exactly one
   user-created row — an admin — so Postgres looked empty and, with a stale
   SQLite file present, the boot would switch away and the admin's credentials
   and configuration would disappear.

   core/001_init.js seeds must_change_password=true; setupService writes false
   once a human completes setup. The FLAG, not the table, distinguishes them,
   and a legacy NULL counts as a real admin.

Verified against a real PostgreSQL 15: a Postgres holding only the seeded row
loses to real SQLite data, the same Postgres wins once setup is completed, and
a server started directly with DATABASE_CLIENT=pg and a pin present comes up on
SQLite with the warning.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 8 — reset admins, CLI config, JSON nulls (#1038)

1. must_change_password is mutable: resetAdminPassword() re-raises it on REAL
   accounts (userManagementService.js:474). Round 7's discriminator therefore
   read a gallery-less Postgres whose only admin had been reset as an untouched
   bootstrap seed — and with a stale SQLite file present, the boot would have
   switched away and hidden those live credentials. The rule is layered now:
   more than one admin, any admin that has logged in, or must_change_password
   false all count as use. Only core/001_init.js's exact leftovers — one admin,
   never logged in, still flagged — read as a seed.

2. The CLI read process.env directly but never loaded the configuration the
   child phases get through knexfile, so running it directly (or via
   `docker exec`, which does not inherit wait-for-db.sh's exports) failed the
   pre-flight checks even with valid settings in backend/.env or
   /run/secrets/db_password. Both sources are loaded up front now.

3. The migration's target check counted a seeded bootstrap admin as user data
   while probePgData classified the identical row as empty, so migrating into a
   previously-initialised-but-unused Postgres demanded --force. Same rule on
   both sides.

4. Cross-engine JSON handling is simpler and no longer lossy. SQLite keeps json
   columns as TEXT holding valid JSON and Postgres accepts JSON text directly,
   so the correct action is to pass them through untouched. Round 1 parsed then
   re-serialised them to undo a double-stringify; that round-tripped the JSON
   literal `null` into SQL NULL, changing data and breaking NOT NULL json
   columns. Not serialising at all fixes both.

Verified against a real PostgreSQL 15: a migrated install now carries
json_typeof = null for a JSON null, object for a nested object, and boolean for
a boolean — matching a native install exactly.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): close review round 9 — probe error classes, marker ordering (#1038)

1. probePgData() answered every failure with "Postgres has data". That is right
   for an unreachable server — the app cannot run on it either way, and
   diverting a healthy pg install to a stale SQLite file over a transient blip
   would be worse — but wrong for a server that answers and then fails the
   query, which is what a half-built or damaged schema looks like. That is not
   evidence of data, and reporting it as such booted the empty Postgres and hid
   a populated SQLite file: the exact failure this guard exists to prevent.

   Reachability is now established with SELECT 1 first, so the two cases get
   opposite answers: unreachable → leave the configured engine alone;
   reachable-but-uninspectable → unproven, and the SQLite side wins if it
   actually holds data.

2. The success marker was written after the SQLite file was renamed away. A
   failure in between — a full disk — left the source retired with no marker:
   the next attempt reported "No SQLite database", the in-progress pin stayed,
   and the operator never saw the rollback path. The marker is written first
   and updated with the retired filename once the rename succeeds, so a failure
   at any point leaves everything recoverable.

Verified against a real PostgreSQL 15: a reachable database whose admin_users
table lacks the probed column now resolves to sqlite3 rather than hiding the
data, while an unreachable host still resolves to pg.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): don't fail the migration on empty SQLite-only tables (#1038)

Review round 10. The final verification flagged every source table missing from
Postgres, regardless of whether it held rows — and SQLite-only tables do exist:
initializeDatabase() creates an `events_new` scratch table and, when its legacy
column copy throws, the catch swallows the error and leaves the empty table
behind (db.js:236). The importer correctly skips tables Postgres does not have,
so verification then reported a mismatch AFTER the data had already landed,
exited 1, and left the install pinned to SQLite with no way to finish.

An absent target table only matters if the source actually had rows. Empty ones
are now listed and skipped.

Reproduced both ways against a real PostgreSQL 15 with an events_new table
present: without the fix the run ends in "ROW COUNTS DO NOT MATCH" and leaves
the in-progress pin; with it, the table is reported as skipped, the migration
completes and the pin is released.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): a completed migration overrides an implicit SQLite config (#1038)

Review round 11. The migration allowed the one configuration it should have
worried about most: DATABASE_CLIENT unset AND NODE_ENV not "production", which
resolves to the development block — i.e. sqlite3. That is precisely the state
the affected installs are in, since it is why they ended up on SQLite at all,
so an operator can easily run the migration before fixing it. The script then
renames the source database away, and the next start resolved to the implicit
sqlite3, created a NEW empty database and served it — after reporting success.

The success marker now overrides an IMPLICITLY resolved sqlite3 when Postgres
settings are present, because the marker is durable proof of where the data
actually went. An explicit DATABASE_CLIENT=sqlite3 still wins: that is the
documented rollback.

The script says something rather than refusing — refusing would block exactly
the population this exists for.

Reproduced with NODE_ENV and DATABASE_CLIENT both empty, against a real
PostgreSQL 15: the migration completes, the source is renamed away, and the
next boot resolves to pg with the data intact. Before this it resolved to
sqlite3 and would have served an empty database.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* refactor(db): drop the dead reachability flag in probePgData (#1038)

github-code-quality flagged `if (reachable)` as always true, and it is right:
the unreachable branch returns, so everything below it runs only when the probe
connected. The variable and the conditional were leftovers from a first draft
that used a single catch for both failure classes.

No behaviour change — the two error paths still return opposite answers.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): refuse to choose when both databases hold data (#1038)

Review round 12.

1. An install that ran on PostgreSQL, lost NODE_ENV/DATABASE_CLIENT, and kept
   working on SQLite has REAL data on both sides: old rows in Postgres, newer
   ones in SQLite. The stranded-data rule only protected SQLite when Postgres
   was empty, so pulling this fix would have booted Postgres and hidden every
   gallery created since the switch — the exact failure this PR exists to
   prevent, in a variant I had not considered.

   A completed migration leaves a marker saying which side is current. Without
   one, two populated databases are a conflict: the boot stops and prints both
   targets, the two DATABASE_CLIENT values that resolve it, and the migration
   command that merges them. This is the only deliberate refusal in the change —
   guessing here would hide data AND split subsequent writes across two
   databases.

2. probePgData was handed knexConfig.connection even when knexfile had resolved
   to SQLite (a completed migration whose environment still says sqlite3), so
   node-postgres dialled its own localhost defaults instead of DB_HOST/DB_NAME —
   false "unreachable" diagnostics and a needless delay on every boot. The probe
   target is now built from the environment when the config is not pg.

The conflict is honoured by all three entry points: the resolver exits 3 with an
empty stdout, wait-for-db.sh stops the container, and server.js refuses to start.

Two existing tests asserted that Postgres wins when both sides hold data. They
encoded the pre-conflict assumption and described a state that cannot occur
after a real migration (which always leaves a marker); both now pass the marker.

Found while testing: the resolver's logger shim had no .error, so the conflict
path threw, was swallowed by the fallback, and silently chose Postgres — the
precise outcome this refuses to make. The shim is complete now.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): symmetric bootstrap rule, one resolved Postgres target (#1038)

Review round 13. Both findings are consequences of earlier rounds.

1. The conflict rule added in round 12 counted an untouched SQLite bootstrap
   admin as data. core/001_init.js seeds one whenever ADMIN_PASSWORD is set —
   including into the accidental SQLite database — so a healthy Postgres install
   that had ever started once without NODE_ENV would have had a seeded-only
   SQLite file beside it, been declared a both-populated conflict, and REFUSED
   TO BOOT. The bootstrap discrimination is applied on both sides now; a
   setup-completed or logged-in admin still counts as real use on either.

2. The CLI's child phases inherited whichever knexfile block NODE_ENV selected.
   The development block defaults Postgres to localhost/postgres/photo_sharing,
   production to db/picpeak/picpeak — and this script is explicitly meant to run
   with NODE_ENV unset. With DB_USER/DB_NAME left to defaults it would therefore
   have migrated into `photo_sharing`, after which following the script's own
   advice to set NODE_ENV=production pointed the app at an empty `picpeak`.

   The target is resolved once, with production defaults, and passed explicitly
   to every phase — so the block knexfile happens to pick can no longer decide
   which database the data lands in. The pin and success marker record that same
   resolved identity.

Verified against a real PostgreSQL 15: a live Postgres beside a seeded-only
SQLite file now boots pg rather than refusing, flipping that admin to
setup-completed restores the conflict, and a migration records
localhost:7102/picpeak_r13b as its target rather than a defaulted guess.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): one Postgres identity everywhere; protect the credentials file (#1038)

Review round 14. Three of the six findings were the same defect as round 13's,
surfacing through paths that fix did not cover: the connection used to PROBE or
MIGRATE could differ from the one the application then OPENS, because
knexfile's development block points Postgres at localhost/postgres/photo_sharing
while production uses db/picpeak/picpeak.

1. server.js exported only DATABASE_CLIENT=pg after the resolver decided, so
   knexfile filled in host/user/database from whichever block NODE_ENV selected.
   With SQLite already retired by a migration, that meant opening an empty
   database. The whole connection is pinned now.

2. Two defaults existed for DB_HOST: wait-for-db.sh resolves and exports
   `postgres`, knexfile's production block says `db`. Since the entrypoint
   exports its value, `postgres` is what a running container actually uses — so
   a `docker exec` migration, which inherits neither, has to agree with that,
   not with the default that is only reached when the entrypoint did not run.

3. The migration's Postgres phases inherited an unset NODE_ENV and therefore the
   development block, which ignores DB_SSL entirely — a managed Postgres
   requiring TLS could never be migrated into. The phases run with production
   semantics now.

4. core/001_init.js writes data/ADMIN_CREDENTIALS.txt, and that data directory
   belongs to the SOURCE install. Bootstrapping the Postgres schema replaced the
   operator's real credentials file with ones for a temporary admin the import
   immediately discards. The file is preserved across the phase, including when
   it fails.

5. The boot line described knexConfig, so an install redirected to Postgres by a
   migration marker still logged "Database engine: sqlite (...)", contradicting
   the warning printed one line earlier.

6. On a both-populated conflict resolveBootEngine returns client:null, and both
   migration runners told the operator their data was in "null" and to set
   DATABASE_CLIENT=null. They now present the two real choices.

Verified against a real PostgreSQL 15: a migrated install started directly with
NODE_ENV unset now logs `postgres (localhost:7102/picpeak_r14)` and opens it,
where before it would have gone to the development block's photo_sharing.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* refactor(db): resolve the PostgreSQL target in exactly one place (#1038)

Rounds 13 and 14 both traced back to the same thing, each time through a caller
the previous fix had not covered: three different defaults existed for the same
connection.

  knexfile development : localhost / postgres / photo_sharing
  knexfile production  : db        / picpeak  / picpeak
  wait-for-db.sh       : postgres  / picpeak  / picpeak   (and it EXPORTS them)

So a process that probed or migrated against one could hand over to a process
that opened another. Patching each caller was not converging — the guard, then
the CLI's child phases, then server.js — so this deletes the divergence instead.

`src/utils/pgConnection.js` now owns the resolution and knexfile's development
and production blocks both derive from it, as does the engine guard. Same shape
as the earlier sqlitePath.js extraction, for the same reason.

The database NAME is what made this dangerous: a wrong host or user fails
loudly at connect time, while a wrong name connects fine and presents an empty
installation.

BEHAVIOUR CHANGE: with DATABASE_CLIENT=pg and no DB_* variables, a
non-production environment now resolves to postgres/picpeak/picpeak instead of
localhost/postgres/photo_sharing. Deployments are unaffected — compose sets
these explicitly and wait-for-db.sh exports them — but a local machine running
Postgres bare now needs DB_HOST=localhost DB_USER=postgres DB_NAME=photo_sharing
(or DATABASE_CLIENT=sqlite3, which is what backend/.env already uses). The
failure mode of getting this wrong is a refused connection, not a silently empty
database.

Side effect worth having: DB_SSL is now honoured whatever NODE_ENV says, so the
managed-Postgres case is fixed at the root rather than by forcing production
semantics onto the migration's child phases.

The test block keeps its own photo_sharing_test default — isolation is the point
there.

Verified: every block plus the guard resolve identically from the same
environment; explicit DB_* still wins; production's pool tuning is preserved;
and a full SQLite → PostgreSQL migration with NODE_ENV unset lands in the right
database with JSON shapes intact.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): two more components that guessed the database instead of asking (#1038)

Both found while sweeping for copies of the connection defaults. Checked in
detail first — one of my suspicions about them was wrong.

scripts/set-admin-password.js hand-rolled its own knex config while all four
sibling scripts (reset-admin-password, create-admin, show-admin-credentials,
reset-admin-mfa) use the application's connection. Two consequences:

  - it read DB_CLIENT, a variable nothing else in this codebase sets, so it
    defaulted to Postgres and could not work on a SQLite install at all;
  - it defaulted to database `picpeak_dev`, a name no other component uses.

It now uses `require('../src/database/db')` like its siblings, so it follows
whatever engine the install actually runs on. Timestamps are written as ISO
strings because it reaches SQLite now, where raw Date objects are the documented
landmine.

NOT changed: the script's "all existing sessions have been invalidated" notice
is accurate — auth.js compares token iat against password_changed_at — and it
deliberately leaves must_change_password alone, which is right for an operator
choosing a password rather than being issued one.

routes/adminSystem.js re-derived three things the live connection already knows,
and each could disagree with it:

  - the engine, from DATABASE_CLIENT || 'sqlite3' — so a Postgres install
    without an explicit DATABASE_CLIENT took the SQLite branch;
  - the Postgres database, from DB_NAME || 'picpeak';
  - the SQLite file, from a hardcoded ../../data/photo_sharing.db that ignored
    DATABASE_PATH entirely.

All three now come from db.client.config, with pg_database_size(current_database()).

Verified: set-admin-password works on SQLite (new hash verifies, old rejected)
and still on PostgreSQL; and on a SQLite install with a custom DATABASE_PATH the
size logic reports the real database (1,748,992 bytes) where the old code
reported a different file entirely (1,851,392) — or 0 where that path does not
exist.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

* fix(db): bind the migration marker to its target; fix a phantom table (#1038)

Review round 15.

1. The marker records `host:port/database`, but only its EXISTENCE was checked.
   Repoint DB_NAME or DB_HOST at a different, empty PostgreSQL after migrating
   and the marker would vouch for that one too — booting it, presenting an empty
   installation, and suppressing the SQLite fallback while the real data sits in
   the recorded target and the renamed rollback copy. The marker is compared
   against the current connection now, and a mismatch stops the boot with both
   targets named and the two ways out.

2. `incoming_invoices` is not a table — supplier documents live in
   `inbound_documents` (core migration 124). Both occupancy lists skip tables
   that do not exist, so those records were silently not protecting anything:
   an install whose only remaining data was inbound documents could be switched
   away from, or overwritten without --force. Verified every other name in the
   lists against the live schema at the same time.

Verified: a marker naming picpeak_original with picpeak_mk configured refuses
with exit 3 and prints both; making them agree boots pg.

Claude-Session: https://claude.ai/code/session_0168gubtwYYacJv8weAjy8DM

---------

Co-authored-by: Paul Nothaft <paul@MacStudio-von-Paul.local>
2026-08-13 18:51:20 +02:00

485 lines
17 KiB
JavaScript

const express = require('express');
const { db, withRetry } = require('../database/db');
const { adminAuth } = require('../middleware/auth');
const { requirePermission } = require('../middleware/permissions');
const fs = require('fs').promises;
const path = require('path');
const os = require('os');
const { formatBoolean } = require('../utils/dbCompat');
const logger = require('../utils/logger');
const { resolveSqlitePath } = require('../utils/databaseEngine');
const { checkForUpdates, getCurrentChannel, getCurrentVersion, getReleasesSince, compareVersions } = require('../services/updateCheckService');
const { getAppSetting, upsertAppSetting } = require('../utils/appSettings');
const { parseWhatsNew } = require('../utils/whatsNew');
const { detectEnvironment, generateUpdateInstructions } = require('../services/environmentService');
const {
checkAndNotifyUpdates,
sendTestUpdateNotification,
getUpdateNotificationSettings
} = require('../services/updateNotificationService');
const router = express.Router();
// Get system version
router.get('/version', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
// Read backend version from package.json
let backendVersion = '1.0.0';
try {
const packagePath = path.join(__dirname, '../../package.json');
const packageContent = await fs.readFile(packagePath, 'utf8');
const packageJson = JSON.parse(packageContent);
backendVersion = packageJson.version || '1.0.0';
} catch (err) {
logger.error('Could not read package.json:', err);
}
const channel = getCurrentChannel(backendVersion);
res.json({
backend: backendVersion,
frontend: '1.0.0', // This will be set by frontend
node: process.version,
environment: process.env.NODE_ENV || 'production',
channel: channel
});
} catch (error) {
logger.error('Error fetching version:', error);
res.status(500).json({ error: 'Failed to fetch version information' });
}
});
// Check for updates
router.get('/updates', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
// Check if update checking is enabled
const updateCheckEnabled = process.env.UPDATE_CHECK_ENABLED !== 'false';
if (!updateCheckEnabled) {
return res.json({
enabled: false,
message: 'Update checking is disabled'
});
}
const forceRefresh = req.query.refresh === 'true';
const updateInfo = await checkForUpdates(forceRefresh);
// Pre-update teaser: the target version's top highlights, so the
// "Update Available" banner can show "New features include …".
let latestHighlights = [];
if (updateInfo.updateAvailable) {
try {
const newer = await getReleasesSince(updateInfo.current, updateInfo.channel);
if (newer[0]) latestHighlights = parseWhatsNew(newer[0].body);
} catch (_) { /* teaser is best-effort */ }
}
res.json({
enabled: true,
...updateInfo,
latestHighlights
});
} catch (error) {
logger.error('Error checking for updates:', error);
res.status(500).json({ error: 'Failed to check for updates' });
}
});
// What's New — after-update highlights. Returns the curated bullets for
// every release the instance moved THROUGH since it last acknowledged one
// (lastSeen < version <= running). Seen-tracking is per-INSTANCE: the first
// admin to dismiss clears it for everyone (a single app_settings row). A
// brand-new install initialises the marker silently so it never pops
// "what's new" with nothing to compare against. Best-effort: any failure
// (GitHub unreachable, etc.) returns hasNews:false, never errors.
router.get('/updates/whatsnew', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
if (process.env.UPDATE_CHECK_ENABLED === 'false') {
return res.json({ enabled: false, hasNews: false });
}
const running = await getCurrentVersion();
const channel = getCurrentChannel(running);
const lastSeen = await getAppSetting('whatsnew_last_seen_version', null);
if (!lastSeen) {
await upsertAppSetting('whatsnew_last_seen_version', JSON.stringify(running), 'system');
return res.json({ enabled: true, hasNews: false, running });
}
if (compareVersions(running, lastSeen) <= 0) {
return res.json({ enabled: true, hasNews: false, running });
}
// Releases in (lastSeen, running], newest-first, with their highlights.
const releases = (await getReleasesSince(lastSeen, channel))
.filter((r) => compareVersions(r.version, running) <= 0);
const versions = releases
.map((r) => ({
version: r.version,
name: r.name,
publishedAt: r.publishedAt,
htmlUrl: r.htmlUrl,
bullets: parseWhatsNew(r.body),
}))
.filter((v) => v.bullets.length > 0);
return res.json({
enabled: true,
hasNews: versions.length > 0,
fromVersion: lastSeen,
toVersion: running,
versions,
});
} catch (error) {
logger.error('Error building what\'s-new:', error);
res.json({ enabled: true, hasNews: false });
}
});
// Acknowledge the What's New — advance the per-instance marker to the
// running version so it stops showing for every admin.
router.post('/updates/whatsnew/seen', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
const running = await getCurrentVersion();
await upsertAppSetting('whatsnew_last_seen_version', JSON.stringify(running), 'system');
res.json({ ok: true, lastSeen: running });
} catch (error) {
logger.error('Error marking what\'s-new seen:', error);
res.status(500).json({ error: 'Failed to update marker' });
}
});
// Aggregated changelog — every release between current and latest in
// the user's channel. Powers the update-available modal (#567) so the
// admin can read release notes for ALL versions they're behind on, not
// just the latest. Body is raw GitHub-flavoured markdown; rendering is
// the client's job (frontend uses `marked`).
router.get('/updates/changelog', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
const updateCheckEnabled = process.env.UPDATE_CHECK_ENABLED !== 'false';
if (!updateCheckEnabled) {
return res.json({ enabled: false, releases: [] });
}
const updateInfo = await checkForUpdates();
const releases = await getReleasesSince(updateInfo.current, updateInfo.channel);
res.json({
enabled: true,
current: updateInfo.current,
channel: updateInfo.channel,
releases,
});
} catch (error) {
logger.error('Error fetching update changelog:', error);
res.status(500).json({ error: 'Failed to fetch changelog' });
}
});
// Get update instructions for current environment
router.get('/updates/instructions', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
// Check if update checking is enabled
const updateCheckEnabled = process.env.UPDATE_CHECK_ENABLED !== 'false';
if (!updateCheckEnabled) {
return res.json({
enabled: false,
message: 'Update checking is disabled'
});
}
const env = await detectEnvironment();
const updateInfo = await checkForUpdates();
if (!updateInfo.updateAvailable) {
return res.json({
updateAvailable: false,
currentVersion: updateInfo.current,
message: 'You are running the latest version'
});
}
const instructions = generateUpdateInstructions(env, updateInfo.latest.forChannel);
res.json({
updateAvailable: true,
currentVersion: updateInfo.current,
targetVersion: updateInfo.latest.forChannel,
channel: updateInfo.channel,
environment: env,
instructions,
releaseNotesUrl: `https://github.com/PicPeak/picpeak/releases/tag/v${updateInfo.latest.forChannel}`
});
} catch (error) {
logger.error('Error generating update instructions:', error);
res.status(500).json({ error: 'Failed to generate update instructions' });
}
});
// Get comprehensive system status
router.get('/status', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
// Database size. Read the LIVE connection rather than re-deriving any of
// this from the environment (#1038): DATABASE_CLIENT is not the only thing
// that decides the engine, DB_NAME is not the only thing that decides the
// database, and DATABASE_PATH was ignored outright here — so a SQLite
// install with a custom path, or a Postgres install without an explicit
// DATABASE_CLIENT, reported the size of something it was not using.
let dbSize = 0;
const liveConnection = db.client.config.connection || {};
if (db.client.config.client === 'pg') {
try {
const result = await db.raw('SELECT pg_database_size(current_database()) as size');
dbSize = result.rows[0]?.size || 0;
} catch (error) {
logger.error('Error getting PostgreSQL database size:', error);
}
} else {
try {
const stats = await fs.stat(liveConnection.filename || resolveSqlitePath());
dbSize = stats.size;
} catch (error) {
logger.error('Error getting SQLite database size:', error);
}
}
// Count various entities
const [eventsCount] = await db('events').count('* as count');
const [photosCount] = await db('photos').count('* as count');
const [adminsCount] = await db('admin_users').count('* as count');
const [categoriesCount] = await db('photo_categories').count('* as count');
// Email queue status
const [pendingEmails] = await db('email_queue').where('status', 'pending').count('* as count');
const [processableEmails] = await db('email_queue')
.where('status', 'pending')
.where('retry_count', '<', 3)
.count('* as count');
const [sentEmails] = await db('email_queue').where('status', 'sent').count('* as count');
const [failedEmails] = await db('email_queue').where('status', 'failed').count('* as count');
const [stuckEmails] = await db('email_queue')
.where('status', 'pending')
.where('retry_count', '>=', 3)
.count('* as count');
// Activity logs count
const [activityCount] = await db('activity_logs').count('* as count');
// Storage info
const [{ totalPhotoStorage }] = await db('photos')
.sum('size_bytes as totalPhotoStorage');
const archives = await db('events')
.where('is_archived', formatBoolean(true))
.whereNotNull('archive_path')
.select('archive_path');
let archiveStorage = 0;
const storagePath = process.env.STORAGE_PATH || path.join(__dirname, '../../../storage');
for (const archive of archives) {
if (archive.archive_path) {
try {
const fullArchivePath = path.join(storagePath, archive.archive_path);
const stats = await fs.stat(fullArchivePath);
archiveStorage += stats.size;
} catch (error) {
logger.error('Archive file not found:', archive.archive_path);
}
}
}
const totalStorage = (parseInt(totalPhotoStorage) || 0) + archiveStorage;
// System info
const systemInfo = {
platform: os.platform(),
arch: os.arch(),
hostname: os.hostname(),
uptime: Math.floor(process.uptime()),
nodeVersion: process.version,
memory: {
total: os.totalmem(),
free: os.freemem(),
used: os.totalmem() - os.freemem()
},
cpu: {
model: os.cpus()[0]?.model || 'Unknown',
cores: os.cpus().length
}
};
// Build response
const status = {
database: {
size: dbSize,
tables: {
events: eventsCount.count,
photos: photosCount.count,
admins: adminsCount.count,
categories: categoriesCount.count,
activityLogs: activityCount.count
}
},
storage: {
totalUsed: totalStorage,
photoStorage: parseInt(totalPhotoStorage) || 0,
archiveStorage: archiveStorage
},
emailQueue: {
pending: pendingEmails.count,
processable: processableEmails.count,
stuck: stuckEmails.count,
sent: sentEmails.count,
failed: failedEmails.count
},
system: systemInfo,
services: {
fileWatcher: { status: 'active' }, // These would ideally check actual service status
expirationChecker: { status: 'active' },
emailProcessor: { status: 'active' }
},
timestamp: new Date()
};
res.json(status);
} catch (error) {
logger.error('Error fetching system status:', error);
res.status(500).json({ error: 'Failed to fetch system status' });
}
});
// Get database statistics
router.get('/database', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
// Get table info
const tables = [
'events', 'photos', 'admin_users', 'photo_categories',
'cms_pages', 'email_templates', 'email_queue', 'activity_logs',
'app_settings', 'email_configs', 'access_logs', 'migrations'
];
const tableInfo = [];
for (const table of tables) {
try {
const [count] = await db(table).count('* as count');
// Get last update time
let lastUpdate = null;
try {
const lastRow = await db(table)
.orderBy('updated_at', 'desc')
.orOrderBy('created_at', 'desc')
.orOrderBy('timestamp', 'desc')
.orOrderBy('applied_at', 'desc')
.first();
if (lastRow) {
lastUpdate = lastRow.updated_at || lastRow.created_at || lastRow.timestamp || lastRow.applied_at;
}
} catch (e) {
// Table might not have timestamp columns
}
tableInfo.push({
name: table,
rows: count.count,
lastUpdate
});
} catch (error) {
// Table might not exist
logger.warn('Failed to retrieve table info', {
table,
error: error.message
});
tableInfo.push({
name: table,
rows: 0,
error: 'Unable to retrieve table details'
});
}
}
res.json({
tables: tableInfo,
timestamp: new Date()
});
} catch (error) {
logger.error('Error fetching database info:', error);
res.status(500).json({ error: 'Failed to fetch database information' });
}
});
// Get update notification settings
router.get('/updates/notifications', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
const settings = await getUpdateNotificationSettings();
res.json(settings);
} catch (error) {
logger.error('Error fetching update notification settings:', error);
res.status(500).json({ error: 'Failed to fetch update notification settings' });
}
});
// Update notification settings
router.put('/updates/notifications', adminAuth, requirePermission('settings.edit'), async (req, res) => {
try {
const { enabled, recipients } = req.body;
if (typeof enabled !== 'undefined') {
await db('app_settings')
.where('setting_key', 'update_email_notifications_enabled')
.update({
setting_value: JSON.stringify(enabled === true),
updated_at: db.fn.now()
});
}
if (typeof recipients !== 'undefined') {
await db('app_settings')
.where('setting_key', 'update_email_recipients')
.update({
setting_value: JSON.stringify(recipients || ''),
updated_at: db.fn.now()
});
}
const updatedSettings = await getUpdateNotificationSettings();
res.json({ success: true, settings: updatedSettings });
} catch (error) {
logger.error('Error updating notification settings:', error);
res.status(500).json({ error: 'Failed to update notification settings' });
}
});
// Manually trigger update notification email
// Send a test update notification email. Uses the dedicated
// `version_update_test` template (migration 087) rather than reusing
// `version_update_available`, so admins on the latest version can still
// verify their SMTP + recipient config — the previous handler bailed
// with "No updates available" when nothing was pending (#418).
router.post('/updates/notifications/send', adminAuth, requirePermission('settings.edit'), async (req, res) => {
try {
const result = await sendTestUpdateNotification();
res.json(result);
} catch (error) {
logger.error('Error sending test update notification:', error);
res.status(500).json({ error: 'Failed to send test update notification' });
}
});
// Check and send update notifications (called on admin login or periodically)
router.post('/updates/notifications/check', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
const result = await checkAndNotifyUpdates();
res.json(result);
} catch (error) {
logger.error('Error checking for update notifications:', error);
res.status(500).json({ error: 'Failed to check for update notifications' });
}
});
module.exports = router;