Refined from the original #530 framing after a dry-run uncovered that the
"bootstrap vs migration chain" diff produces mostly noise — most of the
~200 lines of difference are expected (migrations add new tables and
columns over time). initializeDatabase() isn't a parallel path that
diverges from migrations; it's invoked by migration 001 itself, so every
normal install/upgrade runs both.
The genuine drift hazard surfaced during the dry-run: a DB with the
modern bootstrap tables but an empty `migrations` table (which happens
when a backup was restored that lost the migrations table, or someone
invoked initializeDatabase() outside the runner, or the DB was moved
between systems without copying the migrations row) fails to upgrade.
Failure mode:
1. detectExistingSchema sees the bootstrap tables + empty migrations,
treats it as an "existing deployment".
2. Runs the legacy chain first.
3. legacy/008 renames email_templates.subject → subject_en.
4. core/029 (later in the chain) inserts email templates referencing
the pre-rename `subject` column.
5. Postgres rejects: column "subject" doesn't exist; subject_en is
NOT NULL with no default.
Fresh installs avoid this because they only run core/* (and core/059
handles the rename AFTER core/029 has inserted). Real legacy upgrades
avoid it because their migrations table already records legacy/008–028
as applied historically.
Fix in detectExistingSchema:
- Detect the modern bootstrap fingerprint (photo_categories + cms_pages
both present, which initializeDatabase produces as part of the
consolidated post-004-era bootstrap).
- When matched, enumerate every file in migrations/legacy/ and mark
each as applied. This puts the recovery state on the same code path
fresh installs use — only core migrations run, in core order.
- Real legacy upgrades that already have entries in the migrations
table hit no-op markings (markMigrationAsApplied skips duplicates),
so their behaviour is unchanged.
New CI workflow (`.github/workflows/schema-drift.yml`):
- Boots fresh postgres.
- Seeds via `node -e \"require('./src/database/db').initializeDatabase()\"`
— reproduces the recovery state in one line.
- Runs `npm run migrate:safe`.
- Asserts: precondition (bootstrap fingerprint + empty migrations
table), migrate:safe exits 0, final schema has ≥40 tables (soft floor,
not exact pin so future migrations don't force workflow edits),
legacy migrations marked applied (confirms the fingerprint check
actually fired vs. the chain silently bailing).
- Triggers only on PRs that touch backend/migrations/**,
src/database/db.js, knexfile.js, or this workflow.
Manually verified end-to-end before this commit:
Before fix: migrate:safe dies at core/029 with NOT NULL violation
on email_templates.subject_en (17/48 tables present).
After fix: 82 migrations applied + 27 marked applied = 109 total,
final state has all 48 tables matching fresh-install.
Issue body in #530 has been updated to match this refined scope.
Refs: #530, #484, #519
Two latent install-time issues that emitted scary postgres ERROR lines
on every fresh start but didn't actually break anything. MrGabri flagged
them after #494 had already cleared the FK-ordering crash.
1. Migration 035 builds three `CREATE INDEX` statements against
`backup_runs(created_at, …)`, but 029 creates the table with
`started_at` and no `created_at`. The wrapping try/catch silently
swallowed the resulting `column "created_at" does not exist` ERROR,
so the migration "succeeded" without ever creating the indexes.
Switched 035 to reference `started_at` (same chronological semantics)
and added migration 105 to create the same indexes idempotently for
deployments whose 035 already ran and silently failed.
2. `run-migrations-safe.js` snapshots `appliedFilenames` *before*
`detectExistingSchema()` runs. When `detectExistingSchema()` inserts a
row for e.g. `004_add_categories_and_cms.js` (because its tables exist
from a partially-completed prior install), the subsequent migration
loop still doesn't know about that insert, attempts the legacy
migration anyway, and its transaction-internal
`insert into migrations` conflicts with the row already there.
Re-query the applied set after detectExistingSchema so the loop sees
the corrected snapshot.
No behavioural change for healthy installs. New installs no longer log
the `column "created_at" does not exist` or `duplicate key value
violates unique constraint "migrations_filename_unique"` ERRORs.
- Added check in markMigrationAsApplied to prevent duplicate inserts
- Now checks if migration is already marked before inserting
- Prevents 'duplicate key value violates unique constraint' error
The error occurred when detectExistingSchema() marked a migration
as applied, then the migration runner caught a 'schema exists' error
and tried to mark it as applied again.
- Removed must_change_password field that doesn't exist in admin_users table
- Changed from using db to knex parameter for database operations
- Fixed require statement that was accidentally changed
- Updated security message to reflect no forced password change
- Removed debug logging after identifying the issue
The error occurred because 001_init.js was trying to insert a column
that doesn't exist in the admin_users table schema created by
initializeDatabase().
- Changed from standalone script to proper migration with exports.up/down
- Removed process.exit() calls that were terminating the migration runner
- Removed immediate execution of runMigrations()
- Now properly exports migration functions like other migrations
This was the root cause - 001_init.js was executing immediately when
required and calling process.exit(), preventing it from being run as
a migration and causing 029 to run first on an empty database.
- Changed from string sort to numeric sort for migration files
- String sort was causing '029' to run before '001'
- Now properly extracts and compares numeric prefixes
- Applied fix to both run-migrations.js and run-migrations-safe.js
This ensures 001_init.js runs first and creates all necessary tables
before other migrations try to use them.
- Check for essential tables (events, photos, admin_users, activity_logs)
to determine if it's truly a new deployment
- Only run detectExistingSchema() for actual existing deployments
- Remove obsolete init.js references (now 001_init.js)
- Fix migration filters to handle renamed init file
The issue was that detectExistingSchema() was marking migrations as
applied from previous failed runs, causing the system to incorrectly
treat new deployments as existing ones and run legacy migrations that
expect tables to already exist.
- Renamed core/init.js to core/001_init.js to ensure it runs first
- Updated detectExistingSchema() to reference 001_init.js
- This fixes the issue where backup migrations tried to access
app_settings table before it was created
- Migrations now run in correct order: init first, then numbered
The error occurred because alphabetical sorting put 029 before init,
causing migrations to fail on new deployments.
- Created core/ directory for essential migrations that always run
- Created legacy/ directory for migrations only needed when upgrading
- New deployments will only run core migrations for a clean database
- Existing deployments will run all migrations in proper sequence
- Fixed duplicate migration numbers (014 and 027)
- Updated migration runners to handle new directory structure
- Added README explaining the migration organization
This change optimizes deployment for new users who will get a clean
schema without running unnecessary upgrade migrations.
Original: feat: enhance security logging and ensure rate limit blocks are properly tracked
- Add comprehensive logging for rate limit blocks with full request details
- IP address (with proper proxy detection), user agent, headers, timestamps
- Rate limit info (current count, limit, remaining, reset time)
- Separate tracking for auth vs general endpoints
- Enhance authentication failure logging
- JWT validation failures with detailed error info
- Admin auth attempts without token
- Failed token validation with user context
- All events include IP, path, method, user agent
- Improve Winston logger configuration for production
- Add automatic log rotation (10MB errors, 50MB combined)
- Create separate security.log for auth/rate limit events
- Ensure logs directory exists automatically
- Add structured JSON format for log aggregation
- Support container logging with LOG_TO_CONSOLE env var
- Create comprehensive documentation
- Security logging guide with examples
- Monitoring recommendations
- Configuration reference
- Add test script to verify logging functionality
All rate limit settings remain configurable via admin panel:
- Window duration, max requests, auth limits
- Skip authenticated requests option
- Public endpoints only option
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
Paul Nothaft
paul