paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(backup): config-driven walker via backup_paths table

Browse Source 
Stage B of the three-stage backup-hardening plan (Stage A:
inline-DB-dump + fail-loud guard already landed). The file-backup
walker used to hard-code its subdirectory list inside
`getFilesToBackupInternal`, which is the same footgun that hid the
`business-docs` gap for ~6 months — a new feature drops artefacts
under STORAGE_PATH and the maintainer has to remember to edit the
walker.

Now driven by a `backup_paths` table:

  - Migration 108 creates the table and seeds the 7 canonical
    defaults (events/active, events/archived, thumbnails, previews,
    heroes, uploads, business-docs). Seed data lives on the
    migration as `DEFAULT_PATHS` so the boot self-heal can re-use it.
  - `_backupPathsBoot.js` mirrors `_emailTemplateBoot.js`: on every
    boot it diffs the canonical list against the current rows and
    `INSERT ... ON CONFLICT DO NOTHING`s the missing ones. Keeps
    admin edits intact, picks up new defaults shipped after the
    install (Knex won't re-run migration 108). Wired into server.js
    just before `startBackupService()`.
  - Walker now calls `resolveBackupPaths(config)` which:
      * reads `backup_paths WHERE include_in_default=true ORDER BY
        display_order`
      * falls back to a hard-coded `LEGACY_BACKUP_PATHS` if the
        table is missing OR empty (defense in depth — never silently
        scans nothing)
      * gates each row by its `feature_flag` column (matches how
        `backup_include_archived` already worked; data-driven now)
  - Backward compatible: `getFilesToBackup(true|false)` still works
    for legacy callers and the existing businessDocs test. New
    callers should pass the full config object so feature gates
    other than `backup_include_archived` evaluate correctly.

Tests:
  - new: `backupService.configurableWalker.test.js` — 7 cases
    covering canonical seed, toggling include_in_default, runtime
    INSERT picked up without restart, feature_flag gating both on
    and off, empty-table → LEGACY fallback, boolean backward compat
  - all 15 backup-walker integration tests pass
    (configurableWalker 7 + inlineDbDump 5 + businessDocs 3)
  - frontend build clean
  - 4 pre-existing integration failures (webhookDelivery, storage
    backend, adminPhotos.reference, imageProcessor.storage) confirmed
    unrelated via `git stash` baseline run

Stage C (CRM feature coverage audit + diagnostic UI) follows
in a separate commit.
This commit is contained in:
Luca
2026-05-29 22:09:23 +02:00
parent 7fdf01ad21
commit 302fc6b937
5 changed files with 518 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/services/_backupPathsBoot.js
+90
View File
@@ -0,0 +1,90 @@
/**
* Boot-time self-heal for the `backup_paths` table.
*
* **Why this exists**
*
* Knex won't re-run an applied migration, so once migration
* 108_add_backup_paths.js has run, any later default we want to add
* (a new subdirectory shipped by a future feature) would never reach
* already-deployed installs. The historical fix for this kind of
* "schema is fine, seed drifted" problem is the boot-time self-heal
* pattern documented in [[feedback_self_heal_pattern]] — we just
* re-apply the canonical seed on every boot with `onConflict.ignore()`
* so admin edits stay intact and new rows trickle in.
*
* **Authoritative list**
*
* The list of defaults lives on migration 108 itself
* (`DEFAULT_PATHS` export) — one source of truth that both the
* migration and this seeder read. Tests assert these two stay in
* lockstep.
*
* **Failure semantics**
*
* If the table doesn't exist yet (migrations haven't run, fresh
* install before migration 108 lands, etc.) we no-op and log. The
* walker has a hard-coded `LEGACY_DEFAULTS` fallback for the same
* reason — defense in depth so "Run Backup Now" can never silently
* ship a files-only manifest because of a seed issue. See
* `backupService.js` getFilesToBackupInternal.
*/
const { DEFAULT_PATHS } = require('../../migrations/core/108_add_backup_paths');
let booted = false;
/**
* Idempotently re-seed `backup_paths` with the canonical defaults.
*
* @param {object} db knex instance
* @param {object} logger app logger (must expose .info / .warn)
* @returns {Promise<{ seeded: string[] }>} paths newly inserted on this boot.
*/
async function seedBackupPathsAtBoot(db, logger) {
const log = logger || { info: () => {}, warn: () => {} };
if (booted) return { seeded: [] };
if (!(await db.schema.hasTable('backup_paths'))) {
log.warn('backup_paths table missing at boot — self-heal skipped (migration 108 may not have run yet)');
return { seeded: [] };
}
// Diff: which canonical paths are missing from the table right now?
// We can't easily get "what got inserted by onConflict.ignore" out of
// knex on both backends, so we just compute the diff ourselves and log
// it — admins benefit from seeing exactly what got auto-added when a
// new feature ships.
const existing = await db('backup_paths').select('path');
const existingSet = new Set(existing.map((r) => r.path));
const missing = DEFAULT_PATHS.filter((p) => !existingSet.has(p.path));
if (missing.length === 0) {
booted = true;
return { seeded: [] };
}
try {
await db('backup_paths')
.insert(missing.map((row) => ({
...row,
created_at: new Date(),
updated_at: new Date(),
})))
.onConflict('path')
.ignore();
log.info(`backup_paths self-heal added ${missing.length} row(s): ${missing.map((m) => m.path).join(', ')}`);
} catch (err) {
log.warn(`backup_paths self-heal failed: ${err.message}`);
}
booted = true;
return { seeded: missing.map((m) => m.path) };
}
// Test-only: reset the module-level boot flag so jest can re-exercise
// the seeder against a fresh test DB inside a single worker.
function _resetBootForTests() {
booted = false;
}
module.exports = { seedBackupPathsAtBoot, _resetBootForTests };
backend/src/services/backupService.js
+105 -33
View File
@@ -417,43 +417,111 @@ async function scanDirectory(dirPath, fileList, basePath, excludePatterns = [])
}
}
async function getFilesToBackupInternal(includeArchived = true) {
/**
* Hard-coded fallback when `backup_paths` is missing/empty. Mirrors
* the canonical seed in migration 108 — kept here as defense in depth
* so the walker can never silently degrade to "no directories scanned"
* because of a seed problem.
*
* Order matches the legacy behavior of the inlined sequence this
* function used to contain.
*/
const LEGACY_BACKUP_PATHS = [
{ path: 'events/active', feature_flag: null },
{ path: 'events/archived', feature_flag: 'backup_include_archived' },
{ path: 'thumbnails', feature_flag: null },
{ path: 'previews', feature_flag: null },
{ path: 'heroes', feature_flag: null },
{ path: 'uploads', feature_flag: null },
{ path: 'business-docs', feature_flag: null },
];
/**
* Resolve the walker's target subdirectories from `backup_paths`.
*
* Layered fallback (defense in depth — no scenario where the walker
* silently scans nothing):
* 1. Read `backup_paths` rows where include_in_default = true,
* ordered by display_order.
* 2. If the table is missing OR returns zero rows, fall back to
* LEGACY_BACKUP_PATHS. Logged loudly so the admin sees it.
*
* Per-row gating: when `feature_flag` is set, the corresponding
* config key in `app_settings` must resolve truthy for that path to
* be included. Mirrors the historical `includeArchived` parameter,
* but now driven by data instead of a hard-coded boolean.
*
* @param {object} config resolved backup config (parseSettingValue'd).
* Used to evaluate feature_flag gates.
* @returns {Promise<Array<{ path: string, feature_flag: string|null }>>}
*/
async function resolveBackupPaths(config) {
let rows;
try {
if (!(await db.schema.hasTable('backup_paths'))) {
logger.warn('backup_paths table missing — falling back to LEGACY_BACKUP_PATHS');
rows = LEGACY_BACKUP_PATHS;
} else {
rows = await db('backup_paths')
.where('include_in_default', formatBoolean(true))
.orderBy('display_order', 'asc')
.select('path', 'feature_flag');
if (!rows.length) {
logger.warn('backup_paths has no rows with include_in_default=true — falling back to LEGACY_BACKUP_PATHS');
rows = LEGACY_BACKUP_PATHS;
}
}
} catch (err) {
logger.warn(`Failed to query backup_paths (${err.message}) — falling back to LEGACY_BACKUP_PATHS`);
rows = LEGACY_BACKUP_PATHS;
}
// Apply feature_flag gating. A row with feature_flag='backup_include_archived'
// requires config.backup_include_archived to be truthy (same semantics as
// the historical `includeArchived` parameter).
return rows.filter((row) => {
if (!row.feature_flag) return true;
const flagValue = config ? config[row.feature_flag] : undefined;
return normalizeBoolean(flagValue);
});
}
async function getFilesToBackupInternal(configOrIncludeArchived = true) {
const files = [];
const storagePath = getStoragePath();
await scanDirectory(path.join(storagePath, 'events/active'), files, storagePath);
if (normalizeBoolean(includeArchived)) {
await scanDirectory(path.join(storagePath, 'events/archived'), files, storagePath);
// Backward-compatible call signature:
// - Boolean `true|false` → legacy `includeArchived` argument. We
// forge a config-shaped object so the feature-flag gating
// resolves the same way the old code path did.
// - Object → full resolved backup config (preferred).
// - Anything else → treated as "include archived" (truthy).
let config;
if (typeof configOrIncludeArchived === 'object' && configOrIncludeArchived !== null) {
config = configOrIncludeArchived;
} else {
config = { backup_include_archived: normalizeBoolean(configOrIncludeArchived) };
}
await scanDirectory(path.join(storagePath, 'thumbnails'), files, storagePath);
// Lightbox preview tier (#492). Cheap to back up — typically a few
// hundred KB per photo — and saves admins the regenerate cycle on
// a restore. Tolerated when missing (admins who never enabled the
// feature won't have the folder; scanDirectory short-circuits on
// ENOENT cleanly).
await scanDirectory(path.join(storagePath, 'previews'), files, storagePath);
// Heroes too — same logic; admins who picked a hero photo for the
// gallery header had its 1920x1080 file generated and was missed
// by the original backup walk before this addition.
await scanDirectory(path.join(storagePath, 'heroes'), files, storagePath);
await scanDirectory(path.join(storagePath, 'uploads'), files, storagePath);
// CRM document estate — every PDF and signature artefact the
// service persists for legal-evidence purposes:
// - business-docs/quote/<year>/*.pdf
// - business-docs/contract/<year>/*.pdf (system-rendered + wet uploads)
// - business-docs/contract/signatures/<contract_id>/*.{png,jpg}
// (drawn signatures, forensic-preserved per Date.now() filename)
// - business-docs/invoice/<year>/*.pdf (issued invoices + Storno)
// - business-docs/invoice-imports/<year>/*.pdf (admin-imported
// historical invoices — irrecoverable if not backed up)
// Without this scan, the audit trail (signed_pdf_sha256, signed_*
// _ip, accepted_at, etc.) survives the restore but the documents
// those values refer to do not, leaving every CRM *_path column a
// broken FK. scanDirectory short-circuits on ENOENT so installs
// that never used CRM features won't error.
await scanDirectory(path.join(storagePath, 'business-docs'), files, storagePath);
const targets = await resolveBackupPaths(config);
for (const target of targets) {
// CRM document estate is special-cased in the comment block below
// because it's the most expensive omission to recover from:
// - business-docs/quote/<year>/*.pdf
// - business-docs/contract/<year>/*.pdf (system-rendered + wet uploads)
// - business-docs/contract/signatures/<contract_id>/*.{png,jpg}
// (drawn signatures, forensic-preserved per Date.now() filename)
// - business-docs/invoice/<year>/*.pdf (issued invoices + Storno)
// - business-docs/invoice-imports/<year>/*.pdf (admin-imported
// historical invoices — irrecoverable if not backed up)
// Without this scan, the audit trail (signed_pdf_sha256, signed_*
// _ip, accepted_at, etc.) survives the restore but the documents
// those values refer to do not, leaving every CRM *_path column a
// broken FK. scanDirectory short-circuits on ENOENT so installs
// that never used CRM features won't error.
await scanDirectory(path.join(storagePath, target.path), files, storagePath);
}
return files;
}
@@ -887,7 +955,11 @@ async function runBackupInternal(isManual = false) {
// for the full rationale.
const verifiedDatabaseInfo = await ensureDatabaseDumpForBackup(config);
const files = await service.getFilesToBackup(config.backup_include_archived);
// Pass the full config so the walker can evaluate any feature_flag
// gates declared in the backup_paths table (e.g. `events/archived`
// gated by `backup_include_archived`). Boolean signature is still
// supported for legacy callers and tests — see getFilesToBackupInternal.
const files = await service.getFilesToBackup(config);
logger.info(`Found ${files.length} files to check for backup`);
let result;