paul/picpeak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(admin): make "Storage used" report storage used (#1164) (#1177)

Browse Source 
* fix(admin): make "Storage used" report storage used (#1164)

Stable twin of #1170.

The tile summed photos.size_bytes — the catalogued size of the ORIGINALS,
which in reference mode live on external storage and have no relationship to
the disk PicPeak runs on. The reporter's tile read ~80 GB against 21 GB of
real usage. Worse than the label: the same number drove the soft-limit warning
bar and, via /storage/info, the recommended soft limit — so a reference-mode
install got a disk-capacity recommendation computed from bytes that are not on
the disk.

- new localStorageUsage service walks the storage root and reports the total
  plus a breakdown. Walking rather than summing DB columns is the point:
  thumbnail/preview/hero rows record a key and never a byte count, and orphans
  from a deleted event or an interrupted import are real bytes.
- the external media root is excluded when it sits inside the storage root.
  Its compose default is <storage>/external-media, where the NAS is
  bind-mounted — a plain directory, not a symlink — so walking it would put
  every referenced original back into a figure whose purpose is to leave them
  out. Symlinks are not followed either.
- .download-cache gets its own line: it lives inside the event directory, so
  the naive rule files a multi-GB zip as photography.
- concurrent cold-cache callers share one walk; the dashboard, /storage/info
  and the sidebar are routinely requested together.
- S3 installs keep the catalogued figure and the walk is skipped before it
  runs, since the objects are in the bucket and STORAGE_PATH holds only
  incidental local files.
- an absent measurement reads as "unavailable" and a partial one is marked
  `+` across the dashboard, analytics, sidebar and status tab — a floor
  silently compared against a soft limit reads as "safely under".

Verified on this branch: 11 new service tests, dashboardScope updated for the
changed contract, full suite leaves the same 5 pre-existing failures as
origin/stable. Frontend 20 files / 104 tests, tsc clean.

* fix(admin): tell "no disk to measure" apart from "the measurement failed" (#1164)

External review found both of these on this branch.

Both were reported as `storage_measurement: 'catalog'`, so a failed local walk
made the dashboard claim the objects live in S3. They are different things —
one is a fact about the install, the other is a fault — and there is now an
`unavailable` state for the second.

The analytics percentage could reach the billions. `safeSoftLimit` fell back to
`storageUsed || 1`, and on S3 that is null → 1, while the figure beside it came
from `catalogedBytes`. An editor or viewer holds `analytics.view` but not
`settings.view`, so `/storage/info` 403s for them and `storageInfo` is
undefined — which is exactly when that fallback fires. It now falls back to the
measured figure, and suppresses the percentage entirely when there is no real
limit rather than dividing usage by itself and always reading 100%.

---------

Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
2026-08-26 09:14:48 +02:00
committed by GitHub
co-authored by Paul Nothaft
parent 9ffbe2f98f
commit ac7ef266dc
15 changed files with 634 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/adminDashboard.js
+38 -3
View File
@@ -7,6 +7,7 @@ const { formatBoolean } = require('../utils/dbCompat');
const { resolveAdapter } = require('../services/trackers');
const logger = require('../utils/logger');
const { errorResponse, getPagination } = require('../utils/routeHelpers');
const { measureLocalStorageUsage } = require('../services/localStorageUsage');
const router = express.Router();
/**
@@ -88,11 +89,33 @@ router.get('/stats', adminAuth, requirePermission('analytics.view'), async (req,
.count('id as count')
.first();
// Get storage usage (sum of all photo sizes)
const storageUsed = await applyEventScope(db('photos'), req.admin, 'event_id')
// The catalogued size of the ORIGINALS. Kept, and still worth showing —
// it answers "how much photography is in here" — but it is emphatically
// NOT storage used, which is what it was labelled for years (#1164).
const catalogedBytes = await applyEventScope(db('photos'), req.admin, 'event_id')
.sum('size_bytes as total')
.first();
// Storage used: what is actually on this machine. In reference mode the
// originals above live on a NAS and contribute nothing here; conversely
// this counts what the sum never did — thumbnails, previews, hero
// renditions, watermarks and the per-event download cache.
//
// Deliberately NOT event-scoped, unlike everything else on this endpoint:
// it is a disk measurement, and disk is not divisible by which admin owns
// which event.
//
// Skipped entirely on an S3 backend: the objects are in the bucket and a
// walk of STORAGE_PATH would report near-zero, which is worse than the
// catalogued figure those installs had before #1164.
const usesLocalBackend = (process.env.STORAGE_BACKEND || 'local').toLowerCase() !== 's3';
let localStorage = null;
try {
if (usesLocalBackend) localStorage = await measureLocalStorageUsage();
} catch (err) {
logger.warn(`Dashboard storage measurement failed: ${err.message}`);
}
// Get total views (last 30 days)
const thirtyDaysAgo = new Date();
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
@@ -154,7 +177,19 @@ router.get('/stats', adminAuth, requirePermission('analytics.view'), async (req,
activeEvents: activeEvents.count || 0,
expiringEvents: expiringEvents.count || 0,
totalPhotos: totalPhotos.count || 0,
storageUsed: storageUsed.total || 0,
// Real bytes on this disk. Null when the measurement failed or was
// skipped, which the UI shows as "unavailable" rather than substituting
// a number that means something else.
storageUsed: localStorage ? localStorage.total : null,
// Three states, not two: 'catalog' means the backend is S3 and the
// objects are in the bucket, which is a fact about the install;
// 'unavailable' means the walk failed, which is a fault. Collapsing them
// made a failed local measurement claim the objects live in S3.
storageMeasurement: localStorage ? 'disk' : (usesLocalBackend ? 'unavailable' : 'catalog'),
storageBreakdown: localStorage ? localStorage.breakdown : null,
storagePartial: localStorage ? localStorage.partial : false,
// Catalogued original bytes — what `storageUsed` used to report (#1164).
catalogedBytes: Number(catalogedBytes.total) || 0,
totalViews: totalViews.count || 0,
totalDownloads: totalDownloads.count || 0,
viewsTrend: Math.round(viewsTrend * 10) / 10,
backend/src/routes/adminSettings.js
+29 -1
View File
@@ -24,6 +24,7 @@ const { clearShareLinkSettingsCache } = require('../services/shareLinkService');
const { resetSecurityConfigCache } = require('../utils/authSecurity');
const { errorResponse } = require('../utils/routeHelpers');
const logger = require('../utils/logger');
const { measureLocalStorageUsage } = require('../services/localStorageUsage');
const router = express.Router();
const { clearMaxFilesPerUploadCache, MAX_ALLOWED_FILES_PER_UPLOAD } = require('../services/uploadSettings');
const watermarkService = require('../services/watermarkService');
@@ -1174,6 +1175,8 @@ router.put('/seo', adminAuth, requirePermission('settings.edit'), async (req, re
router.get('/storage/info', adminAuth, requirePermission('settings.view'), async (req, res) => {
try {
// Get total storage used
// Catalogued original bytes. Reported, but no longer as "used" (#1164) —
// in reference mode those files are on a NAS and none of them are here.
const totalStorage = await db('photos')
.sum('size_bytes as total')
.first();
@@ -1254,7 +1257,25 @@ router.get('/storage/info', adminAuth, requirePermission('settings.view'), async
}
}
const totalUsed = totalStorage?.total || 0;
// What is actually on this disk. This is what the soft limit is compared
// against and what the recommendation below is derived from, so getting it
// from the catalogued originals was the load-bearing half of #1164: a
// reference-mode install got a disk-capacity recommendation computed from
// bytes that are not on the disk.
//
// Gated BEFORE the walk: this endpoint is polled by the sidebar, and an S3
// install with a large local tree would otherwise pay a full traversal on
// every cold cache only to discard the result.
const catalogedBytes = Number(totalStorage?.total) || 0;
const usesLocalBackend = (process.env.STORAGE_BACKEND || 'local').toLowerCase() !== 's3';
let localUsage = null;
try {
if (usesLocalBackend) localUsage = await measureLocalStorageUsage();
} catch (err) {
logger.warn(`Storage measurement failed, falling back to catalogued bytes: ${err.message}`);
}
const measuredFromDisk = usesLocalBackend && !!localUsage;
const totalUsed = measuredFromDisk ? localUsage.total : catalogedBytes;
const parseBytesValue = (value) => {
const numeric = Number(value);
@@ -1376,6 +1397,13 @@ router.get('/storage/info', adminAuth, requirePermission('settings.view'), async
res.json({
total_used: totalUsed,
// What total_used used to be, kept so the UI can show both and the
// difference stops being invisible.
cataloged_bytes: catalogedBytes,
storage_measurement: measuredFromDisk ? 'disk' : (usesLocalBackend ? 'unavailable' : 'catalog'),
storage_breakdown: measuredFromDisk ? localUsage.breakdown : null,
storage_partial: measuredFromDisk ? localUsage.partial : false,
excluded_external_root: measuredFromDisk ? localUsage.excludedExternalRoot : null,
archive_storage: archiveStorage,
storage_by_event: storageByEvent,
storage_limit: effectiveSoftLimit,
backend/src/services/localStorageUsage.js
+216
View File
@@ -0,0 +1,216 @@
/**
* What PicPeak actually occupies on this machine (#1164).
*
* The dashboard's "Storage used" tile summed photos.size_bytes, which is the
* catalogued size of the ORIGINALS — a number with no relationship to the disk
* PicPeak runs on:
*
* - in reference mode the originals are never copied. Those bytes are on the
* NAS. The reporter's tile read ~80 GB against 21 GB of real local usage.
* - duplicate rows counted the same file twice (#1162).
* - it ignored everything PicPeak genuinely does write: thumbnails, previews,
* hero renditions, and the per-event download cache — an 11.8 GB
* `.download-cache/all.zip` sat outside the figure entirely.
*
* So the one number an admin reaches for when asking "am I running out of
* disk" pointed away from the answer and omitted exactly the things filling
* the disk. This walks the storage root instead and reports what is there.
*
* Walking rather than summing DB columns is deliberate: thumbnail/preview/hero
* rows record a key, never a byte count, and orphans (a deleted event's
* leftovers, an interrupted import's thumbnails) are real bytes on a real
* disk. A `du` is the only honest answer, and the only one that notices what
* PicPeak has forgotten about.
*
* The external media root is EXCLUDED, and that is the whole point rather than
* a detail. Its compose default is `<storage>/external-media`, where the NAS is
* bind-mounted — a plain directory, not a symlink — so walking it would add
* every referenced original back into a figure that exists to leave them out,
* and compare NAS bytes against statfs() of the local disk. That is the
* over-count this replaces, reintroduced by the fix for it.
*
* Cached, because it is one stat per file. On a large install that is seconds,
* and the dashboard is polled — and concurrent misses share one walk rather
* than each starting their own.
*/
const path = require('path');
const fsp = require('fs').promises;
const logger = require('../utils/logger');
const { getStoragePath } = require('../config/storage');
const TTL_MS = 5 * 60 * 1000;
let cache = null;
// The walk in flight, if any. Two admins loading the dashboard, or the sidebar
// and the storage tab on one page, hit the same cold cache and would otherwise
// each stat every file on the disk.
let inFlight = null;
/**
* The external media root, resolved, when it lies inside the storage root.
* Returns null when it is elsewhere (the usual production case) or cannot be
* resolved — nothing to exclude then.
*/
function nestedExternalRoot(storageRoot) {
let externalRoot;
try {
externalRoot = require('./externalMediaService').getExternalMediaRoot();
} catch (err) {
return null;
}
if (!externalRoot) return null;
const resolvedExternal = path.resolve(externalRoot);
const resolvedStorage = path.resolve(storageRoot);
if (resolvedExternal === resolvedStorage) return null;
return resolvedExternal.startsWith(resolvedStorage + path.sep) ? resolvedExternal : null;
}
/**
* Which line of the breakdown a path belongs to.
*
* The names are the ones the writers actually use — `heroes` from
* imageProcessor, `watermarks` from watermarkService, and so on — rather than
* the ones the layout docs imply. Getting one wrong is not a crash, it is a
* silent 30 MB in "other", which is the least useful place for it to land.
*
* `.download-cache` is the case that needs the explicit check: it lives INSIDE
* an event directory, so the naive rule files an 11.8 GB zip as photography —
* and it is the one bucket that is pure disposable cache, which makes it the
* one an admin most wants to see on its own.
*
* There is no external-media bucket: that subtree is not walked at all (see
* nestedExternalRoot). Those bytes live on the media share, and counting them
* is the exact over-count this measurement exists to end.
*/
function categorize(relPath) {
const segments = relPath.split(path.sep);
if (segments.includes('.download-cache')) return 'downloadCache';
switch (segments[0]) {
case 'thumbnails': return 'thumbnails';
case 'previews': return 'previews';
case 'heroes': return 'heroes';
case 'watermarks': return 'watermarks';
case 'uploads': return 'uploads';
case 'temp': return 'temp';
case 'business-docs': return 'businessDocs';
case 'events':
return segments[1] === 'archived' ? 'archives' : 'originals';
default:
return 'other';
}
}
const EMPTY_BREAKDOWN = () => ({
originals: 0,
archives: 0,
thumbnails: 0,
previews: 0,
heroes: 0,
watermarks: 0,
uploads: 0,
businessDocs: 0,
downloadCache: 0,
temp: 0,
other: 0,
});
async function walk(absDir, relDir, acc) {
// The media share, bind-mounted under the storage root. Walking it would put
// every referenced original back into a local-usage figure, and on a real
// NAS the traversal alone would take far longer than the measurement is
// worth.
if (acc.excludeRoot && path.resolve(absDir) === acc.excludeRoot) {
acc.excludedExternalRoot = acc.excludeRoot;
return;
}
let entries;
try {
entries = await fsp.readdir(absDir, { withFileTypes: true });
} catch (err) {
// A directory that is not there yet (a fresh install has no /previews) is
// not an error. Anything else is worth knowing about but must not abort
// the measurement — a partial number beats no number, and `partial` says
// so to the caller.
if (err.code !== 'ENOENT') {
acc.partial = true;
logger.debug?.(`localStorageUsage: skipped ${absDir}: ${err.message}`);
}
return;
}
for (const entry of entries) {
const abs = path.join(absDir, entry.name);
const rel = relDir ? path.join(relDir, entry.name) : entry.name;
// Symlinks are not followed: a link into the external media mount would
// otherwise add the NAS to the local total, which is the exact confusion
// this replaces.
if (entry.isDirectory()) {
await walk(abs, rel, acc);
} else if (entry.isFile()) {
try {
const stats = await fsp.stat(abs);
acc.total += stats.size;
acc.files += 1;
acc.breakdown[categorize(rel)] += stats.size;
} catch (err) {
// Raced with a delete, most likely. Nothing to add.
if (err.code !== 'ENOENT') acc.partial = true;
}
}
}
}
/**
* @param {{ force?: boolean }} [opts] force skips the TTL cache.
* @returns {Promise<{total:number, files:number, breakdown:object, partial:boolean, measuredAt:string, root:string}>}
*/
async function measureLocalStorageUsage(opts = {}) {
const now = Date.now();
if (!opts.force && cache && now - cache.at < TTL_MS) return cache.value;
// Share a walk already underway rather than starting a second one.
if (!opts.force && inFlight) return inFlight;
inFlight = runMeasurement()
.then((value) => { cache = { at: Date.now(), value }; return value; })
.finally(() => { inFlight = null; });
return inFlight;
}
async function runMeasurement() {
const root = getStoragePath();
const acc = {
total: 0,
files: 0,
breakdown: EMPTY_BREAKDOWN(),
partial: false,
excludeRoot: nestedExternalRoot(root),
excludedExternalRoot: null,
};
await walk(root, '', acc);
return {
total: acc.total,
files: acc.files,
breakdown: acc.breakdown,
partial: acc.partial,
// Set when the media share sits inside the storage root and was skipped,
// so the UI can say why the figure is smaller than `du` would report.
excludedExternalRoot: acc.excludedExternalRoot,
measuredAt: new Date().toISOString(),
root,
};
}
/** Test seam — the TTL cache would otherwise outlive a temp storage root. */
function resetLocalStorageUsageCache() {
cache = null;
inFlight = null;
}
module.exports = {
measureLocalStorageUsage,
resetLocalStorageUsageCache,
};