* fix(admin): make "Storage used" report storage used (#1164) Stable twin of #1170. The tile summed photos.size_bytes — the catalogued size of the ORIGINALS, which in reference mode live on external storage and have no relationship to the disk PicPeak runs on. The reporter's tile read ~80 GB against 21 GB of real usage. Worse than the label: the same number drove the soft-limit warning bar and, via /storage/info, the recommended soft limit — so a reference-mode install got a disk-capacity recommendation computed from bytes that are not on the disk. - new localStorageUsage service walks the storage root and reports the total plus a breakdown. Walking rather than summing DB columns is the point: thumbnail/preview/hero rows record a key and never a byte count, and orphans from a deleted event or an interrupted import are real bytes. - the external media root is excluded when it sits inside the storage root. Its compose default is <storage>/external-media, where the NAS is bind-mounted — a plain directory, not a symlink — so walking it would put every referenced original back into a figure whose purpose is to leave them out. Symlinks are not followed either. - .download-cache gets its own line: it lives inside the event directory, so the naive rule files a multi-GB zip as photography. - concurrent cold-cache callers share one walk; the dashboard, /storage/info and the sidebar are routinely requested together. - S3 installs keep the catalogued figure and the walk is skipped before it runs, since the objects are in the bucket and STORAGE_PATH holds only incidental local files. - an absent measurement reads as "unavailable" and a partial one is marked `+` across the dashboard, analytics, sidebar and status tab — a floor silently compared against a soft limit reads as "safely under". Verified on this branch: 11 new service tests, dashboardScope updated for the changed contract, full suite leaves the same 5 pre-existing failures as origin/stable. Frontend 20 files / 104 tests, tsc clean. * fix(admin): tell "no disk to measure" apart from "the measurement failed" (#1164) External review found both of these on this branch. Both were reported as `storage_measurement: 'catalog'`, so a failed local walk made the dashboard claim the objects live in S3. They are different things — one is a fact about the install, the other is a fault — and there is now an `unavailable` state for the second. The analytics percentage could reach the billions. `safeSoftLimit` fell back to `storageUsed || 1`, and on S3 that is null → 1, while the figure beside it came from `catalogedBytes`. An editor or viewer holds `analytics.view` but not `settings.view`, so `/storage/info` 403s for them and `storageInfo` is undefined — which is exactly when that fallback fires. It now falls back to the measured figure, and suppresses the percentage entirely when there is no real limit rather than dividing usage by itself and always reading 100%. --------- Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
co-authored by
Paul Nothaft
Paul Nothaft
parent
9ffbe2f98f
commit
ac7ef266dc
@@ -7,6 +7,7 @@ const { formatBoolean } = require('../utils/dbCompat');
|
||||
const { resolveAdapter } = require('../services/trackers');
|
||||
const logger = require('../utils/logger');
|
||||
const { errorResponse, getPagination } = require('../utils/routeHelpers');
|
||||
const { measureLocalStorageUsage } = require('../services/localStorageUsage');
|
||||
const router = express.Router();
|
||||
|
||||
/**
|
||||
@@ -88,11 +89,33 @@ router.get('/stats', adminAuth, requirePermission('analytics.view'), async (req,
|
||||
.count('id as count')
|
||||
.first();
|
||||
|
||||
// Get storage usage (sum of all photo sizes)
|
||||
const storageUsed = await applyEventScope(db('photos'), req.admin, 'event_id')
|
||||
// The catalogued size of the ORIGINALS. Kept, and still worth showing —
|
||||
// it answers "how much photography is in here" — but it is emphatically
|
||||
// NOT storage used, which is what it was labelled for years (#1164).
|
||||
const catalogedBytes = await applyEventScope(db('photos'), req.admin, 'event_id')
|
||||
.sum('size_bytes as total')
|
||||
.first();
|
||||
|
||||
// Storage used: what is actually on this machine. In reference mode the
|
||||
// originals above live on a NAS and contribute nothing here; conversely
|
||||
// this counts what the sum never did — thumbnails, previews, hero
|
||||
// renditions, watermarks and the per-event download cache.
|
||||
//
|
||||
// Deliberately NOT event-scoped, unlike everything else on this endpoint:
|
||||
// it is a disk measurement, and disk is not divisible by which admin owns
|
||||
// which event.
|
||||
//
|
||||
// Skipped entirely on an S3 backend: the objects are in the bucket and a
|
||||
// walk of STORAGE_PATH would report near-zero, which is worse than the
|
||||
// catalogued figure those installs had before #1164.
|
||||
const usesLocalBackend = (process.env.STORAGE_BACKEND || 'local').toLowerCase() !== 's3';
|
||||
let localStorage = null;
|
||||
try {
|
||||
if (usesLocalBackend) localStorage = await measureLocalStorageUsage();
|
||||
} catch (err) {
|
||||
logger.warn(`Dashboard storage measurement failed: ${err.message}`);
|
||||
}
|
||||
|
||||
// Get total views (last 30 days)
|
||||
const thirtyDaysAgo = new Date();
|
||||
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
|
||||
@@ -154,7 +177,19 @@ router.get('/stats', adminAuth, requirePermission('analytics.view'), async (req,
|
||||
activeEvents: activeEvents.count || 0,
|
||||
expiringEvents: expiringEvents.count || 0,
|
||||
totalPhotos: totalPhotos.count || 0,
|
||||
storageUsed: storageUsed.total || 0,
|
||||
// Real bytes on this disk. Null when the measurement failed or was
|
||||
// skipped, which the UI shows as "unavailable" rather than substituting
|
||||
// a number that means something else.
|
||||
storageUsed: localStorage ? localStorage.total : null,
|
||||
// Three states, not two: 'catalog' means the backend is S3 and the
|
||||
// objects are in the bucket, which is a fact about the install;
|
||||
// 'unavailable' means the walk failed, which is a fault. Collapsing them
|
||||
// made a failed local measurement claim the objects live in S3.
|
||||
storageMeasurement: localStorage ? 'disk' : (usesLocalBackend ? 'unavailable' : 'catalog'),
|
||||
storageBreakdown: localStorage ? localStorage.breakdown : null,
|
||||
storagePartial: localStorage ? localStorage.partial : false,
|
||||
// Catalogued original bytes — what `storageUsed` used to report (#1164).
|
||||
catalogedBytes: Number(catalogedBytes.total) || 0,
|
||||
totalViews: totalViews.count || 0,
|
||||
totalDownloads: totalDownloads.count || 0,
|
||||
viewsTrend: Math.round(viewsTrend * 10) / 10,
|
||||
|
||||
@@ -24,6 +24,7 @@ const { clearShareLinkSettingsCache } = require('../services/shareLinkService');
|
||||
const { resetSecurityConfigCache } = require('../utils/authSecurity');
|
||||
const { errorResponse } = require('../utils/routeHelpers');
|
||||
const logger = require('../utils/logger');
|
||||
const { measureLocalStorageUsage } = require('../services/localStorageUsage');
|
||||
const router = express.Router();
|
||||
const { clearMaxFilesPerUploadCache, MAX_ALLOWED_FILES_PER_UPLOAD } = require('../services/uploadSettings');
|
||||
const watermarkService = require('../services/watermarkService');
|
||||
@@ -1174,6 +1175,8 @@ router.put('/seo', adminAuth, requirePermission('settings.edit'), async (req, re
|
||||
router.get('/storage/info', adminAuth, requirePermission('settings.view'), async (req, res) => {
|
||||
try {
|
||||
// Get total storage used
|
||||
// Catalogued original bytes. Reported, but no longer as "used" (#1164) —
|
||||
// in reference mode those files are on a NAS and none of them are here.
|
||||
const totalStorage = await db('photos')
|
||||
.sum('size_bytes as total')
|
||||
.first();
|
||||
@@ -1254,7 +1257,25 @@ router.get('/storage/info', adminAuth, requirePermission('settings.view'), async
|
||||
}
|
||||
}
|
||||
|
||||
const totalUsed = totalStorage?.total || 0;
|
||||
// What is actually on this disk. This is what the soft limit is compared
|
||||
// against and what the recommendation below is derived from, so getting it
|
||||
// from the catalogued originals was the load-bearing half of #1164: a
|
||||
// reference-mode install got a disk-capacity recommendation computed from
|
||||
// bytes that are not on the disk.
|
||||
//
|
||||
// Gated BEFORE the walk: this endpoint is polled by the sidebar, and an S3
|
||||
// install with a large local tree would otherwise pay a full traversal on
|
||||
// every cold cache only to discard the result.
|
||||
const catalogedBytes = Number(totalStorage?.total) || 0;
|
||||
const usesLocalBackend = (process.env.STORAGE_BACKEND || 'local').toLowerCase() !== 's3';
|
||||
let localUsage = null;
|
||||
try {
|
||||
if (usesLocalBackend) localUsage = await measureLocalStorageUsage();
|
||||
} catch (err) {
|
||||
logger.warn(`Storage measurement failed, falling back to catalogued bytes: ${err.message}`);
|
||||
}
|
||||
const measuredFromDisk = usesLocalBackend && !!localUsage;
|
||||
const totalUsed = measuredFromDisk ? localUsage.total : catalogedBytes;
|
||||
|
||||
const parseBytesValue = (value) => {
|
||||
const numeric = Number(value);
|
||||
@@ -1376,6 +1397,13 @@ router.get('/storage/info', adminAuth, requirePermission('settings.view'), async
|
||||
|
||||
res.json({
|
||||
total_used: totalUsed,
|
||||
// What total_used used to be, kept so the UI can show both and the
|
||||
// difference stops being invisible.
|
||||
cataloged_bytes: catalogedBytes,
|
||||
storage_measurement: measuredFromDisk ? 'disk' : (usesLocalBackend ? 'unavailable' : 'catalog'),
|
||||
storage_breakdown: measuredFromDisk ? localUsage.breakdown : null,
|
||||
storage_partial: measuredFromDisk ? localUsage.partial : false,
|
||||
excluded_external_root: measuredFromDisk ? localUsage.excludedExternalRoot : null,
|
||||
archive_storage: archiveStorage,
|
||||
storage_by_event: storageByEvent,
|
||||
storage_limit: effectiveSoftLimit,
|
||||
|
||||
Reference in New Issue
Block a user