* fix(admin): make "Storage used" report storage used (#1164) The tile summed photos.size_bytes — the catalogued size of the ORIGINALS, which has no relationship to the disk PicPeak runs on. In reference mode those files are never copied and sit on the NAS; duplicate rows counted the same file twice (#1162); and it ignored everything PicPeak genuinely does write locally: thumbnails, previews, hero renditions, watermarks and the per-event download cache. The reporter's tile read ~80 GB against 21 GB of real usage. Worse than the label: the same number drove the storage soft-limit warning bar and, via /storage/info, the recommended soft limit — so a reference-mode install got a disk-capacity recommendation computed from bytes that are not on the disk. - new localStorageUsage service walks the storage root and reports the total plus a breakdown. Walking rather than summing DB columns is the point: thumbnail/preview/hero rows record a key and never a byte count, and orphans from a deleted event or an interrupted import are real bytes. Symlinks are not followed, so a link into the media mount cannot put the NAS back in the total. Cached for 5 minutes, since the dashboard polls. - the dashboard tile and /storage/info now report that, with the catalogued figure kept and labelled as such next to it. A failed measurement reads as "unavailable" rather than substituting a number that means something else. On the local rig: 64.37 MB used against 15.75 MB catalogued, of which 27.9 MB is watermarks and 6.8 MB is download cache — none of which the old figure could see. Not addressed here: `.download-cache/all.zip` still has no TTL or size cap. It is now at least visible in the breakdown, which is what makes the case for capping it. * fix(admin): exclude the media share from local storage usage (#1164) External review found the walk could reintroduce the exact over-count it replaces. EXTERNAL_MEDIA_ROOT's compose default is `<storage>/external-media`, where the NAS is bind-mounted. That is a plain directory, not a symlink, so the symlink guard did not cover it and the walk descended into the share — putting every referenced original back into a figure whose whole purpose is to leave them out, and comparing NAS bytes against statfs() of the local disk. On the reference-mode installs this issue is about, that is the failure mode reappearing inside its own fix. The configured root is now skipped when it lies inside the storage root, and the result reports which path was excluded. A directory that merely shares the name is still counted, because those really are local bytes. Also from the review: - concurrent cold-cache callers now share one walk. /dashboard/stats, /storage/info and the sidebar are routinely requested together, and each was starting its own stat-per-file traversal of the whole library. - storage_partial is surfaced in the StorageInfo type and the sidebar tile, not just the dashboard and analytics cards. An unreadable subtree makes the total a floor, and a floor silently compared against a soft limit reads as "safely under". * fix(admin): do not report a disk walk on an S3 backend (#1164) Second review round. S3 installs were regressed. With STORAGE_BACKEND=s3 the originals, renditions, archives and download caches are objects in the bucket and STORAGE_PATH holds only incidental local files — so the walk reported near-zero and the soft-limit recommendation was derived from it. Those installs now keep the catalogued figure, which is the approximation they had before this PR, and the response says which measurement it is (`storage_measurement: 'disk' | 'catalog'`) so the UI labels it instead of implying a disk measurement that never happened. The Settings → Status storage card ignored storage_partial, formatting a lower bound as exact and deriving the limit percentage from it — so an unreadable subtree could read as safely under the limit. It now carries the same `+` marker as the sidebar and dashboard. * fix(admin): stop rendering an absent measurement as zero usage (#1164) Third review round, two findings. The analytics storage bar coerced a null measurement to 0, drawing an empty bar labelled "0% of limit" and suppressing the over-limit state — reading as plenty of room at exactly the moment nothing is known. It now shows the catalogued figure on S3, where that IS the available answer, and says "no measurement available" rather than inventing a percentage when there is none. /storage/info walked the filesystem before checking the backend and then threw the result away on S3. The sidebar polls that endpoint, so a migrated install still holding a large local tree paid a full stat-per-file traversal on every cold cache for nothing. Gated before the walk, as the dashboard route already was. * fix(admin): tell "no disk to measure" apart from "the measurement failed" (#1164) External review of the stable twin. Both were reported as `storage_measurement: 'catalog'`, so a failed local walk made the dashboard claim the objects live in S3. They are different things — one is a fact about the install, the other is a fault — and there is now an `unavailable` state for the second. The analytics percentage could reach the billions. `safeSoftLimit` fell back to `storageUsed || 1`, and on S3 that is null → 1, while the figure beside it came from `catalogedBytes`. An editor or viewer holds `analytics.view` but not `settings.view`, so `/storage/info` 403s for them and `storageInfo` is undefined — which is exactly when that fallback fires. It now falls back to the measured figure, and suppresses the percentage entirely when there is no real limit rather than dividing usage by itself and always reading 100%. Also lands the AnalyticsPage half of the previous round, which the commit message claimed but the commit did not contain — only its backend counterpart was staged. The stable twin has carried it since it was written, so this is the parity gap in the unusual direction. --------- Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
co-authored by
Paul Nothaft
Paul Nothaft
parent
1366d6d14c
commit
849a5807b7
@@ -127,7 +127,44 @@ describe('dashboard scoping (GHSA-c2jj / gqx7 / jhcf)', () => {
|
||||
expect(res.status).toBe(200);
|
||||
expect(Number(res.body.totalEvents)).toBe(1);
|
||||
expect(Number(res.body.totalPhotos)).toBe(1);
|
||||
expect(Number(res.body.storageUsed)).toBe(1000);
|
||||
// The catalogued original bytes — this is what carries the per-event
|
||||
// scoping, and what `storageUsed` reported before #1164.
|
||||
expect(Number(res.body.catalogedBytes)).toBe(1000);
|
||||
});
|
||||
|
||||
it('/stats reports disk usage unscoped, because disk is not per-event', async () => {
|
||||
// storageUsed is a measurement of the storage root (#1164), so it is the
|
||||
// same number for every admin by design. Pinned so a future reviewer
|
||||
// reading "everything on this endpoint is scoped" does not turn it into a
|
||||
// sum of this editor's photos again — which is the bug that was fixed.
|
||||
const res = await request(app)
|
||||
.get('/api/admin/dashboard/stats')
|
||||
.set('Authorization', `Bearer ${editorToken}`);
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.storageUsed).not.toBe(1000);
|
||||
expect(res.body).toHaveProperty('storageBreakdown');
|
||||
});
|
||||
|
||||
it('/stats reports the catalogued figure on an S3 backend, not a near-zero disk walk', async () => {
|
||||
// STORAGE_PATH holds only incidental local files when objects live in a
|
||||
// bucket, so walking it would report near-zero and drag the soft-limit
|
||||
// recommendation with it (#1164 review).
|
||||
const prev = process.env.STORAGE_BACKEND;
|
||||
process.env.STORAGE_BACKEND = 's3';
|
||||
try {
|
||||
const res = await request(app)
|
||||
.get('/api/admin/dashboard/stats')
|
||||
.set('Authorization', `Bearer ${editorToken}`);
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.storageUsed).toBeNull();
|
||||
expect(res.body.storageMeasurement).toBe('catalog');
|
||||
expect(Number(res.body.catalogedBytes)).toBe(1000);
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.STORAGE_BACKEND;
|
||||
else process.env.STORAGE_BACKEND = prev;
|
||||
}
|
||||
});
|
||||
|
||||
it('/analytics does not expose a foreign gallery name or slug', async () => {
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
/**
|
||||
* "Storage used" has to mean storage used (#1164).
|
||||
*
|
||||
* The tile summed photos.size_bytes, so on a reference-mode install it
|
||||
* reported the size of files sitting on a NAS — the reporter's read ~80 GB
|
||||
* against 21 GB of real local usage — while omitting everything PicPeak does
|
||||
* write locally, including an 11.8 GB download-cache zip.
|
||||
*
|
||||
* These pin the measurement against a real directory tree, since the whole
|
||||
* point is counting bytes that are actually there.
|
||||
*/
|
||||
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
|
||||
const {
|
||||
measureLocalStorageUsage,
|
||||
resetLocalStorageUsageCache,
|
||||
} = require('../../src/services/localStorageUsage');
|
||||
|
||||
describe('localStorageUsage (#1164)', () => {
|
||||
let root;
|
||||
|
||||
const write = async (rel, bytes) => {
|
||||
const full = path.join(root, rel);
|
||||
await fs.promises.mkdir(path.dirname(full), { recursive: true });
|
||||
await fs.promises.writeFile(full, Buffer.alloc(bytes));
|
||||
};
|
||||
|
||||
beforeEach(async () => {
|
||||
root = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'picpeak-usage-'));
|
||||
process.env.STORAGE_PATH = root;
|
||||
delete process.env.EXTERNAL_MEDIA_ROOT;
|
||||
jest.resetModules();
|
||||
resetLocalStorageUsageCache();
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await fs.promises.rm(root, { recursive: true, force: true }).catch(() => {});
|
||||
delete process.env.STORAGE_PATH;
|
||||
delete process.env.EXTERNAL_MEDIA_ROOT;
|
||||
resetLocalStorageUsageCache();
|
||||
});
|
||||
|
||||
it('counts every byte under the storage root', async () => {
|
||||
await write(path.join('events', 'active', 'wed', 'individual', 'a.jpg'), 1000);
|
||||
await write(path.join('thumbnails', 'a.jpg'), 100);
|
||||
await write(path.join('previews', 'a.jpg'), 300);
|
||||
|
||||
const usage = await measureLocalStorageUsage();
|
||||
|
||||
expect(usage.total).toBe(1400);
|
||||
expect(usage.files).toBe(3);
|
||||
});
|
||||
|
||||
it('breaks the total down by what the bytes are', async () => {
|
||||
// The specific complaint: the derived artefacts PicPeak writes were
|
||||
// invisible, so "what is filling my disk" had no answer in the UI.
|
||||
await write(path.join('events', 'active', 'wed', 'individual', 'a.jpg'), 1000);
|
||||
await write(path.join('events', 'archived', 'old.zip'), 5000);
|
||||
await write(path.join('thumbnails', 'a.jpg'), 100);
|
||||
await write(path.join('previews', 'a.jpg'), 300);
|
||||
await write(path.join('heroes', 'a.jpg'), 200);
|
||||
await write(path.join('watermarks', 'a.jpg'), 700);
|
||||
await write(path.join('uploads', 'logo.png'), 50);
|
||||
|
||||
const { breakdown } = await measureLocalStorageUsage();
|
||||
|
||||
expect(breakdown).toMatchObject({
|
||||
originals: 1000,
|
||||
archives: 5000,
|
||||
thumbnails: 100,
|
||||
previews: 300,
|
||||
heroes: 200,
|
||||
watermarks: 700,
|
||||
uploads: 50,
|
||||
});
|
||||
});
|
||||
|
||||
it('files the download cache separately from the originals it sits among', async () => {
|
||||
// `.download-cache` lives INSIDE the event directory, so the naive rule
|
||||
// files an 11.8 GB zip as photography. It is the one bucket that is pure
|
||||
// disposable cache and the one an admin most needs to see.
|
||||
await write(path.join('events', 'active', 'wed', 'individual', 'a.jpg'), 1000);
|
||||
await write(path.join('events', 'active', 'wed', '.download-cache', 'all.zip'), 9000);
|
||||
|
||||
const { breakdown, total } = await measureLocalStorageUsage();
|
||||
|
||||
expect(breakdown.downloadCache).toBe(9000);
|
||||
expect(breakdown.originals).toBe(1000);
|
||||
expect(total).toBe(10000);
|
||||
});
|
||||
|
||||
it('counts orphans no database row knows about', async () => {
|
||||
// A deleted event's leftovers and an interrupted import's thumbnails are
|
||||
// real bytes on a real disk. Summing DB columns would miss them, which is
|
||||
// half of why this walks instead.
|
||||
await write(path.join('thumbnails', 'ext999_gone.jpg'), 777);
|
||||
|
||||
expect((await measureLocalStorageUsage()).total).toBe(777);
|
||||
});
|
||||
|
||||
it('reports zero on a fresh install rather than failing', async () => {
|
||||
const usage = await measureLocalStorageUsage();
|
||||
|
||||
expect(usage.total).toBe(0);
|
||||
expect(usage.partial).toBe(false);
|
||||
});
|
||||
|
||||
it('survives a storage root that does not exist', async () => {
|
||||
process.env.STORAGE_PATH = path.join(root, 'nope');
|
||||
resetLocalStorageUsageCache();
|
||||
|
||||
const usage = await measureLocalStorageUsage();
|
||||
|
||||
// ENOENT on the root is a fresh/misconfigured install, not a partial read.
|
||||
expect(usage.total).toBe(0);
|
||||
expect(usage.partial).toBe(false);
|
||||
});
|
||||
|
||||
it('does not walk the media share bind-mounted under the storage root', async () => {
|
||||
// The compose default puts EXTERNAL_MEDIA_ROOT at <storage>/external-media,
|
||||
// where the NAS is bind-mounted — a plain directory, not a symlink. Walking
|
||||
// it would put every referenced original back into a figure that exists to
|
||||
// leave them out, which is the over-count this measurement replaces.
|
||||
await write(path.join('thumbnails', 'a.jpg'), 100);
|
||||
await write(path.join('external-media', 'nas', 'huge.jpg'), 50000);
|
||||
process.env.EXTERNAL_MEDIA_ROOT = path.join(root, 'external-media');
|
||||
jest.resetModules();
|
||||
const svc = require('../../src/services/localStorageUsage');
|
||||
svc.resetLocalStorageUsageCache();
|
||||
|
||||
const usage = await svc.measureLocalStorageUsage();
|
||||
|
||||
expect(usage.total).toBe(100);
|
||||
expect(usage.excludedExternalRoot).toBe(path.join(root, 'external-media'));
|
||||
});
|
||||
|
||||
it('still counts a directory that merely looks like the media share', async () => {
|
||||
// Only the CONFIGURED root is skipped. An install whose media lives
|
||||
// elsewhere keeps whatever is in this directory in the total, because
|
||||
// those really are local bytes.
|
||||
await write(path.join('external-media', 'leftover.jpg'), 700);
|
||||
// The production shape: the share is mounted well outside the storage root.
|
||||
const elsewhere = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'picpeak-nas-elsewhere-'));
|
||||
process.env.EXTERNAL_MEDIA_ROOT = elsewhere;
|
||||
jest.resetModules();
|
||||
const svc = require('../../src/services/localStorageUsage');
|
||||
svc.resetLocalStorageUsageCache();
|
||||
|
||||
const usage = await svc.measureLocalStorageUsage();
|
||||
|
||||
expect(usage.total).toBe(700);
|
||||
expect(usage.excludedExternalRoot).toBeNull();
|
||||
await fs.promises.rm(elsewhere, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('shares one walk between concurrent cold-cache callers', async () => {
|
||||
// /dashboard/stats and /storage/info are routinely requested together, and
|
||||
// the sidebar adds a third. Each starting its own full stat-per-file walk
|
||||
// multiplies the cost on exactly the large libraries where it hurts.
|
||||
await write(path.join('thumbnails', 'a.jpg'), 100);
|
||||
const readdir = jest.spyOn(fs.promises, 'readdir');
|
||||
|
||||
const [a, b, c] = await Promise.all([
|
||||
measureLocalStorageUsage(),
|
||||
measureLocalStorageUsage(),
|
||||
measureLocalStorageUsage(),
|
||||
]);
|
||||
|
||||
expect([a.total, b.total, c.total]).toEqual([100, 100, 100]);
|
||||
// One walk: the storage root plus its one subdirectory.
|
||||
expect(readdir).toHaveBeenCalledTimes(2);
|
||||
readdir.mockRestore();
|
||||
});
|
||||
|
||||
it('caches, and honours force', async () => {
|
||||
await write(path.join('thumbnails', 'a.jpg'), 100);
|
||||
expect((await measureLocalStorageUsage()).total).toBe(100);
|
||||
|
||||
await write(path.join('thumbnails', 'b.jpg'), 400);
|
||||
// One stat per file is not free; the dashboard polls.
|
||||
expect((await measureLocalStorageUsage()).total).toBe(100);
|
||||
expect((await measureLocalStorageUsage({ force: true })).total).toBe(500);
|
||||
});
|
||||
|
||||
it('does not follow a symlink out of the storage root', async () => {
|
||||
// A link into EXTERNAL_MEDIA_ROOT would add the NAS back into the local
|
||||
// total — reinstating the exact confusion this replaces.
|
||||
const outside = await fs.promises.mkdtemp(path.join(os.tmpdir(), 'picpeak-nas-'));
|
||||
await fs.promises.writeFile(path.join(outside, 'huge.jpg'), Buffer.alloc(50000));
|
||||
await fs.promises.mkdir(path.join(root, 'events'), { recursive: true });
|
||||
await fs.promises.symlink(outside, path.join(root, 'events', 'nas'), 'dir');
|
||||
|
||||
const usage = await measureLocalStorageUsage();
|
||||
|
||||
expect(usage.total).toBe(0);
|
||||
await fs.promises.rm(outside, { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
@@ -7,6 +7,7 @@ const { formatBoolean } = require('../utils/dbCompat');
|
||||
const { resolveAdapter } = require('../services/trackers');
|
||||
const logger = require('../utils/logger');
|
||||
const { errorResponse, getPagination } = require('../utils/routeHelpers');
|
||||
const { measureLocalStorageUsage } = require('../services/localStorageUsage');
|
||||
const router = express.Router();
|
||||
|
||||
/**
|
||||
@@ -88,11 +89,38 @@ router.get('/stats', adminAuth, requirePermission('analytics.view'), async (req,
|
||||
.count('id as count')
|
||||
.first();
|
||||
|
||||
// Get storage usage (sum of all photo sizes)
|
||||
const storageUsed = await applyEventScope(db('photos'), req.admin, 'event_id')
|
||||
// The catalogued size of the ORIGINALS. Kept, and still worth showing —
|
||||
// it answers "how much photography is in here" — but it is emphatically
|
||||
// NOT storage used, which is what it was labelled for years (#1164).
|
||||
const catalogedBytes = await applyEventScope(db('photos'), req.admin, 'event_id')
|
||||
.sum('size_bytes as total')
|
||||
.first();
|
||||
|
||||
// Storage used: what is actually on this machine. In reference mode the
|
||||
// originals above live on a NAS and contribute nothing here; conversely
|
||||
// this counts what the sum never did — thumbnails, previews, hero
|
||||
// renditions and the per-event download cache.
|
||||
//
|
||||
// Deliberately NOT event-scoped, unlike everything else on this endpoint:
|
||||
// it is a disk measurement, and disk is not divisible by which admin owns
|
||||
// which event. A scoped admin gets the same number a super_admin does,
|
||||
// which is the honest answer to "is this box running out of space" and
|
||||
// leaks nothing beyond the aggregate they can already infer from the
|
||||
// system-health page.
|
||||
// Skipped entirely on an S3 backend: the objects are in the bucket and a
|
||||
// walk of STORAGE_PATH would report near-zero, which is worse than the
|
||||
// catalogued figure those installs had before #1164. `storageUsed` is null
|
||||
// there and the tile shows the catalogued number it already carries.
|
||||
const usesLocalBackend = (process.env.STORAGE_BACKEND || 'local').toLowerCase() !== 's3';
|
||||
let localStorage = null;
|
||||
try {
|
||||
if (usesLocalBackend) localStorage = await measureLocalStorageUsage();
|
||||
} catch (err) {
|
||||
// The dashboard must render without it; the tile falls back to showing
|
||||
// the catalogued figure, labelled as such.
|
||||
logger.warn(`Dashboard storage measurement failed: ${err.message}`);
|
||||
}
|
||||
|
||||
// Get total views (last 30 days)
|
||||
const thirtyDaysAgo = new Date();
|
||||
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
|
||||
@@ -154,7 +182,21 @@ router.get('/stats', adminAuth, requirePermission('analytics.view'), async (req,
|
||||
activeEvents: activeEvents.count || 0,
|
||||
expiringEvents: expiringEvents.count || 0,
|
||||
totalPhotos: totalPhotos.count || 0,
|
||||
storageUsed: storageUsed.total || 0,
|
||||
// Real bytes on this disk. Null when the measurement failed, which the
|
||||
// UI shows as "unavailable" rather than substituting a number that
|
||||
// means something else.
|
||||
storageUsed: localStorage ? localStorage.total : null,
|
||||
// Three states, not two: 'catalog' means the backend is S3 and the
|
||||
// objects are in the bucket, which is a fact about the install;
|
||||
// 'unavailable' means the walk failed, which is a fault. Collapsing them
|
||||
// made a failed local measurement claim the objects live in S3.
|
||||
storageMeasurement: localStorage ? 'disk' : (usesLocalBackend ? 'unavailable' : 'catalog'),
|
||||
storageBreakdown: localStorage ? localStorage.breakdown : null,
|
||||
// True when part of the storage root could not be read, so the total is
|
||||
// a floor rather than the answer.
|
||||
storagePartial: localStorage ? localStorage.partial : false,
|
||||
// Catalogued original bytes — what `storageUsed` used to report (#1164).
|
||||
catalogedBytes: Number(catalogedBytes.total) || 0,
|
||||
totalViews: totalViews.count || 0,
|
||||
totalDownloads: totalDownloads.count || 0,
|
||||
viewsTrend: Math.round(viewsTrend * 10) / 10,
|
||||
|
||||
@@ -25,6 +25,7 @@ const { clearShareLinkSettingsCache } = require('../services/shareLinkService');
|
||||
const { invalidateSiteUrlCache, isEnvPinned, envPinnedBase } = require('../utils/frontendUrl');
|
||||
const { resetSecurityConfigCache } = require('../utils/authSecurity');
|
||||
const { errorResponse } = require('../utils/routeHelpers');
|
||||
const { measureLocalStorageUsage } = require('../services/localStorageUsage');
|
||||
const logger = require('../utils/logger');
|
||||
const router = express.Router();
|
||||
const { clearMaxFilesPerUploadCache, MAX_ALLOWED_FILES_PER_UPLOAD, clearMaxFileSizeCache, MAX_ALLOWED_FILE_SIZE_MB } = require('../services/uploadSettings');
|
||||
@@ -1717,7 +1718,8 @@ router.put('/seo', adminAuth, requirePermission('settings.edit'), async (req, re
|
||||
// Get storage info
|
||||
router.get('/storage/info', adminAuth, requirePermission('settings.view'), async (req, res) => {
|
||||
try {
|
||||
// Get total storage used
|
||||
// Catalogued original bytes. Reported, but no longer as "used" (#1164) —
|
||||
// in reference mode those files are on a NAS and none of them are here.
|
||||
const totalStorage = await db('photos')
|
||||
.sum('size_bytes as total')
|
||||
.first();
|
||||
@@ -1798,7 +1800,32 @@ router.get('/storage/info', adminAuth, requirePermission('settings.view'), async
|
||||
}
|
||||
}
|
||||
|
||||
const totalUsed = totalStorage?.total || 0;
|
||||
// What is actually on this disk. This is what the soft limit is compared
|
||||
// against and what the recommendation below is derived from, so getting it
|
||||
// from the catalogued originals was the load-bearing half of #1164: a
|
||||
// reference-mode install got a disk-capacity recommendation computed from
|
||||
// bytes that are not on the disk.
|
||||
const catalogedBytes = Number(totalStorage?.total) || 0;
|
||||
// Gated BEFORE the walk, not after. This endpoint is polled by the sidebar,
|
||||
// and an S3 install that still has a large local tree from before the
|
||||
// migration would otherwise pay a full stat-per-file traversal on every
|
||||
// cold cache only to discard the result.
|
||||
const usesLocalBackend = (process.env.STORAGE_BACKEND || 'local').toLowerCase() !== 's3';
|
||||
let localUsage = null;
|
||||
try {
|
||||
if (usesLocalBackend) localUsage = await measureLocalStorageUsage();
|
||||
} catch (err) {
|
||||
logger.warn(`Storage measurement failed, falling back to catalogued bytes: ${err.message}`);
|
||||
}
|
||||
// On an S3 backend the originals, renditions, archives and download caches
|
||||
// are all objects in the bucket, and STORAGE_PATH holds only incidental
|
||||
// local files — so the walk would report near-zero and drag the soft-limit
|
||||
// recommendation down with it. Those installs keep the catalogued figure,
|
||||
// which is the approximation they had before #1164, and the response says
|
||||
// which one this is so the UI can label it rather than implying a disk
|
||||
// measurement it never made.
|
||||
const measuredFromDisk = usesLocalBackend && !!localUsage;
|
||||
const totalUsed = measuredFromDisk ? localUsage.total : catalogedBytes;
|
||||
|
||||
const parseBytesValue = (value) => {
|
||||
const numeric = Number(value);
|
||||
@@ -1920,6 +1947,13 @@ router.get('/storage/info', adminAuth, requirePermission('settings.view'), async
|
||||
|
||||
res.json({
|
||||
total_used: totalUsed,
|
||||
// What total_used used to be, kept so the UI can show both and the
|
||||
// difference stops being invisible.
|
||||
cataloged_bytes: catalogedBytes,
|
||||
storage_measurement: measuredFromDisk ? 'disk' : (usesLocalBackend ? 'unavailable' : 'catalog'),
|
||||
storage_breakdown: measuredFromDisk ? localUsage.breakdown : null,
|
||||
storage_partial: measuredFromDisk ? localUsage.partial : false,
|
||||
excluded_external_root: measuredFromDisk ? localUsage.excludedExternalRoot : null,
|
||||
archive_storage: archiveStorage,
|
||||
storage_by_event: storageByEvent,
|
||||
storage_limit: effectiveSoftLimit,
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
/**
|
||||
* What PicPeak actually occupies on this machine (#1164).
|
||||
*
|
||||
* The dashboard's "Storage used" tile summed photos.size_bytes, which is the
|
||||
* catalogued size of the ORIGINALS — a number with no relationship to the disk
|
||||
* PicPeak runs on:
|
||||
*
|
||||
* - in reference mode the originals are never copied. Those bytes are on the
|
||||
* NAS. The reporter's tile read ~80 GB against 21 GB of real local usage.
|
||||
* - duplicate rows counted the same file twice (#1162).
|
||||
* - it ignored everything PicPeak genuinely does write: thumbnails, previews,
|
||||
* hero renditions, and the per-event download cache — an 11.8 GB
|
||||
* `.download-cache/all.zip` sat outside the figure entirely.
|
||||
*
|
||||
* So the one number an admin reaches for when asking "am I running out of
|
||||
* disk" pointed away from the answer and omitted exactly the things filling
|
||||
* the disk. This walks the storage root instead and reports what is there.
|
||||
*
|
||||
* Walking rather than summing DB columns is deliberate: thumbnail/preview/hero
|
||||
* rows record a key, never a byte count, and orphans (a deleted event's
|
||||
* leftovers, an interrupted import's thumbnails) are real bytes on a real
|
||||
* disk. A `du` is the only honest answer, and the only one that notices what
|
||||
* PicPeak has forgotten about.
|
||||
*
|
||||
* The external media root is EXCLUDED, and that is the whole point rather than
|
||||
* a detail. Its compose default is `<storage>/external-media`, where the NAS is
|
||||
* bind-mounted — a plain directory, not a symlink — so walking it would add
|
||||
* every referenced original back into a figure that exists to leave them out,
|
||||
* and compare NAS bytes against statfs() of the local disk. That is the
|
||||
* over-count this replaces, reintroduced by the fix for it.
|
||||
*
|
||||
* Cached, because it is one stat per file. On a large install that is seconds,
|
||||
* and the dashboard is polled — and concurrent misses share one walk rather
|
||||
* than each starting their own.
|
||||
*/
|
||||
|
||||
const path = require('path');
|
||||
const fsp = require('fs').promises;
|
||||
const logger = require('../utils/logger');
|
||||
const { getStoragePath } = require('../config/storage');
|
||||
|
||||
const TTL_MS = 5 * 60 * 1000;
|
||||
|
||||
let cache = null;
|
||||
// The walk in flight, if any. Two admins loading the dashboard, or the sidebar
|
||||
// and the storage tab on one page, hit the same cold cache and would otherwise
|
||||
// each stat every file on the disk.
|
||||
let inFlight = null;
|
||||
|
||||
/**
|
||||
* The external media root, resolved, when it lies inside the storage root.
|
||||
* Returns null when it is elsewhere (the usual production case) or cannot be
|
||||
* resolved — nothing to exclude then.
|
||||
*/
|
||||
function nestedExternalRoot(storageRoot) {
|
||||
let externalRoot;
|
||||
try {
|
||||
externalRoot = require('./externalMediaService').getExternalMediaRoot();
|
||||
} catch (err) {
|
||||
return null;
|
||||
}
|
||||
if (!externalRoot) return null;
|
||||
const resolvedExternal = path.resolve(externalRoot);
|
||||
const resolvedStorage = path.resolve(storageRoot);
|
||||
if (resolvedExternal === resolvedStorage) return null;
|
||||
return resolvedExternal.startsWith(resolvedStorage + path.sep) ? resolvedExternal : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Which line of the breakdown a path belongs to.
|
||||
*
|
||||
* The names are the ones the writers actually use — `heroes` from
|
||||
* imageProcessor, `watermarks` from watermarkService, and so on — rather than
|
||||
* the ones the layout docs imply. Getting one wrong is not a crash, it is a
|
||||
* silent 30 MB in "other", which is the least useful place for it to land.
|
||||
*
|
||||
* `.download-cache` is the case that needs the explicit check: it lives INSIDE
|
||||
* an event directory, so the naive rule files an 11.8 GB zip as photography —
|
||||
* and it is the one bucket that is pure disposable cache, which makes it the
|
||||
* one an admin most wants to see on its own.
|
||||
*
|
||||
* There is no external-media bucket: that subtree is not walked at all (see
|
||||
* nestedExternalRoot). Those bytes live on the media share, and counting them
|
||||
* is the exact over-count this measurement exists to end.
|
||||
*/
|
||||
function categorize(relPath) {
|
||||
const segments = relPath.split(path.sep);
|
||||
if (segments.includes('.download-cache')) return 'downloadCache';
|
||||
switch (segments[0]) {
|
||||
case 'thumbnails': return 'thumbnails';
|
||||
case 'previews': return 'previews';
|
||||
case 'heroes': return 'heroes';
|
||||
case 'watermarks': return 'watermarks';
|
||||
case 'uploads': return 'uploads';
|
||||
case 'temp': return 'temp';
|
||||
case 'business-docs': return 'businessDocs';
|
||||
case 'events':
|
||||
return segments[1] === 'archived' ? 'archives' : 'originals';
|
||||
default:
|
||||
return 'other';
|
||||
}
|
||||
}
|
||||
|
||||
const EMPTY_BREAKDOWN = () => ({
|
||||
originals: 0,
|
||||
archives: 0,
|
||||
thumbnails: 0,
|
||||
previews: 0,
|
||||
heroes: 0,
|
||||
watermarks: 0,
|
||||
uploads: 0,
|
||||
businessDocs: 0,
|
||||
downloadCache: 0,
|
||||
temp: 0,
|
||||
other: 0,
|
||||
});
|
||||
|
||||
async function walk(absDir, relDir, acc) {
|
||||
// The media share, bind-mounted under the storage root. Walking it would put
|
||||
// every referenced original back into a local-usage figure, and on a real
|
||||
// NAS the traversal alone would take far longer than the measurement is
|
||||
// worth.
|
||||
if (acc.excludeRoot && path.resolve(absDir) === acc.excludeRoot) {
|
||||
acc.excludedExternalRoot = acc.excludeRoot;
|
||||
return;
|
||||
}
|
||||
|
||||
let entries;
|
||||
try {
|
||||
entries = await fsp.readdir(absDir, { withFileTypes: true });
|
||||
} catch (err) {
|
||||
// A directory that is not there yet (a fresh install has no /previews) is
|
||||
// not an error. Anything else is worth knowing about but must not abort
|
||||
// the measurement — a partial number beats no number, and `partial` says
|
||||
// so to the caller.
|
||||
if (err.code !== 'ENOENT') {
|
||||
acc.partial = true;
|
||||
logger.debug?.(`localStorageUsage: skipped ${absDir}: ${err.message}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
const abs = path.join(absDir, entry.name);
|
||||
const rel = relDir ? path.join(relDir, entry.name) : entry.name;
|
||||
// Symlinks are not followed: a link into the external media mount would
|
||||
// otherwise add the NAS to the local total, which is the exact confusion
|
||||
// this replaces.
|
||||
if (entry.isDirectory()) {
|
||||
await walk(abs, rel, acc);
|
||||
} else if (entry.isFile()) {
|
||||
try {
|
||||
const stats = await fsp.stat(abs);
|
||||
acc.total += stats.size;
|
||||
acc.files += 1;
|
||||
acc.breakdown[categorize(rel)] += stats.size;
|
||||
} catch (err) {
|
||||
// Raced with a delete, most likely. Nothing to add.
|
||||
if (err.code !== 'ENOENT') acc.partial = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{ force?: boolean }} [opts] force skips the TTL cache.
|
||||
* @returns {Promise<{total:number, files:number, breakdown:object, partial:boolean, measuredAt:string, root:string}>}
|
||||
*/
|
||||
async function measureLocalStorageUsage(opts = {}) {
|
||||
const now = Date.now();
|
||||
if (!opts.force && cache && now - cache.at < TTL_MS) return cache.value;
|
||||
// Share a walk already underway rather than starting a second one.
|
||||
if (!opts.force && inFlight) return inFlight;
|
||||
|
||||
inFlight = runMeasurement()
|
||||
.then((value) => { cache = { at: Date.now(), value }; return value; })
|
||||
.finally(() => { inFlight = null; });
|
||||
return inFlight;
|
||||
}
|
||||
|
||||
async function runMeasurement() {
|
||||
|
||||
const root = getStoragePath();
|
||||
const acc = {
|
||||
total: 0,
|
||||
files: 0,
|
||||
breakdown: EMPTY_BREAKDOWN(),
|
||||
partial: false,
|
||||
excludeRoot: nestedExternalRoot(root),
|
||||
excludedExternalRoot: null,
|
||||
};
|
||||
await walk(root, '', acc);
|
||||
|
||||
return {
|
||||
total: acc.total,
|
||||
files: acc.files,
|
||||
breakdown: acc.breakdown,
|
||||
partial: acc.partial,
|
||||
// Set when the media share sits inside the storage root and was skipped,
|
||||
// so the UI can say why the figure is smaller than `du` would report.
|
||||
excludedExternalRoot: acc.excludedExternalRoot,
|
||||
measuredAt: new Date().toISOString(),
|
||||
root,
|
||||
};
|
||||
}
|
||||
|
||||
/** Test seam — the TTL cache would otherwise outlive a temp storage root. */
|
||||
function resetLocalStorageUsageCache() {
|
||||
cache = null;
|
||||
inFlight = null;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
measureLocalStorageUsage,
|
||||
resetLocalStorageUsageCache,
|
||||
};
|
||||
Reference in New Issue
Block a user