feat(external-media): watch reference folders and import new files automatically (#1345)
* feat(external-media): watch reference folders and import new files automatically Managed uploads dropped into storage/events/active are picked up by the chokidar watcher; external media had no equivalent, so a NAS folder that keeps growing needed an admin to open the event and press Import every time. Relates to issue 1187. - The import pass moves out of the route into services/externalImportService.js. The watcher and the Import button now run the identical function; the route only validates and maps errors to status codes. - Mutual exclusion is the per-event claim from maintenanceJobState (`external_import:<id>`, seeded on demand by the new ensure()) instead of the in-process Set. The Set stopped a double-click in one process; the claim also stops the watcher on a second replica, or an admin clicking while the watcher is mid-run elsewhere. The run heartbeats so a claim from a dead process is taken over. - services/externalMediaWatcher.js: per-event opt-in via the new events.external_watch column (migration 208), chokidar with awaitWriteFinish so a copy in flight is not imported half-written, debounced full pass per change, a timer sweep every 15 minutes as the fallback for NFS/SMB mounts that deliver no inotify events, optional stat-polling via EXTERNAL_MEDIA_WATCH_POLLING. The set of watched events is re-read every minute, so the toggle works from any replica. A watcher that just started runs one pass immediately. - Deletions are ignored on purpose: a file vanishing from a NAS is at least as likely to be a reorganisation or a dropped mount as an intentional removal, and acting on it would delete a guest-visible photo. Rows whose file is gone stay, as they do today. - Not gated on STORAGE_BACKEND: EXTERNAL_MEDIA_ROOT is always local. - Quiet system passes stay out of the activity log; runs that imported something are logged with actor external-media-watcher. - Frontend: "Watch folder for new files" checkbox under the external folder picker, status line in view mode, EN/DE strings. * fix(external-media): close the review gaps in the folder watcher Codex review of the watcher, round 1. All six findings were real: - Enabling the watcher, or pointing an enabled one at another folder, now requires photos.upload — the permission the manual Import already requires. events.edit alone was a way around it. Only the transition is checked, so a role without photos.upload can still edit an already-watched event. The checkbox is disabled for such roles. - Automatic passes defer files that are still changing: anything modified inside the stability window, or whose size moves across one wait of that window, is left for the next pass. chokidar's awaitWriteFinish only settles the file that fired the event, and the sweep sees no events at all, so a sibling still being copied could be inserted half-written and then skipped forever. - Photos an admin deleted are not brought back by the sweep. The delete routes record the file in external_import_exclusions (migration 209); automatic passes skip the list, the manual Import ignores it and clears it for what it imports. - The six EXTERNAL_MEDIA_WATCH* variables are forwarded in all three compose files; they were documented but the backend services use explicit environment lists, so the kill switch did nothing. - A pass re-checks is_active / is_archived at run time, not only in the minutely reconcile. - The lease is renewed on a timer for the whole run, walk included, and ownership is checked before the event row is touched. * fix(external-media): make automatic passes follow the row, not rewrite it Codex review round 2, four findings, all applied: - The event update route drops non-canonical spellings of external_watch and external_path before the permission guard. SQLite resolves column names case-insensitively, so `External_Watch` reached the column while the guard only looked at the lowercase key. - Exclusions are checked per file at insert time, not against a snapshot taken before the settle wait. A photo deleted during the wait was present in the snapshot and got re-inserted by the loop. - An automatic pass no longer writes source_mode / external_path. It re-reads the row after the walk and the settle wait and stops if the folder changed or the event went managed; the manual Import is the only writer. The options are now `automatic` + `settleMs`. - A pass that deferred files re-arms the debounced import, so a file copied just before the watcher started is not stranded when the sweep is disabled. * fix(external-media): keep exclusions for replaced photos, stop a pass whose event stopped qualifying Codex review round 3, both findings applied: - recordExclusions keys on external_relpath alone. A replaced external photo becomes managed but keeps its relpath on purpose, and deleting that replacement must not republish the NAS original. - An automatic pass checks the full watcher predicate (reference mode, same folder, watch on, active, not archived) before it inserts and on every heartbeat tick during the loop, and stops as soon as the event no longer qualifies. --------- Co-authored-by: Paul Nothaft <[email protected]>
This commit is contained in:
Paul Nothaft
co-authored by
Paul Nothaft
Paul Nothaft
parent
1b3f721d10
commit
8cc7d7d14a
@@ -1876,6 +1876,10 @@
|
||||
"sourceModeHelp": "Nutzen Sie den verwalteten Modus für direkte Uploads oder verweisen Sie auf einen gemounteten /external-media Ordner.",
|
||||
"externalFolder": "Externer Ordner",
|
||||
"externalFolderHint": "Diese Ordner stammen aus dem /external-media Mount innerhalb des Containers oder Hosts.",
|
||||
"externalWatch": "Ordner auf neue Dateien überwachen",
|
||||
"externalWatchHint": "Neue Bilder, die in diesen Ordner kopiert werden, werden automatisch importiert – genau wie über den Import-Button. Aus dem Ordner entfernte Dateien werden nie aus der Galerie gelöscht.",
|
||||
"externalWatchActive": "Ordner wird überwacht – neue Dateien werden automatisch importiert.",
|
||||
"externalWatchNoPermission": "Erfordert die Berechtigung, Fotos hochzuladen.",
|
||||
"externalFolderRequired": "Bitte wählen Sie vor dem Speichern einen externen Ordner aus.",
|
||||
"externalFolderEmpty": "Keine Unterordner",
|
||||
"clearSelection": "Löschen",
|
||||
|
||||
@@ -1373,6 +1373,10 @@
|
||||
"sourceModeHelp": "Use managed mode for direct uploads or point to a mounted /external-media folder when using local storage.",
|
||||
"externalFolder": "External Folder",
|
||||
"externalFolderHint": "These folders are read from the /external-media mount inside your container or host.",
|
||||
"externalWatch": "Watch folder for new files",
|
||||
"externalWatchHint": "New images copied into this folder are imported automatically, the same way the Import button does it. Files removed from the folder are never deleted from the gallery.",
|
||||
"externalWatchActive": "Folder is watched — new files are imported automatically.",
|
||||
"externalWatchNoPermission": "Requires the permission to upload photos.",
|
||||
"externalFolderRequired": "Please select an external folder before saving.",
|
||||
"externalFolderEmpty": "No subfolders",
|
||||
"clearSelection": "Clear",
|
||||
|
||||
@@ -424,6 +424,7 @@ export const EventDetailsPage: React.FC = () => {
|
||||
customer_phone: event.customer_phone || '',
|
||||
source_mode: event.source_mode === 'reference' ? 'reference' : 'managed',
|
||||
external_path: event.external_path || '',
|
||||
external_watch: Boolean(event.external_watch),
|
||||
require_password: normalizeRequirePassword(event.require_password),
|
||||
new_password: '',
|
||||
confirm_new_password: '',
|
||||
@@ -626,6 +627,9 @@ export const EventDetailsPage: React.FC = () => {
|
||||
updateData.external_path = editForm.source_mode === 'reference'
|
||||
? externalPathToSave
|
||||
: null;
|
||||
// Always sent, like og_image_share_enabled: the backend writes through
|
||||
// formatBoolean, so a save can switch the watcher off again.
|
||||
updateData.external_watch = editForm.source_mode === 'reference' && editForm.external_watch;
|
||||
if (editForm.customer_name !== undefined && editForm.customer_name !== null) {
|
||||
updateData.customer_name = editForm.customer_name;
|
||||
}
|
||||
|
||||
@@ -27,6 +27,7 @@ import type { AdminPhoto } from '../../../services/photos.service';
|
||||
import type { FeedbackSettings as FeedbackSettingsType } from '../../../services/feedback.service';
|
||||
import { ExternalFolderPicker } from './ExternalFolderPicker';
|
||||
import { safeParseDate } from './utils';
|
||||
import { usePermission } from '../../../hooks/usePermission';
|
||||
import type { EditFormState } from './types';
|
||||
|
||||
interface EventInformationCardProps {
|
||||
@@ -64,6 +65,11 @@ export const EventInformationCard: React.FC<EventInformationCardProps> = ({
|
||||
onRevealNow
|
||||
}) => {
|
||||
const { t } = useTranslation();
|
||||
// Enabling the watcher makes the server import on the admin's behalf, which
|
||||
// the backend gates on photos.upload like the Import button. Mirror that
|
||||
// here rather than letting the save bounce with a 403.
|
||||
const canEnableWatch = usePermission('photos.upload');
|
||||
|
||||
const { format } = useLocalizedDate();
|
||||
const queryClient = useQueryClient();
|
||||
const [logoUploading, setLogoUploading] = useState(false);
|
||||
@@ -356,6 +362,28 @@ export const EventInformationCard: React.FC<EventInformationCardProps> = ({
|
||||
<p className="text-xs text-neutral-500 dark:text-neutral-400 mt-1">
|
||||
{t('events.externalFolderHint', 'These folders come from the /external-media mount inside the container. Ensure it is accessible to the backend process.')}
|
||||
</p>
|
||||
<label className={`flex items-start gap-2 mt-3 ${canEnableWatch || editForm.external_watch ? 'cursor-pointer' : 'opacity-60 cursor-not-allowed'}`}>
|
||||
<input
|
||||
type="checkbox"
|
||||
className="mt-0.5 rounded border-neutral-300 dark:border-neutral-600 text-accent focus:ring-primary-500"
|
||||
checked={editForm.external_watch === true}
|
||||
disabled={!canEnableWatch && !editForm.external_watch}
|
||||
onChange={(e) => setEditForm(prev => ({ ...prev, external_watch: e.target.checked }))}
|
||||
/>
|
||||
<span className="text-sm">
|
||||
<span className="font-medium text-neutral-900 dark:text-neutral-100">
|
||||
{t('events.externalWatch', 'Watch folder for new files')}
|
||||
</span>
|
||||
<span className="block text-xs text-neutral-500 dark:text-neutral-400 mt-0.5">
|
||||
{t('events.externalWatchHint', 'New images copied into this folder are imported automatically, the same way the Import button does it. Files removed from the folder are never deleted from the gallery.')}
|
||||
</span>
|
||||
{!canEnableWatch && !editForm.external_watch && (
|
||||
<span className="block text-xs text-neutral-500 dark:text-neutral-400 mt-0.5">
|
||||
{t('events.externalWatchNoPermission', 'Requires the permission to upload photos.')}
|
||||
</span>
|
||||
)}
|
||||
</span>
|
||||
</label>
|
||||
</div>
|
||||
)}
|
||||
|
||||
@@ -850,6 +878,11 @@ export const EventInformationCard: React.FC<EventInformationCardProps> = ({
|
||||
{event.source_mode === 'reference' && event.external_path ? (
|
||||
<span className="text-neutral-500 dark:text-neutral-400 ml-2">/external-media/{event.external_path}</span>
|
||||
) : null}
|
||||
{event.source_mode === 'reference' && event.external_watch ? (
|
||||
<span className="block text-xs text-neutral-500 dark:text-neutral-400 mt-1">
|
||||
{t('events.externalWatchActive', 'Folder is watched — new files are imported automatically.')}
|
||||
</span>
|
||||
) : null}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
|
||||
@@ -16,6 +16,7 @@ export type EditFormState = {
|
||||
customer_phone: string;
|
||||
source_mode: 'managed' | 'reference';
|
||||
external_path: string;
|
||||
external_watch: boolean;
|
||||
require_password: boolean;
|
||||
new_password: string;
|
||||
confirm_new_password: string;
|
||||
@@ -71,6 +72,7 @@ export const INITIAL_EDIT_FORM: EditFormState = {
|
||||
customer_phone: '',
|
||||
source_mode: 'managed',
|
||||
external_path: '',
|
||||
external_watch: false,
|
||||
require_password: true,
|
||||
new_password: '',
|
||||
confirm_new_password: '',
|
||||
|
||||
@@ -79,6 +79,7 @@ interface UpdateEventData {
|
||||
hero_photo_id?: number | null;
|
||||
source_mode?: 'managed' | 'reference';
|
||||
external_path?: string | null;
|
||||
external_watch?: boolean;
|
||||
photo_cap?: number | null;
|
||||
default_photo_sort?: string;
|
||||
// Per-event opt-in for hero photo as social-share preview (#474).
|
||||
|
||||
@@ -39,6 +39,8 @@ export interface Event {
|
||||
unique_visitors?: number;
|
||||
source_mode?: 'managed' | 'reference' | string;
|
||||
external_path?: string | null;
|
||||
// Folder watcher opt-in (issue 1187). SQLite hands back 0/1, Postgres a boolean.
|
||||
external_watch?: boolean | number | null;
|
||||
// Download protection fields
|
||||
allow_downloads?: boolean;
|
||||
protection_level?: 'basic' | 'standard' | 'enhanced' | 'maximum';
|
||||
|
||||
Reference in New Issue
Block a user