pyscript-Backend zur echten HA-Integration umgebaut (HACS-fähig)

Das Backend liegt jetzt als custom_components/audi_dashboard/ vor - eine
normale Home-Assistant-Integration mit Config-Flow, einer sensor-Plattform
und 18 Diensten. Damit ist die App über HACS installierbar; bis das Repo auf
GitHub gespiegelt ist (HACS spricht ausschließlich mit GitHub), installiert
homeassistant/installationspaket/install.ps1 denselben Ordner ohne HACS.

Fünf Installationsschritte entfallen ersatzlos: der pyscript:-Block, der
panel_custom:-Block, das Kopieren der Oberfläche nach www/, das langlebige
Zugriffstoken (der Verlauf wird direkt über die recorder-API gelesen) und
"pip install pypdf" (steht in manifest.json). Das Fahrzeugprofil legt die
Integration beim ersten Start aus ihrer Vorlage an.

Drei alte Schwächen sind dabei mit erledigt:
- Die Nutzlast landet nicht mehr in der Recorder-Datenbank
  (_unrecorded_attributes - das kann nur eine echte Entität).
- Eine laufende Fahrt überlebt einen Neustart (Store statt Arbeitsspeicher);
  fiel sie während eines Ausfalls ins Ende, schließt
  nach_neustart_fortsetzen() sie beim letzten aufgezeichneten Zeitpunkt.
- Sensor-Zuordnungen wirken sofort - die Zustandsbeobachter werden neu
  gebunden, der Neustart-Hinweis und der Neustart-Dienst sind weg.

Namensvertrag geändert, beide Oberflächen mitgezogen:
pyscript.audi_dashboard_x -> sensor.audi_dashboard_x,
pyscript.audi_dashboard_y -> audi_dashboard.y. Eine Companion-App vom alten
Stand findet nach dem Umstieg nichts mehr und muss neu gebaut werden; das
Panel liegt in der Integration und kann nicht driften.

Der selbstgebaute Updater entfällt - HACS ist die Update-Mechanik, die Home
Assistant kennt. Die Versionierung schrumpft auf eine Quelle: manifest.json.

Geprüft am laufenden Testcontainer (Container byteweise identisch mit dem
Repo): alle 18 Dienste, Panel, Config-Entry neu laden, Historienimport,
echter Shell-Beleg in-process, Neuinstallation im Wegwerf-Container blank mit
automatisch nachinstalliertem pypdf. Companion-App: tsc sauber, 112/112
Tests, beide Rauchtests gegen das laufende Backend grün. Belegparser 8/8.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
2026-08-23 23:53:56 +02:00
co-authored by Claude Opus 5
parent 99cef7c393
commit d8b12da36d
136 changed files with 5289 additions and 17641 deletions
+154 -71
View File
@@ -90,7 +90,7 @@ build`, since companion-app needs a live backend connection the local dev server
2026-08-23: **retroactive data import** — see section F below for the full entry. In short: HA's
recorder keeps sensor history only 10 days by default, which silently capped how far back anything
could ever be reconstructed; `recorder_snippet.yaml` raises that to a year, and a new
`pyscript/historienimport.py` + "Daten importieren aus Home Assistant" button (both codebases)
`historienimport.py` + "Daten importieren aus Home Assistant" button (both codebases)
rebuilds trips, refuels and battery history for any past window out of that recorder history.
This file is the entry point for every new agent
session: what this repo is, what is finished, what is missing, and how to work here. Detail lives in
@@ -188,7 +188,8 @@ assets. Never mix these the other way around.
| Area | What | Status |
|---|---|---|
| `homeassistant/` | HA panel (`panel_custom`): pyscript backend + vanilla-JS frontend | ✅ **finished, in use** to be replaced by the app |
| `custom_components/audi_dashboard/` | The HA integration: backend + panel, installable via HACS | ✅ **finished, in use** — replaced the pyscript backend on 2026-08-23 (section H) |
| `homeassistant/` | Install package, recorder snippet, guides — **no code any more** | ✅ |
| `testumgebung/` | Script that rebuilds a throwaway Home Assistant with the real backend | ✅ new, reproducible |
| `design-system/` | React component library `@audi-dash/ui`, brand-free, feeds Claude Design | ✅ done as a kit (20 components, 1,690 lines) |
| `companion-app/` | **DataMetric360** — successor app (web, PWA, native iOS/Android via Capacitor, HA iframe); will **replace** the panel | ✅ **all 21 screens built and tested**; runs natively on iOS with real data |
@@ -198,17 +199,20 @@ Root files: `dashboard-muster*.html` = original static prototype (superseded, re
`bauauftrag.md`/`.html` = original build brief (historical), `DESIGN_BRIEF_DATAMETRIC360.md` = the
prompt for the Claude Design project.
**`homeassistant/` in one paragraph:** 10 pyscript scripts + 5 modules (`pyscript/modules/`); trip
detection via the FMM003 ignition sensor with pause tolerance (the iPhone WLAN sensor is gone — see
section B), two-stage trip completion via HA history screening (odometer often updates only on the
next trip), fill-up detection on fuel-level rise, Shell PDF parser (`data/shell_beleg_parser.py`,
subprocess, the only tested part of the repo), tire km counter, backup, self-update, image
management. Frontend: one file `www/audi-dashboard-app.js`
(~4,000 lines, custom element, no framework/bundler), 5 tabs + ~19 detail routes, cache-busting via
`audi-dashboard-version.json` + loader stub. Entity IDs are assigned **in the UI** (Settings →
Fahrzeug einrichten → Setup), stored as overrides in `data/entitaeten.json`;
`pyscript/modules/einstellungen.py` holds only the built-in defaults. Deploy: `update.ps1`
(robocopy to Samba share) or — still inactive — self-update from git.
**`custom_components/audi_dashboard/` in one paragraph:** a normal HA integration with a config
flow, one `sensor` platform and 18 services. `koordinator.py` owns the runtime state and the
wiring; the domain modules beside it carry the logic: trip detection via the FMM003 ignition
sensor with pause tolerance (`fahrterkennung.py`), two-stage trip completion via recorder history
(`screening.py` — the odometer often updates only on the next trip), fill-up detection on
fuel-level rise (`tankerkennung.py`), Shell PDF parser (`shell_beleg_parser.py`, in-process via
`pypdf`, the only tested part of the repo), tire km counter, backup, image management, retroactive
history import. `ablage.py` is the only place that touches files. Frontend: `frontend/audi-dashboard-app.js`
(~4,900 lines, custom element, no framework/bundler), 5 tabs + ~19 detail routes, served by the
integration itself under `/audi_dashboard_static/` with the manifest version as the cache breaker.
Entity IDs are assigned **in the UI** (Settings → Fahrzeug einrichten → Setup), stored as overrides
in `/config/audi_dashboard/entitaeten.json`; `einstellungen.py` holds only the built-in defaults
(all empty). Deploy: HACS, or `homeassistant/installationspaket/install.ps1` where HACS can't reach
the repo.
**`companion-app/` — what exists:** the full app. Data layer (`src/api/`: REST, WebSocket with
reconnect backoff, persistent offline write queue, credential storage), domain logic
@@ -356,10 +360,13 @@ vendored MIT library served from Home Assistant itself, which satisfies the actu
## Pending: this branch has diverged from `main` (noted 2026-08-13)
`ha_install.md` (root) plans the move from pyscript to a **native HA integration** with a config
flow — UI-only setup, no YAML, and updates via a self-reporting `UpdateEntity` against the Gitea
repo (HACS is GitHub-only, so it is not an option). All APIs in it were verified against the
running 2026.8.1 instance. It also records which review findings that move eliminates by design.
`ha_install.md` (root) planned the move from pyscript to a **native HA integration** with a config
flow — UI-only setup, no YAML. **That move happened on 2026-08-23; see section H.** The plan
document is now historical: it still argued for a self-reporting `UpdateEntity` against the Gitea
repo because HACS is GitHub-only, and that part was dropped — the built-in updater was removed
rather than reimplemented, and updates go through HACS (or the install script where HACS cannot
reach the repo). All APIs in it were verified against the running 2026.8.1 instance. It also
records which review findings that move eliminates by design.
A full review of `main`'s 18 new commits is in `REVIEW_main_2026-08-13.md` — 15 findings, the
three most serious in the new setup menu (saving with an unloaded catalogue wipes the whole
@@ -1793,71 +1800,137 @@ especially for how long a full year takes (the UI waits up to two minutes before
in the background). Address reverse-geocoding is not attempted for imported trips (`start_address`/
`end_address` stay null, same as live-detected ones before the user edits them).
### H) HACS: deferred by decision, and the deployed-parity problem it exposed (2026-08-23)
### H) HACS: the pyscript backend became a real integration (2026-08-23)
**Decision: no HACS conversion for now.** Owner asked whether the app could be installed via HACS.
Checked against the actually installed HACS 2.0.5 rather than from memory — it cannot, for two
independent reasons:
Owner asked for it directly ("build the hacs") after deferring it earlier the same day. The
deferral reasoning still held — converting first means debugging the rewrite and the real
environment at once — but the owner overrode it, so the conversion happened.
**What was actually blocking HACS.** Two independent things, both checked against the installed
HACS 2.0.5 rather than from memory:
- **HACS is GitHub-only.** `github.com`/`api.github.com` are hardcoded throughout; no Gitea, GitLab
or self-hosted support. This repo lives on `gitea.nothaft.cloud`.
- **No HACS category installs to `/config/pyscript/`.** The six categories resolve to
`custom_components/<domain>/`, `www/community/<name>/`, `python_scripts/`, `themes/`,
`custom_templates/` and `appdaemon/apps/<name>/` (verified in `repositories/*.py``localpath`).
The app needs `/config/pyscript/` **and** `/config/audi_dashboard/` **and** `configuration.yaml`
entries — none of which HACS can do.
The app needed `/config/pyscript/` **and** `/config/audi_dashboard/` **and** `configuration.yaml`
entries. This is what the conversion solved.
- **HACS is GitHub-only.** `github.com`/`api.github.com` are hardcoded throughout; the repo lives
on `gitea.nothaft.cloud`. This is **not** solved and cannot be solved from this side — it needs
the repo mirrored to GitHub. Until then `homeassistant/installationspaket/install.ps1` installs
the identical folder without HACS, so the conversion is useful either way.
The only path is converting the pyscript backend into a real custom integration
(`custom_components/audi_dashboard/`), which would also kill `panel_custom:`, the `www/` copying,
the `data/``audi_dashboard/` rename, `allow_all_imports`/`hass_is_global` and the pyscript
dependency itself. Scope measured: 2.911 lines, 21 services, 15 state triggers, 10 time triggers,
39 `task.executor`, 12 `state.set`. Owner chose to defer it until after the app has actually run on
the real HA — sensible: converting first means debugging the rewrite and the real environment at the
same time. **Do not start this without the owner raising it again.**
**The shape.** `custom_components/audi_dashboard/` at repo root (HACS looks for
`custom_components/<first dir>` there and nowhere else), `hacs.json` beside it. Config flow, one
`sensor` platform, 18 services, ~2,900 lines of pyscript ported to ~2,400 lines of integration.
`koordinator.py` holds the runtime state and does the wiring that `@service`/`@state_trigger`/
`@time_trigger` used to do implicitly; the domain modules keep their old names and structure so the
diff stays readable. `modules/` was flattened (it only existed because pyscript treats that folder
specially). Three renames earned their keep: `profil.py``ablage.py` (it handles trips and
fill-ups too, the old name was actively wrong), `entitaeten.py``zuordnung.py` (avoids collision
with HA "entities"), `frontend_veroeffentlichung.py``veroeffentlichung.py`.
**The open question this surfaced — deployed parity of the iOS app.** The parity rule (binding,
above) guarantees *source* parity: both codebases change in the same session. It guarantees nothing
about what is *running*. Today those two are wired completely differently:
**Five install steps disappeared, and none of them by hand-waving:**
- the panel fetches `/local/audi-dashboard-version.json` with `cache: "no-store"` on every page load
and reloads its assets when the number changed — current within one page load, no user action
| Was | Now |
|---|---|
| `pyscript:` block with `allow_all_imports` + `hass_is_global` | gone — no YAML at all |
| `panel_custom:` block | the integration registers its own panel |
| copy the frontend to `/config/www/` | served from `frontend/` under `/audi_dashboard_static/` |
| long-lived token in `audi_dashboard/ha_token.txt` | recorder API read directly (`verlauf.py`) |
| `pip install pypdf` in the container | `requirements` in `manifest.json`, HA installs it |
| copy `fahrzeugprofil.example.json` by hand | seeded from `vorlage/` on first setup |
Verified on a genuinely fresh throwaway container, not reasoned about: 12 entities, zero trips,
zero fill-ups, zero battery days, profile created from the template, and **pypdf 6.16.2 installed
by HA itself**.
**The naming contract changed, and both frontends had to follow.** `pyscript.audi_dashboard_x`
`sensor.audi_dashboard_x`, `pyscript.audi_dashboard_y``audi_dashboard.y`. The prefix only ever
existed because pyscript owned the domain. Both frontends now carry the names in one table each
(`const.py``frontend/audi-dashboard-app.js` `E`/`DOMAIN``companion-app/src/api/types.ts`
`ENTITAETEN`/`DIENST_DOMAIN`). **A companion app from before this change finds nothing after the
upgrade** — it must be rebuilt in the same step. The panel can't drift: it ships inside the
integration.
**Three long-standing weaknesses fixed, because the conversion made them cheap:**
1. **The payload no longer hits the recorder database.** `_unrecorded_attributes = {"daten"}` on the
entity. Only a real entity belonging to an integration can do this — pyscript's bare `state.set()`
can't, so every publish wrote the complete trip archive to the DB, every 60 seconds. Confirmed by
querying the history API: only `icon` and `friendly_name` come back. Note the mechanism moved:
exclusion now comes from `state_info["unrecorded_attributes"]` on the Entity, **not** from a
per-domain `recorder.py` platform hook (checked in `db_schema.py` of the installed 2026.8.0).
2. **A running trip survives a restart.** `fahrt_start_ts` and the tank low-water mark live in a
`Store` instead of only in memory. This was a documented data-loss bug.
3. **Trigger rebinding is live.** `@state_trigger` baked the entity ID in at module load, so
changing ZUENDUNG/KM/TANK_SENSOR needed an HA restart — the setup dialog had a warning and a
restart button for exactly that. `trigger_neu_binden()` replaces all three; the warning, the
button and the `neustart` service are gone.
**Fixing (2) exposed a new failure mode, and it was found by testing rather than reasoning.** If the
ignition goes off *while HA is down*, nobody sees the transition — the resumed trip would stay open
forever and the next real trip would be appended to its start. `nach_neustart_fortsetzen()` closes
it at the last recorded ignition timestamp (not "now" — HA may have been down for days), and runs
via `async_at_started` so a not-yet-restored source can't be mistaken for "off". Verified live: an
orphaned trip was closed at 21:12:43, the last value the recorder held.
**The self-updater is gone.** `updateverwaltung.py` cloned a git repo into a staging folder and
copied `pyscript/` and `www/` over the running code — necessary only as long as the app had no
install shape Home Assistant knows. HACS is that shape: it checks, downloads, installs, shows a
changelog and can roll back. A second, homemade update mechanism beside it is more attack surface
than value. The Settings tile now shows the installed version and points at HACS.
**Versioning collapsed to one number.** `manifest.json` `version` is the single source: HA lists it,
HACS compares it, the integration publishes it as `sensor.audi_dashboard_app_version`, it hangs on
the panel URL as `?v=…`, and vite reads it for `__APP_VERSION__` (hard-failing if absent). The
`VERSION` file and the Unix-seconds cache-buster in `audi-dashboard-version.json` are both gone —
two sources for one fact would have drifted, and the comparison would have gone silently wrong. The
new cache breaker is strictly better: it changes only on real changes. `VERSIONIERUNG.md` rewritten.
**Installer got smaller and much safer.** It copies one folder and touches nothing else. The
`configuration.yaml` manipulation — the one way an installer can stop HA from starting, and the
reason the old script needed timestamped backups, a conflict refusal and a read-back rollback — is
gone with the YAML it used to write. What remains: refuses to delete a folder whose `manifest.json`
doesn't claim `domain: audi_dashboard`, reads the manifest back after copying. Tested on a mock
config dir: dry run, first install, update run, and the refusal path.
**Removed from the repo** (recoverable via `git log`): `homeassistant/pyscript/`,
`homeassistant/www/audi-dashboard-*` + `badges/`, `homeassistant/data/`, `homeassistant/install.sh`,
`homeassistant/update.ps1`, the `installationspaket/` copy of everything (the duplicate-and-sync
arrangement is gone — the installer reads the real folder), `configuration_snippet.yaml` (nothing
left to add), `ANLEITUNG.pdf` (stale and not regenerable here; the `.md` beside it is current).
`data/tests/``tests/belegparser/`, pointing at the integration's parser copy; 8/8 still pass.
**Do not resurrect the pyscript tree.** If something is missing, port it — running both backends
writes two sets of trips into the same files.
**Still open, unchanged by this:** the repo must reach GitHub before HACS can install it. Everything
else about the HACS path is built and tested.
### The deployed-parity problem, and what was built for it (2026-08-23)
The parity rule (binding, above) guarantees *source* parity: both codebases change in the same
session. It guarantees nothing about what is *running*. After the conversion the two sides are
wired differently in a new way:
- the panel ships **inside** the integration — one HACS update moves backend and panel together,
and it cannot go stale at all
- the companion app is a Capacitor shell with `webDir: "dist"` and **bundled** assets, sideloaded
through Xcode. `package.json` says `0.1.0` and **nothing in `src/` ever checks a version.** It
stays on whatever was bundled at signing time, silently, indefinitely
through Xcode. It stays on whatever was bundled at signing time, silently, indefinitely
So the iOS app can be weeks behind the panel with nothing anywhere making that visible. That is a
real gap, not a hypothetical one, and it gets worse once the app is genuinely in daily use.
**Built 2026-08-23 — the version identity and the drift warning.** See `VERSIONIERUNG.md` for the
owner-facing version.
The first finding was that the number everyone assumed was "the version" is not one:
`audi-dashboard-version.json`'s integer is a **cache-buster**, rewritten with `UtcNow` by
`install.ps1`/`update.ps1` on every deploy regardless of whether any code changed. Two builds of
identical source get different numbers; the same build deployed twice gets two more. It can never
answer "are these the same state?". So the two jobs were separated instead of conflated:
- **`VERSION`** (new, repo root, e.g. `2026.08.23.1`) — the identity, bumped by hand on change
- the existing integer — unchanged, still only breaks the browser cache
`VERSION` now flows to both sides: into `audi-dashboard-version.json` as a second field `app`
(carried over by all three deploy scripts — they previously overwrote the whole file and would have
silently destroyed it), and into the companion build via vite `define``__APP_VERSION__`. The
backend reads the file and publishes `pyscript.audi_dashboard_app_version`; the companion compares
that against its own compiled-in stamp and, on mismatch, says so in the existing `Hinweisleiste`
whose stated principle is already "nie eine stille Veraltung", which is exactly this case with the
staleness moved from the data to the app itself.
The version comparison above is what makes that visible: the app compares its compiled-in stamp
against `sensor.audi_dashboard_app_version` and says so in the existing `Hinweisleiste`, whose
stated principle is already "nie eine stille Veraltung".
Deliberate choices worth keeping: the comparison is **equality only**, never greater/less — the
version is an identifier, not a number, and ordering it would be false precision that breaks
silently on a format change. Either side missing yields `"unbekannt"` and **no** warning, so an
older backend or an offline start cannot produce a false alarm. And the vite build **fails hard**
if `VERSION` is absent rather than emitting an app that cannot detect its own staleness. The panel
needs none of this: it re-fetches on every page load and cannot go stale.
older backend or an offline start cannot produce a false alarm. And the vite build **fails hard** if
the manifest version is absent rather than emitting an app that cannot detect its own staleness.
**OTA delivery: verified viable, not yet built.** `@capgo/capacitor-updater` 8.51.14 checked
against the real package: MPL-2.0, peer `@capacitor/core: ^8.0.0` against our `^8.5.0`, and
self-hosting is first-class (`updateUrl`, or manual mode entirely). Manual mode is the good fit —
**OTA delivery: verified viable, not yet built.** `@capgo/capacitor-updater` 8.51.14 checked against
the real package: MPL-2.0, peer `@capacitor/core: ^8.0.0` against our `^8.5.0`, and self-hosting is
first-class (`updateUrl`, or manual mode entirely). Manual mode is the good fit —
`download({version, url})` + `set()` against a plain zip under `/local/`, needing **no** custom
endpoint at all, with automatic rollback to the last good bundle via `notifyAppReady()`. It pairs
exactly with the version entity above: that entity is already the signal that a newer bundle exists.
@@ -1869,11 +1942,14 @@ Capacitor's `server.url` pointed at HA was considered and **rejected**: it would
current as the panel, but the shell then cannot boot without reaching HA, gutting the deliberately
built offline queue (`api/warteschlange.ts`).
The whole mechanism survives the later integration conversion unchanged — that end state serves
panel assets and companion bundle from the same component, so one HACS update moves both.
### G) Fresh-install audit + installer hardening (2026-08-23, before the first real deployment)
> **Historical from here on.** This describes the pyscript-era package, which section H replaced
> later the same day. The *findings* still matter — the blank-delivery rule and the safety
> properties carried over into the new installer — but the files it names (`pyscript/`, `www/`,
> `data/`, the `configuration.yaml` block) no longer exist. Kept as the record of what was wrong
> and why the current shape looks the way it does.
Owner is about to install on the real HA OS instance and asked for two things: that the one-click
package produce a genuinely blank app, and that it carry no risk to the existing HA system.
@@ -1974,6 +2050,13 @@ which the parity rule exempts as panel-only.
- Decisions are logged, including rejected ones (see the Traccar section of the architecture
doc); superseded sections stay in place marked "ÜBERHOLT" rather than being deleted.
- Numbers in `de-DE` format; font weights 300/400 only, never ≥600 (only those cuts exist).
- Real vehicle/movement data stays local (`data/` contents and receipt PDFs are gitignored).
- Real vehicle/movement data stays local: it lives in Home Assistant under
`/config/audi_dashboard/`, never in this repo. Only the placeholder template
(`custom_components/audi_dashboard/vorlage/fahrzeugprofil.json`) is versioned; the real receipt
PDFs under `tests/belegparser/belege/` are gitignored.
- **There is no second copy of the code to keep in sync any more.** Until 2026-08-23,
`installationspaket/` duplicated `pyscript/`, `www/` and `data/`, and every change had to be
mirrored by hand — the older entries below still say so. The installer now reads
`custom_components/audi_dashboard/` directly. Do not reintroduce a copy.
- Security principle: HA is never publicly exposed; only narrowly scoped surfaces (MQTT broker,
proxy allowlist) may be exposed, each by explicit decision.