Architecture¶
HTMX templates Ninja API endpoints (grow as needed)
\ /
\ /
Services layer (thin views call into this)
|
v
Django ORM
|
v
SQLite (Fly volume) → Postgres when needed
Server-rendered templates with HTMX for partial updates, Alpine.js for small client-side state (dropdowns, modals). Services are plain Python functions in apps/<app>/services.py. Models are Django ORM with custom managers for household scoping.
Apps¶
| App | What lives here |
|---|---|
apps/common/ |
Middleware (current-household context), household-scoped managers, encrypted text field, shared context processors, static template checks |
apps/households/ |
Household, HouseholdMembership, Person; signup service; dashboard view; invite tokens |
apps/auth_google/ |
"Sign in with Google" flow (openid email profile); GoogleIdentity keyed by Google sub. Independent from emailscan's Gmail OAuth. |
apps/documents/ |
Document model (with source field flagging upload / email_in / gmail_triage to drive per-source review surfaces); upload + confirm views; topic / category / search / timeline / person / named-item views; bulk operations; auto-assign; activity timeline |
apps/classifier/ |
Claude Sonnet integration. Stub mode keyword-based; live mode sends pdfplumber-extracted page-1 text inline with the email context, with prompt caching. Page-1 text replaces base64 PDF binaries as a measured ~5x cost reduction with no precision/recall hit (validated on the spike's hand-labelled set). Non-PDF and parse-failure cases degrade to email metadata + filename signal alone. Doctype vocabulary is loaded from apps/classifier/taxonomy.yaml via taxonomy.py — single source of truth with three columns per doctype (classifier_examples, heuristic_fingerprint, suggested_tags); the latter two are populated in subsequent slices. |
apps/emailscan/ |
Gmail OAuth + scan pipeline. Heuristic pre-filter, thread + content-hash dedup, recurring-email dedup, Claude classification, lifecycle bucketing. Scans run via the run_gmail_scan django-tasks task with three callers — OAuth callback (first scan), weekly cron, manual "Scan now" button — all funnelling through enqueue_scan which holds the single rate-limit guard (scan_state != "running"). Ongoing scans pull a delta window via scan_window_start (last_scan_at − 3 days overlap). Also the Postmark inbound webhook — persists the raw payload as InboundEmailPayload and enqueues process_inbound_email so the webhook returns <100ms regardless of attachment count (issue #85). |
apps/triage/ |
Triage UI: two-tab nav (Gmail findings + Email-in Review). Gmail tab is the original accept/dismiss queue; Email-in Review lists already-filed source='email_in' docs that the user hasn't confirmed yet — Confirm sets reviewed_at, Edit opens the standard per-doc form. Per-tab count chips + summed nav badge. Adapter delegates to emailscan in prod, stub in tests. |
apps/profiles/ |
PropertyProfile + VehicleProfile sidecars. Schedule definitions for houses (schedules/house.py) and vehicles (schedules/vehicle.py). DVLA + DVSA lookup service. |
apps/activity/ |
ActivityEvent model + per-event undo (auto-assign undo, confirm-review undo). |
apps/api/ |
Django Ninja API surface. One /api/health endpoint today; grows as consumers need them. |
Data model highlights¶
- Household / Person / NamedItem — the multi-tenant root. Every Document, EmailScanFinding, ActivityEvent has a
householdFK.NamedItem.kindishouseorvehicle.NamedItemediting surfaces a kebab actions menu next to the page title via the sharedtemplates/_partials/entity_actions_menu.htmlpartial: Rename (POST/documents/named-items/<id>/rename/→rename_named_itemservice, pre-checks theunique_together = ("household", "name", "kind")constraint and raisesValueErrorinstead ofIntegrityError, recordsnamed_item_renamed); Same as another … (POST/documents/named-items/<id>/merge/→merge_named_itemsservice, reassigns linkedDocument.named_itemfrom source to target, optionally renames target to a user-chosen canonical name, deletes source, recordsnamed_items_merged); and Delete (POST/documents/named-items/<id>/delete/→delete_named_itemservice, unlinks linked Documents viaDocument.named_item.on_delete=SET_NULLrather than cascading, recordsnamed_item_deleted). The same partial now serves Person pages too (issue #99): Rename (POST/documents/people/<id>/rename/→rename_person) and Delete (POST/documents/people/<id>/delete/→delete_person, which reassigns the person's Documents to the household's SELF person inside a transaction). The SELF person and any Person with alinked_usercan't be deleted — removing a member who can sign in is an access decision (revoke their membership), not a tile deletion (issue #108). Joining a household via invite creates a linked Person (ensure_member_person), so every member has exactly one Person withlinked_userset iff they can authenticate. People are first-class on the dashboard (issue #112): a pill strip for solo households graduating to tiles with per-person document counts, pluscreate_personbehind/documents/people/new/.Document.personis user-editable via the "Filed under" person chip — by the file preview on the confirm form, beside the Topic chip on triage review (a filing decision, not document metadata, so it renders as switchable chrome rather than a form field; the chip drives a hiddenperson_idinput) — the classifier's free-textperson_subjectguess is persisted on the Document and pre-selects the picker viaresolve_person_subject, but is never silently honoured; pickerless paths (upload, inline accept) attach to SELF.PersonProfile(issue #116) is the 1:1 facts sidecar on Person —date_of_birthplus a schema-lightfactsdict (curated labels inFACT_LABELS, custom keys allowed) edited from the person page; activity events record fact labels only, andbuild_classifier_contextnever reads profile values. The review screen's unregistered-person hint is actionable (issue #117): an inline mini-form (HTMLform=attribute, since the review page is one big form) POSTs totriage:add_person_from_hint, duplicate-guarded viaresolve_person_subject; the confirm-form variant links to/documents/people/new/?name=…&next=…with an open-redirect-guarded return. The account page (/account/, issue #114) holds login-level details — display name (written toUser.first_nameand the SELF Person together), email, password (Django's PasswordChangeView), connected Google — and links the household settings pages (invites, suppression). - Document — the unit of filing. Has
topic,document_type_id,provider,issue_date,expiry_date,lifecycle_state(internal — not user-visible),filing_status,reviewed_at,named_item,person, an M2M toTag, andproposed_tags(JSONField — AI suggestions awaiting user review, cleared when the confirm form is saved). Carries the AI's proposal AND the user's eventual confirmation. - Tag — household-scoped, free-form,
name+slug. Cross-cutting axis the user defines themselves (kids, rental, claim-2025). Slug is derived from name and is the URL/uniqueness key — "Kids" and "kids" collapse to the same Tag. Service helpers inapps/documents/tagging.py. The classifier proposes 2–5 tags per document on upload / email-in / triage-accept; suggestions render as one-click ✨-chips on the confirm form alongside chips for existing household tags the user can reuse. - EmailScanFinding — a candidate document found in Gmail awaiting Triage decision. Holds the AI's proposed metadata, plus enough Gmail identifiers (
gmail_message_id,attachment_id) to fetch the actual file bytes later. On accept it becomes a Document; in the future the bytes will be downloaded ontoDocument.fileat that point. - PropertyProfile / VehicleProfile — 1:1 sidecars on NamedItem. Property is wizard-driven (country + ownership/tenure/type/age/facts — the country step branches the questions and the checklist, issue #94); Vehicle is lookup-driven (DVLA + DVSA from the registration plate).
- Invite — server-side household invitation (issue #98): UUID-addressed, single-use, revocable, 14-day expiry, optional
invited_emailbinding. Minted from/household/invites/, accepted at/join/<uuid>/(which also creates the member's linked Person), revoked from the same settings page. Replaced the signed bearer token; the oldinvite_urlcontext processor is gone so no page render carries a join link. - DismissedSignal — learned dismissal suppression (issue #93). Dismissing a finding records/increments a household-scoped fingerprint (sender domain, normalised subject, attachment SHA-1). At
dismiss_count >= 2, matching scan candidates are dropped before the classifier runs. Listed + revocable at/triage/suppression/. - ActivityEvent — append-only log;
is_undoableflag drives the timeline Undo affordance.
The classification pipeline¶
Upload Gmail Triage
| |
v v
Claude (live mode) Heuristic pre-filter (per-doctype fingerprints + household context tokens)
| |
v v
ClassifierProposal Thread + content-hash + recurring-email + learned-suppression dedup
| |
v v
Document (filing_status=KEPT, Claude (live mode, with email context + lazy-fetched bytes)
reviewed_at=null) |
v
topic=null suppressed; the rest →
EmailScanFinding (status=pending) → Triage UI → Document
Both paths share the classifier service. Both produce a Document with the same shape.
The taxonomy file¶
apps/classifier/taxonomy.yaml is the single place a document type is defined (issue #96) — 104 doctypes across the nine topics, each with three columns: classifier_examples (fed to the prompt vocabulary), heuristic_fingerprint (keywords + sender domains the pre-Claude scoring layer matches), and suggested_tags (tag priors surfaced at confirm time). Add a doctype once and all three behaviours pick it up. python manage.py generate_doctype_fingerprints drafts fingerprints for new doctypes (offline-deterministic by default, --live asks Claude) — review the YAML diff before committing.
The heuristic + the gate¶
Scan-time scoring (apps/emailscan/heuristic.py) matches each candidate against every doctype fingerprint: sender-domain hit +2, keyword hit +2 in the subject / +1 in the body, capped at 4 per doctype, summed across the catalog; plus +3 per household-context token and +2 for a useful attachment, threshold ≥ 5. Household context tokens are derived at call time from registered data (apps/emailscan/context.py, issue #95): house names + postcodes, vehicle names + plates, family names, employer hints from connected email addresses. The same derived context is injected into the classifier prompt as a HOUSEHOLD CONTEXT block, along with the household's existing tag names (tag priors, slice 2e). Post-classification, refine_suggested_tags tops up suggestions from the matched doctype's curated suggested_tags — skipped when the classifier's doctype confidence is low, since wrong-doctype priors read as irrelevant recommendations (#119) — and drops tags that duplicate structured fields. On the confirm form only the AI's per-document ✨ suggestions render as chips — existing household tags surface through the AI's reuse-verbatim prompt or the input's autocomplete datalist, never as a chips row of everything. The confirm page's category field is a dropdown whose options narrow live to the selected topic (taxonomy-sourced via a json_script map; legacy free-text values stay selectable), expiry/recurrence inputs only render when they carry a value (quiet '+ add' toggles reveal them), and the document header renders on the navy shell: thumbnail (click to open) + "Filed under" chip + a quiet download link — no white file row.
The classifier runs the spike-validated 9-topic prompt (SPIKE_PROMPT_V2, Phase C cutover): it returns topic ∈ the nine topics or null. topic=null candidates are suppressed at scan time — no finding row reaches the triage UI — which is what lets Medical / Finance / Family-IDs findings surface instead of everything being squeezed through the old is_property_relevant boolean. Confidence buckets findings within a topic (a low-confidence "final" lands in Needs your eye, not Signed and final).
The schedule + completeness panel¶
Lives in apps/profiles/services.py:
_schedule_for(named_item)returns the right schedule list for the kind (house or vehicle) — and, for houses, for the property's country: UK houses get the full TA6-shaped schedule, French houses getschedules/france.py(titre de propriété, taxe foncière, DPE, diagnostic immobilier…), anywhere else gets a short generic list (issue #94)._profile_for(named_item)returns the right profile sidecar.evaluate_schedule(named_item)matches the schedule against the household's filed documents, classifies each entry ashave/missing/not_applicable, and returns counts + a percentage.
Schedule items are dataclasses with a condition callable that reads the profile's facts dict — that's how the MOT slot gets suppressed for vehicles under 3 years old, or how leasehold-only items get hidden for freehold houses.
The operator backend¶
The Django admin (/admin/) is restyled to the front-end design system via templates/admin/base_site.html, which overrides the admin's CSS custom properties (navy chrome, teal accent, light high-contrast content) — no admin templates are forked beyond that thin shell, so it survives upgrades. Every colour pair is WCAG-AA-checked (body text ≥ 4.5:1); the load-bearing rule, shared with the front end, is that teal is too pale for white text, so admin buttons use navy text on teal and body links use a darker teal (#0A6E62), never the pale accent. The brand variables are applied to the light/dark theme selectors alike so the theme toggle and OS preference both resolve to the one accessible theme.
Running Maud as a business lives in two apps — see Operator console for the full reference:
apps/billing—Plan(seeded Free / First Sweep / Ongoing) +Subscription(one per household; absent row = default plan). Limits are display-only (limits_enforcedoff); Stripe is planned in viastripe_*fields and a 501-until-configured webhook stub at/billing/webhooks/stripe/.apps/ops— the staff-only console at/ops/(dashboard, tenant directory + detail, plan management, suspend/reactivate/wipe, read-only view-as) and the append-onlyOpsAuditLog.OpsMiddleware(afterCurrentHouseholdMiddleware) implements the view-as household override and the suspension gate.- Metering — every live classifier call writes a
ClassifierUsagerow (household, source, token counts; never content), priced by an in-code table so the console reports Claude cost per tenant per month. Suspended households are refused byrun_scanand the inbound-email task.
Multi-tenant isolation¶
The HouseholdScopedManager reads the current request's household from a thread-local set by apps/common/middleware.py:CurrentHouseholdMiddleware. Every default Document.objects.all() is therefore implicitly scoped — a view that forgets to filter still won't leak.
Document.all_tenants is the explicit cross-household manager; reserved for management commands and audit code.
Contract tests in apps/common/tests/test_isolation.py and per-app test files exercise both correctness and the failure case (cross-household reads must return empty).
Interesting concrete details¶
Document.reviewed_atis stamped by both the per-block "Confirm all" and by bulk-Move-to-property — moving a doc is an assertion that it belongs there, no second-confirm step needed.- Topic page is a review queue. Property blocks only render if they have at least one unreviewed doc. Confirmed blocks vanish until new docs arrive. Browse-all-of-this-property's-docs lives on the property detail page.
- Email scan dedup is four-keyed: by
gmail_message_id(re-runs), by(thread_id, attachment_name)(forwarded threads), byattachment_sha1(forwarded duplicates across threads), by(from_header, normalised_subject)(recurring marketing). On top of that, learned suppression (DismissedSignal) silently drops anything the household has dismissed twice — before the classifier is paid to look. - Per-document size cap.
settings.MAX_DOCUMENT_BYTES(25 MB) is enforced on every Document-create path: uploads get a 413, email-in attachments are skipped withskipped_count++. Django'sFILE_UPLOAD_MAX_MEMORY_SIZE/DATA_UPLOAD_MAX_MEMORY_SIZEare pinned to the same value. - Triage-accepted documents include the file bytes.
accept_findingcallsattachments().get()against the captured(gmail_message_id, attachment_id)and writes the result ontoDocument.file. Failure (revoked OAuth, deleted message, network) is non-fatal — the Document persists metadata-only and the user can re-runpython manage.py backfill_triage_fileslater, which retries every empty-file Document linked to a finding. - Boot-time migrations.
entrypoint.shrunspython manage.py migrateagainst the volume-backed DB on every container boot. Fly'srelease_commandruns on a separate machine without the volume — that path silently no-op'd against an empty DB until we noticed. - Background task queue. Async work uses
django-taskswith the db-backeddjango-tasks-dbbackend. Dev / tests default toImmediateBackend(synchronous in-process). Prod setsTASK_BACKENDto the DB backend infly.tomland runspython manage.py db_workeras a co-process alongside gunicorn (entrypoint.sh). The worker shares the SQLite volume — co-located until the Postgres migration lets us split it out onto a separate machine. - Token usage logging. The classifier logs every Claude call with input / output / cache-read / cache-creation tokens at INFO level so we can audit cost. Filenames are redacted in these (and inbound-attachment) log lines via
apps/common/logging.py:redact_filename— they routinely carry personal data ("Jane_Smith_payslip.pdf") andfly logsretains whatever we emit, so we log a stable short hash plus the bare extension instead of the name. - Uploaded files and thumbnails are served through household-scoped views, never
MEDIA_URL.MEDIAisn't served in prod (DEBUG=False) and a raw/media/URL would be world-readable across tenants. Sodocuments:file_viewanddocuments:thumbnailstream the bytes from a@login_requiredview that filters byrequest.householdand 404s otherwise. Page-1 thumbnails (generated async bygenerate_thumbnail, blank until the worker runs) render in the shared_partials/document_row.htmllisting and on the confirm page, falling back to a generic icon when absent. - Inline serving is allowlisted to prevent stored XSS.
file_viewonly sendsContent-Disposition: inlinefor a narrow set of safe types (INLINE_SAFE_CONTENT_TYPES: PDF + common raster images). Anything else — including an uploaded or emailed-in.html/.svg, which would otherwise run script in the app's origin against whoever opens it — is forced to download with a neutralapplication/octet-streamtype. Every response also carriesX-Content-Type-Options: nosniffso the browser honours the declared type rather than guessing. - Thumbnail generation is bomb-guarded.
generate_thumbnailcapsImage.MAX_IMAGE_PIXELSand treats Pillow's decompression-bomb error and warning as a skip, so a few-KB malicious image can't expand to gigapixels and exhaust the 512 MB worker. - Rate limiting is two-layered. Login brute-force goes through django-axes (DB-backed so lockout state is shared across gunicorn workers): 5 failures per (username, IP) → 1-hour 429. Signup and invite-accept use a lighter per-IP fixed-window counter in the default cache (
apps/common/throttle.py) — LocMem per-process today, which is fine at these limits on a single machine. Client IP comes from Fly'sFly-Client-IPheader (client_ip), shared by both layers. - Email changes are verified.
/account/email edits mail a signed confirmation link (django.core.signing, 3-day expiry, salt-bound) to the new address; the change applies only when the same signed-in user opens it, with uniqueness re-checked at apply time. Outbound email is env-driven (EMAIL_BACKEND— console in dev; point it at SMTP + creds in prod, e.g. Postmark outbound).