Architecture¶
HTMX templates Ninja API endpoints (grow as needed)
\ /
\ /
Services layer (thin views call into this)
|
v
Django ORM
|
v
SQLite (Fly volume) → Postgres when needed
Server-rendered templates with HTMX for partial updates, Alpine.js for small client-side state (dropdowns, modals). Services are plain Python functions in apps/<app>/services.py. Models are Django ORM with custom managers for household scoping.
Apps¶
| App | What lives here |
|---|---|
apps/common/ |
Middleware (current-household context), household-scoped managers, encrypted text field, shared context processors, static template checks |
apps/accounts/ |
Custom User model — email is the login identifier (username retained as an optional legacy handle). Auth runs on django-allauth (email/password + Google sign-in, signup, email management). SocialAccountAdapter preserves the email-verified linking guard; the user_signed_up signal bootstraps a household. See Authentication. |
apps/households/ |
Household, HouseholdMembership, Person; household-bootstrap service; dashboard view; invite tokens |
apps/auth_google/ |
Retired as a sign-in path (allauth owns Google now). The GoogleIdentity model/table is kept one release as a rollback net for the GoogleIdentity → SocialAccount data migration. |
apps/documents/ |
Document model (with source field flagging upload / email_in / gmail_triage to drive per-source review surfaces); upload + confirm views; topic / category / search / timeline / person / named-item views; bulk operations; auto-assign; activity timeline |
apps/classifier/ |
Claude Sonnet integration. Stub mode keyword-based; live mode sends pdfplumber-extracted page-1 text inline with the email context, with prompt caching. Page-1 text replaces base64 PDF binaries as a measured ~5x cost reduction with no precision/recall hit (validated on the spike's hand-labelled set). Word and Excel attachments get the same treatment (#137): _docx_text (python-docx; paragraphs + table cells, since fees/dates/grades live in tables) and _xlsx_text (openpyxl; first sheet, first ~50 cells) extract text locally and embed it in the same untrusted-content block — no binary is ever sent, so the cost posture is identical to the PDF path. Legacy .doc/.xls are OLE2, which neither library reads; they're labelled legacy format — text extraction not supported so Claude knows a file was attached rather than judging as if there were none. Format is decided by magic bytes (OOXML is a ZIP container) over the declared MIME type, because senders mislabel attachments routinely. Other non-PDF and parse-failure cases degrade to email metadata + filename signal alone — including syntactically-valid JSON that isn't an object (assessment M2). Scanned/image PDFs no longer degrade: when page-1 extraction yields <50 chars, page 1 is rendered to a PNG (pypdfium2 via pdfplumber) and sent as a vision block (#215) — typed PDFs never pay for vision. The assembled system prompt also carries three worked few-shot examples (keepable final / marketing / draft-vs-final, #216), appended at assembly time like the injection guard. Attacker-controlled content (email From/Subject/snippet + attachment page-1 text — anyone can mail a connected inbox or a forward address) travels between BEGIN/END UNTRUSTED CONTENT markers, and an injection guard appended to the system prompt at assembly time tells Claude to treat it as data only, never instructions (assessment M1); SPIKE_PROMPT_V2 itself stays byte-for-byte the spike's. Doctype vocabulary is loaded from apps/classifier/taxonomy.yaml via taxonomy.py — single source of truth with three columns per doctype (classifier_examples, heuristic_fingerprint, suggested_tags); the latter two are populated in subsequent slices. |
apps/emailscan/ |
Gmail OAuth + scan pipeline. Heuristic pre-filter, thread + content-hash dedup, recurring-email dedup, Claude classification, lifecycle bucketing. Scans run via the run_gmail_scan django-tasks task with three callers — OAuth callback (first scan), weekly cron, manual "Scan now" button — all funnelling through enqueue_scan which holds the single rate-limit guard (scan_state != "running"). Ongoing scans pull a delta window via scan_window_start (last_scan_at − 3 days overlap). Also the Postmark inbound webhook — persists the raw payload as InboundEmailPayload and enqueues process_inbound_email so the webhook returns <100ms regardless of attachment count (issue #85). |
apps/triage/ |
Triage UI: two-tab nav (Gmail findings + Email-in Review). Gmail tab is the original accept/dismiss queue; Email-in Review lists already-filed source='email_in' docs that the user hasn't confirmed yet — Confirm sets reviewed_at, Edit opens the standard per-doc form. Per-tab count chips + summed nav badge. Adapter delegates to emailscan in prod, stub in tests. |
apps/profiles/ |
PropertyProfile + VehicleProfile sidecars. Schedule definitions for houses (schedules/house.py) and vehicles (schedules/vehicle.py). DVLA + DVSA lookup service. |
apps/activity/ |
ActivityEvent model + per-event undo (auto-assign undo, confirm-review undo). |
apps/renewals/ |
The proactive layer's read side (#173/#175): upcoming_renewals() computes what's due from Document.expiry_date, recurrence cadences rolled forward from issue_date, and VehicleProfile MOT/tax dates — deduped (newest doc per doctype slot; same provider + same item + dates within 31 days collapse to one obligation), banded (soon ≤30d / upcoming ≤90d / later; past = a gentle "looks due"). RenewalRule records dismiss/snooze curation, soft-revoked and undoable. Surfaces: the dashboard heroes' at-a-glance rows, the house/vehicle page feeds, /renewals/ (maintenance calendar) and /renewals/rules/. Delivery (digest email) is deferred — #176. |
apps/api/ |
Django Ninja API surface. One /api/health endpoint today; grows as consumers need them. |
Authentication¶
Sign-in runs on django-allauth (issue #142), which consolidates password login, Google sign-in, signup and email management onto one maintained library. This is the foundation for MFA (#143) and passkeys (#144), which become config + template changes on top of it.
- Custom user model (
apps/accounts/User): email is theUSERNAME_FIELDand is unique;usernameis kept as an optional, nullable legacy handle (not the credential). The model reuses the existingauth_usertable (db_table = "auth_user") — the swap was done in place, preserving every account and password hash, with no cross-table data copy. - URLs: the legacy
/login/ /signup/ /logout/paths are preserved (served by allauth views); allauth's own routes (email, password reset, confirmations, social callback) live under/accounts/. - Household bootstrap: the
allauth.account.signals.user_signed_upsignal callsbootstrap_new_user_household, so both email and Google signups still create a Household + OWNER membership + SELF person. This is the multi-tenancy join point. - Google linking guard:
apps/accounts/adapters.SocialAccountAdapter.pre_social_loginonly auto-connects a Google login to an existing account when Google reports the email as verified — preserving the guarantee the retiredGoogleOAuthBackendprovided. - Brute-force lockout:
django-axesstays first inAUTHENTICATION_BACKENDS;AXES_USERNAME_FORM_FIELD = "login"points it at allauth's email field. Signup rate-limiting moved to allauth'sACCOUNT_RATE_LIMITS. - Two-factor auth (
allauth.mfa, issues #143/#144): opt-in passkeys (WebAuthn), TOTP, and recovery codes (MFA_SUPPORTED_TYPES = ["totp", "webauthn", "recovery_codes"], issuer "Maud"), enrolled from the account page, which links to allauth's/accounts/2fa/views. Once enrolled, allauth challenges for the second factor after primary auth; axes still guards the password step. Passwordless passkey login is on (MFA_PASSKEY_LOGIN_ENABLED = True): the login page has a "Sign in with a passkey" button (allauth'smfa_login_webauthnflow) so you can sign in with just a passkey — no identifier typed. New passkeys default to discoverable resident credentials so the no-identifier flow can find them (MFA_FORMS["add_webauthn"] = apps.accounts.forms.AddPasskeyFormpre-checks allauth's "passwordless" option). The email/password form stays as a fallback, so removing your last passkey can't lock you out. The WebAuthn Relying Party ID is derived from the request host (maud.fly.devtoday; a later move toheymaud.comwould require re-enrolling passkeys).MFA_WEBAUTHN_ALLOW_INSECURE_ORIGINis on only in local dev. allauth's WebAuthn JS is served by WhiteNoise (collected at build). - Gmail triage OAuth (
apps/emailscan) is a separate flow and is unchanged — it only shares the Google OAuth client with sign-in.
Data model highlights¶
- Household / Person / NamedItem — the multi-tenant root. Every Document, EmailScanFinding, ActivityEvent has a
householdFK.NamedItem.kindishouseorvehicle.NamedItemediting surfaces a kebab actions menu next to the page title via the sharedtemplates/_partials/entity_actions_menu.htmlpartial: Rename (POST/documents/named-items/<id>/rename/→rename_named_itemservice, pre-checks theunique_together = ("household", "name", "kind")constraint and raisesValueErrorinstead ofIntegrityError, recordsnamed_item_renamed); Same as another … (POST/documents/named-items/<id>/merge/→merge_named_itemsservice, reassigns linkedDocument.named_itemfrom source to target, optionally renames target to a user-chosen canonical name, deletes source, recordsnamed_items_merged); and Delete (POST/documents/named-items/<id>/delete/→delete_named_itemservice, unlinks linked Documents viaDocument.named_item.on_delete=SET_NULLrather than cascading, recordsnamed_item_deleted). The same partial now serves Person pages too (issue #99): Rename (POST/documents/people/<id>/rename/→rename_person) and Delete (POST/documents/people/<id>/delete/→delete_person, which reassigns the person's Documents to the household's SELF person inside a transaction). The SELF person and any Person with alinked_usercan't be deleted — removing a member who can sign in is an access decision (revoke their membership), not a tile deletion (issue #108). Joining a household via invite creates a linked Person (ensure_member_person), so every member has exactly one Person withlinked_userset iff they can authenticate. People are first-class on the dashboard (issue #112): a pill strip for solo households graduating to tiles with per-person document counts, pluscreate_personbehind/documents/people/new/.Document.personis user-editable via the "Filed under" person chip — by the file preview on the confirm form, beside the Topic chip on triage review (a filing decision, not document metadata, so it renders as switchable chrome rather than a form field; the chip drives a hiddenperson_idinput) — the classifier's free-textperson_subjectguess is persisted on the Document and pre-selects the picker viaresolve_person_subject, but is never silently honoured; pickerless paths (upload, inline accept) attach to SELF.PersonProfile(issue #116) is the 1:1 facts sidecar on Person —date_of_birthplus a schema-lightfactsdict (curated labels inFACT_LABELS, custom keys allowed) edited from the person page; activity events record fact labels only, andbuild_classifier_contextnever reads profile values. The review screen's unregistered-person hint is actionable (issue #117): an inline mini-form (HTMLform=attribute, since the review page is one big form) POSTs totriage:add_person_from_hint, duplicate-guarded viaresolve_person_subject; the confirm-form variant links to/documents/people/new/?name=…&next=…with an open-redirect-guarded return. The account page (/account/, issue #114) holds login-level details — display name (written toUser.first_nameand the SELF Person together), email, password (Django's PasswordChangeView), connected Google — and links the household settings pages (invites, suppression). - Document — the unit of filing. Has
topic,document_type_id,provider,issue_date,expiry_date,lifecycle_state(internal — not user-visible),filing_status,reviewed_at,named_item,person, an M2M toTag, andproposed_tags(JSONField — AI suggestions awaiting user review, cleared when the confirm form is saved). Carries the AI's proposal AND the user's eventual confirmation. - Tag — household-scoped, free-form,
name+slug. Cross-cutting axis the user defines themselves (kids, rental, claim-2025). Slug is derived from name and is the URL/uniqueness key — "Kids" and "kids" collapse to the same Tag. Service helpers inapps/documents/tagging.py. The classifier proposes 2–5 tags per document on upload / email-in / triage-accept; suggestions render as one-click ✨-chips on the confirm form alongside chips for existing household tags the user can reuse. - EmailScanFinding — a candidate document found in Gmail awaiting Triage decision. Holds the AI's proposed metadata, plus enough Gmail identifiers (
gmail_message_id,attachment_id) to fetch the actual file bytes later. On accept it becomes a Document; in the future the bytes will be downloaded ontoDocument.fileat that point. - PropertyProfile / VehicleProfile — 1:1 sidecars on NamedItem. Property is wizard-driven (country + ownership/tenure/type/age/facts — the country step branches the questions and the checklist, issue #94); Vehicle is lookup-driven (DVLA + DVSA from the registration plate).
- Invite — server-side household invitation (issue #98): UUID-addressed, single-use, revocable, 14-day expiry, optional
invited_emailbinding. Minted from/household/invites/, accepted at/join/<uuid>/(which also creates the member's linked Person), revoked from the same settings page. Replaced the signed bearer token; the oldinvite_urlcontext processor is gone so no page render carries a join link. - DismissedSignal — learned dismissal suppression (issue #93). Dismissing a finding records/increments a household-scoped fingerprint (sender domain, normalised subject, attachment SHA-1). At
dismiss_count >= 2, matching scan candidates are dropped before the classifier runs. Listed + revocable at/triage/suppression/. - ActivityEvent — append-only log;
is_undoableflag drives the timeline Undo affordance.
The classification pipeline¶
Upload Gmail Triage
| |
v v
Claude (live mode) Heuristic pre-filter (per-doctype fingerprints + household context tokens)
| |
v v
ClassifierProposal Thread + content-hash + recurring-email + learned-suppression dedup
| |
v v
Document (filing_status=KEPT, Claude (live mode, with email context + lazy-fetched bytes)
reviewed_at=null) |
v
topic=null suppressed; the rest →
EmailScanFinding (status=pending) → Triage UI → Document
Both paths share the classifier service. Both produce a Document with the same shape.
The taxonomy file¶
apps/classifier/taxonomy.yaml is the single place a document type is defined (issue #96) — 104 doctypes across the nine topics, each with three columns: classifier_examples (fed to the prompt vocabulary), heuristic_fingerprint (keywords + sender domains the pre-Claude scoring layer matches), and suggested_tags (tag priors surfaced at confirm time). Add a doctype once and all three behaviours pick it up. python manage.py generate_doctype_fingerprints drafts fingerprints for new doctypes (offline-deterministic by default, --live asks Claude) — review the YAML diff before committing.
The heuristic + the gate¶
Scan-time scoring (apps/emailscan/heuristic.py) matches each candidate against every doctype fingerprint: sender-domain hit +2, keyword hit +2 in the subject / +1 in the body, capped at 4 per doctype, summed across the catalog; plus +3 per household-context token and +2 for a useful attachment, threshold ≥ 5. "Useful attachment" (USEFUL_MIME_PREFIXES) means PDF, image, Word or Excel — it is a hard gate, not just a score contribution: is_candidate returns False without one whatever the score. Word/Excel were absent until #137, so a solicitor's .docx engagement letter could never become a candidate at any score. The tuple is deliberately shared with _parse_gmail_message, which held a second hardcoded copy of the same prefixes and dropped the attachment at parse time even once the heuristic knew about it. Household context tokens are derived at call time from registered data (apps/emailscan/context.py, issue #95): house names + postcodes, vehicle names + plates, family names, employer hints from connected email addresses. The same derived context is injected into the classifier prompt as a HOUSEHOLD CONTEXT block, along with the household's existing tag names (tag priors, slice 2e). Post-classification, refine_suggested_tags tops up suggestions from the matched doctype's curated suggested_tags — skipped when the classifier's doctype confidence is low, since wrong-doctype priors read as irrelevant recommendations (#119) — and drops tags that duplicate structured fields. On the confirm form only the AI's per-document ✨ suggestions render as chips — existing household tags surface through the AI's reuse-verbatim prompt or the input's autocomplete datalist, never as a chips row of everything. The confirm page's category field is a dropdown whose options narrow live to the selected topic (taxonomy-sourced via a json_script map; legacy free-text values stay selectable), expiry/recurrence inputs only render when they carry a value (quiet '+ add' toggles reveal them), and the document header renders on the navy shell: thumbnail (click to open) + "Filed under" chip + a quiet download link — no white file row.
The classifier runs the spike-validated 9-topic prompt (SPIKE_PROMPT_V2, Phase C cutover): it returns topic ∈ the nine topics or null. topic=null candidates are suppressed at scan time — no finding row reaches the triage UI — which is what lets Medical / Finance / Family-IDs findings surface instead of everything being squeezed through the old is_property_relevant boolean. Confidence buckets findings within a topic (a low-confidence "final" lands in Needs your eye, not Signed and final).
The schedule + completeness panel¶
Lives in apps/profiles/services.py:
_schedule_for(named_item)returns the right schedule list for the kind (house or vehicle) — and, for houses, for the property's country: UK houses get the full TA6-shaped schedule, French houses getschedules/france.py(titre de propriété, taxe foncière, DPE, diagnostic immobilier…), anywhere else gets a short generic list (issue #94)._profile_for(named_item)returns the right profile sidecar.evaluate_schedule(named_item)matches the schedule against the household's filed documents, classifies each entry ashave/missing/not_applicable, and returns counts + a percentage.
Schedule items are dataclasses with a condition callable that reads the profile's facts dict — that's how the MOT slot gets suppressed for vehicles under 3 years old, or how leasehold-only items get hidden for freehold houses.
The operator backend¶
The Django admin (/admin/) is restyled to the front-end design system via templates/admin/base_site.html, which overrides the admin's CSS custom properties (navy chrome, teal accent, light high-contrast content) — no admin templates are forked beyond that thin shell, so it survives upgrades. Every colour pair is WCAG-AA-checked (body text ≥ 4.5:1); the load-bearing rule, shared with the front end, is that teal is too pale for white text, so admin buttons use navy text on teal and body links use a darker teal (#0A6E62), never the pale accent. The brand variables are applied to the light/dark theme selectors alike so the theme toggle and OS preference both resolve to the one accessible theme.
Running Maud as a business lives in two apps — see Operator console for the full reference:
apps/billing—Plan(seeded Free / First Sweep / Ongoing) +Subscription(one per household; absent row = default plan). Limits are display-only (limits_enforcedoff); Stripe is planned in viastripe_*fields and a 501-until-configured webhook stub at/billing/webhooks/stripe/.apps/ops— the staff-only console at/ops/(dashboard, tenant directory + detail, plan management, suspend/reactivate/wipe, read-only view-as) and the append-onlyOpsAuditLog.OpsMiddleware(afterCurrentHouseholdMiddleware) implements the view-as household override and the suspension gate.- Metering — every live classifier call writes a
ClassifierUsagerow (household, source, token counts; never content), priced by an in-code table so the console reports Claude cost per tenant per month. Suspended households are refused byrun_scanand the inbound-email task.
Multi-tenant isolation¶
The HouseholdScopedManager reads the current request's household from a thread-local set by apps/common/middleware.py:CurrentHouseholdMiddleware. Every default Document.objects.all() is therefore implicitly scoped — a view that forgets to filter still won't leak.
Document.all_tenants is the explicit cross-household manager; reserved for management commands and audit code.
Contract tests in apps/common/tests/test_isolation.py and per-app test files exercise both correctness and the failure case (cross-household reads must return empty).
Interesting concrete details¶
Document.reviewed_atis stamped by both the per-block "Confirm all" and by bulk-Move-to-property — moving a doc is an assertion that it belongs there, no second-confirm step needed.- Topic page is a review queue. Property blocks only render if they have at least one unreviewed doc. Confirmed blocks vanish until new docs arrive. Browse-all-of-this-property's-docs lives on the property detail page.
- Email scan dedup is four-keyed: by
gmail_message_id(re-runs), by(thread_id, attachment_name)(forwarded threads), byattachment_sha1(forwarded duplicates across threads), by(from_header, normalised_subject)(recurring marketing). On top of that, learned suppression (DismissedSignal) silently drops anything the household has dismissed twice — before the classifier is paid to look. - Per-document size cap.
settings.MAX_DOCUMENT_BYTES(25 MB) is enforced on every Document-create path: uploads get a 413, email-in attachments are skipped withskipped_count++. Django'sFILE_UPLOAD_MAX_MEMORY_SIZE/DATA_UPLOAD_MAX_MEMORY_SIZEare pinned to the same value. - Triage-accepted documents include the file bytes.
accept_findingcallsattachments().get()against the captured(gmail_message_id, attachment_id)and writes the result ontoDocument.file. Failure (revoked OAuth, deleted message, network) is non-fatal — the Document persists metadata-only and the user can re-runpython manage.py backfill_triage_fileslater, which retries every empty-file Document linked to a finding. - Boot-time migrations.
entrypoint.shrunspython manage.py migrateagainst the volume-backed DB on every container boot. Fly'srelease_commandruns on a separate machine without the volume — that path silently no-op'd against an empty DB until we noticed. - Background task queue. Async work uses
django-taskswith the db-backeddjango-tasks-dbbackend. Dev / tests default toImmediateBackend(synchronous in-process). Prod setsTASK_BACKENDto the DB backend infly.tomland runspython manage.py db_workeras a co-process alongside gunicorn (entrypoint.sh). The worker shares the SQLite volume — co-located until the Postgres migration lets us split it out onto a separate machine. - Token usage logging. The classifier logs every Claude call with input / output / cache-read / cache-creation tokens at INFO level so we can audit cost. Filenames are redacted in these (and inbound-attachment) log lines via
apps/common/logging.py:redact_filename— they routinely carry personal data ("Jane_Smith_payslip.pdf") andfly logsretains whatever we emit, so we log a stable short hash plus the bare extension instead of the name. - Uploaded files and thumbnails are served through household-scoped views, never
MEDIA_URL.MEDIAisn't served in prod (DEBUG=False) and a raw/media/URL would be world-readable across tenants. Sodocuments:file_viewanddocuments:thumbnailstream the bytes from a@login_requiredview that filters byrequest.householdand 404s otherwise. Page-1 thumbnails (generated async bygenerate_thumbnail, blank until the worker runs) render in the shared_partials/document_row.htmllisting and on the confirm page, falling back to a generic icon when absent. - Inline serving is allowlisted to prevent stored XSS.
file_viewonly sendsContent-Disposition: inlinefor a narrow set of safe types (INLINE_SAFE_CONTENT_TYPES: PDF + common raster images). Anything else — including an uploaded or emailed-in.html/.svg, which would otherwise run script in the app's origin against whoever opens it — is forced to download with a neutralapplication/octet-streamtype. Every response also carriesX-Content-Type-Options: nosniffso the browser honours the declared type rather than guessing. - Thumbnail generation is bomb-guarded.
generate_thumbnailcapsImage.MAX_IMAGE_PIXELSand treats Pillow's decompression-bomb error and warning as a skip, so a few-KB malicious image can't expand to gigapixels and exhaust the 512 MB worker. - Rate limiting is two-layered. Login brute-force goes through django-axes (DB-backed so lockout state is shared across gunicorn workers): 5 failures per (username, IP) → 1-hour 429. Signup and invite-accept use a lighter per-IP fixed-window counter in the default cache (
apps/common/throttle.py) — LocMem per-process today, which is fine at these limits on a single machine. Client IP comes from Fly'sFly-Client-IPheader (client_ip), shared by both layers. - Email changes are verified.
/account/email edits mail a signed confirmation link (django.core.signing, 3-day expiry, salt-bound) to the new address; the change applies only when the same signed-in user opens it, with uniqueness re-checked at apply time. Outbound email is env-driven (EMAIL_BACKEND— console in dev; point it at SMTP + creds in prod, e.g. Postmark outbound).
Guard-rails added by the 2026-08 assessment remediation¶
all_tenantsis ratcheted.apps/common/tests/test_all_tenants_usage.pyfails CI if a file outside its audited allowlist referencesall_tenants— new request-path code must use the household-scoped default managers (Model.objects). Migrating the grandfathered files is tracked in issue #199.ActivityEvent.Kindis the documented vocabulary of every kind string production emits (record()takes plain strings; the enum documents, not validates). If you emit a new kind, add it to the enum.- Docs-drift mapping now covers auth (
apps/accounts), the taxonomy files, household membership/roles, activity, the emailscan plumbing (cron/tasks/context/models), andapps/common— the load-bearing areas the original mapping missed (assessment L7).