Triage pipeline¶

How a Gmail message becomes a triage finding, and how a finding becomes a Document. This page is the visual companion to the prose in Architecture — same pipeline, drawn end-to-end with a clear split between deterministic Python and the single LLM call.

File and line references reflect the code at the time of writing; the structure (heuristic gate → dedup → classifier → finding → user accept/dismiss) is stable, but exact line numbers may drift.

End-to-end diagram¶

flowchart TD
    %% Entry points
    U1([User clicks Connect Gmail]):::user --> O1
    U2([Manual / future cron rescan]):::user --> S1

    %% OAuth kickoff
    O1[OAuth callback<br/>oauth.py:94]:::code --> O2[Spawn daemon thread<br/>oauth.py:162]:::code
    O2 --> S1

    %% Gmail fetch
    S1[run_scan<br/>emailscan/services.py:203]:::code --> S2[List messages<br/>q=has:attachment, since=180d<br/>services.py:466]:::api
    S2 --> S3[Parse headers + MIME parts<br/>services.py:502]:::code
    S3 --> S4{"attachment &gt; ~10KB?"}:::code
    S4 -->|yes| S5[Fetch bytes via attachment_id<br/>services.py:142]:::api
    S4 -->|no, inline base64| S6[ScanMessage dataclass]:::code
    S5 --> S6

    %% Heuristic gate
    S6 --> H1[Score = domain + keywords<br/>+ context tokens<br/>+ attachment present<br/>heuristic.py:150]:::code
    H1 --> H2{"score &ge; MIN_SCORE 5<br/>+ useful PDF/image?"}:::code
    H2 -->|no| DROP1[Drop, no LLM call]:::drop

    %% Dedup
    H2 -->|yes| D1[4-way dedup vs household findings:<br/>gmail_id / sha1 /<br/>thread+name / from+norm_subject<br/>services.py:336]:::db
    D1 --> D2{Already seen?}:::code
    D2 -->|accepted or dismissed| DROP2[Skip]:::drop
    D2 -->|pending, newer copy| D3[Replace pending row]:::db
    D2 -->|new| C1
    D3 --> C1

    %% Classifier — the only LLM call
    C1[Build classifier prompt:<br/>filename + MIME + pdfplumber page-1 text,<br/>subject/from/snippet,<br/>household named items<br/>classifier/services.py:230]:::code
    C1 --> C2{{"<b>LLM call</b><br/>claude-sonnet-4-5<br/>system prompt cached<br/>classifier/services.py:250"}}:::llm
    C2 -->|success| C3[Parse JSON proposal:<br/>topic, doctype, lifecycle_state,<br/>is_worth_surfacing,<br/>is_property_relevant,<br/>title, provider, dates, tags]:::code
    C2 -->|API fail / bad JSON| C4[classify_stub<br/>regex on filename<br/>classifier/services.py:269]:::code
    C4 --> C3

    %% Persist
    C3 --> P1[Create EmailScanFinding<br/>status=PENDING<br/>proposed_metadata=full proposal<br/>services.py:399]:::db

    %% Triage UI
    P1 --> V1["/triage/ overview<br/>triage/views.py:24"]:::code
    V1 --> V2[Bucket by lifecycle_state +<br/>is_worth_surfacing +<br/>is_property_relevant<br/>services.py:669]:::code
    V2 --> V3[Render 4 buckets:<br/>signed / look / draft / low]:::code

    %% User decision
    V3 --> UA{User decision}:::user
    UA -->|Accept| A1
    UA -->|Dismiss| DM1

    %% Accept path
    A1[accept_finding<br/>merge user edits into proposed_metadata<br/>services.py:748]:::code
    A1 --> A2[_create_document_from_finding<br/>maps proposal to Document fields<br/>get_or_create NamedItem<br/>services.py:924]:::db
    A2 --> A3[Re-download attachment<br/>services.py:877]:::api
    A3 --> A4[Save bytes to Document.file<br/>finding.status=ACCEPTED]:::db

    %% Dismiss path
    DM1[dismiss_finding<br/>status=DISMISSED<br/>activity event<br/>services.py:1011]:::db

    %% Legend styles
    classDef user fill:#d4f4dd,stroke:#2d7a3e,color:#1a3d20
    classDef code fill:#e8eefc,stroke:#3b5bdb,color:#1c2c66
    classDef api fill:#fff3bf,stroke:#b08900,color:#5c4400
    classDef db fill:#cfe7ff,stroke:#1e40af,color:#1e3a8a
    classDef llm fill:#ffd6a5,stroke:#d97706,color:#7c2d12,stroke-width:3px
    classDef drop fill:#f3f4f6,stroke:#9ca3af,color:#4b5563,stroke-dasharray: 3 3

Legend¶

What's code, what's the LLM¶

• Exactly one LLM call in the pipeline. apps/classifier/services.py:250, model claude-sonnet-4-5. Everything else is deterministic Python or a Gmail/database round-trip. The roadmap item to switch to claude-haiku-4-5-20251001 would change the model id at this single site.
• The heuristic gate is a hard pre-filter. apps/emailscan/heuristic.py:192 — anything below MIN_SCORE = 5 never reaches Claude. Outlook signature image001*.jpg files and oversized camera photos are rejected by filename pattern before scoring. This is the cost lever today.
• Dedup is also pre-LLM. Four checks against the household's existing findings (apps/emailscan/services.py:336): gmail_message_id exact match, attachment_sha1 hash match, (thread_id, attachment_name) pair, (from_header, normalized_subject) recurring series. The first backlog item in the Triage cluster extends this to also check existing Document rows, not just other findings.
• Classifier output is stored verbatim. EmailScanFinding.proposed_metadata is the entire ClassifierProposal dict serialised as JSON. The booleans is_worth_surfacing and is_property_relevant live under raw_signals inside that dict.
• Only page 1 of each PDF is sent to the LLM. apps/classifier/services.py:374 extracts text from the first page via pdfplumber and embeds it inline in the user message — earlier production sent the full base64-encoded PDF binary as an Anthropic document block. The page-1-text mode is a measured ~5× cost reduction with no precision/recall hit (validated on the spike's hand-labelled set; page 1 is where document type, sender, and subject almost always live). When pdfplumber can't extract anything (scanned PDFs, parser failure, oversize), the classifier still runs and falls back to email metadata + filename signal alone — the document is classified, just with weaker evidence. Non-PDF attachments (images) take the same metadata-only path. Tuning lever: MAX_ATTACHMENT_BYTES (defaults to 4 MB) caps which PDFs get extracted at all; oversized ones skip extraction.
• Bucketing is deterministic. apps/emailscan/services.py:669 maps (lifecycle_state, is_worth_surfacing, is_property_relevant) to one of four UI buckets. No second LLM pass.
• Stub fallback is regex on filename. If the Anthropic call fails or returns unparseable JSON, classify_stub (apps/classifier/services.py:269) keeps the pipeline alive with a deterministic guess (e.g. "boiler" → boiler service cert, "insurance" → buildings policy). Tests run with the stub by default.
• Accept does the real work. accept_finding merges the user's edits onto proposed_metadata, then _create_document_from_finding (apps/emailscan/services.py:924) maps proposal fields onto a fresh Document, get-or-creates the NamedItem if property-relevant, re-downloads the attachment from Gmail, and writes the bytes to Document.file. Attachment download failure is non-fatal — the Document still persists.

Where to read the code¶

Trust-aware bucketing and negative heuristic evidence (2026-08)¶

Two quality changes from the triage-pipeline analysis:

• Signed-and-final requires a trust anchor (#217). A final that isn't low-confidence lands in the highest-trust bucket only when the sender's domain has an accepted finding in this household's history or its proposed named item resolves to a registered house/vehicle. With neither signal, it downgrades to Needs-your-eye. Recency is deliberately not a signal — first scans look back months by design. The context is computed once per listing (no per-finding queries).
• Negative heuristic evidence (#218). Identity-free filenames (scan.pdf, document.pdf…) score −2 and bulk-mail platform senders (Mailchimp, HubSpot, SendGrid…) score −3 before the classifier is paid. Taxonomy fingerprint domains already scored +2, so no boost was added. MIN_SCORE stays at 5 — raising it is deferred until validated against the spike archive, because a silently-dropped candidate is invisible to the user.

Both apply to every scan type (first connect, manual "Scan now", weekly cron) — they live in the shared run_scan/list_findings_grouped path. Email forwarding is unaffected by design: a forwarded document is explicit user intent and gets no noise filtering, and filed documents have no triage buckets.

Cross-finding supersession pass (2026-08, #219)¶

Every candidate is classified in isolation, so a DocuSign "Please sign:" draft and its "Completed:" final used to surface as two separate documents — the spike's #1 quality finding. At the end of every run_scan (first connect, manual, weekly cron), resolve_supersessions groups pending findings by (sender domain, subject with reply and workflow prefixes stripped — "Please sign:"/"Completed:" group together); groups with conflicting lifecycle states that include a final go to supersession_verdict (stub: newest final wins deterministically; live: one small Claude call per group, capped at 10 per scan). Superseded members drop to the low-signal bucket with a "Superseded by…" note — still pending, visible, and recoverable, never deleted. Verdict ids outside the judged group are discarded wholesale, so a hallucinated response can't touch other findings. Inbound email forwarding is out of scope: it files one document at a time and has no sibling candidates.

Mailbox identity (#223)¶

EmailConnection.mailbox_address records which inbox the OAuth grant actually covers, captured best-effort from users.getProfile(userId="me") at the start of run_scan whenever the field is blank — so pre-existing connections backfill on their next scan (including manual "Scan now"). Surfaced on the triage banner and Account settings. A service without getProfile (older test fakes) or a failed profile call never breaks the scan; the field just stays blank.

Honest scan-state on failure (#226)¶

run_gmail_scan's error path used to stamp last_scan_at to unstick the UI — silently advancing the delta window over mail the failed scan never read. It now clears scan_started_at instead (same UI effect, no window advance), and auth-shaped failures (RefreshError, invalid_grant, 401s — see _is_auth_failure) flip the connection to Status.EXPIRED. Expired connections: are skipped by request_household_scan and the weekly cron, surface as a Reconnect banner on triage (with owner + mailbox), and can still be disconnected. Reconnecting a non-ACTIVE connection resets last_scan_at, so the first scan after recovery is the full 180-day sweep. Context: Google expires refresh tokens for Testing-status OAuth apps after ~7 days — until the CASA track moves the app to production status, connections structurally die weekly and this machinery is what makes that visible.

Per-scan trace (#139)¶

ScanTraceEntry records one row per message run_scan examines — heuristic score + reasons, then the terminal outcome (junk_attachment / below_threshold / the four dedup kinds / suppressed / already_filed / not_keepable with the classifier's reasoning / finding_created with a finding FK). Metadata only: subject truncated to 200 chars, sender header, filename — never bodies or bytes. Written best-effort (a trace failure never breaks a scan). Surfaced at /triage/scan-trace/ (searchable, linked from the triage banner); rows purge after SCAN_TRACE_RETENTION_DAYS (default 30) via the weekly cron. This is the tool for every future "why didn't my email show up?" — read the row instead of re-deriving the pipeline from cost tiles.

Catalog: employment coverage (2026-08-09)¶

employment_contract joined the legal topic (keywords: employment contract, contract of employment, offer of employment, statement of particulars, terms of employment) — the catalog covered employment's edges (payslips, P60/P45) but not the contract itself, discovered when a real one arrived by DocuSign. Taxonomy.yaml remains the single source: the doctype flows to the prompt vocabulary, heuristic fingerprints, and suggested tags automatically.

Heuristic noise fixes (2026-08-09 trace evidence)¶

The first real scan-trace session exposed three scoring defects, all fixed the same evening: fingerprint keywords now match on word boundaries (mot scored inside "Rightmove"; the wills doctype matched the auxiliary verb "will"); context tokens that appear in the scanned mailbox's own address score nothing (the account holder's name is in their own To-address on every email — a uniform +3 that made MIN_SCORE meaningless); and the trace distinguishes no_useful_attachment (score fine, but link-only email — invisible to the scan by design) from genuinely below_threshold.