FIN Central

TZ-5: Living Intelligence Design — Email Intel Fusion → Commodity Overlay → Drill-Through Wiki

13 min read

TZ-5: Living Intelligence Design — Email Intel Fusion → Commodity Overlay → Drill-Through Wiki

Status: design (2026-05-16). Implementation pending in T7 follow-ups + EIF Phase 2.

Author: Rob Garcia, with Claude Code session 01.

Authority basis: Engineering Spec [§3.4 mixed-section model], [§12.5 embedded charts], [§13.6 cite-then-claim], [ADR-006 two-layer rendering], [project_living_wiki_drill_through_vision memory], [project_email_intel_fusion_program memory], [project_world_class_search_drill_through memory].

1. Vision Statement

A salesman types “avocado” in the FIN search bar. The drill-through page is not a static wiki entry. It is a living intelligence brief that fuses three layers in real time:

LayerSourceExample content for “avocado”
Universal (Plane 3)knowledge.domain=commodity GDF v2.1.1Scientific name Persea americana, AGROVOC URI, post-harvest specs (chilling injury 41°F, freezing point 30.6°F), market share by variety (Hass 95%)
Tenant operationalevents + signal.* + knowledge aggregates”8,855 cases received this week across 6 RP581 samples, 42-44”
Role-aware emphasisactors.context + actor.roleSalesman sees: customer demand snapshot, Blair pricing trend, what’s in stock today.
Warehouse worker sees: where the pallets are, which lots are on hold, what’s in the van.
QC inspector sees: defect history, recent inspection outcomes, pending re-inspections.

Every claim on the page has its own provenance chain. Every chart pulls from live event data. Every photo links back to the lot, the inspection, the receiving event that captured it.

This is the agentic ERP experience. Not navigation. Not a CRUD form. A living wiki that knows what you do, what’s happening right now, and what the world (Blair, USDA, vendor harvest comms) is saying about the commodity in front of you.

2. Why This Design Doc Exists

The architecture of the living intelligence experience touches multiple programs that have so far been designed in isolation:

  • Engineering Spec §3.4 — projected vs authored sections, never mixed
  • Engineering Spec §11.5 — doc-CQRS projection (doc_backlink_projection, entity_doc_projection)
  • Engineering Spec §12.5 — embedded charts (Vega-Lite specs in projected sections)
  • Engineering Spec §13.6 — cite-then-claim contract for agent generation
  • ADR-006 — two-layer rendering: Central=universal Plane 3, Tenant=overlay+fusion
  • EIF Program — email intel fusion (intel@fintail.net → signal.* events)
  • Living-wiki memory — drill-through fuses KB + tenant operational + role emphasis
  • World-class search memory — search and drill-through are decision-augmentation, not navigation
  • Anonymization Policy v1 — k≥5 cross-tenant aggregates, hashed vendor IDs, ISO-week resolution

Without a single document mapping HOW these compose into one user-visible experience, future sessions will re-derive the architecture and risk drifting.

This doc is that map.

3. The Three-Layer Composition

3.1 Layer 1 — Universal (Plane 3)

Source of truth: knowledge.domain = commodity (Central FIN) + _taproot_mirror/ (read-only sync to tenants).

Rendered by: commodity_profile_renderer (today, ships v1.0.0) → fin-central-intranet/docs/entity_profiles/commodities/<id>.md.

Content: Identity & Botany (scientific name, classification, AGROVOC), Varieties (cultivar list with market share), Post-Harvest Handling (chilling injury, freezing point, ethylene class, mixed-load groups), Market Intelligence (pack sizes, demand elasticity, organic profile).

Confidence: external_authoritative for identity/botany (sourced from FAO AGROVOC + GDF curation); observed for varieties + post-harvest + market intel (knowledge-forge GDF v2.1.1 data extraction).

Visibility: tenant_private at the doc level; data_plane: universal_intelligence (federation-eligible). Tenants mirror via Taproot sync, not direct read.

Mutation cadence: rare. New AGROVOC URIs, GDF re-curation events, post-harvest spec updates from FDA/USDA. Snapshot pattern (§11.6) preserves W19/M05 immutable versions.

3.2 Layer 2 — Tenant Operational Overlay

Source of truth: tenant-side events table filtered by commodity_id + tenant-side knowledge aggregates (vendor scorecards, customer demand curves) + _taproot_mirror/ for universal facts.

Rendered by: commodity_overlay_renderer (PENDING — T7 follow-up; design here) → nathel-intranet/docs/commodities/<id>_overlay.md.

Content sections (per ADR-006, with implementation specifics added below):

SectionSource eventsRender shape
Recent Arrivalslot.received filtered by commodity_id, last 30 daysChart (Vega-Lite line: cases/day) + table (last 10 lots with vendor, brand, count, dock)
Pricing Trendsignal.market_pricing_observed (Blair) + sales.invoice_generated filtered by commodityTwo-line chart: Blair wholesale benchmark vs Nathel sell-through; per-variety drill-down
Vendor Availabilitysignal.vendor_quote_observed (intel@ extracted vendor offers)Table: vendor, variety, pack size, price, harvest window, last-quoted date
Active Inspectionsqc.inspection_completed + qc.inspection_failed filtered by commodity, last 14 daysTable linking to lot, inspector, defect summary, hold status
Best Recent Lots Galleryartifact.uploaded events with entity_type=lot, commodity_id=<id> filtered to inspection photosPhoto grid (customer-visibility-stripped per CLAUDE.md HARD RULES — no vendor names, no cost annotations)
Customer Demand Snapshotsales.order_received + sales.quote_requested last 7 daysAggregates: top 5 customers by case volume, current backlog, repeat-order rate
Operator Notesauthored (per CLAUDE.md author convention)Markdown free-form, version-controlled

Confidence: observed for all event-derived sections (every claim cites event_ids per §13.6); authored for Operator Notes.

Visibility: tenant_private; data_plane: relationship_data (NOT federation-eligible without anonymization gate).

Mutation cadence: continuous. Every lot.received, qc.inspection_completed, signal.market_pricing_observed event invalidates the relevant section’s content_hash. Doc-CQRS rebuild queue (§11.5.3) handles re-render.

3.3 Layer 3 — Role-Aware Emphasis

Source of truth: actors.actor_id, actors.role, actors.memory.context (current shift, current zone, current task).

Applied at: drill-through page composition time (NOT at render time — same overlay markdown, different sections promoted to “above the fold”).

Examples (from project_role_and_actor_aware_pulse memory):

RoleAbove-the-fold sectionsSuppressed/collapsed
Salesman (sales_rep)Pricing Trend, Vendor Availability, Customer Demand Snapshot, Recent Arrivals (cases-available framing)Active Inspections (collapsed unless hold blocking sale)
Box Man (warehouse_worker)Recent Arrivals (with location grid), Best Recent Lots Gallery (showing pack quality)Pricing Trend, Customer Demand (irrelevant to picking)
QC Inspector (qc_inspector)Active Inspections, Best Recent Lots Gallery (defect-focused), Recent Arrivals (with vendor pattern)Customer Demand, Pricing Trend
Foreman (foreman)Recent Arrivals (zone-filtered), Active Inspections rollup, Vendor Availability (incoming pipeline)per-customer demand drill-downs
Exec (admin)Pricing Trend (margin view), Customer Demand Snapshot (aggregate), Vendor Availability (cost view)individual lot/inspection rows (rolled up)

Implementation: Quartz composer reads actor.role from auth context, applies a role-specific section ordering + collapse-state preset. Same overlay doc, different presentation.

4. Email Intel Fusion → Commodity Overlay Pipeline

This is the new pipe. Today, intel@fintail.net receives Blair daily reports, vendor harvest comms, and ad-hoc market chatter, but only the EIF S0–S5 chain processes them. The output (signal.* events) is not yet consumed by any commodity overlay renderer.

4.1 The end-to-end path

intel@fintail.net inbox  (Cloudflare Email Workers)
    ↓
[EIF S0: ingest]        signal.email_received event emitted
    ↓
[EIF S0.5: classify]    signal.email_classified
                        body_kind ∈ {market_report, vendor_quote, harvest_update,
                                     pp_arrival, customer_inquiry, generic}
                        poster_role ∈ {blair, vendor, customer, partner, internal}
    ↓
[EIF S1: extract]       signal.market_pricing_observed
                        signal.vendor_quote_observed
                        signal.availability_observation
                        signal.eta_observation
                        signal.market_chatter
                        signal.pp_arrival_logged
                        (subset of 15 signal.* subtypes — see project_signal_event_family memory)
    ↓
[EIF S3: reconcile]     for vendor quotes: link to existing vendor knowledge row;
                        for arrivals: reconcileArrivalCandidates wires to load.expected
    ↓
[EIF S4: route]         extractor.* consumers + authority-router send signals to
                        appropriate downstream consumers
    ↓
[EIF S5: synthesize]    deriveSeasonalPattern + vendor scorecard updates
                        write back to knowledge.domain={vendor,seasonal_pattern}
    ↓
[NEW — TZ-5 scope]      commodity_overlay_renderer subscribes to signal.* events,
                        marks affected overlay docs `stale` in entity_doc_projection
                        rebuild queue (§11.5.2 status field)
    ↓
[Doc-CQRS]              rebuild queue worker re-renders <commodity>_overlay.md
                        section-by-section (only sections whose source events changed)
    ↓
[Vault commit]          renderer commits to nathel-intranet (via vault webhook in
                        reverse — push not pull); webhook invalidates Quartz cache
    ↓
[Drill-through]         next user navigation to /commodities/avocado sees fresh data

4.2 Per-section refresh triggers

Overlay sectionTriggering eventRe-render scope
Recent Arrivalslot.received, lot.unloaded, inventory.lot_frozenSection only (other sections untouched)
Pricing Trendsignal.market_pricing_observed, signal.price_observation, sales.invoice_generatedSection only
Vendor Availabilitysignal.vendor_quote_observed, signal.availability_observationSection only
Active Inspectionsqc.inspection_completed, qc.inspection_failed, quality.hold_placed, quality.hold_releasedSection only
Best Recent Lots Galleryartifact.uploaded (with commodity-tagged lot scope)Section only
Customer Demand Snapshotsales.order_received, sales.quote_requested, sales.order_cancelledSection only

This per-section model maps directly to Engineering Spec §3.4 (each section is independently rendered with its own body_section_map[].fact_citations) and §11.7 (incremental embedding via content_hash — only sections whose hash changed get re-embedded).

4.3 Provenance chain example (cite-then-claim per §13.6)

A user clicks the “Pricing Trend” chart line for “Hass avocado, week 19, $44.00 wholesale”:

walk_provenance(
  doc_id: "commodity_overlay_avocado",
  section_id: "pricing_trend",
  claim: "Hass W19 wholesale = $44.00"
) →
  fact_citations: [
    {
      event_id: "ev_signal_pricing_2026w19_blair_001",
      event_type: "signal.market_pricing_observed",
      timestamp: "2026-05-12T08:30:00Z",
      data: {
        source: "blair_daily_report_2026-05-12",
        commodity: "avocado", variety: "hass",
        pack: "48s", region: "MX_uruapan",
        wholesale_per_case: 44.00, units: "USD",
        actor_id: "system_eif_extractor"
      }
    }
  ]
  upstream_signal: signal.email_received#ev_email_intel_001
  upstream_artifact: blair_2026-05-12.pdf (S3 key: artifacts/intel/...)

The user can drill from the chart line → the signal event → the email → the original PDF. Every layer has a citation. No claim is unsourced.

5. Drill-Through Page Composition

Composing the universal layer (Central) + overlay (tenant) + role emphasis into one page is a render-time merge, not a build-time merge.

5.1 The composer

A Quartz plugin (fin-living-intel-composer) wraps the default page renderer:

// Pseudo-code — actual implementation in T7+
async function composeCommodityPage(commodityId: string, ctx: AuthContext) {
  const universal = await loadDoc(`_taproot_mirror/commodities/${commodityId}.md`);
  const overlay = await loadDoc(`docs/commodities/${commodityId}_overlay.md`);
  const role = ctx.actor.role;
  const rolePreset = ROLE_PRESETS[role] ?? ROLE_PRESETS.default;
 
  return {
    title: universal.frontmatter.title, // "Avocado — Commodity Profile"
    sections: orderSections(
      [...universal.body_section_map, ...overlay.body_section_map],
      rolePreset.sectionOrder,
      rolePreset.collapsedSections,
    ),
    sideDock: buildSideDock(overlay, role), // ambient intelligence per role
    breadcrumb: ["Commodities", universal.frontmatter.title],
    factCitations: [...universal.fact_citations, ...overlay.fact_citations],
  };
}

5.2 Side-dock: ambient intelligence

Per project_world_class_search_drill_through memory, drill-through pages have a side-dock that surfaces ambient intelligence the agent thinks the user might want next. For commodities:

RoleSide-dock content
Salesman”Customer X usually orders Hass weekly — last order was 14 days ago"
"Blair shows MX prices easing; suggest Y for promo"
"3 customers asked about organic last week — current organic stock: 0 cases”
Box Man”Pallet positions in van V-3 for Hass: A2, A4, B1"
"Lot L-2026-05-15-AVOC-001 due for QC re-inspection at 14:00”
QC Inspector”Last 5 Mission Produce loads had soft-fruit defect rate 3.2% (above your 2.0% threshold) — consider stricter sampling"
"Pending re-inspections: 2 lots (L-2026-05-14-AVOC-002 at 09:00, L-2026-05-14-AVOC-003 at 11:00)“

5.3 Linked records — clickable provenance

Per Rob’s vision: every lot, inspection, photo, sale on the page is a typed wikilink. Clicking → drill into THAT entity’s living intelligence page.

Recent Arrivals table row:
| Date       | Vendor          | Brand    | Cases | Dock | Inspection |
|------------|-----------------|----------|-------|------|------------|
| 2026-05-15 | [[vendor:mission_produce\|Mission Produce]] | [[brand:mission\|Mission]] | 4,200 | [[door:dock_face_3]] | [[qc_case:qc_2026-05-15-001\|✓ Passed]] |

Each typed wikilink resolves via the wikilink resolver (T7 follow-up):

  1. First try: entity_doc_projection lookup for that domain+key → standalone doc
  2. Fallback: anchor on parent doc (e.g., [[variety:hass]] → commodity_profile_avocado#hass)

ADR-004’s embedded-by-default rule is the resolver’s default fallback.

6. Implementation Roadmap

6.1 Done as of 2026-05-15

  • ✅ commodity_profile_renderer v1.0.0 — Central layer Plane 3 (16/16 tests passing)
  • ✅ 87 commodity profile docs rendered in vault
  • ✅ 244 variety profile docs rendered in vault (matches Rob’s 331 corpus target)
  • ✅ Hand-authored avocado.md exemplar shows the fusion vision (TZ-2)
  • ✅ DocContent + EntityProfileContent schemas (packages/shared/src/schemas/contracts/doc.ts)
  • ✅ Doc parser + transactional upsert (T7 §11.5.3)
  • ✅ Migrations 049 (doc_backlink_projection) + 050 (entity_doc_projection)
  • ✅ EIF S0–S5 program scaffolded (per project_email_intel_fusion_program memory)
  • ✅ 15 signal.* event subtypes registered
  • ✅ ADR-006 locks the two-layer pattern

6.2 Pending (this design’s contribution to T7+)

ItemOwnerBlocker
commodity_overlay_renderer module (packages/mcp-server/src/ingestion/renderers/commodity-overlay-renderer.ts)T7 follow-upNone — design complete
Tenant-side bulk overlay seeder (mirror of bulk-render-knowledge-forge.ts for tenant)T7 follow-upNeeds tenant Neon read access from tenant Worker
Doc-CQRS rebuild queue worker (consumes entity_doc_projection.status='stale')T7 follow-upNone — interface declared in entity-doc-renderer-worker.ts
signal.* → overlay.section staleness mapperT7 follow-upEIF S5 must be deployed first
Quartz composer plugin (fin-living-intel-composer)Phase 5 §3 (Rob)Phase 5 §3 + §4 wire-up
Role-preset config (ROLE_PRESETS per §5.1)Phase 5 §3 (Rob)Composer plugin
Side-dock ambient intel modulePhase 6+Composer + signal.* event subscription
Wikilink resolver (entity_doc_projection lookup + fallback)T7 follow-upMigration 050 applied (currently pending)
Vega-Lite chart embedding (§12.5 schema field embedded_charts[])T7 follow-upNone — schema exists, no consumer yet
Photo gallery embedding (extension to schema or linked_artifact_ids[])T7 follow-upSchema decision: new field vs reuse linked_artifact_ids
intel@fintail.net → signal.market_pricing_observed pipeline (extends EIF S5)EIF Continuation routineEIF Phase 2 surface (EIF-RM-030..034)
Anonymization gate (Wave 3 Harvest Directory)DESIGN onlyWave 3 not started
Snapshot pattern for overlays (e.g., avocado_overlay_W19.md immutable)T7+ follow-upNone — §11.6 already supports

6.3 Dependency graph

                Migration 050 applied
                       │
                       ▼
          wikilink resolver implemented
                       │
                       ▼
        commodity_overlay_renderer + worker
              │              │
              ▼              ▼
       EIF S5 deployed   Quartz composer + role presets
              │              │
              └──────┬───────┘
                     ▼
       end-to-end signal.* → overlay → drill-through
                     │
                     ▼
       side-dock ambient intel + chart/photo embedding
                     │
                     ▼
      anonymization gate (Wave 3 cross-tenant aggregates)

7. Open Architectural Questions

These are NOT blockers for T7 follow-up but should be resolved before Wave 3 Harvest Directory. Capturing here so future sessions don’t re-derive.

7.1 Where does the role preset config live?

Options:

  • (a) actors.memory.context.role_preset — per-actor override, defaults from role
  • (b) knowledge.domain=role_config, key=<role> — tenant-wide config
  • (c) Hardcoded in Quartz composer plugin

Recommendation: (b) for the default; (a) for per-actor override. This mirrors how Pulse audience routing works today.

7.2 How fresh is “fresh” for the overlay?

Options:

  • (a) Render on every event (worst case ~10s/min during heavy receiving)
  • (b) Render on a 5-minute debounce per commodity_id
  • (c) Render on user navigation (lazy)

Recommendation: (b) for high-traffic commodities (avocado, strawberry); (c) for long-tail (kohlrabi, quince). Configure via entity_doc_projection.refresh_policy jsonb field.

7.3 Should overlay docs be in fin-central-intranet OR a separate tenant intranet?

Per ADR-006: tenant overlay lands in nathel-intranet, NOT fin-central-intranet. Sovereignty boundary.

Open: does each tenant need its own GitHub repo (<tenant>-intranet)? Or one repo with tenant_id-scoped subdirectories?

Recommendation: Separate repo per tenant. Matches the sovereignty model. nathel-intranet, <future-tenant>-intranet. Vault webhook routes by repo → tenant.

7.4 What happens when Central re-renders avocado.md and tenant overlay is mid-edit?

Scenario: AGROVOC URI updates. commodity_profile_renderer re-runs and re-pushes commodities/avocado.md. Meanwhile tenant overlay renderer was building avocado_overlay.md from a stale view of the universal facts.

Recommendation: Overlay renderer reads from _taproot_mirror/ snapshot at render-start, not live. Mirror sync is its own pipeline (per ADR-006). Composer always shows the latest overlay + the latest universal independently.

7.5 How do we handle commodities WITHOUT a universal Plane 3 profile?

Scenario: Tenant receives cantaloupe_melon_special_variety but no GDF profile exists.

Recommendation: Overlay renders standalone. Composer hides the universal layer panel. Trigger a knowledge-forge ingestion task to add the GDF profile. The 109 unlinked varieties from the bulk render (e.g., Quince) are this case in miniature.

8. Anti-Patterns to Avoid

These were considered and explicitly rejected:

Anti-patternWhy rejected
Render once, never refreshDefeats “living” property; turns into static wiki
One renderer fuses everything centrallyADR-006: tenant data sovereignty boundary
Frontmatter-only data fusion (no body sections)Engineering Spec §3.4: agent reads body sections, frontmatter is metadata only
Inline fact citations as markdown footnotes§13.6 cite-then-claim requires structured fact_citations[] array per section, not freeform
Bespoke React component per commodity pagefeedback_agent_native_drill_through: drill-through must use CardRenderer with tool-sourced CardSpecs OR projected markdown — never bespoke per-entity components
Mutating the projected sections AFTER render§3.4: projected sections are deterministic output of the renderer — mutation breaks reproducibility + provenance chain
Allowing Central to query tenant DB directlyADR-006 + sovereignty: Central never reaches into tenant; tenant pushes anonymized aggregates upward via signal.*

9. Success Criteria

This design is “done” when:

  1. ✅ A salesman can type “avocado” and land on a page with universal facts + RP581 arrivals + Blair pricing chart + customer demand snapshot, all with provenance
  2. ✅ A QC inspector on the same URL sees defect history + recent inspection photos at the top, pricing trend collapsed
  3. ✅ Every claim on the page has a walk_provenance chain back to events / artifacts / external sources
  4. ✅ Updating a Blair PDF in intel@ inbox propagates to the Pricing Trend chart in <5 minutes (debounced refresh)
  5. ✅ The page renders mobile-first (warehouse workers use phones — per project_mobile_is_primary_for_qc memory)
  6. ✅ Cost-basis and vendor-identity hard rules are enforced at render time (not at composer — defense in depth)
  7. ✅ A new tenant onboarding renders the universal layer instantly (Taproot sync) and the overlay layer fills in as their events accumulate
  • Engineering Spec: FIN_INTRANET_HUMAN_AGENT_PROJECTION_LAYER_DESIGN_2026-05-13.md §3.4, §11.5, §12.5, §13.6
  • ADR-006: Tenant Zero rendering strategy — locks the two-layer pattern
  • ADR-004: Variety embedded by default — wikilink resolver fallback
  • ADR-003: Vault authoritative canon — fin-central-intranet is source of truth for universal layer
  • Anonymization Policy: v1 — Wave 3 cross-tenant aggregate gate
  • Hand-authored exemplar: avocado.md — shows the rich-fusion vision in static form
  • Memory pointers (in ~/.claude/projects/-Users-robert-Projects-FIN--03-Dev-fin-agentic/memory/MEMORY.md):
    • project_living_wiki_drill_through_vision.md
    • project_email_intel_fusion_program.md
    • project_world_class_search_drill_through.md
    • project_role_and_actor_aware_pulse.md
    • project_signal_event_family.md
    • project_mobile_is_primary_for_qc.md
    • project_orchestrated_intelligent_frictionless_elegant.md
  • Code anchors (existing):
    • packages/mcp-server/src/ingestion/renderers/commodity-profile-renderer.ts (Central layer, shipped)
    • packages/mcp-server/src/ingestion/renderers/entity-doc-renderer-worker.ts (interface declared, impl pending)
    • packages/shared/src/events/payloads/signal/registry.ts (15 signal.* subtypes)
    • db/migrations/049_doc_backlink_projection.sql
    • db/migrations/050_entity_doc_projection.sql
  • Code anchors (pending):
    • packages/mcp-server/src/ingestion/renderers/commodity-overlay-renderer.ts (T7)
    • packages/quartz/plugins/fin-living-intel-composer.ts (Phase 5 §3)
    • packages/agent/src/wikilink-resolver/index.ts (T7)

Graph View