FIN Central

Recall & Proof Projection Architecture

12 min read

Recall & Proof Projection Architecture

Companion to: docs/architecture/QC_PROJECTION_ARCHITECTURE_2026-05-18.md + docs/architecture/SALES_MARKET_PROJECTION_ARCHITECTURE_2026-05-18.md. Source strategy: docs/product/FIN_DATA_SIGNAL_OFFERINGS_2026-05-15.md (G-1 / G-3 / G-4 federal offerings + B-5 custom reports + Tier 2 proof packages). Source catalog: docs/projections/PROJECTION_CATALOG_2026-05-15.md §6.1 + §6.4 + §6.7.

0. TL;DR

Every recall-trigger or proof-window event ripples into 4 vault MDs + 3 JSONB projections + 1 evidence bundle + ~10 backlink edges. All renderers in this family are DETERMINISTIC (identical input → byte-identical output) because the outputs are legal-grade citation targets for federal recall investigations + FSMA enforcement + customer dispute resolution + Tier 2 customer proof reporting.

trigger event                          → ripple
─────────────────────────────────────────────────────────────────
compliance.recall_initiated           → recall_trace.<lot_id> (on-demand)
                                       + recall_evidence_bundle (R2 attachments)
                                       + notifies impacted customers via
                                         customer_commodity_matrix (Sales arch §2.3)

quarter_end + 5d (scheduled)          → proof_package.<tenant>.<quarter>
recall_drill.fsma-204-test request    → recall_drill MD + drill_metrics row

knowledge.fsma_completeness_updated   → fsma_traceability_completeness.<tenant>
                                       (live dashboard for FDA-facing view)

agent invokes assemble_recall_trace   → on-demand walk via po_timeline_projection
                                       + vendor_scorecard + qc_inspection_history
                                       + customer_commodity_matrix

Why determinism matters here: these outputs are subpoena material. A federal investigator must be able to re-run the renderer 6 months later against frozen event-store state and get the same bytes back. No LLM in the renderer path. No timestamps-of-now in the body. No random IDs. Source-events list + renderer_version + content_hash + frozen_input_event_set are mandatory.

0.5 Central-overlay reads contract (3-rung taproot — see AOL §13.6)

Every projector in this family is a rung-3 tenant overlay renderer that ALSO captures Central knowledge state at frozen-input time. This is stricter than QC + Sales arches: because outputs are legal-grade citation targets, the projector MUST snapshot the exact Central MD content_hash at frozen_input_event_id_range time and pin it into the output’s frontmatter. A re-render at T+6mo reads the same Central MD versions, NOT the current ones.

Per Recall & Proof projector:

ProjectorCentral MDs read from _taproot_mirror/Pinned in output asWhy
recall_trace.<lot_id>docs/commodities/<slug>.md, docs/vendors/<slug>.md, docs/grower-profiles/<slug>.md (when populated), docs/certifications/<slug>.mdcentral_md_pins[]: [{path, content_hash, captured_at}]Upstream chain + grower terminus + cert-as-of-recall-time evidence
proof_package.<tenant>.<quarter>docs/commodities/<slug>.md (every commodity tenant traded in quarter), docs/regions/<slug>.mdcentral_md_pins[]: ...”Industry-standard baseline” for KPI claims (what the cited commodity profile SAID this commodity should do during this quarter)
recall_drill.fsma-204-testdocs/commodities/<slug>.md for synthetic-scenario commodity, docs/certifications/<slug>.mdcentral_md_pins[]: ...Synthetic scenario must use frozen Central content for replay reproducibility
fsma_traceability_completeness.<tenant>docs/certifications/<slug>.md (each cert held by tenant’s vendors)central_md_pins[]: ... (monthly snapshot variant)Cert MD lists required-KDE fields; completeness score is computed against frozen-at-snapshot rules

Pinning contract: central_md_pins[] becomes part of the content_hash input. A change to a Central MD at T+1 does NOT invalidate the pinned recall_trace — that’s the whole point. To force re-evaluation, bump renderer_version AND re-run with the new Central MD content_hash captured.

Read graceful-degradation: same as other arches — central_pending: true flag if mirror missing. But for federal-audience outputs (recall_trace variant=federal, proof_package variant=federal), central_pending triggers system.federal_render_degraded alert and the federal variant DOES NOT ship until Central is present. Internal + customer variants ship anyway.

Write contract: projectors write to tenant paths (docs/recall/...) only. Never write to _taproot_mirror/.

1. Vault MD projectors (rung 3, DETERMINISTIC)

All 4 emit byte-identical outputs for identical inputs. Use crypto.subtle.digest('sha-256', ...) over the source-events list + renderer_version + central_md_pins[] to compute content_hash. Mismatched hash on re-render fires system.deterministic_renderer_drift_detected.

1.1 recall_trace.<lot_id>

  • doc_kind: recall_trace
  • audience: FIN operator (internal) + customer (customer-safe filtered) + federal (regulator)
  • cadence: on-demand via assemble_recall_trace MCP tool (privileged) + auto-render on compliance.recall_initiated
  • path: nathel-intranet:docs/recall/traces/<lot_id>/<iso_timestamp>.md
  • determinism: MANDATORY
  • sources:
    • Provenance walk via walk_provenance (P1-08 — reads doc_backlink_projection + fact_citations)
    • lot_lifecycle_reconciliation_projection (already live) for receipt → adjustment → reconciliation chain
    • po_timeline_projection (Sales arch §2.2) for procurement chain
    • customer_commodity_matrix (Sales arch §2.3) for customer reach
    • qc_inspection_history (QC arch §2.2) for quality-event timeline
    • vendor_scorecard + vendor_label_evidence (QC arch + Sales arch) for provenance back to grower
    • Comms history (operational_memory keyed by lot_id)
    • Corrective action events (compliance.corrective_action_taken, inventory.lot_quarantined)
  • output shape (deterministic structure):
    • Section 1: scope statement (lot_id, commodity, vendor, harvest_date, total_cases, contamination_class)
    • Section 2: receipt chain (PO → vendor → arrival → lot → QC outcome — each step with event_id citation)
    • Section 3: outbound chain (lot → sales orders → invoices → customers → delivery confirmations)
    • Section 4: customer reach (count, customer_ids, dollar exposure, primary contacts)
    • Section 5: root-cause candidates (causally-linked events: vendor history, temperature exceptions, QC patterns)
    • Section 6: mitigation timeline (when quarantined, when notified, when corrective action filed)
    • Section 7: recordkeeping evidence (links to PDFs, photos, comms threads — all stored in R2 with content-addressed paths)
  • audience filters:
    • federal: full chain, all event_ids, all PII redacted per FedRAMP rules; no commercial fields
    • customer: their orders only, their invoices only; vendor identity redacted; cost basis stripped; other customers invisible
    • FIN operator: full chain unredacted

1.2 proof_package.<tenant>.<quarter>

  • doc_kind: proof_package
  • audience: FIN operator + customer (executive level on share) + federal + partner (Anthropic / investor on share)
  • cadence: quarterly (quarter_end + 5d) + on-demand
  • path: nathel-intranet:docs/recall/proof-packages/<tenant>/<YYYY-Q[1-4]>.md
  • determinism: MANDATORY (frozen input event set = events with timestamp < quarter_end_timestamp)
  • sources:
    • Tenant KPIs from daily_company_pulse_projection, sales_team_dashboard, buyer_team_dashboard, data_quality_dashboard (all live JSONB)
    • Event volume by family (SELECT type, count(*) FROM events WHERE tenant_id=? AND timestamp BETWEEN ?)
    • Agent action counts (agent.tool_called event family aggregated per agent)
    • Outcomes attributed to FIN intervention (per agent_enrichment_metrics projection — when shipped)
    • Customer testimonials linked from comms threads (operational_memory category = testimonial)
    • Cited evidence for every claim (each KPI claim cites its event-id source set)
  • output shape:
    • Section 1: executive summary (3-bullet leadership read)
    • Section 2: KPI deltas vs prior quarter (with embedded sparklines / bar charts as inline SVG)
    • Section 3: narrative of operational improvements (LLM-AUTHORED but cached — see §3.1 LLM-author exception)
    • Section 4: cited evidence ledger (every claim links to source event_ids)
    • Section 5: attachments ledger (R2 paths + content hashes)
    • Section 6: signing metadata (renderer_version, content_hash, frozen_input_event_id_range)

1.3 recall_drill.fsma-204-test

  • doc_kind: runbook
  • audience: federal + FIN operator
  • cadence: on-demand (quarterly drill recommended; recall_drill_requested event triggers)
  • path: nathel-intranet:docs/recall/drills/<YYYY-MM-DD>-<drill_id>.md
  • determinism: MANDATORY
  • sources:
    • Synthetic recall scenario events (created at drill start with data.is_drill: true)
    • Time-to-trace measurements (timestamp deltas across the recall walk)
    • Renderer of recall_trace.<synthetic_lot_id> invoked against the drill’s synthetic events
  • output shape:
    • Section 1: drill scope statement (commodity, hypothetical contamination, target time-to-trace)
    • Section 2: actual drill plan (steps tested, agents involved, tools invoked)
    • Section 3: expected step durations vs measured durations
    • Section 4: pass/fail per FSMA 204 24-hour rule (assemble_recall_trace completion within 24h of drill start)
    • Section 5: evidence of completion (cited synthetic events + the produced recall_trace.<synthetic_lot_id> link)
    • Section 6: gaps identified (any incomplete provenance, slow walks, missing evidence)

1.4 fsma_traceability_completeness.<tenant> (vault MD form)

  • doc_kind: system_projected (live dashboard) + proof_package (monthly snapshot)
  • audience: FIN operator + employee (compliance) + federal (FDA-facing)
  • cadence: live (event-driven on KDE events) + monthly snapshot
  • path: nathel-intranet:docs/recall/fsma-completeness/<tenant>/<YYYY-MM>.md
  • determinism: MANDATORY for monthly snapshot; live variant non-deterministic (just-in-time view)
  • sources:
    • fsma_traceability_completeness_projection (existing JSONB substrate per project_fsma_204_compliance_envelope)
    • All KDE-bearing events (pallet.*, location.*, sensor.*, compliance.*) — verify full envelope
    • Lot-level walks: every active lot must have full provenance chain end-to-end
  • output shape:
    • Completeness score (0-100% per CTE — Critical Tracking Event)
    • Per-commodity breakdown
    • Per-vendor breakdown (which vendors’ lots have gaps)
    • List of incomplete lots (with the specific KDE field missing)
    • 24-hour rule readiness check (would assemble_recall_trace complete within 24h for every active lot?)
    • Monthly snapshot: frozen input event set + renderer_version + content_hash

2. JSONB projections (event-driven)

2.1 recall_evidence_chain_projection (NEW)

  • migration: 072_recall_evidence_chain_projection.sql
  • schema: tenant_id × lot_id → { upstream_chain[], downstream_chain[], evidence_artifacts[], walk_completion_status, last_walked_at }
  • source events: any event with this lot_id (read-through) + compliance.recall_initiated (triggers materialization)
  • replay command: pnpm cli replay-projection --projection=recall_evidence_chain_projection --tenant=<tenant_id>
  • RLS: required
  • fed-by: event-handlers/projection-recall-evidence-chain.ts (NEW)
  • consumed-by: recall_trace.<lot_id> renderer

2.2 drill_metrics_projection (NEW)

  • migration: 073_drill_metrics_projection.sql
  • schema: tenant_id × drill_id → { drill_started_at, drill_completed_at, total_walk_duration_ms, step_durations[], pass_fail, gaps_identified[] }
  • source events: recall_drill.started, recall_drill.step_completed, recall_drill.completed
  • replay command: pnpm cli replay-projection --projection=drill_metrics_projection --tenant=<tenant_id>
  • fed-by: event-handlers/projection-drill-metrics.ts (NEW)
  • consumed-by: recall_drill.fsma-204-test renderer + federal_readiness_tracker

2.3 proof_package_evidence_index (NEW)

  • migration: 074_proof_package_evidence_index.sql
  • schema: tenant_id × quarter → { kpi_event_id_sets{}, testimonial_message_ids[], frozen_at, frozen_input_event_id_range, content_hash, renderer_version }
  • source events: quarter-boundary cron writes once; immutable after write
  • replay command: pnpm cli replay-projection --projection=proof_package_evidence_index --tenant=<tenant_id>
  • fed-by: event-handlers/projection-proof-package-evidence-index.ts (NEW)
  • consumed-by: proof_package.<tenant>.<quarter> renderer

Note: existing proof_package_projection (per audit registry — kb_health_summary siblings) is a separate JSONB substrate for raw KPI rollups. The new proof_package_evidence_index is the immutable-citation-index layer that makes the package itself deterministic and re-renderable. Keep both.

3. Determinism contract (non-negotiable)

3.1 Renderer rules

RuleWhy
NO Date.now() in the body. All timestamps come from source events.Re-render at T+6mo must produce same bytes
NO Math.random() / crypto.randomUUID(). Use content-addressed IDs (sha-256 of input).Determinism
NO LLM in the rendering path (with one exception §3.2).Determinism + auditability
Input is the frozen event-id set, not a query at render time.Re-render must use same input
content_hash is computed and emitted as part of the doc.Drift detection
renderer_version is bumped any time the renderer changes; bump triggers re-render with new hash.Versioned reproducibility
Renderer is a pure function (deps injected: events fetch, attachment fetch).Testable + replayable

3.2 LLM-authored exception (§3.3 of catalog spec — “narrative of operational improvements”)

ONE section of proof_package (§1.2 Section 3 — narrative summary) may be LLM-authored because it’s editorial prose summarizing the KPI deltas. To preserve determinism:

  1. LLM call runs ONCE at quarter-end render time
  2. Output is stored in proof_package_evidence_index (§2.3) keyed to that exact frozen_input_event_id_range + renderer_version
  3. Re-renders read the cached LLM output from the index, never re-call the LLM
  4. New LLM output requires explicit renderer_version bump
  5. proof_package_evidence_index.llm_narrative_cache_id is part of content_hash input

This is the only LLM call permitted in this family. All other renderers are pure deterministic.

4. Evidence bundle (R2 attachments)

Recall trace + proof package outputs reference attachments stored in R2 with content-addressed paths:

r2://<tenant>/recall-evidence/
  └── <lot_id>/
      └── <sha256-of-content>/
          ├── qc-photo-<message_id>.jpg
          ├── vendor-label-extract-<message_id>.json
          ├── temperature-log-<sensor_id>-<window>.csv
          ├── pp-arrival-report-<load_id>.pdf
          └── comms-thread-<thread_id>.json (export of operational_memory rows)

Path components are deterministic (sha256 of content → path), so the same bytes always land at the same path. The recall_trace MD references attachments by R2 path; federal investigators can fetch by URL and verify hash matches.

R2 retention policy: evidence bundles for any lot involved in a compliance.recall_initiated event are RETAINED INDEFINITELY. All other evidence follows FSMA 204 7-year retention.

5. Native comms integration

SurfaceWhere Recall & Proof projections land
Compliance role railfsma_traceability_completeness daily snapshot pinned; recall_drill results posted on completion
FIN operator railproof_package weekly preview during quarter-end window
Customer portal (when shipped)recall_trace customer-facing variant: their orders only, vendor redacted
Federal portal (FedRAMP, Wave 5+)recall_trace federal variant: full chain + evidence bundle access
EmailCritical recall: urgent_recall_notification email to affected customers + tenant compliance role + federal liaison; uses customer_commodity_matrix (Sales arch §2.3) to compute recipient list
Slackcompliance.recall_initiated → instant Slack ping to #compliance channel with link to in-flight recall_trace.<lot_id>
SMS / VoiceCritical recall: SMS to compliance role primary contact + FIN operator on-call

6. Frontmatter contract (same as QC + Sales arches)

---
schema_version: 1
doc_id: <projector-output-doc-id>
doc_kind: recall_trace | proof_package | runbook | system_projected
title: <human-readable>
slug: <kebab-case>
source_attribution:
  - producer:<projector-name>
  - source_events:<comma-separated event_ids in frozen input set>
  - renderer_version:<semver>
  - content_hash:<sha256>
  - frozen_input_event_id_range:<min>-<max>
lifecycle_status: active
created_at: <iso>
updated_at: <iso>
maintained_by: <projector-name>
visibility: restricted | federal_only | customer_visible | company
data_plane: relationship_data
linked_entities:
  - { type: lot, id: <lot_id> }
  - { type: commodity, slug: <slug> }
  - { type: vendor, slug: <slug> }
  - { type: customer, slug: <slug> }
  - { type: po, id: <po_id> }
  - ... etc
tags: [recall, proof, fsma_204, ...]
determinism_required: true
re_render_hash_drift_alert: true
---

Note determinism_required: true flag — projectors with this flag set hit the determinism-drift check on every re-render.

7. Build sequence

Sequence R1 — 3 JSONB projections (~2-3 days)

  • R1-A recall_evidence_chain_projection (migration 072 + handler + dispatcher + test)
  • R1-B drill_metrics_projection (migration 073 + handler + dispatcher + test)
  • R1-C proof_package_evidence_index (migration 074 + handler + dispatcher + test)

Sequence R2 — 4 vault MD renderers (~5-7 days, DETERMINISTIC)

  • R2-A recall_trace renderer (event-driven on compliance.recall_initiated + on-demand MCP tool) + content_hash machinery + audience-filter machinery
  • R2-B proof_package renderer (quarterly cron) + LLM-narrative cache (§3.2) + 4-audience output variants
  • R2-C recall_drill renderer (on-demand) + drill metrics aggregation
  • R2-D fsma_traceability_completeness renderer (live + monthly snapshot dual mode)

Sequence R3 — Evidence bundle pipeline (~2 days)

  • R3-A R2 content-addressed path resolver + attachment fetcher + retention policy enforcer
  • R3-B Federal investigator MCP tool: fetch_recall_evidence(lot_id, attachment_hash) returns the bytes with audit log

Sequence R4 — Agent surface + UI (~2 days)

  • R4-A assemble_recall_trace MCP tool (privileged role only)
  • R4-B request_recall_drill MCP tool (compliance role)
  • R4-C get_proof_package MCP tool (operator + customer-share)
  • R4-D Compliance role rail integration

Sequence R5 — Federal hardening (~1 week, FedRAMP gated)

  • R5-A Federal audience filter audit + redaction completeness verification
  • R5-B Federal MCP endpoint deployment (FedRAMP infrastructure)
  • R5-C Agency-specific data handling protocols + IdP integration

R5 ships only when first federal contract clears (per feedback_federation_publisher_frozen analog — code can be ready, activation gated on Robert approval + federal contract).

8. Coordination with other arch docs

Other arch docRecall & Proof dependency
QC archReads qc_inspection_history JSONB for inspection events; reads vendor_label_evidence for label-verification chain; uses QC photos as recall-evidence-bundle attachments
Sales & Market archReads po_timeline_projection for procurement chain; reads vendor_scorecard for vendor history; reads customer_commodity_matrix for customer-reach computation in recall_trace §4
Agent Audit arch (forthcoming)Recall investigations are themselves agent sessions → tracked via agent_harness_session projection

9. Open questions

IDQuestionDefault until answered
RP-Q1LLM-author cache (§3.2) — store in proof_package_evidence_index or separate llm_narrative_cache table?Same index — keeps a single re-renderable source-of-truth per quarter
RP-Q2Determinism-drift alert (§3 last bullet) — fail the render or warn-and-continue?Fail loud — federal-grade outputs cannot drift silently
RP-Q3Recall trace customer-facing variant — pre-render or on-demand?On-demand — recall is rare; pre-render is wasteful
RP-Q4Quarterly proof package cadence — fixed quarter-end + 5d or tenant-configurable?Fixed; multi-tenant later may override per-tenant
RP-Q5FSMA completeness score threshold for tenant-readiness — what % triggers tenant.fsma_below_threshold alert?95% — gives 5% buffer for in-flight lots
RP-Q6R2 retention — recall-involved lots stored forever; how is “involved” defined?Any lot transitively reachable via 2 hops from a compliance.recall_initiated event
RP-Q7Federal MCP endpoint (R5-B) on Cloudflare FedRAMP or separate AWS GovCloud?Defer until federal contract; FedRAMP-on-Cloudflare per existing strategy

10. Productization mapping

OfferingRecall & Proof projector that enables it
G-1 Real-time Recall Trace Access (1M/agency)recall_trace.<lot_id> (§1.1) federal-audience variant + R3 evidence bundle
G-3 Outbreak Investigation Support (500K/year + incident fees)assemble_recall_trace + fetch_recall_evidence privileged MCP tools
G-4 FSMA 204 Compliance Dashboards (250K/agency + Tier 2 add-on)fsma_traceability_completeness.<tenant> (§1.4)
B-5 Custom Reports (50K/report)proof_package + ad-hoc reads of evidence_chain JSONB
C-3 Food Safety Alerts (consumer free)Customer notification triggered by compliance.recall_initiated via customer_commodity_matrix (Sales arch)
Tier 2 quarterly customer proof (current Nathel value)proof_package.<tenant>.<quarter> (§1.2) customer-share variant

This arch ships the substrate for 6 of the 17 strategy doc offerings.

11. Sign-off

This arch is the canonical reference for any future recall, proof, FSMA-compliance, evidence-bundle, or federal-audience projection work. Future projectors in this family MUST cite this doc in their PR description. Cross-mirrored to fin-central-intranet:docs/architecture/RECALL_AND_PROOF_PROJECTION_ARCHITECTURE_2026-05-18.md.

— claude_orchestrator (Opus 4.7 1M-context, session 7612a827-b892-47e3-86cd-08835737208e)

Graph View