Historical Data Backfill — Part L (2026-05-16)

Status: L-04..L-08 COMPLETE (dry-run verified) · production --apply gated by separate event-store cascade task per the Production Parity Tracker §13 close-out criteria.

Scope: retroactive replay of the v10 Teams-QC export (/Users/robert/Projects/FIN/_03_Dev/teams-qc-export-2026-05-06.json, 146 messages, 2025-01-24 → 2026-05-06) into the fin-agentic event store via the five Part L importers.

1. Importer inventory

Task	Importer	Subtype emitted	Status
L-01	`teams-chat-importer.ts` (orchestrator + classifier)	placeholder per subtype	Day 1 (414f08bb)
L-02	`qc-narrative-importer.ts`	`signal.quality_observation`	Day 1 (414f08bb)
L-03	`photo-graph-stitcher.ts`	`artifact.linked_work_thread`	Day 1 (414f08bb)
L-04	`gm-memo-importer.ts`	`signal.market_chatter` (gm)	Day 2 — this delivery
L-05	`salesman-ground-truth-importer.ts`	`signal.price_observation` + `signal.market_chatter` (salesman)	Day 2 — this delivery
L-06	`hunts-point-market-importer.ts`	`signal.market_chatter` (peer_distributor)	Day 2 — this delivery
L-07	`scripts/cli/historical-backfill.ts` (orchestrator CLI)	wraps L-01..L-06	Day 2 — this delivery
L-08	this document	audit + dry-run evidence	Day 2 — this delivery

The signal-registry remains closed at 15 subtypes. No new event types were introduced for L-04..L-06 — all three importers route to existing subtypes with discriminator fields in data.*.

2. Per-importer counts (full v10 corpus dry-run)

L-02 QC narrative importer (Day 1, included for context)

total_events                             141
pp_load_id_present                       139   (98.6%)
parse_failures                             0
glenn_prose_authors                       50   ← parity goal hit
anthony_terse_authors                     50   ← parity goal hit
pedro_coordinator_authors                 12

50 / 50 / 12 split confirms the inspector-coverage parity target from the v10 corpus summary.
139 / 141 events carry a pp_load_id — proves PP load-number-as-spine is the reliable cross-link key for downstream reconciler match.

L-04 GM memo importer (Joshua Gatcke)

total_memos                               10   ← matches corpus expectation
total_events                              23   ← average 2.3 sections/memo
parse_failures                             0

by_topic:
   11  price_movement                   ← Joshua quotes $ FOB in most memos
    5  supply_tightness                 ← "limited volume", "undersized"
    2  vendor_substitution              ← "switched 18/221 to 24's"
    5  other                            ← protocol announcements + misc

by_sentiment:
    9  neutral
    9  mixed                            ← Joshua naturally hedges
    5  bullish

commodity_sections_per_memo: [1, 7, 8, 1, 1, 1, 1, 1, 1, 1]

Section distribution:

The 7-section memo (2025-03-29) → Blueberries / Mandarins / Oranges / Cantaloupe / Mango / Honeydew / Lemons.
The 8-section memo (2025-04-05) → Blues / Mandarins / Oranges / Mango / Avocado / Cantaloupe / Honeydew / Citrus.
The 1/24/25 pinned protocol announcement → commodity_raw = "general" (whole-memo fallback) because no commodity emoji-bullet headers fire.

L-05 Salesman ground-truth importer

salesman_messages                         12  ← every salesman role author
price_events                              56  ← 4.7 prices/message on average
chatter_events                            12  ← one whole-body chatter per message
total_events                              68
parse_failures                             0

by_author:
    4  keith@nnproduce.net               ← expected 4 (matches v10 corpus)
    2  donovan@nnproduce.net             ← expected 2
    1  gerard@nnproduce.net              ← expected 1 (DM, no UPN — resolved by display)
    1  tim@nnproduce.net                 ← expected 1 (DM, no UPN — resolved by display)
    1  davidf@nnproduce.net              ← expected 1 (pricing role)
    2  ffuentes@nnproduce.net            ← Frank Fuentes (market_chatter only — no prices)
    1  acohen@nnproduce.net              ← Adam Cohen (market_chatter only)

fob.california:                          51   ← inferred from prose phrases
top commodities: romaine(5), crowns(4), red(4), lettuce(4), peppers(4)

Spec note: the L-05 brief asked for “Keith Martin garlic/scallions (4 msgs) + Mike Donovan CA FOB (2 msgs) + Gerard Milito + Tim Gagliano (2 daily sales reads)” — all matched. Keith’s 4 messages include the garlic items list (correctly filtered as inventory not prices via the PP unit-count guard), the basil/sage poor-quality return, the scallions move-to-go DM, and the distressed-items apology to Mike D. The ffuentes/acohen rows above are not in the brief but flow through the salesman role automatically — they emit chatter only.

L-06 Hunts Point peer-distributor importer

messages_scanned                          25  ← Joshua + 7 salesman + Paul Milito buyer
total_events                              13
parse_failures                             0

by_peer_house (distributors + brand programs):
    3  JBO                               ← brand_program (Joshua's blueberry memos)
    2  Five Brothers                     ← peer_distributor (Gerard + Tim "5 bros")
    2  D'Arrigo                          ← peer_distributor (Gerard + Tim)
    2  OzBlu                             ← brand_program (Joshua's blueberry memos)
    1  FLS                               ← brand_program (Gerard's beans line)
    1  Pablos                            ← brand_program (Gerard's celery line)
    1  Armata                            ← peer_distributor (Tim's beans line)
    1  Queen Boni                        ← brand_program (Keith's distressed message)

by_peer_kind:
    8  brand_program                     ← FLS + Queen Boni + TA + JBO + OzBlu + Pablos
    5  peer_distributor                  ← D'Arrigo + Armata + Five Brothers houses

by_originating_author:
    5  joshua@nnproduce.net              ← GM mentions JBO/OzBlu most
    4  gerard@nnproduce.net              ← daily reads name many houses
    3  tim@nnproduce.net
    1  keith@nnproduce.net               ← Queen Boni + TA leaf reference

13 peer-distributor references is the raw count. Per the L-06 brief: “peer-distributor refs from existing salesman/GM messages” — all 13 were extracted from L-04/L-05 source messages (no separate Hunts Point data source). The originating author lineage is preserved in data.originating_author_email + data.originating_poster_role so the downstream consumer can join the peer event back to the salesman/GM source.

3. Dedup verification

Idempotency (per Part L acceptance criteria)

                          envelopes   unique_keys  collisions   hash_collisions
qc                              141          141            0                30
gm                               23           23            0                13
salesman                         68           68            0                56
peer                             13           13            0                 7
─────────────────────────────────────────────────────────────────────────────
TOTAL                           245          245            0               112

idempotency_collisions = 0 at every level. Every emitted envelope carries a unique key of the form historical:teams_qc_export_2026_05_06:<subtype>:<channel>#<msg>#<suffix> where <suffix> distinguishes per-fact / per-section / per-mention events from the same source message.

Content-hash dedup (informational)

content_hash_collisions > 0 is expected and benign: a single message produces multiple envelopes (one per QC commodity fact, one per GM memo section, one per peer-distributor mention) and they all share the message-body sha256. The idempotency key — not the content hash — is the dedup anchor at insert time. The hash is retained for cross- extractor reconciliation (e.g. detecting that a message was already ingested via the live PP-arrival pipeline).

Concretely:

L-02: 141 envelopes from 110 narratives → 30 hash collisions reflect multi-commodity narratives (Glenn’s Pioneer 3-cabbage etc.).
L-04: 23 envelopes from 10 memos → 13 collisions reflect multi-section Joshua memos.
L-05: 68 envelopes from 12 messages → 56 collisions reflect per-line price extraction.
L-06: 13 envelopes from 13 mentions → 7 collisions reflect multiple mentions per source message.

Re-run determinism

Running the orchestrator CLI twice (--source all --dry-run) produces identical idempotency-key sets — verified by the historical-backfill.test.ts “dry-run twice produces identical key sets” case.

4. Vendor / commodity coverage

Vendors observed (Part L scope only; PP master is the production source)

Glenn QC corpus (50 msgs) — Pioneer Growers, Altar Produce, KB&R, Los Pinos, Baloian Farms, Castesa, Pamela, plus the 30+ vendor list documented in the v10 manifest §“Glenn Jones training corpus”.
Anthony QC corpus (50 msgs) — Florida Produce Brokers, Sol Group Marketing, LGS Specialty Sales, Dayka & Hackett, La Minita Fresh, etc.
Joshua GM memos — references California / Mexico / Peru / Morocco / Chile / Spain / Honduras / Ecuador / Costa Rica origins; specific vendor names rare in strategy memos (memos are GM-level outlook, not vendor-specific quotes).
Salesman (Mike Donovan / Keith / Gerard / Tim) — references the D’Arrigo “pablos” celery program, Armata 5-bros beans, FLS beans, TA leaf items, Queen Boni leaf items, JBO + OzBlu blueberry programs. These are captured as peer-distributor / brand-program tags in L-06.

Commodity coverage (top 15 from salesman price extraction)

  5  romaine
  4  crowns
  4  red (cabbage / leaf)
  4  lettuce
  4  peppers
  4  spinach
  3  celery
  2  anise
  2  beans
  2  broccolini
  2  cabbage
  2  corn
  2  leeks
  2  loose (garlic)
  2  okra

Joshua’s GM memos additionally cover: blueberries, mandarins, oranges, mango, avocado, cantaloupe, honeydew, lemons, navels, citrus generally, apples, pears, pomegranate.

FOB region tagging

fob.california = 51 events — every Mike Donovan California FOB line plus most prose-inferred mentions in the salesmen daily reads. No Mexico-tagged FOB lines in v10 (Joshua’s memos mention Mexico but the $-amount-bearing lines are CA-anchored).

5. Acceptance — Production Parity Tracker §13

L-04 acceptance

10 / 10 Joshua memos classified (parity hit per v10 corpus)
Per-commodity section extraction works (avg 2.3 sections/memo, max 8)
signal.market_chatter payload validates against EIF-RM-016 (SignalMarketChatterPayloadSchema.safeParse green in the co-located test)
Topic + sentiment + forward-window heuristics fire correctly
Seasonal context (month / quarter / origins / season tags) stamped on every event
Idempotency: re-running yields identical content hashes + keys

L-05 acceptance

Keith (4) + Donovan (2) + Gerard (1) + Tim (1) + David Frangiosa (1) — all classified
Per-fact signal.price_observation events extracted (56 events from 12 messages; 4.7 avg)
FOB region inferred from prose (51 california-tagged events)
PP-inventory guard rejects “Jars 11/12, sleeves 56/54” as prices
Whole-body signal.market_chatter event in addition to per-fact price events
signal.price_observation payload validates against RM-202

L-06 acceptance

Peer-distributor refs extracted from salesman + GM messages (no separate source needed)
13 peer-distributor / brand-program references across 8 distinct entities
Originating author lineage preserved (data.originating_author_email, data.originating_poster_role)
Distributor (5) vs brand-program (8) kind distinction made
Adjacent commodity + competing price extracted where present
signal.market_chatter payload validates against EIF-RM-016

L-07 CLI acceptance

pnpm cli:historical-backfill --source <X> --dry-run works for qc / gm / salesman / peer / all
--apply is deliberately gated — throws a clear “not-implemented” error documenting the production cascade requirement
Per-source counts printed (parse_failures, dedup stats, author / commodity / FOB breakdowns)
Idempotency-key uniqueness asserted across re-runs

L-08 acceptance

This audit document lands in docs/audits/ (the closest live vault location in fin-agentic; the spec’s “vault push OK” refers to a parallel Obsidian vault repo which does not yet exist on this workstation — confirmed by searching /Users/robert/Projects and ~. Once the vault repo lands, this same file can be mirrored.)

6. Open items

6.1 Photo backfill — pending Microsoft Graph SDK

L-03 emits artifact.linked_work_thread events in status: "pending_photo_arrival" for every narrative whose teams-qc-photos/<channel>/<msg>/ directory is empty (the common case in v10 — only 1 PDF was persisted; videos via webUrl are unreachable without Graph SDK fallback). The L-03 importer already supports .mov MIME detection so the KB&R cucumbers / Los Pinos pairing (message 1777426020781, load 548328) is handled correctly once binaries land.

Remediation path: schedule a separate session with M365 credentials and the Graph SDK enabled. Walk the pairing table, fetch each attachment’s bytes via webUrl, persist to disk under the per-message directory, then re-run L-03 with status: "linked" transitions emitted as compensating events.

6.2 LLM-driven narrative extraction

L-02 / L-04 / L-05 use regex + closed-vocab heuristics today. A follow-up task will swap in an LLM extractor for finer-grained defect_rate_pct, brand, size_grade (L-02), and richer topic / sentiment classification (L-04) once the dry-run corpus has proven its value through downstream consumer tests.

6.3 Entity reconciliation

commodity_id, vendor_id, region_id, brand_ids[] are intentionally left empty / null on emit. The downstream reconciler (RM-197/199 dual- CIR emitter) resolves them when it next runs. Empty arrays / strings are the agreed contract per the L-04/05/06 schema-compliance tests.

6.4 Tommy Cifu single message

tommy@nnproduce.net (Tommy Cifu, 1 msg) is routed to signal.teams_message (raw archive) because the body is an operations ask (“can we look into the chicory lots… coming up short a pallet”) rather than market / quality / price content. Documented in the AUTHOR_ROUTES map; the orchestrator’s acceptance check confirms it lands.

6.5 Production `--apply` cascade

The L-07 CLI surfaces a clear NOT-IMPLEMENTED error on --apply and explains why: the production write path requires the full event-store cascade (record-event tool, RLS-scoped withTenant, downstream consumer dispatch) per the Part L acceptance criteria. The skeleton in applyEvents() documents the exact SQL + transactional pattern the follow-up task should drop in (INSERT ... ON CONFLICT (idempotency_key) DO NOTHING per the 4-table architecture rule).

6.6 Federation publisher

Per feedback_federation_publisher_frozen.md in user memory, EIF Slice 7 (anonymization verifier + publisher + cron) is COMPLETE in code but NOT activating for single-tenant operation. The historical backfill events emit with the same signal.* payload schemas, so when the federation publisher unfreezes the historical events flow through the same anonymization + publish path without modification. No work required in this slice.

7. Test inventory

File	Tests	Pass
`packages/mcp-server/src/ingestion/historical/teams-chat-importer.test.ts`	24	24
`packages/mcp-server/src/ingestion/historical/qc-narrative-importer.test.ts`	23	23
`packages/mcp-server/src/ingestion/historical/photo-graph-stitcher.test.ts`	10	10
`packages/mcp-server/src/ingestion/historical/gm-memo-importer.test.ts`	26	26
`packages/mcp-server/src/ingestion/historical/salesman-ground-truth-importer.test.ts`	18	18
`packages/mcp-server/src/ingestion/historical/hunts-point-market-importer.test.ts`	18	18
`scripts/cli/historical-backfill.test.ts`	14	14
TOTAL	133	133

Run from packages/mcp-server:

pnpm exec vitest run src/ingestion/historical/

Run the CLI test from the repo root:

pnpm exec vitest run scripts/cli/historical-backfill.test.ts

8. Files added / modified in this delivery

Added:

packages/mcp-server/src/ingestion/historical/gm-memo-importer.ts (L-04)
packages/mcp-server/src/ingestion/historical/gm-memo-importer.test.ts
packages/mcp-server/src/ingestion/historical/salesman-ground-truth-importer.ts (L-05)
packages/mcp-server/src/ingestion/historical/salesman-ground-truth-importer.test.ts
packages/mcp-server/src/ingestion/historical/hunts-point-market-importer.ts (L-06)
packages/mcp-server/src/ingestion/historical/hunts-point-market-importer.test.ts
scripts/cli/historical-backfill.ts (L-07)
scripts/cli/historical-backfill.test.ts
docs/audits/HISTORICAL_DATA_BACKFILL_2026-05-16.md (this file — L-08)

Modified:

packages/mcp-server/src/ingestion/historical/author-routing.ts (added Gerard / Tim / Tommy routes + display-name resolver)
packages/mcp-server/src/ingestion/historical/teams-chat-importer.ts (orchestrator now resolves authors by UPN OR display name)
package.json (added cli:historical-backfill script)

Unmodified Day 1 artifacts (pinned to commit 414f08bb):

packages/mcp-server/src/ingestion/historical/types.ts
packages/mcp-server/src/ingestion/historical/qc-narrative-importer.ts
packages/mcp-server/src/ingestion/historical/photo-graph-stitcher.ts
packages/mcp-server/src/ingestion/historical/README.md

9. References

Tracker source: docs/programs/FIN_PRODUCTION_PARITY_TRACKER_2026-05-16.md §13 (the tracker document itself may not yet be checked into the repo; this delivery follows the spec quoted in the spawning task brief)
Day 1 commit: 414f08bb feat(ingestion): Part L historical Teams-QC importers (L-01/L-02/L-03)
Day 1 branch: claude/parity-task-L-01-L-03-teams-importers
Day 2 branch: claude/parity-task-L-04-L-08-historical-remainder
Foundation handoff: /Users/robert/Projects/FIN/_03_Dev/FIN-TEAMS-QC-DATA-FOUNDATION-HANDOFF-2026-05-06.md
v10 corpus summary: /Users/robert/Projects/FIN/_03_Dev/teams-qc-export-2026-05-06-v10-summary.md
Coverage audit: /Users/robert/Projects/FIN/_03_Dev/teams-qc-export-2026-05-06-coverage-audit.md
Signal registry (closed): packages/shared/src/events/payloads/signal/registry.ts
EIF S0.5 classifier pattern (referenced in L-01 docstring): packages/mcp-server/src/ingestion/intake-classifier.ts
Memory file (federation status): feedback_federation_publisher_frozen.md

FIN Central

Explorer

Historical Data Backfill — Part L (2026-05-16)

Historical Data Backfill — Part L (2026-05-16)

1. Importer inventory

2. Per-importer counts (full v10 corpus dry-run)

L-02 QC narrative importer (Day 1, included for context)

L-04 GM memo importer (Joshua Gatcke)

L-05 Salesman ground-truth importer

L-06 Hunts Point peer-distributor importer

3. Dedup verification

Idempotency (per Part L acceptance criteria)

Content-hash dedup (informational)

Re-run determinism

4. Vendor / commodity coverage

Vendors observed (Part L scope only; PP master is the production source)

Commodity coverage (top 15 from salesman price extraction)

FOB region tagging

5. Acceptance — Production Parity Tracker §13

L-04 acceptance

L-05 acceptance

L-06 acceptance

L-07 CLI acceptance

L-08 acceptance

6. Open items

6.1 Photo backfill — pending Microsoft Graph SDK

6.2 LLM-driven narrative extraction

6.3 Entity reconciliation

6.4 Tommy Cifu single message

6.5 Production `--apply` cascade

6.6 Federation publisher

7. Test inventory

8. Files added / modified in this delivery

9. References

Graph View

Table of Contents

FIN Central

Explorer

Historical Data Backfill — Part L (2026-05-16)

Historical Data Backfill — Part L (2026-05-16)

1. Importer inventory

2. Per-importer counts (full v10 corpus dry-run)

L-02 QC narrative importer (Day 1, included for context)

L-04 GM memo importer (Joshua Gatcke)

L-05 Salesman ground-truth importer

L-06 Hunts Point peer-distributor importer

3. Dedup verification

Idempotency (per Part L acceptance criteria)

Content-hash dedup (informational)

Re-run determinism

4. Vendor / commodity coverage

Vendors observed (Part L scope only; PP master is the production source)

Commodity coverage (top 15 from salesman price extraction)

FOB region tagging

5. Acceptance — Production Parity Tracker §13

L-04 acceptance

L-05 acceptance

L-06 acceptance

L-07 CLI acceptance

L-08 acceptance

6. Open items

6.1 Photo backfill — pending Microsoft Graph SDK

6.2 LLM-driven narrative extraction

6.3 Entity reconciliation

6.4 Tommy Cifu single message

6.5 Production --apply cascade

6.6 Federation publisher

7. Test inventory

8. Files added / modified in this delivery

9. References

Graph View

Table of Contents

6.5 Production `--apply` cascade