FIN Central

FIN Operational Corpus Ingestion Roadmap

23 min read

FIN Operational Corpus Ingestion Roadmap

0. Revision Summary

First cut. Defines the four-week ingestion sequencing for the Nathel pilot operational corpus, anchored on PO-as-spine. Specifies what to pull from Produce Pro, TURVO (inbound TMS), Samsara (outbound delivery telemetry), and the buyers Org Box authored corpus. Provides field-level pull lists, event schemas, projection migration plan, and join-key discipline. Supersedes ad-hoc ingestion guidance in 2-Week Execution Program by extending it from 2 weeks to 4 and adding the TURVO/Samsara legs explicitly.

1. Why PO Is the Spine

Every operational event in produce — every inspection, every comms thread, every truck movement, every sales transaction, every cold-chain reading, every settlement adjustment — carries an implicit PO context. Without binding events to po_id, the corpus is a pile of disconnected signals. With binding, the corpus is a continuous operational record.

The PO is also the boundary across which provenance flows. Inbound provenance (grower → vendor → carrier → arrival) terminates at the PO. Outbound provenance (PO → inventory lot → sales order → customer) originates from the PO. FSMA 204 KDE compliance, federal traceability, customer-facing certificates of origin — all of these hang off the PO spine.

The discipline this doc enforces: every ingestion path produces events that carry po_id as a hard foreign key. No exceptions. If a signal cannot be bound to a PO at ingest time, it is held in a pending bucket and resolved by an explicit reconciliation pass — it does not enter the operational projections without a PO binding.

2. Four-Week Shape

Week 1 (May 18-22). Produce Pro extractor trio + foundational masters. PO created, PO received, daily sales, daily inventory snapshot, physical inventory adjustments, PD335 (vendors), PD327 (customers), COMMODITY (commodities). Watermelon entity profile renders from real data. QC live test events bind to real PO records. Comms threads bind to real PO records when context is clear.

Week 2 (May 25-29). TURVO inbound integration. Every TURVO shipment binds to a po_id via the customer-reference field. Inbound cold-chain projection becomes possible. Vendor lead-time projection. Carrier scorecard projection seeded.

Week 3 (June 1-5). Samsara outbound integration. Every Samsara trip binds to a sales order, which binds to inventory pulls, which binds to PO. Outbound cold-chain projection. Customer delivery service-level projection. Driver scorecard projection.

Week 4 (June 8-12). Aggregate intelligence projections that fuse the planes. Vendor scorecards (PO + receiving QC + inbound performance). Carrier scorecards (TURVO inbound + Samsara outbound). Customer scorecards (sales + outbound delivery + comms tone). Commodity profiles updated daily with operational signal. The buyers Org Box authored corpus gets pulled into the projection layer as the authored knowledge that operational projections cite against.

By end of Week 4: a working closed loop on a single tenant. Structured data lands → events → CQRS projections → MD projections → agent reads → agent acts → new events. This is the substrate the Self-Improving System Roadmap builds on.

3. Week 1 — Produce Pro Extractor Trio + Masters

Produce Pro is the operational ERP of record at Nathel. It has no API; reports are PDF-exported. The extraction path is PP report PDF → CIR (Canonical Intermediate Representation) → event emission, with deterministic schema enforcement at the CIR boundary. See project memory: produce_pro_architecture and project memory: pp_extractor_order.

3.1 COMMODITY — Commodity Master

PP’s commodity dimension table. Pulled first because every other report references commodity codes.

  • Output events. commodity.master.upserted (one per commodity row)
  • Payload fields. commodity_code, commodity_name, default_uom, default_pack_spec, category, perishability_class, default_grade_standard
  • Projection target. commodity entity in knowledge table (already exists); MD projection: commodity_profile per commodity
  • Cadence. Weekly full re-export + daily delta if any commodity added

3.2 PD335 — Vendor Master

PP’s vendor dimension. Required before PO ingestion (vendor join target).

  • Output events. vendor.master.upserted
  • Payload fields. vendor_code, vendor_name, vendor_type (grower, broker, importer, etc.), default_payment_terms, country_of_origin_codes, certifications (organic, fair-trade, etc.)
  • Projection target. vendor entity; MD projection: vendor_profile
  • Cadence. Weekly full re-export

3.2A PD327 — Customer Master

PP’s customer dimension. Required before sales, service-level, comms, and outbound delivery joins.

  • Output events. customer.master.upserted
  • Payload fields. customer_code, customer_name, address, contact, salesman_code, ship_via, account_metadata
  • Projection target. customer entity; MD projection: customer_profile
  • Cadence. Weekly full re-export

3.3 PO Created (RE101 or equivalent)

The PO at creation time — before it has been received.

  • Output events. po.created
  • Payload fields. po_id (PP PO number, the join key), vendor_code, commodity lines (commodity_code, expected_qty, expected_pack_spec, expected_price, expected_arrival_date), shipping_terms, payment_terms, created_by, created_at
  • Projection target. po_state_projection (new CQRS projection, migration TBD)
  • MD projection target. po_lifecycle (new doc_kind) — one MD doc per PO that grows through its lifecycle, every event on the PO appends to its doc projection
  • Cadence. Daily extraction; lag tolerance: same-day

3.4 PO Received (PD032 active PO received; daily receiving equivalent)

The PO after it has been physically received at the warehouse.

  • Output events. po.received, po.line.received (one per line)
  • Payload fields. po_id, line_no, received_qty, received_pack_spec, weight_variance_pct, accepted/rejected/partial, receiving_clerk, received_at, lot_assignment(s)
  • Projection target. po_state_projection updates; inventory_lot_projection creates new lot rows
  • MD projection target. po_lifecycle doc appended; inventory_lot entity profile created
  • Cadence. Daily; ideally multiple-times-daily (receiving happens throughout the morning)

3.5 RP581 — QC Inspection Report

Already in flight as the foundational QC ingestion path. Critical that every RP581 binds to a po_id.

  • Output events. qc.inspection.completed (one per inspection event)
  • Payload fields. inspection_id, po_id, line_no, commodity_code, lot_id (if assigned), inspector_id, inspection_findings (structured: temperature, brix, defect_classes, defect_pct, grade_assigned, accept/reject), inspection_started_at, inspection_completed_at, attached_photos (R2 refs)
  • Projection target. qc_inspections_projection (migration 048 per existing plan)
  • MD projection target. po_lifecycle doc appended; commodity profile aggregate updated; procedure_profile rendering when applicable (see Self-Improving Roadmap §3.1)
  • Cadence. Real-time during inspection workflow (this is the live-test system)

3.6 Daily Sales (RP-equivalent — sales detail report)

The sales side. Every line of every invoice for the day.

  • Output events. sales.line.created (one per line)
  • Payload fields. sales_order_id, line_no, customer_code, commodity_code, lot_id (sourcing the sale), sold_qty, sold_pack_spec, unit_price, total_price, sold_at, salesperson_id
  • Projection target. sales_velocity_projection, customer_buying_pattern_projection
  • MD projection target. commodity_profile updated with daily sales velocity; customer_profile updated with buying pattern
  • Cadence. Daily end-of-day; aim for same-night extraction

3.7 Daily Inventory Snapshot

The current-state inventory across all lots, end-of-day. Reconciles against derived state.

  • Output events. inventory.snapshot.captured (header) + inventory.lot.snapshotted (one per lot)
  • Payload fields. snapshot_id, snapshot_at, lot_id, commodity_code, on_hand_qty, on_hand_pack_spec, age_days, location
  • Projection target. inventory_state_projection (reconciliation target); divergence between derived state and snapshot is itself an event (inventory.divergence.detected)
  • MD projection target. daily_inventory_digest (new doc_kind, daily projection)
  • Cadence. Daily end-of-day

3.8 Physical Inventory Adjustments

Manual adjustments to inventory — write-offs, dumps, found, transfers.

  • Output events. inventory.adjustment.applied
  • Payload fields. adjustment_id, lot_id, adjustment_qty (signed), adjustment_reason (dump, shrink, found, transfer, correction), adjusted_by, adjusted_at, supporting_notes
  • Projection target. inventory_state_projection; aggregates into shrink_projection per commodity per period
  • MD projection target. Surfaces in daily_inventory_digest; aggregates into vendor_profile (vendor-attributable shrink) and commodity_profile
  • Cadence. As-they-happen; daily extraction batch is acceptable

3.9 Buyers Org Box — Authored Corpus

Already feeding in. Treated as authored knowledge (doc_kind: authored) and serves as the citation target for grower/vendor/commodity background information.

  • Discipline. Org Box docs are pulled, parsed, chunked, embedded, indexed under the same hybrid retrieval contract as projection-rendered docs
  • Anchor. Org Box docs link to vendor and commodity entities via frontmatter or explicit references; the typed wikilink resolver wires them in
  • Use. Operational projections (vendor_profile, commodity_profile) cite against Org Box authored content. Authored content provides the “what is this vendor / commodity in the broader world” context that operational projections layer on top of

4. Week 2 — TURVO Inbound

TURVO is Nathel’s TMS for inbound shipments — the leg from grower/vendor to Nathel warehouse. Public REST API + webhook subscription model.

4.1 Scope Required

  • API token. Read scopes for shipments, tracking events, documents, carriers, stops, customer-references
  • Webhook subscription. Shipment status changes, exception events, document attachments
  • Customer-reference field. Identify which field on the TURVO shipment maps to Nathel’s PP PO number. This is the join key. Without this, no PO binding is possible. Confirm with Alex Cohen / TURVO operations contact before Week 2 starts.

4.2 Field-Level Pull List

Shipments / loads. Pull on creation + each status change.

  • shipment_id, customer_reference (→ po_id), carrier_id, tractor_id, trailer_id, driver_id, pickup_location, pickup_window_start, pickup_window_end, delivery_location, delivery_window_start, delivery_window_end, status, commodities, weights, equipment_type (reefer/dry), temperature_setpoint

Tracking events. Pull via webhook subscription; only event-driven (no polling for telemetry).

  • event_type (created, tendered, dispatched, picked_up, in_transit, arrived, unloaded, closed, exception)
  • event_timestamp, location (lat/lng if available), notes, temperature_at_event (if equipped)

Documents. Pull on attach event.

  • document_id, document_type (BOL, invoice, COA, organic_cert, phytosanitary, packing_slip), attached_at, document bytes (stored in R2 via StorageProvider)

Carriers. Pull once per carrier, refresh weekly.

  • carrier_id, carrier_name, mc_number, dot_number, insurance_status, safety_score, equipment_types

Stops. Multi-stop pickups are common in produce.

  • stop_sequence, stop_type (pickup, dropoff), planned_arrival, actual_arrival, planned_departure, actual_departure, location

4.3 Event Schemas

  • inbound.shipment.created — initial creation
  • inbound.shipment.status_changed — every status transition (one event per transition)
  • inbound.shipment.tracking_event — exception events (delay, breakdown, temperature alarm, missed pickup, detention)
  • inbound.shipment.document_attached — BOL, invoice, certs
  • inbound.shipment.completed — final closure
  • carrier.master.upserted — carrier dimension upsert
  • stop.completed — individual stop completion within multi-stop shipment

All events carry po_id (from customer_reference). All events also carry shipment_id. Shipments may map 1:N to POs if a single TURVO shipment covers multiple POs — model this as one event per PO when a single tracking event spans multiple PO bindings.

4.4 Projection Updates

  • inbound_shipment_projection (new CQRS projection) — current state per shipment
  • inbound_cold_chain_projection (new) — temperature exposure summary per shipment per PO
  • carrier_scorecard_projection (new) — rolling on-time %, reefer compliance %, exception rate, claim rate
  • vendor_lead_time_projection (new) — observed lead time per vendor per commodity vs. PO expected_arrival_date

4.5 MD Projection Updates

  • Append to po_lifecycle doc for every shipment event tied to that PO
  • New doc_kind: carrier_profile — rendered from carrier_scorecard_projection
  • New doc_kind: cold_chain_inbound_report — per-PO inbound cold-chain summary
  • Update to vendor_profile — vendor_lead_time observed metrics appended

4.6 Webhooks vs Polling

Subscribe to webhooks for shipment status changes, exception events, document attachments. Poll daily for: carrier master refresh, completed shipments that may have been missed (reconciliation pass). Do not poll for location telemetry — TURVO is event-driven, not telemetry-stream.

5. Week 3 — Samsara Outbound

Samsara is Nathel’s outbound fleet telemetry — vehicle locations, driver behavior, reefer temperature, geofence events. Different shape from TURVO: high-frequency telemetry rather than event-driven. The discipline is sample-rate management — do not store every ping in the events table.

5.1 Scope Required

  • API token. Fleet read, sensors read, drivers read, geofences read, routes read, webhooks
  • Webhook subscription. Geofence enter/exit events, alarm events, HOS violations
  • Join key discipline. Samsara routes/trips need to link back to PP sales orders. This is typically operator-entered. If the link is unreliable, fixing the operator-entry discipline is itself a Week 3 task before pulling Samsara data into projections.

5.2 Field-Level Pull List

Vehicle locations and trips. Do NOT store every ping. Store derived trip events; reference raw telemetry in R2 or on-demand via Samsara API.

  • Trip-level: trip_id, vehicle_id, driver_id, route_id, started_at, ended_at, mileage, fuel_used
  • Geofence-derived: arrived_at_customer, departed_customer (per stop)

Geofence events. This is the operational gold. Subscribe via webhook.

  • geofence_id, geofence_name, customer_code (mapped), vehicle_id, event_type (enter/exit), event_timestamp

Driver scores. Pull rolling scorecard per driver, daily.

  • driver_id, score_period, harsh_braking_count, harsh_acceleration_count, harsh_turning_count, speeding_minutes, distracted_driving_events, idling_hours, hos_violations, composite_safety_score

Reefer / temperature sensors. Sampled time series. Store summarized per leg + alarms individually.

  • Per leg: vehicle_id, leg_id, setpoint, min_temp, max_temp, mean_temp, temp_excursion_minutes, samples_count
  • Alarms (individual events): alarm_id, vehicle_id, alarm_type (over_temp, under_temp, door_open, sensor_failure), triggered_at, cleared_at

Routes. Plan vs actual.

  • route_id, route_plan (sequence of stops with planned times), actual_completion (sequence of stops with actual times), route_completion_status

HOS data. Daily per driver.

  • driver_id, hos_date, drive_hours_used, drive_hours_available, on_duty_hours_used, cycle_hours_used, cycle_hours_available

5.3 Event Schemas

  • outbound.trip.started
  • outbound.trip.geofence_arrived (customer arrival)
  • outbound.trip.geofence_departed (customer departure)
  • outbound.trip.completed
  • outbound.reefer.alarm_triggered
  • outbound.reefer.alarm_cleared
  • outbound.driver.score_captured (daily)
  • outbound.hos.status_captured (daily)
  • outbound.route.plan_completed

All trip events carry sales_order_id (resolved via Samsara route ↔ PP sales order link), which resolves to po_id via the inventory lot consumed. PO binding may be 1:N for a trip that delivers multiple sales orders (multi-stop outbound route).

5.4 Projection Updates

  • outbound_trip_projection — current state per trip
  • outbound_cold_chain_projection — outbound temperature exposure summary per trip per sales order per PO
  • driver_scorecard_projection — rolling driver safety metrics
  • customer_service_level_projection — on-time delivery, dwell time per customer per period
  • route_efficiency_projection — plan vs actual, route-level

5.5 MD Projection Updates

  • Append to po_lifecycle doc for outbound leg events tied to that PO
  • New doc_kind: driver_profile — rendered from driver_scorecard_projection
  • New doc_kind: cold_chain_outbound_report — per-trip outbound cold-chain summary
  • New doc_kind: customer_service_level_report — per-customer SLA tracking
  • Update to customer_profile — service-level observed metrics appended

5.6 Storage Discipline (Critical)

  • Events table: trip lifecycle events, geofence events, alarm events, daily scorecards. Bounded volume. Append-only.
  • operational_memory: summarized per-leg reefer time series (min/max/mean/excursion minutes), route plan vs actual comparison. Larger payloads, not events.
  • R2: raw telemetry firehose (if ever needed for forensic deep-dive). Referenced by event payloads via storage URIs. NOT pulled by default — pulled on-demand by tools.

Pulling raw telemetry into Postgres is a Postgres anti-pattern. The 4-table law is preserved by keeping raw telemetry out of the events table and using operational_memory for summaries plus R2 for the firehose.

6. Week 4 — Aggregate Intelligence Projections

Week 4 produces the projections that fuse the planes — the projections that make the agent’s “knowing produce” real.

6.1 Vendor Scorecards

Per vendor, rolling 30/60/90 day:

  • On-time arrival % (TURVO actual vs PO expected_arrival_date)
  • Receiving quantity variance (PP po.received qty vs PO expected qty)
  • QC pass rate (qc.inspection.completed accept rate)
  • Inbound cold-chain compliance (temperature excursion minutes per shipment)
  • Vendor-attributable shrink (inventory.adjustment.applied tagged to vendor)
  • Document completeness (BOL, COA, certs attached %)

MD projection: vendor_profile → richer; new section “Operational scorecard (last 90 days)“.

6.2 Carrier Scorecards

Per carrier, rolling 30/60/90 day:

  • Inbound on-time % (TURVO data)
  • Outbound on-time % (Samsara data, if Nathel uses external outbound carriers)
  • Inbound reefer compliance %
  • Exception rate (per shipment)
  • Detention time (per shipment)
  • Claim rate (per shipment)
  • Safety score trend

MD projection: carrier_profile doc with the scorecard rendered.

6.3 Customer Scorecards

Per customer, rolling 30/60/90 day:

  • Sales volume + revenue trend
  • Buying pattern (commodities, seasonality)
  • Outbound on-time delivery %
  • Average dwell time at customer location
  • Comms tone trend (from comms system signals — escalation rate, complaint rate)
  • Claims rate

MD projection: customer_profile doc enriched with operational scorecard.

6.4 Commodity Profiles (daily updates)

Per commodity, daily:

  • Current on-hand position
  • Sales velocity (7/14/30 day)
  • Inbound pipeline (open POs not yet received)
  • Average margin trend
  • QC defect class distribution (last 30d)
  • Vendor mix (last 30d)
  • Customer mix (last 30d)

MD projection: commodity_profile updated daily with the latest aggregate intelligence.

6.5 Sight-Picture Projections

The PO + sales + inventory three-way reconciliation. Per commodity per day:

  • What’s in stock right now
  • What’s expected in (open POs)
  • What’s expected out (sales orders in queue)
  • Projected stock position +1d, +3d, +7d
  • Gaps (where projected stock goes negative or excessive)

MD projection: sight_picture_digest (new doc_kind, daily projection per commodity).

7. TURVO Field-Level Pull Reference

Consolidated reference for the TURVO ingest extractor implementation.

7.1 Shipments Endpoint

GET /api/v2/shipments?modified_since={timestamp} — incremental pull

  • Fields: id, customer_reference, status, carrier, tractor, trailer, driver, pickup_location, delivery_location, equipment_type, temperature_setpoint, commodities, weights, created_at, updated_at

7.2 Tracking Endpoint

GET /api/v2/shipments/{id}/tracking — event chain per shipment

  • Fields: event_type, timestamp, location, notes, temperature_reading, exception_code

7.3 Documents Endpoint

GET /api/v2/shipments/{id}/documents

  • Fields: document_id, document_type, attached_at, url (signed)
  • Document bytes pulled via separate fetch + stored in R2

7.4 Carriers Endpoint

GET /api/v2/carriers?modified_since={timestamp}

  • Fields: id, name, mc_number, dot_number, insurance_status, safety_score, equipment_types

7.5 Webhook Subscription

POST /api/v2/webhooks/subscriptions

  • Events: shipment.created, shipment.status_changed, shipment.tracking_event, shipment.document_attached, shipment.exception
  • Endpoint: https://api.fin.app/webhooks/turvo/{tenant_id} (route to Worker that emits FIN events)
  • Signing: HMAC SHA-256 per webhook subscription
  • Idempotency: dedupe on (event_id, event_type)

8. Samsara Field-Level Pull Reference

Consolidated reference for the Samsara ingest extractor implementation.

8.1 Vehicles / Trips

GET /fleet/trips?startTime={ts}&endTime={ts}

  • Fields: id, vehicleId, driverId, routeId, startTime, endTime, distanceMeters, fuelConsumedMl

8.2 Geofence Events

GET /fleet/geofences/stats?startTime={ts}&endTime={ts} or webhook

  • Fields: geofenceId, geofenceName, vehicleId, eventType (entered/exited), timestamp

8.3 Driver Scores

GET /fleet/drivers/safety/score?driverIds={...}&startMs={...}&endMs={...}

  • Fields: driverId, scorePeriod, harshBraking, harshAcceleration, harshTurning, speedingMinutes, distractedDrivingCount, idlingHours, hosViolations, compositeScore

8.4 Sensors (Reefer)

GET /fleet/assets/sensors/history?assetIds={...}&startMs={...}&endMs={...} (sampled)

  • Fields: assetId, timestamp, temperatureSetpoint, temperatureActual, alarmActive

8.5 Routes

GET /fleet/routes?startTime={ts}&endTime={ts}

  • Fields: id, name, scheduledRouteId, stops (sequence with planned + actual times)

8.6 HOS

GET /fleet/hos/daily-logs?driverIds={...}&startDate={...}&endDate={...}

  • Fields: driverId, logDate, driveTimeHours, onDutyTimeHours, cycleHoursUsed, cycleHoursAvailable

8.7 Webhook Subscription

POST /webhook (Samsara webhook subscription endpoint)

  • Events: Geofence Entry, Geofence Exit, Vehicle Maintenance Alert, Sensor Alarm Active, Sensor Alarm Cleared, HOS Violation
  • Endpoint: https://api.fin.app/webhooks/samsara/{tenant_id}
  • Signing: HMAC SHA-256 per webhook subscription

9. Event Schemas (Consolidated New Event Kinds)

This roadmap introduces the following new event kinds. All are subject to the signal.* event family reservation memory and event-table append-only discipline.

po.created
po.received
po.line.received
commodity.master.upserted
vendor.master.upserted
customer.master.upserted
sales.line.created
inventory.snapshot.captured
inventory.lot.snapshotted
inventory.adjustment.applied
inventory.divergence.detected
inbound.shipment.created
inbound.shipment.status_changed
inbound.shipment.tracking_event
inbound.shipment.document_attached
inbound.shipment.completed
carrier.master.upserted
stop.completed
outbound.trip.started
outbound.trip.geofence_arrived
outbound.trip.geofence_departed
outbound.trip.completed
outbound.reefer.alarm_triggered
outbound.reefer.alarm_cleared
outbound.driver.score_captured
outbound.hos.status_captured
outbound.route.plan_completed

Each event payload conforms to the existing event schema discipline:

  • event_id, event_kind, event_ts, tenant_id, actor_id, payload_json, correlation_id, causation_id
  • po_id is a first-class field on every event that has a PO binding (denormalized for query performance)

10. Projection Migration Plan

The following new CQRS projections amend the migration sequence. Numbering picks up after the planned 045-048 (doc_backlink, entity_doc, mcp_tool_registry, qc_inspections).

  • 049 — po_state_projection. Current state per PO; supports get_po_state(po_id).
  • 050 — inventory_lot_projection. Lot-level inventory; references PO.
  • 051 — inventory_state_projection. Current on-hand position per commodity per location; reconciled against daily snapshots.
  • 052 — sales_velocity_projection. Rolling 7/14/30 day sales per commodity.
  • 053 — customer_buying_pattern_projection. Per customer commodity-mix and seasonality.
  • 054 — inbound_shipment_projection. Per shipment state.
  • 055 — inbound_cold_chain_projection. Per shipment per PO temperature exposure.
  • 056 — carrier_scorecard_projection. Per carrier rolling metrics.
  • 057 — vendor_lead_time_projection. Per vendor per commodity observed lead time.
  • 058 — outbound_trip_projection. Per trip state.
  • 059 — outbound_cold_chain_projection. Per trip per sales order per PO temperature exposure.
  • 060 — driver_scorecard_projection. Per driver rolling safety metrics.
  • 061 — customer_service_level_projection. Per customer SLA tracking.
  • 062 — route_efficiency_projection. Per route plan vs actual.
  • 063 — sight_picture_projection. Per commodity sight-picture daily snapshot.

All projections honor the 4-table law: they sit as derived state in operational_memory, not as new tables. Migrations create indexes on operational_memory keyed by projection_name + projection_key.

11. Join-Key Discipline

The join keys that bind the spine:

  • po_id — primary spine. PP PO number, used as the join key everywhere. Every event carries it where applicable.
  • vendor_code — PP vendor code. Used to join PO → vendor master → vendor profile.
  • commodity_code — PP commodity code. Used everywhere commodity is referenced.
  • customer_code — PP customer code. Used to join sales → customer master.
  • lot_id — Internal lot identifier assigned on PO receipt. Lives as long as the lot lives.
  • sales_order_id — PP sales order ID. Used to join Samsara trips → sales orders → inventory lots → PO.
  • shipment_id — TURVO shipment ID. Used internally for inbound shipment lifecycle tracking.
  • trip_id — Samsara trip ID. Used internally for outbound trip lifecycle tracking.

Discipline. Every ingest extractor MUST emit events with the correct join keys populated. Events emitted without a resolvable po_id (where one is expected) route to a pending bucket (operational_memory.namespace = pending_po_resolution) and trigger a reconciliation pass. Operational projections do NOT consume from the pending bucket — they wait for the reconciliation to bind, OR they emit a signal.po_binding_missing alert if the pending event ages past 48 hours.

12. Webhooks vs Polling Strategy

  • Real-time webhooks: TURVO shipment status changes, exception events, document attachments. Samsara geofence events, alarm events, HOS violations.
  • Daily polling: Carrier master refresh (TURVO), commodity/vendor/customer master refresh (PP), daily inventory snapshot (PP — manual export, no API), daily sales report (PP — manual export, no API), driver scorecards (Samsara), HOS daily logs (Samsara).
  • On-demand only: Raw Samsara telemetry. PP report PDFs (extractor-triggered).
  • Reconciliation passes: Daily sweep of TURVO/Samsara to catch webhook misses; weekly sweep of all sources to validate completeness against expected event counts.

13. Storage Discipline (4-Table Law Compliance)

  • events table: All lifecycle events (immutable, append-only). Bounded volume per source.
  • knowledge table: Master data (commodities, vendors, customers, carriers, drivers as entity rows). Doc projections (commodity_profile, vendor_profile, etc.) as domain = doc rows.
  • actors table: Per-actor identity and authorization (no operational data).
  • operational_memory table: All CQRS projections, all summarized time-series, all reconciliation state. Keyed by projection_name + projection_key.

No new tables. Ever. This is non-negotiable per the 4-table constraint memory.

Raw byte payloads (PDFs, BOLs, photos, raw telemetry firehose) live in R2 via the StorageProvider abstraction, referenced from event payloads by storage URI.

14. Open Decisions

  • TURVO customer-reference field mapping. Confirm with Alex Cohen which TURVO field maps to PP PO number. Block Week 2 start until confirmed.
  • Samsara route ↔ PP sales order link. Currently operator-entered. Audit reliability before pulling Samsara into projections. May need an operator-discipline fix (Week 3 pre-task).
  • PP report extraction cadence and authority. Daily batch (acceptable) or real-time via terminal-emulator polling (preferred for inventory)? Defer to PP runtime sequencing memory: Claude/Codex drives initial dial-in, OpenClaw is Phase B.
  • Document storage retention. BOLs, COAs, certs — 7-year retention per federal substrate. Cold storage transition policy TBD.
  • Multi-PO TURVO shipment modeling. When a single TURVO shipment covers multiple POs, emit one event per PO or one event with multi-PO array? Default: one event per PO (denormalized for query simplicity); revisit if event volume becomes painful.
  • Multi-stop Samsara trip modeling. Same question, outbound side. Default: emit one trip event with stops array, plus individual geofence_arrived events per stop.
  • Reconciliation alert routing. Where does signal.po_binding_missing route? Default: Slack fin-alerts channel + comms thread tagged to ops manager + intranet escalation queue.

15. Acceptance Criteria

End of Week 1.

  • PP extractor trio producing real events bound to real po_id values
  • Watermelon commodity_profile MD projection renders with current operational signal: on-hand, sales velocity, recent receiving, recent QC findings
  • QC live-test events all bind to real PO records
  • Comms threads bind to PO records when context is clear; otherwise stay in pending
  • Acceptance test: agent answers “what’s our watermelon position right now?” with citations to operational projections, in under 5 seconds

End of Week 2.

  • TURVO inbound webhooks subscribed and firing
  • Every inbound shipment in TURVO since subscription start has a corresponding inbound.shipment.* event chain in FIN’s events table
  • inbound_cold_chain_projection and carrier_scorecard_projection rendering for at least 10 shipments
  • Acceptance test: agent answers “which carriers ran cold-chain on our watermelon loads this month and how did they perform?” with citations

End of Week 3.

  • Samsara outbound webhooks subscribed and firing
  • Every outbound trip since subscription start has a corresponding outbound.trip.* event chain in FIN’s events table
  • outbound_cold_chain_projection, driver_scorecard_projection, customer_service_level_projection rendering for at least 50 trips
  • Acceptance test: agent produces grower-to-customer provenance with timestamped cold-chain proof for a unit sold last week. This is the FSMA 204 demo moment.

End of Week 4.

  • Vendor / carrier / customer scorecards rendering daily
  • Sight-picture digest rendering daily per commodity
  • Buyers Org Box authored corpus integrated as authored knowledge citation target
  • Acceptance test: agent produces a Monday-morning sight-picture briefing for the operations manager that fuses PO pipeline, current inventory, sales pipeline, vendor scorecards, and identified gaps — with citations across all four data sources

16. Cross-Doc Dependencies

  • 2-Week Execution Program — Week 1 of this roadmap is the operational corpus portion of the existing 2-week sprint
  • FIN Self-Improving System Roadmap — uses this corpus as the substrate for verifiers, case memory, procedure projections, counterfactual replay
  • FIN Intranet Human-Agent Projection Engineering Spec v0.2 — doc_kind extensions (po_lifecycle, carrier_profile, driver_profile, cold_chain_inbound_report, cold_chain_outbound_report, customer_service_level_report, sight_picture_digest, daily_inventory_digest) amend the projection spec
  • Projection Catalog v0.1 — new projections need catalog entries; tier-tagging required for Wave 2-3 anonymization policy compliance
  • Tenant Demarcation Plan — TURVO and Samsara are tenant-side data sources; ingestion lives on Nathel deployment, NOT FIN Central
  • Anonymization Policy v1 — any of these projections that flow to Plane 3 (public Harvest Directory) MUST pass anonymization at projection time

17. The Punchline

PO is the spine. Bind every signal to it. Ingest order is: PP masters → PP PO lifecycle → PP sales + inventory → TURVO inbound → Samsara outbound → aggregate intelligence projections that fuse the planes. By Week 4 the agent has a continuous operational record from grower to customer, with structural provenance, with cold-chain proof, with scorecards across every dimension Nathel cares about.

That is the corpus the Self-Improving System Roadmap builds verifiers, case memory, procedure projections, and counterfactual replay on top of. No corpus, no self-improvement. Get the corpus right first.

18. Audit Annotation - 2026-05-20

Roadmap audit tracker: FIN Agentic Harness and Knowledge Roadmap Audit Tracker.

Verified correction: current code maps Produce Pro reports as PD335 = vendor, PD327 = customer, and COMMODITY = commodity master. Earlier W1 text had PD335/PD327 reversed for commodity/vendor. This document now uses the verified mapping.

Scope clarification: po_id remains the spine for PO-bound operational facts, but not every source class can carry po_id at ingest time. Master-data rows, background knowledge, and unbound signals must route through entity keys or pending_po_resolution until an explicit reconciliation pass binds them.

Implementation status: active PO Received/PD032 ingestion is partially implemented and tested as event-store facts with freshness/retention metadata. The open W1 execution task is to complete the RE101/PO-created side, corrected master-data joins, and a watermelon commodity_profile that renders from real operational data and is answerable by the agent with citations.

19. Current Mailbox Ingestion Status - 2026-05-20

This roadmap should not be read as if all Produce Pro ingestion starts from zero. Source review verifies that several Produce Pro report paths already enter FIN through intel@fintail.net and arrivals@fintail.net, then flow through an event-driven processing pipeline.

19.1 Verified email intake and artifact path

  • fin-email-worker forwards parsed mailbox payloads to /webhooks/email/inbound and /webhooks/email/intel; the shared payload schema explicitly models mailbox = arrivals | intel | unknown.
  • The email webhook stores attachments, computes SHA-256, writes object storage, indexes knowledge[source_artifact], and emits ingest.artifact_stored.
  • Produce Pro CSV attachments are detected inline from filename/content, then passed to processReport(...).
  • Produce Pro attachment provider traces include produce_pro_report_type, produce_pro_report_code, raw-report retention policy, and active-PO report metadata when applicable.

19.2 Verified CSV report coverage

The current Produce Pro bridge recognizes these report families:

Bridge typeReport codeCurrent treatment
stockRP711Event-backed import; emits inventory snapshot facts, freshness, bridge.report_imported
inventoryRP421Event-backed import; emits expected/received inventory facts, freshness, bridge.report_imported
vendorPD335Actor/master-data upsert; freshness, bridge.report_imported
poRE101 / active PO CSV profileEvent-backed import; active PO CSV path can infer PD032 PO Received freshness and expected-vs-received event plans
customerPD327Actor/master-data upsert; freshness, bridge.report_imported
arrivalRP581Event-backed import; also has PDF arrival routing path below
commodityCOMMODITYKnowledge upsert under Produce Pro commodity bridge
salesSALESEvent-backed import; freshness, bridge.report_imported
physical_inventoryPHYSICAL_INVENTORYEvent-backed import; freshness, bridge.report_imported

This means the W1 task is not “invent email ingestion”; it is “prove and complete the already-wired mailbox-to-operational-truth path, then project it into the agent-visible commodity/PO surfaces.”

19.3 Verified PDF and signal processing path

  • PDF attachments emit ingest.pdf_pending_extraction.
  • The ingestion consumer classifies PDFs as arrival_report, vendor_pricelist, and related intel classes through heuristics or an approved raw-PDF privacy path.
  • For arrival_report, the pipeline routes to arrival-routing.ts: it finds the paired source artifact, fetches bytes through StorageProvider, runs the deterministic RP581 parser first, falls back to the PDF extractor for non-RP581 arrival layouts, adapts to PPInboundArrivalRow, and calls pull_pp_inbound_arrivals.
  • The arrival route can emit ingest.pdf_extracted, signal.pp_arrival_logged, lot.pre_arrival_synthesized, and inbound.load_expected.
  • dispatchIngestionConsumers fans signal.email_extract into correspondence extraction, pricing rail extraction, commodity mention extraction, temporal resolution, cross-linking, arrival update observation, arrival fusion, authority routing, and QC thread creation.

19.4 Remaining proof and completion work

  • Run a fresh staging/live proof for newest intel@ and arrivals@ messages showing the full chain: mailbox receipt → source artifact → CSV import or PDF extraction → normalized events/signals → freshness/knowledge promotion → agent answer with citations.
  • Prove active PO Create / PO Received reports through event-store rows, load-context package signals, Work Thread rows, and PO-bound source attribution.
  • Treat vendor_pricelist PDFs as classified-but-not-yet-fully-operational until a structured price-list PDF extractor/adaptor emits normalized price observations or an explicit terminal backlog event.
  • Add mailbox/report freshness health to the verifier/dashboard work so agents can ask whether the evidence behind a Produce Pro answer is fresh enough before citing it.

Graph View

Backlinks