FIN Central

Central Taproot External-Source Freshness Audit

4 min read

Central Taproot External-Source Freshness Audit

Audit timestamp: 2026-05-20T03:10Z

Scope: verify whether external-source data is refreshed automatically through FIN Central / Knowledge Forge, and whether tenant taproots are receive-only for external data.

Executive Finding

The intended architecture is right: FIN Central Taproot / Knowledge Forge should be the only external-source gatherer. Tenant taproots should receive Central Taproot mirrors plus tenant operational data only.

Current runtime does not fully meet that rule. Knowledge Forge has active scheduled crawlers and a deployed active-agent worker, but several source families are failing after the crawl layer, and fin-agentic still contains tenant-side external fetch paths for USDA AMS, NWS weather, and openFDA recall checks.

Verified Source Status

Source familyCurrent verdictProof
NOAA weather alerts via KF active-agentFresh in Central stagingcrawler_health shows active-agent:weather_alert completed at 2026-05-20T03:01:38Z; source_documents has 23 verified docs, latest crawled 2026-05-19T22:25:00Z; intelligence_events has 94 noaa/weather_alert rows.
Active intelligence v2Fresh in Central stagingGitHub workflow green hourly; D1 crawler_health completed 2026-05-20T03:02:55Z; source_documents latest crawled 2026-05-19T20:30:20.533Z.
USDA AMS Market NewsNot citable/fresh end to endGitHub crawl green on 2026-05-19T07:10:15Z, but D1 extraction status is failed at 2026-05-19T07:12:13Z; raw R2 object raw/usda-ams-market-news/2026-05-19/3324.json contains HTTP 401 body User is not found; last successful D1 extraction is 2026-05-16T21:39:16Z.
Composite USDA crawlPartially fresh, not cleanGitHub crawl-usda.yml green through 2026-05-19T07:18:13Z; D1 has 275 completed usda extraction jobs and 731 entities, but quality crawler_health is failed with FOREIGN KEY constraint failed.
Market news / terminal marketsBrokenLatest crawl-market-news.yml failed on 2026-05-19T22:24:38Z; terminal-markets failed all markets with No address associated with hostname; extraction queue step was skipped.
PACA licensesBroken / not refreshingConfig says weekly, workflow is monthly; latest scheduled run 2026-05-01 was cancelled after 3 hours; prior runs failed; D1 source_documents has no PACA rows.
FDA recallsCrawl green, extraction brokenGitHub crawl-fda-recalls.yml green through 2026-05-19T09:03:02Z; D1 extraction jobs are failed with Claude response was not valid extraction JSON.
Federal RegisterCrawl green, extraction brokenGitHub crawl-federal-register.yml green through 2026-05-19T08:08:06Z; D1 extraction jobs are failed with Claude response was not valid extraction JSON.
Drought MonitorCrawl green, extraction brokenGitHub crawl-drought-monitor.yml green through 2026-05-14T12:46:16Z; D1 extraction jobs for drought-monitor are failed with 0 entities.
Google TrendsCrawl green, no queue publish proofWorkflow is green weekly, but inspected workflow only runs crawl; no extraction queue publish step is present.

Tenant Boundary Finding

fin-agentic still contains live tenant-side external source paths:

  • packages/mcp-server/src/ops/usda-market-multi-tenant.ts runs a multi-tenant USDA market refresh and calls scanSupplyConstraintsWithClient.
  • packages/mcp-server/src/tools/usda-market-refresh.ts can fetch live USDA Market News when USDA_MARKET_NEWS_ENABLED=true.
  • packages/mcp-server/src/integrations/usda-market-news.ts calls https://marsapi.ams.usda.gov/services/v1.2.
  • packages/mcp-server/src/tools/supply-chain-monitor.ts, evaluate-weather-impact.ts, and trigger-weather-cascade.ts can fetch NWS alerts via fetchNWSWeatherAlerts.
  • packages/mcp-server/src/tools/fda-recall-monitor.ts can fetch https://api.fda.gov/food/enforcement.json when enabled.

Even if some switches are off in a given environment, the architecture is not yet enforced. Tenant systems should consume Central-derived signal projections, Central mirror docs, operational tenant data, or explicitly supplied alert payloads. They should not fetch external public data directly.

Tenant Freshness Surface

Existing staging proof confirms the serving gap:

  • .artifacts/staging-verification/operational-data-spine-r2-2026-05-20/operational-data-spine-freshness-gap-staging-proof.json
  • source_class_freshness.usda_ams.status = unknown
  • vendor_quote.status = stale
  • produce_pro, source_artifact, and operational are fresh.

That means tenant operational data is healthy, but Central external market data is not yet landing as a fresh, citable tenant evidence class.

Required Remediation

  1. Enforce Central-only external gathering: disable or migrate tenant-side USDA/NWS/FDA external fetchers; leave tenant tools as consumers of Central signal projections or caller-supplied payloads.
  2. Repair Knowledge Forge AMS auth/extraction: a green crawl that writes a 401 raw object must fail the crawl before queue publication.
  3. Repair PACA: align config/workflow cadence, stop the 3-hour cancellation, and prove staged source docs.
  4. Repair market news: isolate terminal-market DNS failures from AMS, allow partial source success, and publish only valid source prefixes.
  5. Repair post-crawl extraction for FDA, Federal Register, Drought Monitor, and Google Trends queue publication.
  6. Add a Central Taproot freshness dashboard/reader that joins GitHub workflow status, D1 crawler_health, source_documents, extraction jobs, R2 object existence/hash, migration status, and tenant mirror freshness.
  7. Feed Central external signals into MD-first projection evidence manifests so FINN can cite them through stable chunk ids, content hashes, freshness SLA, and source license.

Graph View