FIN Reliability and Ops Roadmap
0. Revision Summary
First cut. Defines reliability and operational discipline for FIN running inside a high-ops-tempo, high-data-volume NYC produce wholesaler. Specifies latency budgets per surface, throughput targets, queue depth budgets, observability stack, SLO/SLI framework, incident response playbooks, DR scenarios, and on-call model. Closes Pillar 6 of the Platform North Star Roadmap.
1. Operating Environment Characterization
FIN’s first deployment is Nathel & Nathel Produce at Hunts Point. Operating realities that drive reliability requirements:
Continuous operations. Hunts Point is effectively a 24/7 facility. Early-morning receiving (3-7 AM), midday QC and fulfillment, late-night outbound loads, weekend operations. The platform cannot have routine maintenance windows that take it offline during operating hours, because there are no non-operating hours. Maintenance must be rolling.
High event volume. Conservative steady-state estimate at Nathel scale:
- ~50 POs created per day
- ~150 inbound shipment events per day (TURVO webhooks)
- ~500-1000 QC inspection events per week
- ~2000-5000 sales line events per day
- ~10000-20000 Samsara geofence and trip events per day
- ~5000+ comms events per day (messages, threads, voice transcriptions)
- Inventory snapshot reconciliation events daily
- Document attachments continuously
This is ~30k-50k events per day per tenant at steady state. Multi-tenant at three tenants is 100k-150k events per day before federated intelligence projections.
Low operator latency tolerance. When a truck is on the dock and the foreman asks “what’s on this load?” — they will not wait 5 seconds for an agent response. The threshold for an agent-mediated answer is 1 second median, 2 seconds p99 from question to first byte rendered. Anything slower and the operator goes around the system.
High criticality. Food safety, perishability (hours-not-days windows), customer claims, financial settlement. Wrong answers are not “annoying” — they cost money, food, or safety. Reliability is a quality dimension, not just an uptime dimension.
High variance. Commodity rotation, seasonality, weather shocks, market price movements, vendor mix changes. The system must absorb variance without operational disruption.
Live test posture (now and through Wave 2). QC live test and comms live test are running against production data. Production has real users (Alex Cohen, foreman, inspectors). Production posture is required from day one. There is no “soft launch.”
These realities define the bar: production-grade reliability discipline starting now, not later.
2. Surface Tiers and Latency Budgets
Different surfaces have different latency budgets driven by user expectation. The harness must enforce these.
2.1 Surface Tier S0 — Real-Time Operator Surfaces
Surfaces operators touch in active workflow. Foreman on dock, inspector mid-inspection, dispatcher on phone, customer service rep on call.
- Examples: QC inspector PWA on mobile, voice agent, comms thread reply, dispatch agent answer
- Latency budget: p50 < 1s, p99 < 2s from input to first byte rendered
- Throughput budget: ~100 concurrent operator sessions sustained
- Backpressure policy: never block; degrade gracefully (smaller context window, smaller model, cached answer with staleness banner)
- Availability target: 99.5% during operating hours (which is effectively 24/7)
2.2 Surface Tier S1 — Near-Real-Time Operator Surfaces
Surfaces operators touch but not on the critical path. Dashboards, daily digests, sight-picture briefings, weekly scorecards.
- Examples: intranet dashboards, daily inventory digest, sight-picture digest, vendor scorecard rendering
- Latency budget: p50 < 3s, p99 < 8s
- Throughput budget: ~50 concurrent sessions
- Backpressure policy: queue with explicit “computing…” UI affordance; eventual completion guaranteed
- Availability target: 99% during operating hours
2.3 Surface Tier S2 — Batch Ingestion and Projection Refresh
Background pipeline work. PP report extractor, TURVO webhook handler, Samsara webhook handler, projection refresh, MD projection re-rendering.
- Examples: RP581 extractor, daily sales extractor, projection migrations
- Latency budget: event-to-event-store < 30s p99 for real-time signals; daily batches complete within the daily window (overnight)
- Throughput budget: must absorb peak event rate (3x steady state) without backpressure to upstream
- Backpressure policy: queue-with-DLQ; alert at queue depth > N; never drop
- Availability target: 99.9% measured over the daily window
2.4 Surface Tier S3 — Asynchronous Compute
Deep analyses, scheduled reports, gradeboard runs, model evaluations, embedding refreshes, counterfactual replays.
- Examples: weekly vendor analysis, gradeboard nightly run, embedding refresh on doc edits
- Latency budget: hours to days; explicit job ID + status
- Throughput budget: elastic — scale workers within cost ceiling
- Backpressure policy: queue; defer; explicit completion notification
- Availability target: completion within stated SLA (e.g., gradeboard nightly within 6h overnight)
3. Throughput Targets
Per-surface throughput targets sized for Nathel-only and Nathel + 2 additional tenants.
| Surface | Nathel-only steady | Nathel-only peak | 3-tenant steady | 3-tenant peak |
|---|---|---|---|---|
| Agent responses / min | 30 | 120 | 90 | 360 |
| Event ingest / sec | 0.5 | 3 | 1.5 | 9 |
| Projection refreshes / hour | 50 | 200 | 150 | 600 |
| Webhook handles / sec | 1 | 10 | 3 | 30 |
| Embedding ops / hour | 100 | 1000 | 300 | 3000 |
| Doc projections rendered / day | 200 | 500 | 600 | 1500 |
Sizing discipline. Worker counts, Hyperdrive pool sizes, Durable Object instance counts, and Postgres connection budgets must comfortably handle 3-tenant peak — not Nathel-only steady. Capacity below 3-tenant peak is a known sizing gap and requires explicit acknowledgment in the deployment runbook.
4. Queue Depth Budgets and Backpressure
Every queue has a depth budget. Exceeding it triggers a graduated response.
| Queue | Normal depth | Warning depth | Alert depth | Action at alert |
|---|---|---|---|---|
| Event ingest (per source) | < 10 | 50 | 200 | Page on-call; pause that source’s webhooks; spin emergency consumer |
| Projection refresh | < 5 | 20 | 100 | Page on-call; degrade dependent surfaces to cached projections with staleness banner |
| Agent response | < 3 | 10 | 30 | Page on-call; degrade Surface S0 to smaller model; queue S1 with explicit affordance |
| Embedding ops | < 100 | 500 | 2000 | Defer non-critical embeddings; alert |
| DLQ (any) | 0 | 1 | 10 | Page on-call; halt deploys; investigate before processing |
Drop policy. Events are never dropped. DLQ catches what can’t be processed inline. DLQ entries trigger on-call attention and replay only after root cause is identified.
Degradation order. When system is under pressure, degrade in this order: S3 jobs paused → S2 batch sizes reduced → S1 surfaces show staleness banners → S0 surfaces switch to smaller/cached responses → if all of those exhausted, S0 surfaces return “system busy, retry in N seconds” rather than failing silently.
5. Observability Stack
The observability discipline that makes the above measurable.
5.1 Logs
Structured JSON logging from every Worker, every Durable Object, every batch job. Centralized to a single sink (Cloudflare Logs → R2 archive → queryable index TBD).
Required fields on every log line: timestamp, level, service, deployment, tenant_id (where applicable), actor_id (where applicable), correlation_id, causation_id, message, structured payload.
Sensitive field handling. PII redacted at log emit time per Anonymization Policy. No raw email bodies, no raw voice transcriptions, no raw chat content in centralized logs — instead, content_hash + storage URI pointing to the encrypted payload in R2.
5.2 Metrics
Time-series metrics for every quantitative SLI.
SLIs tracked per surface:
- Response latency (p50, p95, p99)
- Throughput (requests/sec, events/sec)
- Error rate (5xx, validation failures, verifier failures)
- Queue depth (current, max in window)
- Availability (success / total over rolling window)
Custom metrics:
- Gradeboard score by skill (nightly batch)
- Verifier pass rate by verifier
- Projection freshness (lag from source event to projected state)
- Embedding queue depth and processing time
- PP extractor success/failure rate
5.3 Traces
Distributed traces for every agent session. The harness emits trace spans for every tool call, every model call, every retrieval, every verifier invocation. Traces are correlation-id-keyed and downloadable from gradeboard for debugging.
Trace retention: 30 days hot, 1 year cold (R2).
5.4 Gradeboard
Already exists. Continues as the quality SLO surface. Score floor enforcement gates deploys.
5.5 Harness Telemetry
Specific to the agentic harness:
- Session count, session duration distribution
- Tool call distribution by tool
- Model routing distribution (which model was used for which question type)
- Verifier failure distribution by reason
- Skill version in use per session
- Replay determinism check pass rate
5.6 Dashboards
A handful of focused dashboards rendered in the FIN Central intranet (later: tenant intranet):
- Platform Health — uptime, error rate, queue depth, latency percentiles across all surfaces
- Ingestion Pipeline — per-source event flow, lag, DLQ status
- Agent Quality — gradeboard rolling score, verifier pass rate, model routing, skill usage
- Tenant Health (per tenant) — same as Platform Health, scoped to one tenant
- Incident Timeline — last 30d incidents, severity, resolution time
6. SLO / SLI Framework
Service Level Objectives committed to operators. Service Level Indicators measured continuously. Error budgets defined.
6.1 Initial SLOs (Wave 2, Nathel pilot)
| Surface | SLI | Target | Error Budget |
|---|---|---|---|
| S0 — Inspector PWA | Agent response p99 latency | < 2s | 5% of operating hours |
| S0 — Voice agent | Voice agent response p95 latency | < 1.5s | 5% of operating hours |
| S0 — Comms thread reply | Reply rendered p99 latency | < 2s | 5% of operating hours |
| S1 — Intranet dashboard | Render p99 latency | < 8s | 5% of operating hours |
| S2 — Event ingest | Event-to-store p99 latency | < 30s | 5% of operating hours |
| S2 — Webhook handler | Webhook ack p99 latency | < 5s | 1% of webhooks |
| All — Availability | Surface-up % over week | > 99% | 1.68h / week |
| All — Gradeboard | Rolling 7d gradeboard score | > 5.0/9.0 | Score floor; no error budget |
6.2 Error Budget Discipline
When a surface burns through > 50% of its monthly error budget in a single week, deploys to that surface require explicit go/no-go per the deploy checklist. When it burns through > 80%, the surface is feature-frozen until budget recovers.
6.3 SLO Evolution
SLO targets ratchet up over phases:
- Phase Wave 2 (now): values in §6.1
- Phase Wave 3 (next 6 months): tighten S0 latencies to p99 < 1.5s; raise availability target to 99.5%
- Phase Wave 4 (next 12-18 months): tighten S0 latencies to p99 < 1s; raise availability target to 99.9% for Tier 2+ tenants
7. Incident Response Playbooks
Every surface has an incident playbook. The playbooks are short, decisive, and live in docs/runbooks/incident-*.
7.1 Severity Levels
- SEV1: S0 surface completely unavailable OR active data loss OR security breach. Page on-call immediately. Status page update within 15 minutes. Postmortem within 5 business days.
- SEV2: S0 degraded OR S1/S2 unavailable OR data integrity drift detected. Page on-call within 30 minutes during operating hours. Status page update within 1 hour. Postmortem within 10 business days.
- SEV3: S1/S2 degraded OR S3 unavailable OR localized quality regression. Ticket created; on-call attention within 4 hours during operating hours. No postmortem unless escalates.
- SEV4: Non-blocking quality issue OR cosmetic surface bug. Backlog; addressed in normal flow.
7.2 Playbook Topics (initial set)
incident-event-ingest-stalled.md— what to do when an event source stops flowingincident-projection-divergence.md— what to do when CQRS projection diverges from event storeincident-agent-quality-regression.md— what to do when gradeboard drops below floorincident-verifier-mass-failure.md— what to do when verifier failure rate spikesincident-webhook-outage.md— what to do when an upstream webhook source goes downincident-database-degraded.md— Neon / Hyperdrive degradation playbookincident-cf-worker-degraded.md— Cloudflare Workers degradation playbookincident-pii-leak-suspected.md— suspected PII leak in projections or logs
Each playbook follows the structure: trigger → first 5 minutes → diagnostic checklist → mitigation options → escalation criteria → comms template → postmortem prompts.
8. DR / Recovery
Disaster recovery scenarios and recovery objectives.
8.1 Recovery Objectives
| Failure Mode | RTO | RPO |
|---|---|---|
| Single Worker zone failure | < 5 min (Cloudflare auto-failover) | 0 |
| Neon regional outage | < 30 min (failover to read replica region, write-path degraded) | < 60s |
| R2 bucket loss | N/A (R2 is multi-region) | 0 |
| Cloudflare global outage | RTO undefined; static intranet from R2 archive only | N/A (no writes during) |
| Git vault corruption | < 1h (restore from GitHub) | 0 |
| AWS Secrets Manager outage | < 15 min (cached secret usage) | N/A |
| Catastrophic data loss (events table) | < 24h restore from Neon PITR | < 24h |
| Tenant data exfiltration suspected | Immediate isolation; investigation; reload from event store replay | N/A |
8.2 DR Drill Cadence
- Monthly: runbook walkthrough (no actual failure injection)
- Quarterly: failover drill on staging (chaos engineering — kill a worker, drop a connection, simulate webhook outage)
- Annually: full DR exercise — simulate regional outage with full recovery procedure end-to-end
First DR drill: end of Wave 2 (within 60 days).
8.3 Backup Discipline
- Neon PITR: 7 days continuous; 30 day daily snapshots; 1 year monthly snapshots
- R2: multi-region replication enabled on tenant-bytes buckets; weekly checksum audit
- GitHub vaults: GitHub Enterprise backup + weekly local snapshot to S3 archive
- AWS Secrets Manager: versioned secrets retained 90 days
9. On-Call Model
Production reliability requires on-call. Even at solo-founder + partners scale, on-call has to exist.
9.1 Wave 2 (now): Solo On-Call
Rob is primary and secondary on-call. Pages go to phone + email. Acknowledged within 15 minutes during operating hours, 1 hour outside.
Mitigation: SEV1 escalation list includes Alex Cohen for Nathel-specific operational impact (he can pause workflows on Nathel side to reduce blast radius while Rob diagnoses).
9.2 Wave 3 (3-6 months): Two-Person Rotation
Once a second engineer is on board (FIN Central tech hire OR Anthropic engagement model resolved), establish 1-week-on / 1-week-off rotation.
9.3 Wave 4 (6-18 months): Three+ Person Rotation
Standard SRE-style rotation. Primary + secondary + executive escalation per shift.
9.4 Pager Discipline
- Pages MUST be actionable. A page that says “system slow” without a clear diagnostic is itself a bug — fix the alert.
- Pager fatigue is tracked: pages per week per person. Above 5/week, the page-volume is itself a SEV3 issue.
- Every page produces a one-line log entry in
docs/runbooks/on-call-log-YYYY-MM.md.
10. Capacity and Cost Discipline
Reliability has a cost dimension.
10.1 Capacity Headroom Targets
- Worker concurrency: utilization < 60% steady, < 85% peak
- Hyperdrive connection pool: utilization < 50% steady, < 80% peak
- Postgres CPU: utilization < 50% steady, < 80% peak
- Neon storage: < 80% of provisioned
- R2: no practical limit; monitor cost trend
10.2 Cost Ceilings (initial)
| Category | Wave 2 monthly ceiling | Wave 3 ceiling | Wave 4 ceiling |
|---|---|---|---|
| Frontier API spend | $5,000 | $15,000 | $50,000 |
| Cloudflare | $500 | $1,500 | $5,000 |
| Neon | $500 | $2,000 | $8,000 |
| R2 | $200 | $800 | $3,000 |
| AWS (Secrets Manager etc.) | $100 | $300 | $1,000 |
| Total platform | $7,000 | $20,000 | $70,000 |
Cost overrun > 20% in a month is a SEV3 issue and triggers a cost review.
10.3 Model Routing for Cost
Per harness spec, model routing chooses the right model per question class. Reliability discipline ratchets the cost lever:
- Tier S0 fast responses: smaller/faster model with fallback to larger on verifier failure
- Tier S1 dashboards: larger model OK, but cached aggressively
- Tier S3 deep analyses: largest model acceptable; bounded by per-job cost cap
11. Phased Delivery
Phase R1 (current 2-week sprint). SLO definitions per §6.1 written down. Initial logging structure standardized. Gradeboard already running. On-call protocol documented (Rob solo).
Phase R2 (next 4-6 weeks). Metrics dashboards landed (Platform Health, Ingestion Pipeline, Agent Quality, Tenant Health, Incident Timeline). Initial incident playbooks for the 8 topics in §7.2. First DR drill executed.
Phase R3 (3-6 months). SLO tightening per §6.3. Two-person on-call rotation. Quarterly DR drill. Cost discipline ratcheted to Wave 3 ceilings.
Phase R4 (6-18 months). Multi-region failover ready. 99.9% availability on Tier 2 surfaces. Three-person on-call rotation. Annual full DR exercise.
12. Acceptance Criteria
End of Phase R1.
- SLO targets §6.1 written and visible in this doc
- Gradeboard floor enforced on deploys
- Initial on-call protocol documented
- Logging discipline §5.1 applied to all Workers
End of Phase R2.
- Five dashboards live in FIN Central intranet
- Eight initial incident playbooks written and tested via tabletop
- First DR drill executed; gaps documented and closed
- Surface latency SLOs measured continuously
End of Phase R3.
- Two-person on-call rotation operating
- Tightened SLOs §6.3 Wave 3 values met
- Quarterly DR drills executed
- Cost discipline holding within Wave 3 ceilings
End of Phase R4.
- Multi-region failover demonstrated in DR exercise
- 99.9% availability achieved on at least one Tier 2 tenant
- Three-person on-call rotation operating
- Annual full DR exercise complete
13. Cross-Doc Dependencies
- Platform North Star Roadmap — this doc is Pillar 6
- Agentic Harness Spec v0.1 — harness emits the telemetry; harness implements degradation policy
- Demarcation Plan — infra split affects DR scope per deployment
- Anonymization Policy v1 — logging discipline must honor anonymization
- Self-Improving System Roadmap — verifier failure rate is a SLO; verifiers are part of the reliability bar
- Operational Corpus Ingestion Roadmap — throughput targets must absorb the corpus ingestion peak
14. Open Decisions
- Centralized log query backend. Cloudflare Logs + R2 is the storage; query layer is TBD. Options: Grafana Loki self-hosted, Datadog, Honeycomb, custom on Neon. Default: defer to Phase R2; choose based on team comfort + cost.
- Status page surface. Public status page (status.fin.app) vs intranet-only. Default: intranet-only for Wave 2; public for Wave 3+ when partners depend on FIN.
- PagerDuty vs alternatives. Default: defer to Phase R2; for solo on-call, native phone/email is sufficient.
- Cloudflare zone vs Anycast load distribution. Default: rely on Cloudflare’s existing Anycast; revisit if Tier 2 latency targets are missed.
- DLQ retention and replay policy. Default: DLQ retains 30 days; replay requires explicit operator action plus root-cause documentation; never auto-replay.
15. The Punchline
A high-ops-tempo NYC produce wholesaler doesn’t tolerate a brittle system. The reliability bar is set by what operators actually need: subsecond agent responses on dock, lossless event ingestion, 99%+ availability during the 24/7 operating window, observable enough to debug at 4am when a webhook stops flowing. This doc spells out the latency budgets, the throughput targets, the queue policies, the observability stack, the SLOs, the playbooks, the DR objectives, the on-call model, and the cost ceilings. Phased delivery in four waves; first DR drill within 60 days; full multi-region failover within 18 months.
This is the discipline that turns the agentic OS from a demo into operational infrastructure.
16. Audit Annotation - 2026-05-20
Roadmap audit tracker: FIN Agentic Harness and Knowledge Roadmap Audit Tracker.
Verification status: the S0/S1/S2/S3 budgets are the correct target model, but current proof does not include a fresh gradeboard-backed latency baseline for QC PWA, voice, and comms reply. The checked-in gradeboard artifact is stale (2026-04-07) and reports 9.65, so the external “gradeboard 4.01” claim is not verified in this checkout.
Execution note: before retrieval top-K expansion or model routing changes, capture p50/p95/p99 input-to-first-byte per S0 surface and store the artifact in the gradeboard/proof trail. Provider routing is currently task-type based, not surface-tier based, so S0/S1/S3 routing needs explicit policy and telemetry work.