FIN Central

FIN Self-Improving System Roadmap

17 min read

FIN Self-Improving System Roadmap

0. Revision Summary

First cut. Defines the architectural path from FIN’s current world-knowing system (events + CQRS + MD projections + frontmatter + GitHub vaults + cite-then-claim) to a self-improving system that compounds quality without retraining the base model. Companion to FIN Intranet Human-Agent Projection Engineering Spec v0.2 and FIN Agentic Harness Spec v0.1. Subordinates the question “should we train our own model?” to a strict ordering: deterministic verifiers first, learned verifier second, distilled domain model later, frontier-scale training likely never.

1. Framing: World-Knowing vs Self-Improving

The system FIN has built is excellent at knowing. Events flow in, CQRS projections derive operational state, MD projections render that state for both humans and agents, frontmatter binds the graph, the harness orchestrates retrieval and action. The agent reads a coherent slice of the world and produces grounded answers.

The system FIN has not yet built is one that learns from its own actions. Agent successes and failures are not yet fed back as training signal for the next agent. Drift is detected by humans, not by the system. Skills are authored, not earned. Confidence is per-claim, not per-session. The world model is rich; the policy model is static.

The self-improving system retains the existing substrate and adds a closed-loop layer on top. This doc specifies that layer.

2. Frontier Code-Agent Patterns — What They Tell Us

The qualitative gap between “a chatbot with RAG” and a frontier code-agent (Claude Code, Cursor agent mode, Devin) is not the model. It is four structural choices.

Fat context, not thin context. Code-agents flood the model with structured signal — repo state, diffs, test output, type errors, related files, call trees. They trust the model to integrate. FIN’s MD projection layer is the produce-industry analog. The optimization is more, not less: project at multiple granularities, project more frequently, project more dimensions.

Verifiable feedback loops. Code-agents don’t guess if their output is good. They run tests, linters, type checkers. Every action produces a machine-readable verdict. Wrong is a return code, not a vibe. FIN’s gradeboard runs out-of-band against scenarios — it is evaluation, not in-loop verification. The missing piece is inline, deterministic, per-output verifiers an agent runs against its own response before sending it.

Explicit planning, critique, verification phases. Code-agents don’t just respond. They plan, act, critique, repeat. FIN’s harness has session structure, but the proposal → review → merge loop that already exists for docs has not been extended to skill or prompt edits.

Subagent specialization with replay. Frontier systems decompose problems across specialized small agents coordinated by an orchestrator, with the ability to replay or branch. FIN’s harness spec supports this. It is not yet leaned into operationally.

The takeaway: FIN does not need a new model. FIN needs to extend what it already has along these four axes. The base model can remain whatever frontier API is best at the time. The moat is the harness, the verifiers, the projections, and the learned policy that emerges from observing the harness.

3. The Seven Gaps and How to Close Them

The seven items below close the loop. Listed in priority order — the order in which they unlock the next.

3.1 Procedure Projections (twin to entity profiles)

The projection catalog enumerates entity profiles (what a watermelon is, what a vendor does, what a customer pattern looks like). It does not yet enumerate procedure profiles — what successful past handling of a situation type looks like.

When a QC inspection of a watermelon load arrives 8% over weight in summer with a Class 3 outer rind defect, and the operator successfully routes it (reject, partial accept, price adjustment, claim filing, customer renegotiation), that resolution should accumulate into a procedure projection: “How we handle a watermelon load that arrives 8% over weight in summer with a Class 3 outer rind defect.” Same doc_kind machinery as entity profiles, new entry: doc_kind = procedure_profile.

Mechanism.

  • New doc_kind enum value: procedure_profile
  • Triggered by closed loops on harness sessions tagged with situation features (commodity, defect class, season, severity)
  • Renders as a markdown doc with: situation pattern, observed resolutions, observed outcomes (graded), recommended path, escalation triggers, related entities
  • Indexed under the same hybrid retrieval contract; reachable by agents via get_procedure_profile(situation_features)
  • Maintained by the same proposal-review-merge process as authored docs

Why it matters. Agents already read entity profiles to know what things are. Procedure profiles let them know what to do. Same retrieval pattern, different epistemic role. The corpus of procedure profiles becomes the institutional procedural memory of Nathel, then FIN, then every tenant.

Dependencies. Projection catalog v0.3 update; doc_kind enum extension in projection spec; new MCP tool get_procedure_profile; harness instrumentation to tag sessions with situation features.

3.2 Deterministic Verifier Suite (pytest for produce-domain answers)

The single largest lever between FIN’s current quality and code-agent-level quality is making “good” structurally definable instead of vibes-definable. Every agent output, before it leaves the harness, runs through a verifier suite. Failures route to retry, escalation, or explicit “I cannot answer this within my contract” response.

Verifiers to ship in Phase A (first three).

  • Citation validity. Every cited chunk resolves to a real chunk_id in operational_memory or the doc store. Empty citation lists are rejected at the boundary (already in harness spec; enforce here).
  • Freshness compliance. Every cited chunk is within the freshness window appropriate to the question type. Inventory questions: chunks must be within last 24h. Vendor scorecard questions: within last 90d. Procedure questions: latest version of the procedure_profile.
  • Schema conformance. Numeric claims serialize to the expected schema (units, precision, currency). Free-string returns for typed slots are rejected.

Verifiers to ship in Phase B (next four).

  • Business-rule conformance. Recommendations match sanctioned procedures; novel recommendations are tagged novel_suggestion: true and require explicit human acceptance before they can be acted on by tool-using agents.
  • Numeric reconciliation. Numbers cited from operational projections (e.g. “we have 1,240 cases of watermelon”) match the projection at retrieval time. Drift between answer-time and retrieval-time triggers a rerun.
  • Entity reference validity. Every commodity, vendor, customer, carrier, lot, or PO referenced in the answer resolves to a known entity_id.
  • Provenance walkability. Every claim can be walked back to source via walk_provenance. Unwalkable claims are rejected.

Mechanism.

  • Verifiers live as standalone modules; each is callable as verify_<name>(answer, context) -> {pass, fail, warn} + reason
  • The harness runs the verifier chain after model response generation, before tool execution or user return
  • Verifier failures produce events: verifier.failed with verifier_name, answer_id, reason
  • Verifier pass rates become a harness-health KPI tracked alongside gradeboard
  • Verifier outputs feed the self-critique loop (3.3)

Why this is the lever. Verifiers convert agent quality from a subjective grading problem into a deterministic gate. They also produce the cleanest possible training signal for the learned verifier in Phase D.

3.3 Self-Critique → Proposal Loop (proposal-review-merge for skills and prompts)

The proposal-review-merge pattern already exists for docs (propose_doc_edit). Extend it to skills and prompts.

Trigger conditions.

  • Repeated verifier failure of the same type from the same skill (threshold: N=5 within a rolling window)
  • Gradeboard score regression on a skill (week-over-week drop beyond noise)
  • Drift detection: agent output diverges from the corresponding procedure_profile recommendation N times in a row
  • Explicit operator feedback flagged via comms

Mechanism.

  • New event kinds: skill.edit.proposed, prompt.edit.proposed
  • Proposal payload: skill_id or prompt_id, observed failure pattern, supporting trace_ids, proposed diff, justification
  • Review queue surfaced in the FIN Central intranet (and tenant intranet for tenant-scoped skills)
  • Initial reviewer: human. Later: a privileged reviewer agent that holds the proposer agent accountable but cannot self-merge — human approval still gates skill changes in Wave 1 and Wave 2; agent-only auto-merge gated behind explicit policy graduation per skill
  • Merged proposals bump skill version; old version archived (skills are versioned, never deleted)
  • Pre-merge: the proposed skill must pass its eval set (3.7) at non-regressed score

Why it matters. This is the loop. The system watches itself fail, proposes its own fix, validates that fix against the existing eval set, and merges only on improvement. Once this is running with humans in the loop, the discipline is in place to graduate selectively to agent-only loops on low-risk skill classes.

3.4 Behavioral / Case-Based Memory

actor_memory holds facts about actors. The system also needs a memory of situations seen and resolutions tried, separate from facts. Different retrieval pattern: indexed by situation features (commodity, defect class, customer segment, severity), not by entity.

Mechanism.

  • New projection over harness traces: case_memory_projection (migration TBD, sequence after current planned 045-048)
  • Each closed session with a graded outcome becomes a case_memory row: features, action taken, outcome, gradeboard score
  • Retrievable by get_similar_cases(situation_features, k) MCP tool
  • Surfaced as an MD projection (doc_kind extension: case_digest) for human readability and audit
  • Procedure_profiles (3.1) are the generalized version; case_memory is the specific version. Agents read both.

Why it matters. Generalization (procedure_profile) is brittle when a situation is on the boundary. Case-based memory provides the “have we ever seen anything like this?” fallback, which is how experienced operators actually reason.

3.5 Counterfactual Replay

The event store is already optimal substrate. Almost no one builds the layer on top. Replay isn’t just “what happened” — it’s “what would have happened if a different decision had been made at event N.”

Mechanism.

  • New harness mode: counterfactual_replay
  • Input: session_id, event_id at which to fork, perturbation (decision substituted, fact changed, action different)
  • The harness re-executes downstream agent decisions from the fork point with the perturbed state
  • Output: a divergence report — which downstream events would have differed, what the gradeboard score of the counterfactual run would be, with provenance
  • Stored as events: counterfactual.run.started, counterfactual.run.completed
  • Rendered as MD projection (doc_kind: counterfactual_report)

Use cases.

  • “If we had rejected this load instead of partially accepting, what would the gradeboard say?”
  • “If we had routed the PO to truck B instead of A, what would the cold-chain projection look like?”
  • “If we had flagged this defect class earlier in the inspection workflow, would the customer complaint chain have happened?”

Why it matters. This is the analytical superpower of event-sourced architectures. It produces unique operational insights that no competitor can match, because no competitor sits on a clean event-sourced spine over the PO lifecycle. It is also the foundation for synthetic-data generation if FIN ever does train a learned policy (Phase D+).

3.6 Confidence Calibration with Explicit Escalation

Per-claim confidence (already in projection spec §3.5) is not enough. Sessions need a meta-level uncertainty score and a policy.

Mechanism.

  • Session-level uncertainty score computed from: per-claim confidence aggregation, verifier warning rate, retrieval set diversity, model self-reported confidence (when available)
  • Escalation policy per question type:
    • Below threshold A → answer normally
    • Threshold A to B → answer with explicit “low confidence” tag and surfaced reasoning
    • Above threshold B → refuse, escalate (more expensive model, human review, or “I don’t know — here’s what I’d need to know”)
  • Escalation events: harness.escalation.triggered
  • Calibration audit: compare predicted-confidence histogram to actual-correctness histogram on graded sessions; recalibrate weekly

Why it matters. Trustworthiness. The discipline that converts hallucination from an uncontrolled possibility into a tracked, accountable event. Federal positioning depends on this — auditors need to see that the system knows what it doesn’t know.

3.7 Skill Packaging with Per-Skill Eval Sets

Every learned procedure becomes a versioned skill. Every skill ships with its own eval set — situations it claims to handle, outputs that count as good, verifiers that must pass. The unit of self-improvement is the skill.

Mechanism.

  • Skill packaging discipline (already specified in harness spec §15): every skill is a directory with SKILL.md, evals/, verifiers/, version metadata
  • Eval set is a YAML or JSON file enumerating: scenarios, expected output shapes, required verifier passes, gradeboard score floor
  • Pre-merge gate on any skill change: eval set runs in CI; merge blocked on regression
  • Skill registry projection (doc_kind: skill_catalog) auto-renders from the skill packaging metadata
  • Each skill in the registry has a status (proposed, active, deprecated, archived) and a maintenance owner

Why it matters. Skills become testable artifacts. The agent’s behavior is version-controlled with the same discipline as the code. Self-improvement compounds because every skill change is validated against the corpus of situations the skill claims to handle.

4. Training Strategy

Short answer: lean fully into the architecture for 6+ months before training anything, and even then, train a verifier before training a model.

The four training paths, in order of when (if ever) they make sense for FIN.

4.1 Deterministic Verifiers (now)

Not training. Engineering. Ship the verifier suite (3.2). This is the prerequisite for everything below — without deterministic verifiers, there is no clean training signal.

Cost. Engineering time only. Days to weeks per verifier. Payoff. Direct quality improvement, no model training needed. Decision. Ship.

4.2 Learned Verifier (Phase D, 6-12 months out)

A small classifier (1B–3B params, distilled or fine-tuned from an open weights model like Llama 3.2 or Qwen 2.5) that takes (question, candidate answer, retrieved context) and outputs a 0–1 score: “does this look like a good produce-domain answer that respects FIN’s contracts?”

Training data. Harness traces with gradeboard labels + deterministic verifier outputs. Cost. Single-digit thousands of dollars of compute; weeks of ML engineering. Payoff. Catches soft drift that deterministic verifiers miss. Cheap inline scoring of every agent output. Augments the deterministic suite rather than replacing it. Trigger. When ~10k clean labeled traces exist and the deterministic suite catches most hard violations. Decision. Plan for ~Q4 2026; defer until the corpus is mature.

4.3 Distilled Domain Model (Phase E, 12-18 months out)

A small open-weights model fine-tuned on FIN’s projection corpus + harness traces, suitable for cost-sensitive Tier 0 calls, latency-sensitive inline use, and federal/sovereign on-prem deployments.

Cost. Mid-five-figures compute; months of ML engineering; ongoing eval discipline. Payoff. Latency, cost, sovereignty for federal deployment. Tier 0 public MCP becomes economically viable at higher volumes. Not a quality improvement over frontier APIs for Tier 2-4 use. Trigger. Federal pipeline maturity requiring on-prem; or volumes at Tier 0 that make API costs material. Decision. Plan for 2027; explicit gate on federal commitment or Tier 0 economics.

4.4 Frontier-Scale RLHF / Policy Training (likely never)

The thing Anthropic does to make Claude. Wildly expensive, requires ML infrastructure and team that doesn’t exist at FIN’s scale, and the upside is marginal because the structural quality of FIN’s system is not bottlenecked on the base model.

Decision. Skip indefinitely. Revisit only if FIN-scale changes by 100×.

4.5 The Discipline

The order matters and is non-negotiable. Train a verifier before you train a policy. Build deterministic gates before you train a verifier. Most teams reverse this and burn money on models that the architecture wasn’t ready to use. FIN’s discipline is: architecture first, verifiers second, learned scoring third, distilled inference fourth, policy training last (or never).

5. Phased Delivery

Phase A — Foundation (2-Week Sprint, current).

  • Procedure_profile doc_kind added to projection spec
  • Verifier suite scaffolding in harness
  • First three deterministic verifiers shipped: citation validity, freshness compliance, schema conformance
  • Skill packaging discipline applied to existing skills (QC, comms, intranet rendering)
  • Initial eval sets for the three skills above
  • Acceptance: every agent response in the QC live test runs through the three verifiers; verifier pass rate tracked daily

Phase B — Closed Loop (next 2-4 weeks after Sprint).

  • Remaining four deterministic verifiers shipped (business rules, numeric reconciliation, entity validity, provenance walkability)
  • skill.edit.proposed + prompt.edit.proposed event kinds wired
  • Review queue rendered in FIN Central intranet
  • First counterfactual replay v0 (single-fork, no branching)
  • Acceptance: at least one skill edit proposed by the system, reviewed, merged, and tracked through to gradeboard improvement

Phase C — Behavioral Layer (next 2-3 months).

  • case_memory_projection migration shipped
  • get_similar_cases MCP tool live
  • Confidence calibration meta-layer wired
  • Escalation policy enforced per question type
  • Verifier coverage to 80% of agent output paths
  • Acceptance: agents reliably refuse and escalate above threshold B; calibration histogram matches actual correctness within 10 percentage points

Phase D — Learned Signals (3-6 months).

  • 10k+ clean labeled traces accumulated
  • Learned verifier trained on (Llama 3.2 3B or Qwen 2.5 3B base)
  • Verifier deployed inline in harness
  • Eval set proliferation: every active skill has an eval set
  • Acceptance: learned verifier catches >50% of deterministic-verifier-missed quality issues identified by post-hoc human review

Phase E — Distilled Inference (6-12 months, conditional).

  • Federal pipeline gate met OR Tier 0 economics gate met
  • Distilled domain model trained
  • Tier 0 public MCP migrated to distilled model where economics warrant
  • Federal/sovereign on-prem deployment available
  • Acceptance: distilled model passes the same eval sets at the same gradeboard floor as the frontier API on Tier 0 question types

6. Open Decisions

  • Procedure_profile maintenance cadence. Daily? Weekly? Triggered? Default: triggered (on N closed sessions of the situation type), but with a weekly sanity sweep.
  • Reviewer agent authority graduation. What is the policy for letting a reviewer agent auto-merge skill edits? Default: never in Phase B; explicit per-skill policy in Phase C; never for tool-using or money-moving skills.
  • Counterfactual replay scope. Bounded to read-only re-execution, or allowed to project hypothetical events? Default: read-only re-execution in v0; hypothetical event projection in v0.2.
  • Confidence calibration source. Pure self-reported model confidence is unreliable. Should we use ensemble agreement, retrieval-set entropy, or both? Default: both, with weights tuned against gradeboard.
  • Eval set ownership. Per-skill maintainer, or central eval team? Default: per-skill maintainer with central review.

7. Cross-Doc Dependencies

8. Acceptance Criteria

End of Phase A (end of 2-week sprint).

  • Three verifiers running inline on every QC-agent and comms-agent response
  • Verifier pass rate baseline established and tracked in gradeboard
  • First procedure_profile rendered (watermelon over-weight defect handling)
  • Skill packaging discipline applied to QC, comms, intranet rendering skills

End of Phase B (4-6 weeks total).

  • At least one system-proposed skill edit merged and validated through gradeboard improvement
  • Counterfactual replay v0 demonstrated on a real PO event chain
  • Self-critique loop running continuously, with reviewer queue maintained

End of Phase C (3-4 months total).

  • Case-based memory queryable; at least 100 case_memory rows accumulated
  • Confidence calibration matches actual correctness within 10 percentage points
  • Verifier coverage at 80% of agent output paths

End of Phase D (6-9 months total).

  • Learned verifier in production, catching >50% of soft drift
  • Every active skill has an eval set; all skill changes gated on eval pass

End of Phase E (12-18 months total, conditional).

  • Distilled domain model serving Tier 0 (or federal on-prem) at gradeboard parity with the frontier API

9. The Punchline

What FIN built is the read path for an agentic OS. What’s left is the write-back-into-itself path. Procedure projections are the artifact. Verifiers are the gate. Self-critique is the proposer. Gradeboard is the judge. The event store is the substrate. Counterfactual replay is the analytical engine. Case-based memory is the operator’s intuition. Confidence calibration is the integrity layer. Skill packaging is the unit of compounding.

No new training paradigm. No new model. The base model can be whatever frontier API is best at the time. The moat is the closed loop, and the closed loop is buildable with the architecture FIN already has.

10. Audit Annotation - 2026-05-20

Roadmap audit tracker: FIN Agentic Harness and Knowledge Roadmap Audit Tracker.

Verification status: the verifier suite is correctly prioritized, but the current checkout does not yet show a global inline harness egress verifier that rejects unresolved citations or zero-citation evidence answers before user return. query_knowledge and query_operational_memory return source/freshness metadata, yet the response builders do not enforce citation validity, per-source freshness, or schema conformance as a boundary gate.

Execution note: treat citation validity plus fresh gradeboard/S0 latency baselining as Phase A’s first two engineering tasks. Prompt tuning, model swaps, and retrieval expansion should follow those deterministic gates, not precede them.

Graph View

Backlinks