FIN Agentic Harness Engineering Spec

Date: 2026-05-15 Status: Draft v0.1 Related design: docs/design/FIN_INTRANET_HUMAN_AGENT_PROJECTION_LAYER_DESIGN_2026-05-13.md Related spec: docs/engineering/FIN_INTRANET_HUMAN_AGENT_PROJECTION_ENGINEERING_SPEC_2026-05-13.md (v0.2) Related catalog: docs/projections/PROJECTION_CATALOG_2026-05-15.md Related strategy: docs/strategy/FIN_AGENT_NATIVE_ASSESSMENT_2026-04-30.md

0. Purpose

The agentic harness is the runtime that executes FIN agents in production against the projection layer, MCP tool catalog, event store, and external channels. It is the policy enforcement point that converts model inference into auditable, citable, replayable agent behavior.

This spec defines:

• session model and routing
• tool affordance sequencing
• citation enforcement at the harness boundary
• multi-modal and voice handling
• queue resilience and offline tolerance
• replay determinism
• OpenClaw / NemoClaw mediation
• model routing (federal, cost, sovereignty)
• anonymization proxy integration
• memory shape per actor role
• agent-to-agent communication primitives
• gradeboard / eval integration
• skill packaging contract

The harness is NOT the model. The harness is the structured environment the model executes inside. Models change; the harness’s guarantees do not.

1. Architectural Principles

1. Harness as policy enforcement point. The harness, not the model, enforces visibility, sequencing, citation, and audit. Models cannot bypass harness rules through prompt cleverness.
2. Typed structured generation. Tools that produce doc content return structured arrays of cited claims (§13.6 of the Projection Spec), never free strings. The harness rejects free-string returns from typed tools.
3. No silent model routing. Every model call records provider, model ID, version, prompt version, seed, temperature, and tool-result responses. Replay must produce structurally identical output.
4. Sessions are events; events are sessions. Every harness session is an event stream. Every relevant external event can spawn a session. The harness is event-sourced end-to-end.
5. One harness per audience class. Operator agents, public Tier 0 agents, federal agents do not share harness instances even when calling identically-named tools. Audience isolation is structural.
6. Queue-backed dispatch. All tool calls go through QueueProvider. No live HTTP from the agent loop. Offline-tolerance, retry, and replay come from the queue.
7. Citations carry through the harness. Inputs the agent saw, tools the agent called, claims the agent emitted — all carry citations end-to-end.
8. Replay is a first-class operation. Any session must be replayable from its event log + tool-result log without re-calling live external systems.
9. OpenClaw mediates all privileged actions. The harness cannot directly execute privileged tools; it must propose through OpenClaw.
10. The harness’s behavior is itself patentable IP. Logging and behavior records must support prior-art demonstration and patent enforcement.

2. Session Model

2.1 Session Lifecycle

agent.session.requested
  → agent.session.started
    → agent.message_received | agent.tool_called | agent.tool_returned | ...
      → agent.session.checkpoint
        → agent.session.suspended | agent.session.resumed
          → agent.session.closed
            → doc.projection_generated (agent_harness_session.<session_id>)

Every transition is an immutable event. Session state is a projection over the event stream. Suspended sessions can resume on a different harness instance from the same event log.

2.2 Session Identity

Every session carries:

• session_id (ULID)
• tenant_id
• actor_id (the human or service actor running the session)
• audience_class (operator | tier_0_public | federal | partner | a2a)
• route_id (which MCP endpoint received the session)
• entry_channel (web | mobile | voice | sms | email | webhook | a2a)
• correlation_id (links back to the triggering event when applicable)
• parent_session_id (when this session was spawned by another agent)
• started_at
• tenant_data_plane_scope (which planes the session may read)

audience_class and route_id together determine the MCP tool catalog loaded for the session. Routes are not pluggable from within a session; session must close and reopen to change route.

2.3 Per-Actor Context Isolation

Each session loads:

• the actor’s actors.memory (preferences, expertise tags, working memory)
• the actor’s role’s skill bundle (§16)
• the tenant’s workflow config relevant to the actor’s role
• the tenant’s data plane scope as authorized for this audience class

Sessions never load another tenant’s data. Cross-tenant operations require explicit Tier 0 / federal routes with anonymized data, not tenant-scoped sessions.

3. Tool Affordance State Machine

3.1 Sequencing Discipline

Agents in FIN have ordered tool affordances enforced at the harness level:

retrieve → verify → propose → review → commit

For each privileged operation, the harness rejects calls out of sequence. Examples:

• An operator agent cannot call commit_doc_edit without first calling propose_doc_edit and receiving an approval signal.
• A recall agent cannot call notify_customer without first running assemble_recall_trace and writing the trace doc.
• A pricing agent cannot call update_customer_price without first calling walk_provenance on the cited pricing rationale.

Sequencing rules live in packages/mcp-server/src/harness/affordance-rules.ts as a deterministic state machine per audience class.

3.2 Affordance Discovery

The harness exposes list_affordances() so agents can introspect what they may call now given session state. The list is computed from the state machine + the actor’s skill bundle + the current session step.

3.3 Out-Of-Sequence Handling

Out-of-sequence tool calls receive 409 SequenceViolation with the required predecessor tool listed. The agent is expected to either call the predecessor or fail gracefully. Repeated sequence violations within a session emit agent.sequence_violation_repeated and trigger harness escalation (route to human reviewer or terminate session).

4. Citation Enforcement At The Harness Boundary

4.1 Generation Tool Contract

Per Projection Spec §13.6: any tool that produces doc-bound prose returns a structured array of attributed claims, never a free string. The harness rejects free-string returns from registered generation tools with 400 NoStructuredOutput.

4.2 Conversational Tool Contract

Conversational tools (the agent’s reply to the user) may return strings but MUST also carry a parallel evidence[] field referencing the chunks / events / knowledge keys the response is grounded in. The harness rejects empty-evidence replies for any audience class with non-trivial trust level. For casual operator chat, evidence may be implicit (session_context).

4.3 Inbound Citation Tracking

Every chunk / row / event / external URI the agent retrieves becomes part of the session’s evidence_pool. The pool is appended-to, never trimmed. At session close, the pool is summarized into the session doc projection (agent_harness_session.<session_id>) so the full provenance of every agent action is preserved.

4.4 Citation Conflicts

When the agent retrieves two pieces of evidence that contradict (different events claim different temperatures for the same lot at overlapping times, or external source disagrees with internal observation), the harness emits agent.evidence_conflict_detected and the agent must either:

• choose evidence and record the rationale in inference_notes, or
• escalate the conflict to a human reviewer via the workflow lane.

Silent suppression of conflicting evidence is a CI failure for federal- audience sessions.

5. Multi-Modal Handling

5.1 Input Modalities

The harness accepts:

• text (chat, email, SMS)
• voice (real-time stream + final transcript)
• image (defect photos, receiving photos, scale screenshots, document scans)
• audio file (recorded voicemail, driver radio capture)
• structured event (webhook payload from a connected system)
• agent message (A2A inbound)

Each input creates an agent.input_received event with the modality and a content hash. Binary content goes to StorageProvider; the event carries the artifact reference.

5.2 Modality Routing

Inputs are routed to modality-specific preprocessors before reaching the model:

• voice → speech-to-text + speaker diarization → text input
• image → vision model for content tagging + EXIF extraction
• document scan → OCR + structured extraction
• audio file → transcription
• structured event → JSON validation against event payload schema

Preprocessor outputs carry provenance metadata (model used, confidence, processing time) that flows into the session’s evidence pool.

5.3 Multi-Modal Tool Calls

Tools that consume multi-modal content (vision model for defect classification, voice model for intent detection) MUST declare their input modality in their MCP capability description. The harness rejects malformed cross-modal calls (text-only tool receiving an image reference) with 400 ModalityMismatch.

6. Voice & Telephony Sessions

6.1 Telephony Provider

Per project_telephony_provider memo, Telnyx is the primary provider for SMS and voice. The harness’s telephony adapter sits between Telnyx webhooks and the agent session loop.

6.2 Real-Time Voice Loop

Inbound voice flow:

1. Telnyx call rings → harness agent.session.requested with entry_channel = voice.
2. Audio stream → real-time transcription → text input events.
3. Agent loop produces text response → text-to-speech → audio output to caller.
4. Latency budget: target ≤1.5 seconds for first audio token, ≤3 seconds for full response.

The voice loop runs in a Durable Object instance per session to maintain audio stream state. The Durable Object never holds canonical state; the event log is canonical.

6.3 Voice Interruption Handling

The caller may interrupt the agent’s audio output. The harness must:

• truncate the agent’s current response
• emit agent.voice_interrupted with the partial response
• begin listening for the new input
• accumulate the new input over barge-in pauses (~500ms silence threshold)

6.4 Voice Provenance

Voice sessions emit transcripts. Transcripts are persisted as source artifacts with mime_type = audio/... and the transcribed text as a linked doc. Citations to voice content reference the transcript with a timestamp anchor.

6.5 Voice Permissioning

Voice sessions require additional consent metadata per call: caller identity (verified via callback or PIN), recording consent (per jurisdiction), and any required jurisdiction-specific disclosures (California, GDPR-applicable EU calls). The harness enforces this at session start; missing consent → session refused.

7. Queue Resilience & Offline Tolerance

7.1 QueueProvider Integration

All tool calls dispatch through QueueProvider (the abstraction flagged P0 in the 2026-04-30 data plane audit). The agent loop never directly calls a tool; it enqueues a tool call and awaits a result.

This gives the harness:

• retry on transient failure
• exactly-once semantics for idempotent tools
• offline tolerance (queue persists during connectivity loss)
• replay (queue serves as the tool-call log)
• backpressure (slow tools don’t block the agent loop)

7.2 Operator-Side Offline Mode

Operators on warehouse floors with spotty connectivity work against a client-side queue. Actions queue locally, sync to harness when connectivity returns. The harness deduplicates by client-supplied idempotency_key.

While offline, the operator’s PWA shows a deterministic snapshot of the session state (last successful sync) and explicitly marks queued actions as “pending sync.”

7.3 Harness-Side Tool Queue

Outbound tool calls (LLM provider, MCP server, external API) queue through QueueProvider. Failures retry per per-tool policy (exponential backoff, max retries, dead letter queue).

Dead-lettered tool calls emit agent.tool_dead_lettered events and trigger session-level review.

8. Replay Determinism

8.1 Replay Inputs

Any session can be replayed given:

• the original session event log (all agent.* events for the session)
• the original tool-result log (every tool response captured)
• the original model metadata (provider, model, version, prompt version, seed, temperature, top_p)

8.2 Replay Mode

replay_session(session_id, mode) reruns the session with mode ∈ {tool_replay, model_replay, full_replay}:

• tool_replay: real models called, tool responses served from log. Use for evaluating new model versions against historical tool inputs.
• model_replay: tool responses captured from log, model responses served from log. Use for verifying audit claims about specific past sessions.
• full_replay: both served from log. Use for reproducing exact past output (federal audit, customer dispute).

8.3 Replay Output

Replay produces a replay_run artifact: the new session event stream, diff against the original, and a replay_verdict (identical | diverged | divergence_explained). The verdict feeds the gradeboard.

8.4 Replay Storage

Tool-result logs are stored in operational_memory with a 7-year retention default (federal-grade audit horizon). Model metadata is stored in events with the session events.

9. OpenClaw / NemoClaw Mediation

9.1 Privileged Tool Boundary

Privileged tools (those that move money, send external communication, modify customer pricing, commit doc edits crossing visibility planes, take any action with external real-world effect) are NOT registered directly in the agent’s MCP catalog.

They are registered in the OpenClaw catalog. The harness’s only path to call them is openclaw.propose_action(tool, args, rationale, evidence).

9.2 Proposal Flow

OpenClaw runs:

1. Policy check (does this tool, with these args, by this actor, for this tenant, comply with policy)
2. Evidence sufficiency check (is the rationale grounded enough)
3. Review queue routing (who must approve)
4. Approval → execution
5. Audit log (openclaw.action_approved, openclaw.action_executed)

Rejection emits openclaw.action_rejected with a structured reason that the agent can use to retry or escalate.

9.3 NemoClaw Integration

NemoClaw is the human-in-the-loop interface for reviewing OpenClaw proposals. The harness presents proposals in NemoClaw with full provenance (rationale, evidence chain via walk_provenance, prior similar proposals and their outcomes).

NemoClaw enforces the discipline that humans see the full justification before approving, not just a “approve / reject” button on a summary.

9.4 Auto-Approval Tiers

Some classes of proposal can be auto-approved (low-value, reversible, within policy bounds, with sufficient evidence). The auto-approval rules are versioned policy artifacts reviewable per tenant. Auto-approval still emits the full audit chain.

10. Model Routing

10.1 Multi-Provider Support

The harness supports multiple LLM providers per session, selected by policy:

• Anthropic Claude (default for production tenant agents)
• Gemini (preferred for budget-eval per feedback_eval_costs memo)
• OpenAI (backup, specific tool calls where it outperforms)
• Local/self-hosted (federal-sovereign tenants)

Provider selection is recorded per call. No silent provider switching.

10.2 Federal-Safe Mode

Tenants in federal mode have a stricter provider whitelist:

• Anthropic Bedrock (FedRAMP authorized)
• Self-hosted federal-cleared models
• No external SaaS providers

The harness reads tenant.federal_mode flag and refuses non-whitelisted providers. Refusals emit agent.federal_mode_routing_refused events.

10.3 Cost Budget Enforcement

Sessions and tools carry token / cost budgets. The harness:

• tracks cumulative cost per session
• refuses model calls that would exceed the per-session cap
• routes to cheaper models when cost-elastic tools are called near budget

Per feedback_eval_costs memo, eval runs prefer Gemini Flash for budget; production tenant sessions default to Claude Sonnet/Opus by tier.

10.4 Tenant Sovereignty

Tenants that operate their own model infrastructure (sovereign deployment) configure their endpoint in tenant config. The harness routes all model calls to the tenant’s endpoint. The anonymization proxy is bypassed in this mode (the tenant trusts their own infra).

10.5 Model Version Pinning

Sessions may pin a specific model version for replay stability or regulatory consistency. Pinned sessions refuse provider auto-upgrades.

11. Anonymization Proxy Integration

11.1 Proxy Activation

Per the data plane assessment trio, the anonymization proxy is text-only, session-scoped, with placeholder dictionary. The harness activates the proxy when:

• tenant.privacy_mode === 'proxy_required', OR
• provider.privacy === 'partial', OR
• audience class is tier_0_public and the underlying tenant data is about to cross the public boundary

11.2 Forward Path

Outbound model calls go through the proxy:

1. Identify candidate PII / tenant-private text (customer names, vendor names, exact prices, exact volumes, internal lot IDs).
2. Replace with session-scoped placeholders (CUSTOMER_001, VENDOR_002).
3. Send to model.
4. Receive completion.
5. Reverse-map placeholders.
6. Return to agent loop.

11.3 Bypass Cases

The proxy is bypassed when:

• LLMProvider.privacy === 'full' (tenant-sovereign endpoint)
• tenant.privacy_mode === 'native' (tenant explicitly opted out)

Bypasses are logged so the audit trail shows when the proxy was not used.

11.4 Anonymization Drift Detection

If the proxy returns an unmapped placeholder (model fabricated a new “PLACEHOLDER_999”), the harness emits agent.anonymization_unmapped_placeholder and either drops the response or escalates to review. Fabrication of placeholders is a hallucination signal.

12. Memory Shape Per Actor Role

12.1 actors.memory Structure

Per project_pk_v2_implementation memo, actors.memory is JSONB with:

• working_memory[] — recent interactions, ephemeral
• preferences — durable preferences (notation style, channel, language)
• expertise[] — tags this actor is competent in (used for review routing)
• career_tracking — onboarding state, role progression
• last_active_session_ids[] — for resumption

The harness reads this at session start and writes back at session close.

12.2 Working Memory Lifecycle

Working memory grows during sessions. At session close, the harness summarizes (cite-then-claim) into durable entries. Old working memory is archived to operational_memory as chunks (searchable but not loaded by default).

12.3 Role-Scoped Memory Visibility

Different roles see different slices of memory:

• The actor sees their own full memory.
• A privileged manager actor sees direct-report memory (preferences, career tracking) but not working memory.
• An ops agent operating on behalf of the actor sees only the relevant preferences and expertise — never raw working memory.
• A federal-audience replay sees redacted memory with PII stripped.

13. Agent-To-Agent (A2A) Communication

13.1 A2A Channel

Agents may communicate with other agents (within the same tenant, across tenants for marketplace, or with external partner agents per Tier 0 contracts).

A2A messages are events: agent.a2a_sent and agent.a2a_received. The message carries:

• sender agent identity
• receiver agent identity
• conversation_id (the shared thread)
• message body (structured, not free text)
• citations
• intent (request | response | broadcast | proposal)

13.2 A2A Routing Rules

Cross-tenant A2A goes through a “blind broker” pattern (per marketplace planning memo): agents see opportunity, not counterparty identity, until both sides commit. The harness enforces the blinding.

External-partner A2A uses the Tier 0 public MCP contract with authentication. Partner agents are authenticated through tenant-issued keys, not through tenant data plane access.

13.3 Conversation State

A2A conversation state is event-sourced. Each conversation has its own projection (similar to agent_harness_session but cross-agent). Either party may resume from the event log.

13.4 A2A Failure Handling

A2A messages can fail (unreachable peer, signature failure, policy block). Failures emit agent.a2a_failed and the originating session decides whether to retry, fall back, or escalate.

14. Gradeboard / Eval Integration

14.1 Eval Hooks

The harness exposes eval hooks at every major decision point:

• input received
• evidence retrieved
• tool sequenced
• response generated
• proposal made
• proposal approved/rejected

Eval scenarios subscribe to these hooks via the gradeboard contract.

14.2 Prompt Versioning

Every prompt template carries a version. The harness records which prompt version produced which behavior. A/B testing prompt versions runs through the eval framework, never directly in production.

14.3 Replay-Driven Eval

The gradeboard reruns past sessions through new models / prompts / skill bundles to measure regression and improvement. The replay machinery (§8) is the substrate.

14.4 Gradeboard Stop Conditions

Per the 2026-04-13 PK v2 implementation, certain gradeboard scores gate production deploys. The harness refuses to load a skill bundle that fails its current gradeboard threshold.

15. Skill Packaging Contract

15.1 Skill Bundle

A skill bundle is a versioned set of:

• system prompt
• tool affordances (subset of MCP catalog)
• few-shot examples
• guardrail patterns
• per-task latency / cost budgets
• gradeboard threshold this bundle must pass

Bundles are stored as knowledge.domain = skill_bundle rows.

15.2 Per-Role Bundle Loading

Each actor role has a default bundle. Session start loads the bundle. Skill bundles compose (an operator agent loads operator.base plus any role-specific add-ons like operator.qc, operator.receiving).

15.3 Bundle Evolution

New bundles are proposed through a propose_skill_bundle workflow (authored doc), tested in eval, promoted on gradeboard threshold pass. Demotion follows the reverse path if production metrics regress.

15.4 Skill Bundle Provenance

Every session records the exact skill bundle versions loaded. Replay loads the same versions. New bundle versions cannot retroactively “improve” past sessions.

16. Public Tier 0 Harness

16.1 Distinct Harness Instance

Tier 0 (public produce intelligence) runs on a separate harness instance from tenant-scoped harnesses. Different MCP endpoint, different model routing policy, different rate limiting, different audit retention.

16.2 Tier 0 Tool Catalog

Tier 0 exposes a strict subset of MCP tools:

• search_docs (public docs only)
• get_doc (public docs only)
• get_entity_profile (Harvest Directory, public_anonymized variant)
• list_recent_changes (public stream)
• walk_provenance (public chain only)
• get_aggregate_intelligence (publicly released aggregates)

No write tools. No tenant-data tools. No privileged actions.

16.3 Tier 0 Authentication

Public Tier 0 uses API keys issued per developer. Keys carry quota, rate limit, and usage metering. Heavy users move to paid tiers.

16.4 Tier 0 Audit

Tier 0 calls are audited but not at the same retention as tenant sessions. 60-day default rolling retention. Anomalous patterns (scraping, abuse, attempts to enumerate tenant identity) trigger key suspension.

17. Tests

17.1 Unit Tests

• session lifecycle event ordering
• tool affordance state machine (sequence enforcement, out-of-order rejection)
• typed structured generation contract
• citation conflict detection
• multi-modal preprocessor routing
• voice interrupt handling
• queue retry behavior
• replay determinism (full_replay produces identical output)
• model routing policy (federal mode refusal, cost cap enforcement)
• anonymization proxy forward+reverse roundtrip
• A2A message signature verification
• skill bundle composition

17.2 Integration Tests

• end-to-end operator agent session with tool sequencing
• voice session with interrupt + recovery
• offline operator action queue + sync
• replay of a closed session reproduces session doc projection
• OpenClaw rejection flow round-trip
• federal-mode session refuses external model provider
• Tier 0 session refuses tenant-data tool
• cross-tenant A2A blinding (no counterparty identity until commit)
• gradeboard rerun on new skill bundle vs old

17.3 Browser / Mobile Tests

• PWA operator session resumption from offline queue
• voice barge-in latency target met
• session checkpoint visible during long-running sessions
• NemoClaw review queue renders OpenClaw proposals with provenance walk
• mobile mic + camera flows into multi-modal session

17.4 Adversarial Tests

• agent attempts to bypass sequencing via prompt injection (must fail structurally)
• agent attempts to emit free-text from typed generation tool (must be rejected at boundary)
• agent attempts to call tool from different audience class (must fail at route)
• model returns fabricated placeholder (must trigger drift event)
• model contradicts retrieved evidence (must trigger conflict event)
• replay with substituted tool log (verdict must detect divergence)

18. Proof Gates

Wave 1 harness is not complete until:

• session lifecycle event schemas land in packages/shared/src/events/payloads/agent/
• affordance state machine implemented and tested
• typed structured generation contract enforced at one production tool (start with propose_doc_edit)
• one full-replay test passes for a closed session
• cost budget enforcement test passes
• one production tenant operator session runs end-to-end with all events emitted

Wave 2 harness is not complete until:

• voice session loop runs end-to-end against Telnyx
• multi-modal preprocessor pipeline ships for image and audio
• OpenClaw mediation integrated for one privileged tool family
• offline operator PWA queue + sync runs end-to-end
• skill bundle loading mechanism ships with at least 3 role bundles
• agent_harness_session.<session_id> doc projection writes on every session close

Wave 3 harness is not complete until:

• federal mode routing enforced and audited
• Tier 0 public harness deployed on distinct infrastructure
• replay-driven gradeboard regression detection wired
• A2A primitives ship with cross-tenant blinding
• model version pinning supports per-session pin

Wave 4 harness is not complete until:

• multi-provider model routing (Anthropic + Gemini + OpenAI + sovereign) proven in production
• anonymization proxy drift detection in production
• prompt version A/B testing harness in production
• auto-approval tiers ship for low-risk OpenClaw actions

19. Stop Conditions

Stop implementation and return to architecture review if:

• a fifth runtime table is required to support sessions or memory
• replay cannot be made deterministic for federal-audience sessions
• OpenClaw mediation cannot be enforced at the harness boundary (agents find a bypass)
• voice latency cannot meet the 1.5s/3s target on production infra
• the anonymization proxy leaks tenant-private text in any test case
• Tier 0 harness cannot be isolated from tenant data plane
• session event volume overwhelms the event store cost envelope
• skill bundle evolution cannot be gradeboard-gated

20. Out Of Scope For v0.1

• proactive agent monitoring (agent decides to initiate a session on external signal) — deferred to v0.2
• agent self-modification of skill bundles — never. Bundles evolve through proposed-doc-edit workflow with human approval.
• direct agent-to-agent training data sharing across tenants — deferred to federated learning research track
• emotion / sentiment modeling in voice sessions — deferred
• on-device model inference for offline operators — deferred to Wave 5

21. Relationship To Other Specs

The agentic harness is the runtime for the projection layer’s MCP tools. Specifically:

• §13.6 (cite-then-claim) of the Projection Spec is enforced at this harness’s tool boundary (§4 here).
• §8.12 (walk_provenance) of the Projection Spec is the primary tool for citation-chain audit in this harness.
• §11.5.3 (transactional projection writes) of the Projection Spec is the consistency boundary for any tool the harness calls that writes projections.
• §13.5 (agent enrichment loop) of the Projection Spec is implemented through OpenClaw mediation (§9 here).
• §19 (editing surfaces) of the Projection Spec defines the human side of what this harness’s agent side proposes.

This spec defers all projection-layer concerns to the Projection Spec. This spec owns: sessions, sequencing, citation enforcement, replay, multi-modal, voice, queue, OpenClaw mediation, model routing, anonymization proxy, memory, A2A, gradeboard integration, skill packaging.