Agentic Eval Live Results — V-701..V-703 + Gradeboard
Status: VERIFYING (per orchestration discipline §1)
Target: https://staging-api.fintail.net @ worker SHA 3981fe5
Auth Mode: jwt (eval-auth-adapter with admin persona Josh Garcia)
Source-of-truth real data: nathel-prod Neon (project wispy-mud-49174151)
Artifact branch: claude/parity-artifacts-2026-05-17-eval-subagent
Headline: Gradeboard PASSED at 5.00/5.00 ceiling (structural readiness). CFA probe suite at 17/20 = 85% pass (slight miss vs 28/31 proportional target; 2 real failures + 1 deploy metadata degraded). V-703 surfaced a clean orchestration parity gap: 13 real ORG BLACKBERRY events exist in nathel events table but the agent’s planner only invoked
get_commodity_profilefor the arrivals query — never reached the operational layer.
V-701 — CFA 31-Probe Suite
Runner: pnpm exec tsx test/audit/cfa/cfa-audit.ts
Artifact: .artifacts/tests/cfa-runner-5b3c18c2-20260517-031449.txt + audit/cfa/
Authoritative classification: unique probe IDs in surface reports
| Metric | Value |
|---|---|
| Total unique probes defined | 20 (11 endpoint + 10 tool; spec mentions 31 — probe-count drift to track) |
| Pass | 17 (85.0%) |
| Fail | 2 |
| Degraded | 1 |
| Overall surface score | 7.8/10 |
| Deployed version | 3981fe5 |
| Auth mode | jwt |
Failures (live)
| Probe | Status | Latency | Detail |
|---|---|---|---|
get_actor_context | 403 fail | 368ms | PERMISSION_DENIED: Access denied cross-tenant tool invocation. Matches V-113 SPEC-LOCK known-flaky pattern. |
get_market_brief | 500 fail | 21801ms | canceling statement due to statement timeout. Real DB-side bug — query exceeds statement_timeout. |
Degraded
| Probe | Detail |
|---|---|
Deployment Status | Reports staging as degraded (metadata inconsistency vs the healthy probe surface). |
Findings (synthesized from per-surface scoring)
- [high] All tracked Produce Pro freshness domains are stale (
get_data_freshnessoverall=stale). Affects knowledge_freshness, tool_completeness, workflow_completeness. - [medium] Channel readiness uneven across exposed interfaces. Public + admin healthy; deployment metadata still degraded.
V-701 acceptance evaluation
| Target | Actual | Verdict |
|---|---|---|
| ≥28 of 31 probes pass (90.3%) | 17 of 20 probes pass (85.0%) | VERIFYING-PARTIAL — pass rate ≈5pp below proportional target; probe count differs from spec |
Root-cause notes:
- The cfa-audit codebase has 20 probes, not 31. Either the spec target is aspirational and probe definitions are pending (needs spec/code reconciliation) or the renderer expected an additional 11 probes that were never authored. The §2.B
CFA_SURFACESgroup includes scenarios that scoring uses but probe-counted as a separate concern. get_actor_context403 — the eval persona (Josh Garcia, admin@nathel) is querying for tenant=fin-centralin the probe args; this is the cross-tenant denial pattern. Either probe args need to scope to nathel only, or the runner needs to map persona tenant onto args.tenantId.get_market_brieftimeout — actual DB performance issue, not a SPEC-LOCK drift artifact. Should be fenced for B-followup investigation.
V-702 — parity-agentic-gradeboard
Runner: pnpm exec tsx scripts/audit/parity-agentic-gradeboard.ts
Artifact: .artifacts/tests/gradeboard-31a5472a-20260517-031740.txt + docs/audits/PARITY_G_GRADEBOARD_BASELINE_2026-05-17.{md,json}
Composite score: 5.00 / 5.00 (target 4.51; prior baseline 4.01).
| Axis | Score | Notes |
|---|---|---|
| tool_surface | 5.00 | 4/4 source files present; 4/4 registered in toolRegistry. |
| schema_layering | 5.00 | composeProfileSections=true, EntityProfileLayer type=true |
| provenance | 5.00 | terminate=true, cycles=true, truncate=true |
| graph | 5.00 | bound=true, cycles=true, fanout=true |
| search_recall | 5.00 | recall test=true, merge logic=true |
| composer | 5.00 | package present=true, composeLivingIntelPage exported=true |
| role_presets | 5.00 | 13 role presets wired |
| side_dock_composers | 5.00 | 12/12 composers present |
| eval_suite | 5.00 | 8 test() cases authored |
| side_dock_ui | 5.00 | side_dock=true, drill_breadcrumb=true, next_steps_strip=true |
Caveat (from gradeboard doc): structural readiness only — does the code exist, are the tools wired, are the composers in place. Live drill-through under a real LLM-driven agent run captures semantic recall + UX feel + ambient intelligence freshness (the 4.01 baseline measured that). Runtime 4.51+ lands once entity overlay docs are seeded for Tenant Zero, Phase 5 §3 CF Pages wire-up (BLK-01) lands, and the Playwright suite runs against the live workspace (currently mocked).
V-702 acceptance evaluation
| Target | Actual | Verdict |
|---|---|---|
| ≥4.51 composite | 5.00 | PASS (structural) |
V-703 — ORG BLACKBERRY Watermelon-Substituted Test
Runner: pnpm exec tsx scripts/audit/v703-blackberry-probe.ts (authored this run)
Query: "ORG BLACKBERRY arrivals from HOMEGROWN ORGANIC FARMS this past week"
Artifacts:
.artifacts/lineage/blackberry-4ce4353a-2026-05-17T03-20-42-652Z.json— agent response capture.artifacts/lineage/v703-real-corpus-validation.json— Neon-validated real corpus
Live agent response (POST /agent/command)
| Field | Value |
|---|---|
| HTTP status | 200 |
| Tool calls made | 1 |
| Tool invoked | get_commodity_profile(commodity=blackberry, tenantId=nathel) |
| Tool duration | 301ms |
| Tool succeeded | true |
| Profile returned | Blackberry (Rubus fruticosus) v2.1.1, profile ID 84822208-5abd-4647-929b-e55e764595b6, confidence 0.93 |
| Anonymization active | yes — HOMEGROWN ORGANIC FARMS → [COMPANY_1]; profile UUID → [ACCOUNT_2] |
fact_citations in response | 0 |
claim_anchor in response | 0 |
event_id references | 0 |
The agent’s final message text was contextually correct (“I only have the blackberry commodity profile, not the actual arrival records … pull the actual receiving records … so we can report exact load IDs, case counts, lot IDs, and receipt timestamps”) — it correctly recognized the gap, but did NOT recover by issuing follow-up tool calls.
Real corpus validation (direct Neon query)
Query:
SELECT id, type, timestamp FROM events
WHERE tenant_id = 'nathel'
AND data->>'service' IS DISTINCT FROM 'staging-smoke'
AND timestamp >= '2026-05-09'::timestamptz
AND (data::text ILIKE '%blackberry%' OR data::text ILIKE '%HOMEGROWN ORGANIC%')
ORDER BY timestamp DESC;Result: 13 distinct real events available (well above the ≥3 target):
| Event type | Count | Notable event_ids |
|---|---|---|
signal.pp_arrival_logged (ORG BLACKBERRY / HOMEGROWN) | 2 | 55a94b35-6d78-44ab-8cc7-ed6b92b143df (2026-05-13, 48 cs), 7e0917c5-964d-419e-8b2b-7c3de203c65f (2026-05-12, 48 cs) |
signal.pp_arrival_logged (BLACKBERRY 12 OZ / FAMILY TREE) | 1 | 9b88eecb-c43b-4d22-8c56-97d80675f827 (2026-05-09, 672 cs) |
ingest.pdf_extracted | 2 | 25d6d716..., fd984dc2... (PP RP581 raw extractions for load 00267096) |
resolver.candidate_created | 2 | 04b7d5fb..., 64732928... (org-blackberry entity) |
resolver.pending_review | 2 | 502212ab..., 390c498d... |
resolver.arrival_reconciled | 1 | 1273a108... (HOMEGROWN ORGANIC FARMS / ORG BLACKBERRY / 48 cs) |
inbound.load_conflict_pending | 1 | fb2a6a07... (CONFLICTING quantity_cases 60 vs 48 for ORG BLACKBERRY HOMEGROWN load 00267096) |
lot.pre_arrival_synthesized | 1 | 2abdc23c... (lot 91325068 ORGBLAC TLC-20260513-af5tjqbj) |
inbound.load_expected | 1 | 118e6b50... (load 00267096 HOMEGROWN ORGANIC FARMS ORGANIC) |
work.thread_created | 1 | 2087a2a6... (qc_case thread for load 00267096 HOMEGROWN) |
health.daily_reconciliation | 1 | d4ce44c5... (named HOMEGROWN ORGANIC FARMS load 00267096 with signal.pp_arrival_logged=1) |
| TOTAL | 13 |
V-703 acceptance evaluation
| Target | Actual | Verdict |
|---|---|---|
| Agent cites ≥3 real event_ids that exist in nathel events table | 0 cited / 13 available | VERIFYING-PARTIAL_real_corpus_exists_but_agent_did_not_retrieve_it |
V-703 root cause: agent orchestration parity gap
The agent demonstrates correct domain knowledge (recognized the gap, asked a sensible follow-up) and correct anonymization. But its single-tool plan for an OPERATIONAL ARRIVALS query is a real gap:
Expected tool plan for "X arrivals from Y this past week":
get_events(tenantId=nathel, types=['signal.pp_arrival_logged','inbound.load_expected','ingest.pdf_extracted'], since=2026-05-09, filter_text=BLACKBERRY+HOMEGROWN)→ returns 5+ real events.lookup_inventory(commodity=blackberry, tenantId=nathel)→ returns active lots.walk_provenance(docId, claimAnchor, tenantId=nathel)on each cited event to build lineage.- Then synthesize message citing event_ids with timestamps + case counts + load IDs.
Actual tool plan: get_commodity_profile(commodity=blackberry, tenantId=nathel) and stop.
This is the “Living-Wiki commodity drill-through (the agentic ERP experience)” vision gap (MEMORY.md): the primitives all exist; the audit purpose is to tune + wire + time correctly. V-703 has now produced the canonical proof artifact for this gap.
Per-tool eval breakdown (V-704)
Aggregating across V-701..V-703:
| Tool | V-701 status | V-703 invoked? | V-703 surfaced gap? |
|---|---|---|---|
get_commodity_profile | pass 627ms | YES (the only call) | No (works as designed; returned the profile correctly) |
query_knowledge | pass 584ms | NO | Planner not wired — should have hit operational events |
get_events | pass 680ms | NO | Planner not wired — would have returned 5+ real ORG BLACKBERRY rows |
lookup_inventory | pass 465ms | NO | Planner not wired — would have returned active blackberry lots |
get_data_freshness | pass 464ms (overall=stale) | NO | Surfaced stale data finding in V-701 |
get_workflow | pass 467ms | NO | n/a for this query class |
trace_lot | pass 1447ms | NO | Could have traced lot 91325068 (TLC-20260513-af5tjqbj) |
get_brief | pass 955ms | NO | n/a — agent should compose own brief |
get_market_brief | fail 21801ms statement timeout | NO | Real perf bug — separate issue |
get_actor_context | fail 403 cross-tenant | NO | V-113 known-flaky |
walk_provenance (G-tool) | not in CFA probe set | NO | G-02 surfaced in gradeboard — structurally ready, not exercised live |
get_doc_graph (G-tool) | not in CFA probe set | NO | G-03 surfaced in gradeboard — structurally ready, not exercised live |
search_entity_profiles (G-tool) | not in CFA probe set | NO | G-04 surfaced in gradeboard — structurally ready, not exercised live |
get_entity_profile (G-tool) | not in CFA probe set | NO | G-01 surfaced in gradeboard — structurally ready, not exercised live |
get_intel_inbox_digest | not in CFA probe set | NO | n/a for this query |
Known followups / proposed B-tasks
- B-followup (HIGH):
get_market_briefstatement timeout. 21801ms exceeds default statement_timeout. Investigate query plan — likely missing index or N+1 across tenant_metrics. - B-followup (HIGH): Agent planner needs an “operational arrivals query” route that fans out to
get_events + lookup_inventory + walk_provenanceinstead of stopping atget_commodity_profile. This is the V-703 canonical gap. - B-followup (MED): CFA probe count drift — spec says 31, codebase has 20. Reconcile by either (a) authoring the missing 11 probes (
scenarios.tsextension), or (b) amending spec target to ≥18/20. - B-followup (MED): CFA
get_actor_contextprobe persona/tenant mismatch — Josh Garcia (admin@nathel) probing for tenant=fin-central produces a 403. Probe args should bind to persona tenant. - B-followup (LOW): Deploy metadata reports staging as
degradedwhile all health probes pass — metadata source needs cleanup. - V-704 deferred to follow-up session: the live runtime 4.01 baseline (vs 5.00 structural) gap will narrow as entity overlay docs are seeded for Tenant Zero, BLK-01 (Phase 5 §3 CF Pages wire-up) lands, and the Playwright suite runs against the live workspace (currently mocked per gradeboard CAVEAT).
Tracker proof artifacts
| V-task | Proof type | Artifact | State |
|---|---|---|---|
| V-701 | test_output | .artifacts/tests/cfa-runner-5b3c18c2-20260517-031449.txt + audit/cfa/findings.json | VERIFYING-PARTIAL |
| V-702 | test_output | .artifacts/tests/gradeboard-31a5472a-20260517-031740.txt + docs/audits/PARITY_G_GRADEBOARD_BASELINE_2026-05-17.{md,json} | VERIFYING (target met) |
| V-703 | lineage_trace + curl_response | .artifacts/lineage/blackberry-4ce4353a-2026-05-17T03-20-42-652Z.json + .artifacts/lineage/v703-real-corpus-validation.json | VERIFYING-PARTIAL |
| V-704 | code_commit | this doc | VERIFYING |
Artifact branch: claude/parity-artifacts-2026-05-17-eval-subagent (pushed to origin in same run).