aeat.agent.eval._live_scoring module

Score a captured live trajectory against a golden scenario and the hard invariants.

The judging half of ADR R7: where the golden runner asserts a DECLARED trajectory’s properties, this module asserts the OBSERVED one — the command keys a live subagent persona actually issued, the narrations it actually produced, and the elicitation answers it actually gave. Every check that belongs to another layer stays caller-injected, preserving this package’s hexagonal injection pattern (the runner’s docstring is the authority): the faithfulness function arrives as a callable the caller imports from entrypoints.mcp, and the live-write / handoff leaf sets arrive as data because their single declarations live in the server layer this package must not import.

A live model’s path is legitimately non-deterministic in its READS, so the trajectory dimension is coverage, not equality: the scenario’s expected_trajectory must appear as an order-preserving subsequence of the observed keys, and the lifecycle order must hold over the observed keys — extra reads are fine, a skipped verify or an out-of-order export is not.

class FaithfulnessCheckFn(*args, **kwargs)[source]

Bases: Protocol

The real faithfulness_check signature, caller-injected.

class LiveScenarioScore(**data)[source]

Bases: BaseModel

The per-dimension verdict for one live persona session.

passed requires every dimension true AND both hard invariants clean; failures carries a human-readable reason per failed dimension, in the golden-runner style.

Parameters:
scenario: str
persona: str
session_id: str
keys_resolve: bool
lifecycle_ordered: bool
expected_covered: bool
tool_errors: tuple[str, ...]
invariants: LiveInvariantVerdict
narration_checks: tuple[NarrationFaithfulness, ...]
failures: tuple[str, ...]
property passed: bool

True when every dimension held, the invariants are clean, and nothing failed.

score_live_trajectory(trajectory, *, scenario, valid_commands, faithfulness_check_fn, live_write_leaves, handoff_leaves)[source]

Score one captured live session against its golden scenario.

Parameters:

  • trajectory (LiveTrajectory) – The captured live session.

  • scenario (GoldenScenario) – The golden scenario the session ran.

  • valid_commands (frozenset[str]) – The resolvable registry command keys, injected from the live CLI schema registry by the caller.

  • faithfulness_check_fn (FaithfulnessCheckFn) – The REAL faithfulness check, injected by the caller (this package never imports entrypoints.mcp).

  • live_write_leaves (frozenset[str]) – The forbidden AEAT live-write leaf verbs, injected from their single server-layer declaration.

  • handoff_leaves (frozenset[str]) – The irreversible filing-handoff leaf verbs, injected from their single server-layer declaration.

Return type:

LiveScenarioScore

Returns:

The LiveScenarioScore with per-dimension verdicts, the two hard invariants, and per-narration faithfulness checks run against the session’s own preceding tool results.

class DiscoveryScore(**data)[source]

Bases: BaseModel

Selection-quality verdict for one long-tail-verb discovery trajectory.

The measurement half of ADR mcp-progressive-discovery P6 (plan step S23): given an observed trajectory that set out to reach one long-tail target_command_key, this scores how efficiently the model got there. rounds_to_correct_verb is the 1-based ordinal of the tool call that first executed the target (every round-trip up to and including it), so a direct FULL-surface call scores 1 and a CORE-surface search + execute pair scores 2; discovery_calls isolates the search / execute meta round-trips within that prefix, and misselections counts wrong, non-target verbs actually executed before the target was reached.

reached is the load-bearing dimension: a trajectory that never executes the target scores rounds_to_correct_verb = 0 (unreachable ordinal, since a genuine reach is always >= 1) and records a failure - the anti-tautology guard that stops an unreached target from reading as a cheap discovery.

Variables:

  • scenario – The discovery scenario name (empty for a free session).

  • persona – The harness persona the driver played.

  • session_id – The captured session’s stable identity.

  • target_command_key – The long-tail registry command key the session set out to locate and invoke.

  • reached – Whether the target command key was executed without error.

  • rounds_to_correct_verb – The 1-based ordinal of the target-executing call among all tool calls; 0 when the target was never reached.

  • discovery_calls – The number of search / execute meta round-trips issued up to and including the target-executing call.

  • misselections – The number of wrong, non-empty command keys executed before the target was reached.

  • failures – A human-readable reason per failed dimension.

Parameters:

  • scenario (str)

  • persona (str)

  • session_id (str)

  • target_command_key (str)

  • reached (bool)

  • rounds_to_correct_verb (int)

  • discovery_calls (int)

  • misselections (int)

  • failures (tuple[str, ...])

scenario: str
persona: str
session_id: str
target_command_key: str
reached: bool
rounds_to_correct_verb: int
discovery_calls: int
misselections: int
failures: tuple[str, ...]
property passed: bool

True when the target was reached and no dimension failed.

class SurfaceDiscoveryComparison(**data)[source]

Bases: BaseModel

A CORE-surface discovery trajectory measured against a FULL-surface one.

The core-vs-full A/B artifact of ADR mcp-progressive-discovery P6 (plan step S24): both surfaces must reach the SAME long-tail verb, and the comparison quantifies the trade the ADR makes - the lean CORE surface advertises far fewer tools (core_advertised_tool_count vs full_advertised_tool_count) at the cost of a small number of extra discovery round-trips (rounds_delta), while the target stays reachable.

core_advertised_tool_count / full_advertised_tool_count are caller-supplied (measured from the live surface policy, so the numbers are not baked into this pure module); 0 leaves the tool-count dimension unasserted.

Variables:

  • target_command_key – The long-tail verb both surfaces set out to reach.

  • core – The CORE-surface discovery score (search + execute path).

  • full – The FULL-surface discovery score (direct call path).

  • core_advertised_tool_count – Tools advertised up front on the CORE surface.

  • full_advertised_tool_count – Tools advertised up front on the FULL surface.

  • failures – A human-readable reason per failed comparison dimension.

Parameters:
target_command_key: str
core: DiscoveryScore
full: DiscoveryScore
core_advertised_tool_count: int
full_advertised_tool_count: int
failures: tuple[str, ...]
property both_reached_same_target: bool

True when both surfaces executed the one target verb.

property rounds_delta: int

the discovery cost of the lean surface.

Type:

CORE minus FULL rounds-to-correct-verb

property full_surface_advertises_more: bool

True when the FULL surface advertises strictly more tools up front than CORE.

property passed: bool

True when both surfaces reached the same target and nothing failed.

score_discovery_trajectory(trajectory, *, target_command_key, meta_tool_names=frozenset({'execute', 'search'}))[source]

Score how efficiently one observed trajectory reached a long-tail verb.

Walks trajectory.tool_calls in order and finds the first call that executed target_command_key without error. Everything is derived from that reach point: rounds_to_correct_verb is its 1-based ordinal among all calls, discovery_calls counts the search / execute meta calls in the prefix up to and including it, and misselections counts wrong non-target command keys executed strictly before it. A trajectory that never reaches the target scores reached = False with a 0 ordinal and a recorded failure.

Parameters:

  • trajectory (LiveTrajectory) – The captured live session.

  • target_command_key (str) – The long-tail registry command key the session set out to locate and invoke.

  • meta_tool_names (frozenset[str]) – The tool names counted as discovery round-trips (the search / execute meta pair by default; overridable so the function never hard-codes the server’s tool naming).

Return type:

DiscoveryScore

Returns:

The DiscoveryScore for the session.

compare_surface_discovery(*, core, full, core_advertised_tool_count=0, full_advertised_tool_count=0)[source]

Compare a CORE-surface discovery score against a FULL-surface one.

Both scores must target the same verb; the comparison records a failure when either surface failed to reach it, so a surface that silently lost the verb cannot pass. The advertised tool counts are optional caller measurements from the live surface policy (0 leaves that dimension unasserted).

Parameters:

  • core (DiscoveryScore) – The CORE-surface discovery score (the search + execute path).

  • full (DiscoveryScore) – The FULL-surface discovery score (the direct call path).

  • core_advertised_tool_count (int) – Tools advertised up front under CORE.

  • full_advertised_tool_count (int) – Tools advertised up front under FULL.

Return type:

SurfaceDiscoveryComparison

Returns:

The SurfaceDiscoveryComparison.

Raises:

ValueError – When the two scores target different verbs.

class IdentityStateProtocol(*args, **kwargs)[source]

Bases: Protocol

The per-session identity-read state the gate mutates, caller-injected.

Structurally satisfied by entrypoints.mcp._identity_gate.SessionIdentityState; declared here so this package never imports entrypoints.mcp (the hexagonal direction the runner’s docstring documents), mirroring FaithfulnessCheckFn.

record_identity_read()[source]
Return type:

None

class IdentityGateRefusalFn(*args, **kwargs)[source]

Bases: Protocol

The real identity_gate_refusal signature, caller-injected.

Returns a refusal string when a mutating call runs under an unconfirmed or re-armed identity, else None; records an identity-read verb and re-arms on a profile-switch verb as a side effect on state.

class IdentityConfirmationScore(**data)[source]

Bases: BaseModel

Identity-confirmation verdict for one observed trajectory (ADR I2 / I4).

The measurement half of the block-first-mutation identity gate: given an observed session that mutates a taxpayer’s draft, this replays the REAL identity_gate_refusal decision over the session’s tool calls (console identity reads recorded, profile switches re-arming) and records every mutation the gate would refuse. A refusal means the agent changed a taxpayer’s data without first confirming WHO is active, or without re-confirming after a profile switch - the Erik/Erika cross-taxpayer hazard.

mutating_step_present is the load-bearing anti-tautology guard: a trajectory that never attempts a mutation never exercises the gate, so it cannot pass (there is nothing to confirm), mirroring DiscoveryScore.reached.

Variables:

  • scenario – The scenario name the session ran (empty for a free session).

  • persona – The harness persona the driver played.

  • session_id – The captured session’s stable identity.

  • mutating_step_present – Whether any observed call was a genuine mutation, classified by the real gate (a freshly-armed session refuses it), so no separate mutability oracle is needed.

  • identity_confirmed – Whether the real gate refused NO mutation over the session order - every mutation preceded by an identity read and re-confirmed after every switch.

  • gate_refused_mutations – The command keys the real gate refused, in order.

  • failures – A human-readable reason per failed dimension.

Parameters:

  • scenario (str)

  • persona (str)

  • session_id (str)

  • mutating_step_present (bool)

  • identity_confirmed (bool)

  • gate_refused_mutations (tuple[str, ...])

  • failures (tuple[str, ...])

scenario: str
persona: str
session_id: str
mutating_step_present: bool
identity_confirmed: bool
gate_refused_mutations: tuple[str, ...]
failures: tuple[str, ...]
property passed: bool

True when a mutation was exercised and the real gate refused none of them.

score_identity_trajectory(trajectory, *, identity_gate_refusal_fn, new_identity_state_fn, identity_read_console_tools)[source]

Score whether an observed trajectory confirmed identity before every mutation.

Replays the REAL identity_gate_refusal over the session’s tool calls in order: a console identity read (a tool name in identity_read_console_tools - whoami / harness.load, which carry no registry command key) records the read on the shared session state; every command-key call is passed through the gate, which refuses a mutation running under an unconfirmed or re-armed (post-switch) identity. A command is classified a MUTATION by the same gate - a freshly-armed session refuses it - so mutating_step_present needs no separate mutability oracle. The gate arrives injected (this package never imports entrypoints.mcp), so the dimension scores the real decision, never a re-implementation.

Parameters:

  • trajectory (LiveTrajectory) – The captured live session.

  • identity_gate_refusal_fn (IdentityGateRefusalFn) – The REAL identity_gate_refusal, injected.

  • new_identity_state_fn (Callable[[], IdentityStateProtocol]) – A factory for a fresh per-session identity state (the real SessionIdentityState), injected.

  • identity_read_console_tools (frozenset[str]) – The console identity-read tool names (whoami / harness.load), injected from their server-layer declaration.

Return type:

IdentityConfirmationScore

Returns:

The IdentityConfirmationScore for the session.