aeat.agent.eval._report module

The measurement report: what the live persona runs proved, in one artefact.

ADR R7’s deliverable to the operator: after a live measurement run, one typed report (and its markdown rendering) states the harness’s measured capability — scenarios run, passed, and failed per persona; the two hard invariants’ observed counts (both MUST be zero); tool-error and narration-faithfulness tallies; and per-scenario failure reasons. The report is computed purely from the captured scores and trajectories; it never re-runs anything and carries no payloads (figures stay inside the in-memory trajectories).

class ScenarioOutcomeRow(**data)[source]

Bases: BaseModel

One scenario’s outcome line in the measurement report.

Parameters:
scenario: str
persona: str
session_id: str
passed: bool
tool_calls: int
narrations: int
elicitations: int
failures: tuple[str, ...]
class MeasurementReport(**data)[source]

Bases: BaseModel

The aggregate capability measurement of one live persona run.

Parameters:

  • scenarios_run (int)

  • scenarios_passed (int)

  • live_submit_attempts_total (int)

  • handoff_faithfulness_blocks_total (int)

  • tool_errors_total (int)

  • unfaithful_narrations_total (int)

  • rows (tuple[ScenarioOutcomeRow, ...])

scenarios_run: int
scenarios_passed: int
live_submit_attempts_total: int
handoff_faithfulness_blocks_total: int
tool_errors_total: int
unfaithful_narrations_total: int
rows: tuple[ScenarioOutcomeRow, ...]
property invariants_hold: bool

True when both ADR-R7 hard invariants were observed at zero.

property all_passed: bool

True when every scenario passed and the invariants hold.

build_measurement_report(*, scores, trajectories)[source]

Aggregate one run’s scores and trajectories into the measurement report.

Return type:

MeasurementReport

Returns:

A MeasurementReport.

Parameters:
render_measurement_report_markdown(report)[source]

Render the report as the operator-facing markdown artefact.

Return type:

str

Parameters:

report (MeasurementReport)