aeat.agent.eval._flywheel module

The data flywheel: promote a live failure into a golden regression scenario.

ADR R7’s operating cadence: a live subagent-persona failure is not just a red run — it becomes a NEW golden scenario so the same failure cannot recur unnoticed. Promotion is deliberately conservative: the promoted scenario pins the ORIGINAL scenario’s declared expectations (the correct path), while the observed failing behaviour is preserved verbatim in the scenario’s comment header as the annotation-queue evidence a tax-professional reviews. Names are content-addressed (a hash of scenario + failure signature) so re-promoting the same failure is idempotent — the flywheel dedups by construction, never by clock.

failure_signature(score)[source]

A stable content address for one failure shape (scenario + failed dimensions).

Return type:

str

Parameters:

score (LiveScenarioScore)

promote_failure(*, score, trajectory, scenario)[source]

Render the promoted golden-scenario TOML for one live failure.

The promoted scenario re-declares the ORIGINAL correct expectations; the observed failing trajectory and the failure reasons ride in the comment header as evidence, never as expectations (a failure must not become the new normal).

Raises:

ValueError – When the score did not actually fail — a passing run has nothing to promote.

Return type:

str

Parameters:
write_promoted_scenario(*, score, trajectory, scenario, scenarios_dir)[source]

Write the promoted scenario into scenarios_dir, idempotently.

Return type:

Path

Returns:

The path written (or the existing identical file on re-promotion).

Parameters: