aeat.application.diagnostics_run_health module

Local-only run-health diagnostics: LLM run timing plus auth-session staleness.

Folds two existing local-only signals into one typed operator-facing report so a slow LLM-backed classification run or a stale/expired persisted AEAT auth session is diagnosable without leaving the host:

Nothing here performs a network call or a live AEAT read: the LLM run records are read from encrypted local secure-object storage and the auth probe reads only the locally persisted session token’s metadata. This backs the aeat app diagnostics run-health operator surface (GitHub issue #407).

list_recent_runs() projects the same recorded LLMRunRecord rows individually (most-recent-first, optionally limited) rather than aggregated per-provider, backing the sibling aeat app diagnostics runs listing verb (also GitHub issue #407). It reuses load_records() directly – there is no parallel capture or storage path here.

build_latency_report() and build_error_breakdown() project the same recorded rows into a percentile-latency view and a failed-run error-kind breakdown, backing the aeat app diagnostics latency and aeat app diagnostics errors verbs (also GitHub issue #407). Neither introduces a new capture or storage path – both read load_records() exactly as run-health and runs do, honouring composition-service-no-parallel-write-path.

build_llm_usage_report() projects the same recorded rows into a run-count/duration/success-rate summary grouped by provider AND by model, backing the aeat app diagnostics llm-usage verb (also GitHub issue #407). LLMRunRecord carries only timing and outcome metadata – no token counts are recorded on this store – so the usage summary reports run counts, durations, and success rate rather than token/cost figures (those are covered by the separate build_llm_diagnostics_report() usage/cost/confidence report, which folds the distinct completion-call UsageRecord log). This report again reuses load_records() directly – there is no parallel capture or storage path here either.

See also

LLMRunTelemetryRecorder

Local encrypted recorder that supplies every run row this module reads.

LLMRunRecord

Timing/outcome-only record projected into each diagnostic report.

test_operator_auth()

Local auth-session probe folded into the run-health report.

_app_diagnostics

CLI transport for the run-health, runs, latency, errors, and llm-usage verbs.

diagnostics_telemetry

Remote-telemetry posture/flush service that reuses the aggregate LLM-run signal without widening the payload.

class LlmRunProviderMetrics(**data)[source]

Bases: BaseModel

Per-provider aggregate of recent local LLM run-timing telemetry.

Aggregated from LLMRunRecord rows for a single provider. Carries only timing and outcome metadata – never prompt or response text.

Parameters:

  • provider (str)

  • runs (int)

  • succeeded (int)

  • failed (int)

  • min_duration_ms (int | None)

  • max_duration_ms (int | None)

  • mean_duration_ms (Decimal | None)

provider: str
runs: int
succeeded: int
failed: int
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
class RunHealthReport(**data)[source]

Bases: BaseModel

Typed local-only run-health report: LLM run timing plus auth staleness.

Produced by build_run_health_report(). has_run_data is False when no LLM run telemetry has been recorded yet, so callers can print an instructive empty message. The auth section always carries a verdict (a fresh profile with no configured provider still reports persisted_session_present = False).

Parameters:
since: date | None
until: date | None
llm_providers: tuple[LlmRunProviderMetrics, ...]
total_runs: int
total_succeeded: int
total_failed: int
auth_provider: str
auth_configured: bool
persisted_session_present: bool
persisted_session_expired: bool | None
persisted_session_state: str
probe_summary: str
property has_run_data: bool

Return True when at least one LLM run has been recorded locally.

property session_stale: bool

Return True when a persisted session exists and has expired.

build_run_health_report(*, since=None, until=None, provider=None, run_telemetry_recorder=None, auth_probe=None)[source]

Aggregate local LLM run telemetry and the auth-session probe into one report.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional LLM run-record provider label filter (e.g. "llm:claude:sonnet", "claude"); scopes ONLY the LLM run-timing section. This is distinct from an AEAT auth provider name – the auth-session probe always auto-resolves its provider from workflow state (see auth_probe below) and never receives this filter.

  • run_telemetry_recorder (LLMRunTelemetryRecorder | None) – Injected recorder (dependency injection for tests); defaults to the active-bucket LLMRunTelemetryRecorder.

  • auth_probe (AuthTestResult | None) – Injected AuthTestResult (dependency injection for tests); defaults to a fresh call to test_operator_auth() with no provider override, so it reports whatever AEAT auth provider is configured in workflow state (or “none configured”).

Return type:

RunHealthReport

Returns:

The populated RunHealthReport.

class RunRecordView(**data)[source]

Bases: BaseModel

One individual local LLM run-timing record, as reported to an operator.

Mirrors LLMRunRecord field-for-field; carries only accounting/timing metadata, never prompt or response text.

Parameters:
run_id: str
caller: str
provider: str
model: str
duration_ms: int
succeeded: bool
error_kind: str
started_at: datetime
list_recent_runs(*, since=None, until=None, provider=None, limit=None, run_telemetry_recorder=None)[source]

Return recent local LLM run-timing records, most-recent-first.

Reuses load_records() directly – the same recorder build_run_health_report() reads – so there is no parallel capture or storage path for this listing.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; None returns every provider.

  • limit (int | None) – Optional cap on the number of most-recent rows returned; None returns every matching record.

  • run_telemetry_recorder (LLMRunTelemetryRecorder | None) – Injected recorder (dependency injection for tests); defaults to the active-bucket LLMRunTelemetryRecorder.

Return type:

tuple[RunRecordView, ...]

Returns:

Matching RunRecordView rows ordered most-recent-first (ties broken by run_id descending, mirroring the recorder’s own stable ascending order reversed).

class LatencyPercentiles(**data)[source]

Bases: BaseModel

Percentile and summary latency statistics over a set of run durations.

Percentiles are computed with the nearest-rank method (ceil(p * n / 100), 1-indexed into the ascending-sorted duration list) – a deterministic, interpolation-free method whose outputs always equal a recorded duration value. Populated only when at least one duration is present; entries is 0 (all other fields absent) for an empty input.

Parameters:

  • entries (int)

  • min_duration_ms (int | None)

  • max_duration_ms (int | None)

  • mean_duration_ms (Decimal | None)

  • p50_duration_ms (int | None)

  • p95_duration_ms (int | None)

  • p99_duration_ms (int | None)

entries: int
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
p50_duration_ms: int | None
p95_duration_ms: int | None
p99_duration_ms: int | None
class LatencyReport(**data)[source]

Bases: BaseModel

Typed local-only latency report: overall plus optional per-provider percentiles.

Produced by build_latency_report(). by_provider is populated only when the caller did not scope the query to a single provider filter (a single-provider query makes overall and the sole provider row redundant).

Parameters:
since: date | None
until: date | None
provider: str | None
overall: LatencyPercentiles
by_provider: tuple[tuple[str, LatencyPercentiles], ...]
property has_run_data: bool

Return True when at least one run duration was aggregated.

class ErrorKindCount(**data)[source]

Bases: BaseModel

One error_kind value’s failure count, optionally scoped to a provider.

Parameters:

  • error_kind (str)

  • provider (str)

  • count (int)

error_kind: str
provider: str
count: int
class ErrorsBreakdownReport(**data)[source]

Bases: BaseModel

Typed local-only breakdown of failed LLM runs by provider and error kind.

Produced by build_error_breakdown(). Rows are sorted by descending count, then by provider, then by error_kind for a stable presentation order.

Parameters:
since: date | None
until: date | None
provider: str | None
total_runs: int
total_failed: int
by_error_kind: tuple[ErrorKindCount, ...]
property has_failures: bool

Return True when at least one failed run was recorded.

build_latency_report(*, since=None, until=None, provider=None, run_telemetry_recorder=None)[source]

Aggregate recorded run durations into overall and per-provider percentiles.

Reuses load_records() directly – the same recorder build_run_health_report() and list_recent_runs() read – so there is no parallel capture or storage path for this report.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; when supplied, overall reflects only that provider’s runs and by_provider is left empty (a single-provider breakdown would duplicate overall).

  • run_telemetry_recorder (LLMRunTelemetryRecorder | None) – Injected recorder (dependency injection for tests); defaults to the active-bucket LLMRunTelemetryRecorder.

Return type:

LatencyReport

Returns:

The populated LatencyReport.

build_error_breakdown(*, since=None, until=None, provider=None, run_telemetry_recorder=None)[source]

Group failed recorded runs by provider and error_kind.

Reuses load_records() directly – the same recorder every sibling diagnostics report reads – so there is no parallel capture or storage path for this report.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; None breaks down every provider’s failures.

  • run_telemetry_recorder (LLMRunTelemetryRecorder | None) – Injected recorder (dependency injection for tests); defaults to the active-bucket LLMRunTelemetryRecorder.

Return type:

ErrorsBreakdownReport

Returns:

The populated ErrorsBreakdownReport.

class LlmUsageModelMetrics(**data)[source]

Bases: BaseModel

One provider’s per-model aggregate of recent local LLM run telemetry.

Aggregated from LLMRunRecord rows sharing a single provider (recorded on the owning LlmUsageProviderMetrics) AND model. Carries only run-count, duration, and outcome metadata – LLMRunRecord records no token counts, so this is a run/timing/success-rate summary, not a token-usage summary.

Parameters:

  • model (str)

  • runs (int)

  • succeeded (int)

  • failed (int)

  • min_duration_ms (int | None)

  • max_duration_ms (int | None)

  • mean_duration_ms (Decimal | None)

  • total_duration_ms (int)

model: str
runs: int
succeeded: int
failed: int
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
total_duration_ms: int
property success_rate: Decimal

Return the fraction of runs that succeeded, or 0 when runs is 0.

class LlmUsageProviderMetrics(**data)[source]

Bases: BaseModel

One provider’s aggregate of recent local LLM run telemetry, plus its per-model rows.

models breaks the same provider-scoped records down further by model, so an operator can see which model within a provider drives run volume, duration, or failures.

Parameters:
provider: str
runs: int
succeeded: int
failed: int
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
total_duration_ms: int
models: tuple[LlmUsageModelMetrics, ...]
property success_rate: Decimal

Return the fraction of runs that succeeded, or 0 when runs is 0.

class LlmUsageReport(**data)[source]

Bases: BaseModel

Typed local-only LLM usage summary: run counts, durations, and success rate.

Produced by build_llm_usage_report(). Groups the same recorded LLMRunRecord rows build_run_health_report() reads by provider (by_provider), each provider row carrying its own per-model breakdown (models). has_run_data is False when no LLM run telemetry has been recorded yet.

Parameters:
since: date | None
until: date | None
by_provider: tuple[LlmUsageProviderMetrics, ...]
total_runs: int
total_succeeded: int
total_failed: int
property has_run_data: bool

Return True when at least one LLM run has been recorded locally.

property overall_success_rate: Decimal

Return the fraction of all recorded runs that succeeded, or 0 when empty.

build_llm_usage_report(*, since=None, until=None, provider=None, run_telemetry_recorder=None)[source]

Aggregate recorded LLM run telemetry into a usage summary by provider and model.

Reuses load_records() directly – the same recorder every sibling diagnostics report reads – so there is no parallel capture or storage path for this report (composition-service-no-parallel-write-path). LLMRunRecord carries no token counts, so this is a run-count/duration/success-rate summary rather than a token-usage summary.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; None aggregates every provider.

  • run_telemetry_recorder (LLMRunTelemetryRecorder | None) – Injected recorder (dependency injection for tests); defaults to the active-bucket LLMRunTelemetryRecorder.

Return type:

LlmUsageReport

Returns:

The populated LlmUsageReport.