cadrumo.application.diagnostics_run_health module

Local-only run-health diagnostics: LLM run timing plus auth-session staleness.

Folds two existing local-only signals into one typed operator-facing report so a slow LLM-backed classification run or a stale/expired persisted AEAT auth session is diagnosable without leaving the host:

Nothing here performs a network call or a live AEAT read: the LLM run records are read from encrypted local secure-object storage and the auth probe reads only the locally persisted session token’s metadata. This backs the aeat app diagnostics run-health operator surface.

list_recent_runs() projects the same recorded DiagnosticRunRecord rows individually (most-recent-first, optionally limited) rather than aggregated per-provider, backing the sibling aeat app diagnostics runs listing verb. It reads them through the injected diagnostic telemetry port – there is no parallel capture or storage path here.

build_latency_report() and build_error_breakdown() project the same recorded rows into a percentile-latency view and a failed-run error-kind breakdown, backing the aeat app diagnostics latency and aeat app diagnostics errors verbs. Neither introduces a new capture or storage path – both read through the diagnostic telemetry port exactly as run-health and runs do, honouring aeat-architecture-boundaries.

build_llm_usage_report() projects the same recorded rows into a run-count/duration/success-rate summary grouped by provider AND by model, backing the aeat app diagnostics llm-usage verb. DiagnosticRunRecord carries only timing and outcome metadata – no token counts are recorded on this store – so the usage summary reports run counts, durations, and success rate rather than token/cost figures (those are covered by the separate build_llm_diagnostics_report() usage/cost/confidence report, which folds the distinct completion-call UsageRecord log). This report again reads through the injected diagnostic telemetry port – there is no parallel capture or storage path here either.

See also

DiagnosticRunTelemetryPort

Application-owned read contract that supplies every run row this module reads.

DiagnosticRunRecord

Timing/outcome-only record projected into each diagnostic report.

test_operator_auth()

Local auth-session probe folded into the run-health report.

_app_diagnostics

CLI transport for the run-health, runs, latency, errors, and llm-usage verbs.

diagnostics_telemetry

Remote-telemetry posture/flush service that reuses the aggregate LLM-run signal without widening the payload.

class LlmRunProviderMetrics(**data)[source]

Bases: BaseModel

Per-provider aggregate of recent local LLM run-timing telemetry.

Aggregated from DiagnosticRunRecord rows for a single provider. Carries only timing and outcome metadata – never prompt or response text.

Parameters:
provider: str
runs: NonNegativeInt
succeeded: NonNegativeInt
failed: NonNegativeInt
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
class RunHealthReport(**data)[source]

Bases: BaseModel

Typed local-only run-health report: LLM run timing plus auth staleness.

Produced by build_run_health_report(). has_run_data is False when no LLM run telemetry has been recorded yet, so callers can print an instructive empty message. The auth section always carries a verdict (a fresh profile with no configured provider still reports persisted_session_present = False).

Parameters:
since: date | None
until: date | None
llm_providers: tuple[LlmRunProviderMetrics, ...]
total_runs: int
total_succeeded: int
total_failed: int
auth_provider: str
auth_configured: bool
persisted_session_present: bool
persisted_session_expired: bool | None
persisted_session_state: str
probe_summary: str
property has_run_data: bool

Return True when at least one LLM run has been recorded locally.

property session_stale: bool

Return True when a persisted session exists and has expired.

build_run_health_report(*, since=None, until=None, provider=None, run_telemetry_port, auth_probe_port)[source]

Aggregate local LLM run telemetry and the auth-session probe into one report.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional LLM run-record provider label filter (e.g. "llm:claude:sonnet", "claude"); scopes ONLY the LLM run-timing section. This is distinct from an AEAT auth provider name – the auth-session probe always auto-resolves its provider from workflow state and never receives this filter.

  • run_telemetry_port (DiagnosticRunTelemetryPort) – Injected diagnostic telemetry read port supplied by the outer composition root.

  • auth_probe_port (DiagnosticAuthProbePort) – Injected redacted auth-readiness probe supplied by the outer composition root.

Return type:

RunHealthReport

Returns:

The populated RunHealthReport.

class RunRecordView(**data)[source]

Bases: BaseModel

One individual local LLM run-timing record, as reported to an operator.

Mirrors DiagnosticRunRecord field-for-field; carries only accounting/timing metadata, never prompt or response text.

Parameters:
run_id: str
caller: str
provider: str
model: str
duration_ms: NonNegativeInt
succeeded: bool
error_kind: str
started_at: datetime
list_recent_runs(*, since=None, until=None, provider=None, limit=None, run_telemetry_port)[source]

Return recent local LLM run-timing records, most-recent-first.

Reads through the same injected diagnostic telemetry port build_run_health_report() uses, so there is no parallel capture or storage path for this listing.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; None returns every provider.

  • limit (int | None) – Optional cap on the number of most-recent rows returned; None returns every matching record.

  • run_telemetry_port (DiagnosticRunTelemetryPort) – Injected diagnostic telemetry read port supplied by the outer composition root.

Return type:

tuple[RunRecordView, ...]

Returns:

Matching RunRecordView rows ordered most-recent-first (ties broken by run_id descending, mirroring the recorder’s own stable ascending order reversed).

class LatencyPercentiles(**data)[source]

Bases: BaseModel

Percentile and summary latency statistics over a set of run durations.

Percentiles are computed with the nearest-rank method (ceil(p * n / 100), 1-indexed into the ascending-sorted duration list) – a deterministic, interpolation-free method whose outputs always equal a recorded duration value. Populated only when at least one duration is present; entries is 0 (all other fields absent) for an empty input.

Parameters:

  • entries (int)

  • min_duration_ms (int | None)

  • max_duration_ms (int | None)

  • mean_duration_ms (Decimal | None)

  • p50_duration_ms (int | None)

  • p95_duration_ms (int | None)

  • p99_duration_ms (int | None)

entries: int
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
p50_duration_ms: int | None
p95_duration_ms: int | None
p99_duration_ms: int | None
class LatencyReport(**data)[source]

Bases: BaseModel

Typed local-only latency report: overall plus optional per-provider percentiles.

Produced by build_latency_report(). by_provider is populated only when the caller did not scope the query to a single provider filter (a single-provider query makes overall and the sole provider row redundant).

Parameters:
since: date | None
until: date | None
provider: str | None
overall: LatencyPercentiles
by_provider: tuple[tuple[str, LatencyPercentiles], ...]
property has_run_data: bool

Return True when at least one run duration was aggregated.

class ErrorKindCount(**data)[source]

Bases: BaseModel

One error_kind value’s failure count, optionally scoped to a provider.

Parameters:

  • error_kind (str)

  • provider (str)

  • count (int)

error_kind: str
provider: str
count: int
class ErrorsBreakdownReport(**data)[source]

Bases: BaseModel

Typed local-only breakdown of failed LLM runs by provider and error kind.

Produced by build_error_breakdown(). Rows are sorted by descending count, then by provider, then by error_kind for a stable presentation order.

Parameters:
since: date | None
until: date | None
provider: str | None
total_runs: int
total_failed: int
by_error_kind: tuple[ErrorKindCount, ...]
property has_failures: bool

Return True when at least one failed run was recorded.

build_latency_report(*, since=None, until=None, provider=None, run_telemetry_port)[source]

Aggregate recorded run durations into overall and per-provider percentiles.

Reads through the same injected diagnostic telemetry port build_run_health_report() and list_recent_runs() use, so there is no parallel capture or storage path for this report.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; when supplied, overall reflects only that provider’s runs and by_provider is left empty (a single-provider breakdown would duplicate overall).

  • run_telemetry_port (DiagnosticRunTelemetryPort) – Injected diagnostic telemetry read port supplied by the outer composition root.

Return type:

LatencyReport

Returns:

The populated LatencyReport.

build_error_breakdown(*, since=None, until=None, provider=None, run_telemetry_port)[source]

Group failed recorded runs by provider and error_kind.

Reads through the same injected diagnostic telemetry port every sibling diagnostics report uses, so there is no parallel capture or storage path for this report.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; None breaks down every provider’s failures.

  • run_telemetry_port (DiagnosticRunTelemetryPort) – Injected diagnostic telemetry read port supplied by the outer composition root.

Return type:

ErrorsBreakdownReport

Returns:

The populated ErrorsBreakdownReport.

class LlmUsageModelMetrics(**data)[source]

Bases: BaseModel

One provider’s per-model aggregate of recent local LLM run telemetry.

Aggregated from DiagnosticRunRecord rows sharing a single provider (recorded on the owning LlmRunHealthProviderMetrics) AND model. Carries only run-count, duration, and outcome metadata – DiagnosticRunRecord records no token counts, so this is a run/timing/success-rate summary, not a token-usage summary.

Parameters:
model: str
runs: NonNegativeInt
succeeded: NonNegativeInt
failed: NonNegativeInt
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
total_duration_ms: int
property success_rate: Decimal

Return the fraction of runs that succeeded, or 0 when runs is 0.

class LlmRunHealthProviderMetrics(**data)[source]

Bases: BaseModel

One provider’s aggregate of recent local LLM run telemetry, plus its per-model rows.

models breaks the same provider-scoped records down further by model, so an operator can see which model within a provider drives run volume, duration, or failures.

Parameters:
provider: str
runs: NonNegativeInt
succeeded: NonNegativeInt
failed: NonNegativeInt
min_duration_ms: int | None
max_duration_ms: int | None
mean_duration_ms: Decimal | None
total_duration_ms: int
models: tuple[LlmUsageModelMetrics, ...]
property success_rate: Decimal

Return the fraction of runs that succeeded, or 0 when runs is 0.

class LlmUsageReport(**data)[source]

Bases: BaseModel

Typed local-only LLM usage summary: run counts, durations, and success rate.

Produced by build_llm_usage_report(). Groups the same recorded DiagnosticRunRecord rows build_run_health_report() reads by provider (by_provider), each provider row carrying its own per-model breakdown (models). has_run_data is False when no LLM run telemetry has been recorded yet.

Parameters:
since: date | None
until: date | None
by_provider: tuple[LlmRunHealthProviderMetrics, ...]
total_runs: int
total_succeeded: int
total_failed: int
property has_run_data: bool

Return True when at least one LLM run has been recorded locally.

property overall_success_rate: Decimal

Return the fraction of all recorded runs that succeeded, or 0 when empty.

build_llm_usage_report(*, since=None, until=None, provider=None, run_telemetry_port)[source]

Aggregate recorded LLM run telemetry into a usage summary by provider and model.

Reads through the same injected diagnostic telemetry port every sibling diagnostics report uses, so there is no parallel capture or storage path for this report (aeat-architecture-boundaries). DiagnosticRunRecord carries no token counts, so this is a run-count/duration/success-rate summary rather than a token-usage summary.

Parameters:

  • since (date | None) – Inclusive lower date bound on run records, or None.

  • until (date | None) – Inclusive upper date bound on run records, or None.

  • provider (str | None) – Optional provider label filter; None aggregates every provider.

  • run_telemetry_port (DiagnosticRunTelemetryPort) – Injected diagnostic telemetry read port supplied by the outer composition root.

Return type:

LlmUsageReport

Returns:

The populated LlmUsageReport.