aeat.application.ledger._llm_diagnostics module

LLM usage, cost, and classification-confidence diagnostics aggregation.

Folds the two metric stores the system already persists into one typed, operator-facing report:

  • the encrypted LLM usage log written by UsageRecorder (per call: provider, model, input/output tokens, estimated cost, cache-hit flag); and

  • the classification confidence stamped on each ledger Transaction whose active decision came from an LLM classifier (classified_by shaped llm:<provider>:<model> with a classification_confidence in [0, 1]), loaded from the active bucket’s TransactionCatalogueRepository when callers do not inject transactions directly.

No new tracking is introduced here: every figure is aggregated from records already on disk. The usage store and the classification store use distinct provider namespaces (the completion adapter’s LLMProvider versus the subprocess classifier provenance), so the report presents them as two parallel sections rather than a lossy cross-namespace join.

The report reads only accounting metadata (token counts, cost, confidence scores, provider labels); it never surfaces the persisted response text or any financial content, honouring sensitive-financial-data-secure-storage-only.

See also

build_llm_diagnostics_report():

Public aggregator that folds the usage log and ledger confidence rows.

UsageRecorder:

Storage boundary for provider/token/cost accounting records.

TransactionCatalogueRepository:

Bucket-scoped ledger catalogue reader used for confidence diagnostics.

DEFAULT_LOW_CONFIDENCE_THRESHOLD

Default confidence floor below which an LLM classification is reported as low-confidence. Operator-facing display threshold, overridable per call.

class LlmUsageProviderMetrics(**data)[source]

Bases: BaseModel

Per-provider aggregate of the LLM usage/cost log.

Aggregated from UsageRecord rows for a single provider. calls counts every recorded call (cache hits included); cache_hits counts the subset served from the local cache.

Parameters:

  • provider (str)

  • calls (int)

  • cache_hits (int)

  • input_tokens (int)

  • output_tokens (int)

  • total_tokens (int)

  • cost_estimate_usd (Decimal)

provider: str
calls: int
cache_hits: int
input_tokens: int
output_tokens: int
total_tokens: int
cost_estimate_usd: Decimal
class LlmConfidenceProviderMetrics(**data)[source]

Bases: BaseModel

Per-provider confidence distribution over LLM-classified transactions.

Aggregated from the active ledger catalogue’s transactions whose classified_by carries an llm: provenance and a non-null classification_confidence. low_confidence_count is the number of those decisions below the report’s tunable threshold. high_confidence_count (>= 0.8) and medium_confidence_count ([0.5, 0.8)) are fixed-floor distribution buckets; the remaining decisions fall below 0.5.

Parameters:

  • provider (str)

  • classified_count (int)

  • low_confidence_count (int)

  • high_confidence_count (int)

  • medium_confidence_count (int)

  • min_confidence (Decimal | None)

  • max_confidence (Decimal | None)

  • mean_confidence (Decimal | None)

provider: str
classified_count: int
low_confidence_count: int
high_confidence_count: int
medium_confidence_count: int
min_confidence: Decimal | None
max_confidence: Decimal | None
mean_confidence: Decimal | None
class LlmDiagnosticsReport(**data)[source]

Bases: BaseModel

Typed LLM usage / cost / confidence diagnostics report.

Produced by build_llm_diagnostics_report(). The usage section folds the encrypted usage log; the confidence section folds the classification confidence stamped on ledger transactions. has_data is False when neither store carries any LLM activity, so callers can print an instructive empty message.

Parameters:
since: date | None
until: date | None
low_confidence_threshold: Decimal
usage_providers: tuple[LlmUsageProviderMetrics, ...]
total_calls: int
total_cache_hits: int
total_input_tokens: int
total_output_tokens: int
total_cost_estimate_usd: Decimal
confidence_providers: tuple[LlmConfidenceProviderMetrics, ...]
total_classified: int
total_low_confidence: int
property has_data: bool

Return True when either metric store carried LLM activity.

build_llm_diagnostics_report(*, since=None, until=None, low_confidence_threshold=Decimal('0.5'), usage_recorder=None, bucket_id=None, transactions=None)[source]

Aggregate the existing usage and confidence metric stores into a report.

Parameters:

  • since (date | None) – Inclusive lower usage-record date bound, or None.

  • until (date | None) – Inclusive upper usage-record date bound, or None.

  • low_confidence_threshold (Decimal) – Confidence floor below which a classification counts as low-confidence.

  • usage_recorder (UsageRecorder | None) – Injected recorder; defaults to the active-bucket UsageRecorder.

  • bucket_id (str | None) – Ledger bucket whose TransactionCatalogueRepository supplies confidence rows; defaults to the active bucket. Ignored when transactions is supplied.

  • transactions (Iterable[Transaction] | None) – Injected transaction iterable; when None the active (or bucket_id) ledger catalogue is loaded.

Return type:

LlmDiagnosticsReport

Returns:

The populated LlmDiagnosticsReport.