cadrumo.application.ledger.llm_diagnostics module

LLM usage, cost, and classification-confidence diagnostics aggregation.

Folds the two metric stores the system already persists into one typed, operator-facing report:

  • the encrypted LLM usage log (per call: provider, model, input/output tokens, estimated cost, cache-hit flag); and

  • the classification confidence stamped on each ledger Transaction whose active decision came from an LLM classifier (classified_by shaped llm:<provider>:<model> with a classification_confidence in [0, 1]), loaded from the active bucket when supplied by the outer transaction diagnostics reader.

No new tracking is introduced here: every figure is aggregated from records already on disk. The usage store and the classification store use distinct provider namespaces (the completion usage reader versus the subprocess classifier provenance), so the report presents them as two parallel sections rather than a lossy cross-namespace join.

The report reads only accounting metadata (token counts, cost, confidence scores, provider labels); it never surfaces the persisted response text or any financial content, honouring sensitive-financial-data-secure-storage-only.

See also

build_llm_diagnostics_report():

Public aggregator that folds the usage log and ledger confidence rows.

LlmDiagnosticsPorts:

Required application-owned readers for the two metric stores.

DEFAULT_LOW_CONFIDENCE_THRESHOLD

Default confidence floor below which an LLM classification is reported as low-confidence. Operator-facing display threshold, overridable per call.

class LlmUsageCostProviderMetrics(**data)[source]

Bases: BaseModel

Per-provider aggregate of the LLM usage/cost log.

Aggregated from LlmUsageDiagnosticRecord rows for a single provider. calls counts every recorded call (cache hits included); cache_hits counts the subset served from the local cache.

Parameters:

  • provider (Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)])

  • calls (Annotated[int, Ge(ge=0)])

  • cache_hits (Annotated[int, Ge(ge=0)])

  • input_tokens (Annotated[int, Ge(ge=0)])

  • output_tokens (Annotated[int, Ge(ge=0)])

  • total_tokens (Annotated[int, Ge(ge=0)])

  • cost_estimate_usd (Decimal | None)

  • unpriced_calls (int)

provider: LlmProviderName
calls: NonNegativeInt
cache_hits: NonNegativeInt
input_tokens: NonNegativeInt
output_tokens: NonNegativeInt
total_tokens: NonNegativeInt
cost_estimate_usd: Decimal | None
unpriced_calls: int
class LlmConfidenceProviderMetrics(**data)[source]

Bases: BaseModel

Per-provider confidence distribution over LLM-classified transactions.

Aggregated from the active ledger catalogue’s transactions whose classified_by carries an llm: provenance and a non-null classification_confidence. low_confidence_count is the number of those decisions below the report’s tunable threshold. high_confidence_count (>= 0.8) and medium_confidence_count ([0.5, 0.8)) are fixed-floor distribution buckets; the remaining decisions fall below 0.5.

Parameters:

  • provider (Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=1, max_length=None, pattern=None, ascii_only=None)])

  • classified_count (Annotated[int, Ge(ge=0)])

  • low_confidence_count (Annotated[int, Ge(ge=0)])

  • high_confidence_count (Annotated[int, Ge(ge=0)])

  • medium_confidence_count (Annotated[int, Ge(ge=0)])

  • min_confidence (Decimal | None)

  • max_confidence (Decimal | None)

  • mean_confidence (Decimal | None)

provider: LlmProviderName
classified_count: NonNegativeInt
low_confidence_count: NonNegativeInt
high_confidence_count: NonNegativeInt
medium_confidence_count: NonNegativeInt
min_confidence: Decimal | None
max_confidence: Decimal | None
mean_confidence: Decimal | None
class LlmDiagnosticsReport(**data)[source]

Bases: BaseModel

Typed LLM usage / cost / confidence diagnostics report.

Produced by build_llm_diagnostics_report(). The usage section folds the encrypted usage log; the confidence section folds the classification confidence stamped on ledger transactions. has_data is False when neither store carries any LLM activity, so callers can print an instructive empty message.

Parameters:
since: date | None
until: date | None
low_confidence_threshold: Decimal
usage_providers: tuple[LlmUsageCostProviderMetrics, ...]
total_calls: int
total_cache_hits: int
total_input_tokens: int
total_output_tokens: int
total_cost_estimate_usd: Decimal | None
total_unpriced_calls: int
confidence_providers: tuple[LlmConfidenceProviderMetrics, ...]
total_classified: int
total_low_confidence: int
property has_data: bool

Return True when either metric store carried LLM activity.

build_llm_diagnostics_report(*, ports, since=None, until=None, low_confidence_threshold=Decimal('0.5'))[source]

Aggregate the existing usage and confidence metric stores into a report.

Parameters:

  • since (date | None) – Inclusive lower usage-record date bound, or None.

  • until (date | None) – Inclusive upper usage-record date bound, or None.

  • low_confidence_threshold (Decimal) – Confidence floor below which a classification counts as low-confidence.

  • ports (LlmDiagnosticsPorts) – Required application-owned readers for usage and transaction diagnostics. The executable composition root binds them to concrete encrypted stores.

Return type:

LlmDiagnosticsReport

Returns:

The populated LlmDiagnosticsReport.