aeat.application.corpus_search._embed_build module

Build-time corpus embedding precompute (the semantic half of R3).

The corpus is static and bundled, so its embeddings are precomputed ONCE at build time with model2vec (potion-multilingual-128M — MIT code and MIT weights) and shipped as a plain float32 numpy matrix plus a parallel chunk-id list. Model outputs are shippable per the licence-clean rule; the query model is needed ONLY to embed a live query, so a build that ships the corpus vectors carries a degraded, no-download semantic mode: the “more-like-this-document” primitive here runs over the shipped matrix with numpy alone and never touches the model.

model2vec rides the capability-gated aeat-cli[search] extra. When it is absent this module still imports (the degraded lexical-only mode stays live) and embed_corpus() refuses with an install hint rather than crashing, mirroring every other optional-integration boundary in the tree. numpy is imported function-locally for the same reason, so the lexical-only surface never depends on it at import time.

See also

ensure_corpus_embeddings()

Runtime build-once cache that calls this precompute behind the aeat-cli[search] extra.

QueryEmbedder

Live-query half that embeds operator text into the same vector space.

hybrid_search()

Retrieval fusion consumer of the precomputed matrix.

embed_corpus(chunks, *, matrix_path, chunk_ids_path, model_id='minishlab/potion-multilingual-128M', revision='73908c3438cf03b6a01bcb9611d62b23d0726f08', cache_dir=None)[source]

Precompute and persist corpus embeddings for chunks.

Parameters:

  • chunks (Iterable[CorpusChunk]) – The chunk sequence to embed, typically iter_corpus_chunks(). Order is preserved so the matrix stays row-aligned with the persisted chunk-id list.

  • matrix_path (Path) – Destination .npy file for the float32 matrix.

  • chunk_ids_path (Path) – Destination .json file for the parallel chunk-id list.

  • model_id (str) – The model2vec model to load.

  • revision (str) – The pinned model revision, recorded for provenance.

  • cache_dir (Path | None) – Optional app-controlled model cache directory.

Return type:

CorpusEmbeddingBuildResult

Returns:

A CorpusEmbeddingBuildResult describing the written artifacts.

Raises:

CorpusSearchDependencyError – If the search extra (model2vec) is not installed.

load_embeddings(matrix_path, chunk_ids_path)[source]

Load a precomputed matrix and its parallel chunk-id list.

Parameters:

  • matrix_path (Path) – A .npy matrix written by embed_corpus().

  • chunk_ids_path (Path) – The parallel .json chunk-id list.

Return type:

tuple[ndarray, tuple[str, ...]]

Returns:

The loaded matrix and the chunk-id tuple.

more_like_this(matrix, chunk_ids, query_chunk_id, *, top_k=5)[source]

Return the cosine-nearest chunks to query_chunk_id.

Runs over the precomputed matrix with numpy alone — no model, so this is the degraded, no-download semantic primitive. The query chunk itself is excluded from its own results.

Parameters:

  • matrix (ndarray) – The precomputed embedding matrix, one row per chunk id.

  • chunk_ids (Sequence[str]) – The chunk ids aligned with matrix rows.

  • query_chunk_id (str) – The chunk to find neighbours for.

  • top_k (int) – Maximum number of neighbours to return.

Return type:

tuple[SimilarChunk, ...]

Returns:

Up to top_k SimilarChunk records, most similar first.

Raises:

CorpusSearchInputError – If query_chunk_id is not in chunk_ids, top_k is not positive, or the matrix and id list disagree in length.