cadrumo.application.corpus_search.lexical_index module

FTS5 lexical index over bundled normative HTML extractions.

The index is the ranking half of the grounding surface, alongside the separate exact-citation lookup. It is built from top-level normatives/html/*.html.extracted.json payloads, including content-aware extractions of XML responses, into a caller-supplied SQLite path. PDFs, workbooks, and terminology sources enrolled in the development RAG index are outside this runtime lexical index. No dependency beyond the standard library sqlite3 (FTS5 is present in every standard CPython build) and the core snowballstemmer (a pure-Python Spanish Snowball stemmer) is required, so this module imports and runs on every install.

Two searchable columns ride each chunk: text_folded (searched with the unicode61 remove_diacritics 2 tokenizer, so recargo matches recárgo and accents fold) and text_stemmed (the same prose pre-stemmed with the Spanish Snowball stemmer, so declaraciones matches declaración). FTS5’s built-in porter stemmer is English-only, which is why the Spanish stem rides its own precomputed column.

Exact citation lookup (“art. 27.2 LGT”) does NOT go through this index; it resolves through the signed registry authority (see citation_lookup). This index covers in-prose concept recall only within the normative HTML extraction set.

bundled_corpus_html_root()[source]

Return the on-disk path of the bundled normatives HTML corpus.

Return type:

Path

iter_corpus_chunks(corpus_root=None)[source]

Yield deterministic chunks from normative HTML extraction payloads.

The extraction directory is walked in sorted filename order, and each extracted unit is split into paragraph-bounded chunks, so the same inputs yield the same chunk sequence with the same ids.

Parameters:

corpus_root (Path | None) – Directory holding the *.html.extracted.json payloads. Defaults to the package-bundled normative HTML directory.

Yields:

One CorpusChunk per prose chunk, in document then unit then chunk order.

Return type:

Iterator[CorpusChunk]

build_lexical_index(database_path, chunks)[source]

Build the FTS5 lexical index at database_path from chunks.

The database is created fresh (any existing tables are dropped) so the build is deterministic and idempotent: the same corpus produces a byte-stable chunk id sequence and an equivalent index.

Parameters:
Return type:

CorpusIndexBuildResult

Returns:

A CorpusIndexBuildResult with the document and chunk counts.

search_lexical(database_path, query, *, limit=10)[source]

Return the top lexical-search hits for query.

The query is matched against both the diacritic-folded column (raw terms) and the Spanish-stemmed column (stemmed terms), unioned, and ranked by FTS5 BM25. This is the ranking primitive the retrieval module wraps with the exact-citation short-circuit; nothing is fused on top of it.

Parameters:

  • database_path (Path) – A lexical index built by build_lexical_index().

  • query (str) – Free-text query.

  • limit (int) – Maximum number of hits to return.

Return type:

tuple[LexicalSearchHit, ...]

Returns:

Up to limit LexicalSearchHit records, best first.

Raises:

CorpusSearchInputError – If query carries no searchable terms or limit is not positive.