aeat.application.corpus_search._lexical_index module

Build-time FTS5 lexical index over the bundled BOE/AEAT corpus.

The index is the lexical half of the R3 hybrid grounding surface. It is built from the already-bundled *.extracted.json corpus triples (the same triples the registry legal catalogue grounds against, read through the resources boundary) into a caller-supplied SQLite path. No dependency beyond the standard library sqlite3 (FTS5 is present in every standard CPython build) and snowballstemmer (a pure-Python Spanish Snowball stemmer) is required, so this module is importable in the degraded, no-download mode.

Two searchable columns ride each chunk: text_folded (searched with the unicode61 remove_diacritics 2 tokenizer, so recargo matches recárgo and accents fold) and text_stemmed (the same prose pre-stemmed with the Spanish Snowball stemmer, so declaraciones matches declaración). FTS5’s built-in porter stemmer is English-only, which is why the Spanish stem rides its own precomputed column.

Exact citation lookup (“art. 27.2 LGT”) does NOT go through this index; it is a structured key lookup over the registry legal catalogue (see _citation_lookup). This index covers in-prose concept recall.

bundled_corpus_html_root()[source]

Return the on-disk path of the bundled normatives HTML corpus.

Return type:

Path

iter_corpus_chunks(corpus_root=None)[source]

Yield deterministic CorpusChunk records from the corpus.

The corpus is walked in sorted filename order, and each extracted unit is split into paragraph-bounded chunks, so the same corpus always yields the same chunk sequence with the same ids.

Parameters:

corpus_root (Path | None) – Directory holding the *.html.extracted.json triples. Defaults to the package-bundled corpus.

Yields:

One CorpusChunk per prose chunk, in document then unit then chunk order.

Return type:

Iterator[CorpusChunk]

build_lexical_index(database_path, chunks)[source]

Build the FTS5 lexical index at database_path from chunks.

The database is created fresh (any existing tables are dropped) so the build is deterministic and idempotent: the same corpus produces a byte-stable chunk id sequence and an equivalent index.

Parameters:
Return type:

CorpusIndexBuildResult

Returns:

A CorpusIndexBuildResult with the document and chunk counts.

search_lexical(database_path, query, *, limit=10)[source]

Return the top lexical-search hits for query.

The query is matched against both the diacritic-folded column (raw terms) and the Spanish-stemmed column (stemmed terms), unioned, and ranked by FTS5 BM25. This is the lexical-recall primitive; the R3 hybrid fusion (RRF with the semantic side) is layered on top in the retrieval module.

Parameters:

  • database_path (Path) – A lexical index built by build_lexical_index().

  • query (str) – Free-text query.

  • limit (int) – Maximum number of hits to return.

Return type:

tuple[LexicalSearchHit, ...]

Returns:

Up to limit LexicalSearchHit records, best first.

Raises:

CorpusSearchInputError – If query carries no searchable terms or limit is not positive.