cadrumo.core.text_fold module¶
Canonical accent-folding primitive for diacritic-insensitive text matching.
Nine call sites across core, application, domain, and adapters
independently hand-rolled the same NFKD-decompose-and-strip-combining-marks
routine (label matching, denylist matching, dedup fingerprinting, header
aliasing, terminology search, corpus-text normalisation). This module gives
the shared inner step one canonical home; each caller keeps composing its own
trailing transform (whitespace collapse, non-alphanumeric squeeze, HTML-tag
strip, casing) on top of it.
Where two callers compose the SAME trailing transform, that composition earns a
name here rather than being written twice: fold_for_matching() is the
fold-collapse-casefold pipeline the terminology search and the AEAT marker
matcher had each written out.
- COMBINING_MARK_UNIDATA_VERSION¶
Unicode database version
COMBINING_MARK_RANGESwas derived from.
- COMBINING_MARK_RANGES: tuple[tuple[int, int], ...]¶
Inclusive codepoint ranges of Unicode category
Mn(Mark, nonspacing) as ofCOMBINING_MARK_UNIDATA_VERSION. Checked in because deriving them at import scans all 0x110000 codepoints throughunicodedata.category, which dominated the import of every process that folds text. The text-fold tests re-derive the ranges from the live database and print the literal to paste when it moves.
- combining_mark_ranges_from_unicodedata()[source]¶
Derive the inclusive
Mncodepoint ranges from the running interpreter.
- fold_diacritics(text)[source]¶
Return text with combining diacritical marks removed.
NFKD-decomposes text, then drops every character in Unicode category
Mn(Mark, nonspacing) — the combining accents NFKD decomposition exposes ("ó"->"o"+ U+0301 COMBINING ACUTE ACCENT, which this function then drops). Every other codepoint passes through unchanged, including one with no ASCII-compatible decomposition (an em dash, the euro sign,"ø"): this function folds ACCENTS, it does not transliterate to ASCII. A caller that also needs to discard such codepoints composes its ownstr.encode("ascii", "ignore")pass afterward rather than relying on this function to do it.Case is left untouched. Callers that need case-insensitive matching compose this with their own
str.casefold()(preferred overstr.lower()for caseless comparison) call, since the diacritic fold and the case fold are orthogonal concerns.
- fold_printed_phrase(printed)[source]¶
Return the form a printed multi-word phrase is matched under.
Three transforms and no more, in this order: fold case, fold accents, collapse every run of whitespace to one space and trim the ends. Each earns its place against how a phrase arrives from a document rather than from a keyboard – typography is the issuer’s choice, a phrase set as a line wraps with the break inside it, and a text layer or an OCR pass routinely returns the ASCII spelling of an accented word.
Punctuation is deliberately NOT stripped, which is the transform a fourth one would be. A vocabulary can carry a name whose stops are part of it, and squeezing punctuation generally starts matching strings that are not phrases at all.
Case folds BEFORE accents, and the order is load-bearing rather than incidental: a caller that folds accents first and cases afterward is a different function, because
str.casefold()can itself emit a combining mark that the earlier accent fold is no longer there to remove.
- unicode_compose(text)[source]¶
Return text NFKC-composed, every diacritic and printed character kept.
A DIFFERENT normalization form from
fold_diacritics(), for a different job: two documents printing the same figure can differ in whether a composed character ("ó") arrived pre-composed or as a base letter plus a combining accent, with no difference in what was printed. NFKC folds that representational gap closed without touching accents, case, digits, or punctuation — unlikefold_diacritics(), this must never be used where the accent itself is evidence to preserve.
- fold_for_matching(text)[source]¶
Fold text to the form two matchers compare against.
Drops diacritics, collapses every run of whitespace to one space, trims the ends, and casefolds. This is the exact pipeline the terminology search and the AEAT response-marker matcher had each spelled out.
The casefold runs last, which is equivalent to casefolding first: neither
str.casefoldnor the whitespace collapse affects what the other does. That equivalence is what let the two callers, which ordered the steps differently, converge on one function.
- ascii_slug(text)[source]¶
Return the lowercase ASCII slug of text, or
""when it has none.Folds accents, discards every remaining non-ASCII codepoint, casefolds, then collapses each run of non-slug characters to a single hyphen and trims the hyphens from both ends.
Returning the empty string rather than raising is deliberate: the callers that need a slug disagree about what an absent one MEANS – one is parsing an HTML source and one is compiling a registry declaration, and each owes its own typed error – so the shared code computes and the caller judges.