cadrumo.core.tabular module

Deterministic dialect normalization for arbitrary tabular exports.

Turns a CSV, TSV or delimited-text export of unknown dialect into one typed NormalizedTable: a header row, verbatim data cells, the preamble and summary rows separated out, and the detected dialect recorded. It answers “what shape is this file”, never “what does this column mean” — role mapping is a separate step consuming this output, and no cell value is interpreted here.

The contract lives in cadrumo.core because every layer meets a delimited export and none of them owns it: the inbound financial adapter normalizes a bank statement through it, and the application’s bulk invoice import normalizes an invoice book through the same call. A per-layer copy would mean two delimiter sniffers disagreeing about the same operator file.

Six axes vary across real operator exports and are each detected rather than assumed: the delimiter (;, ,, tab, |), the decimal convention (Spanish comma or English dot), the byte encoding (including a UTF-8 BOM), metadata preamble rows before the header, appended summary rows after the data, and newlines embedded inside quoted fields.

Cell text is stored verbatim. A Spanish 1.234,56 stays 1.234,56 in NormalizedRow.cells; the detected TabularDialect.decimal_separator records how to read it, so the interpretation is carried alongside the printed form rather than replacing it. That is what lets a later step copy a cell without a lossy round trip.

Normalization never refuses a file whole. A row whose field count differs from the table’s is carried with a notice rather than dropped, an unreadable decimal convention is reported and defaulted, and only a source that cannot be decoded at all, or that yields no usable header, raises TabularSourceError. Each consuming layer translates that refusal into its own operator-facing error taxonomy.

See also

FieldRole

The closed vocabulary a normalized column’s meaning is mapped onto.

RawTransaction

The record a projected tabular row eventually becomes.

exception TabularSourceError(message=None, *, context=None, translated_message=None)

Bases: CoreValidationError

Raised when tabular bytes cannot be decoded or carry no usable table.

The three structural refusals normalization makes: no candidate codec decoded the bytes, no candidate delimiter produced a rectangle, or no row in the rectangle reads as a header. Consumers translate it into their own layer’s refusal — the financial providers into InvalidFinancialSourceError, the invoice bulk import into InvoiceValidationError.

Parameters:

• message (str | None)

• context (Mapping[str, object] | None)

• translated_message (str | None)

Return type:

None

code

CANDIDATE_DELIMITERS: tuple[str, ...]

Delimiters a delimited-text export realistically uses. Each is scored against the whole file; the one producing the most consistent rectangle wins, which is what survives a metadata preamble that a sample-window sniffer reads as the table.

TabularNoticeCode

Closed set of structural observations normalization can report.

alias of Literal[‘encoding_fallback’, ‘preamble_skipped’, ‘summary_row_excluded’, ‘ragged_row’, ‘decimal_convention_defaulted’, ‘decimal_convention_mixed’, ‘blank_row_skipped’]

class TabularNotice(**data)

Bases: BaseModel

One structural observation made while normalizing a source.

Variables:

• code – Which observation was made.

• detail – Operator-readable specifics, already carrying the values that justify the observation.

• source_line_number – Physical 1-based line the observation is about, or None for a whole-file observation.

Parameters:

data (Any)

code: TabularNoticeCode

detail: str

source_line_number: int | None

class TabularDialect(**data)

Bases: BaseModel

The detected shape of one tabular source.

Variables:

• delimiter – Field separator the rectangle was parsed with.

• quotechar – Quoting character honoured while parsing.

• encoding – Codec the bytes decoded under.

• decimal_separator – How a numeric cell in this file reads. Recorded, never applied — cells keep their printed form.

• header_line_number – Physical 1-based line the header was found on.

• column_count – Field count of the table’s rectangle.

Parameters:

data (Any)

delimiter: str

quotechar: str

encoding: str

decimal_separator: DecimalSeparatorValue

header_line_number: int

column_count: PositiveCount

class NormalizedRow(**data)

Bases: BaseModel

One source row, cells verbatim.

Variables:

• source_line_number – Physical 1-based line the row started on. A row carrying a newline inside a quoted field spans several physical lines; this is the first.

• cells – The parsed fields exactly as printed, with no stripping, separator rewriting or type coercion applied.

Parameters:

data (Any)

source_line_number: int

cells: tuple[str, ...]

class NormalizedTable(**data)

Bases: BaseModel

One tabular source resolved into a rectangle plus what surrounds it.

Variables:

• dialect – The detected dialect.

• headers – Header cells verbatim, one per column of the rectangle.

• rows – Movement rows, in source order.

• preamble – Rows above the header, kept rather than discarded so an operator can see the account and period metadata they carry.

• summary_rows – Rows below the data recognised as aggregates. Excluded from rows so a TOTAL line is never read as a movement.

• notices – Every structural observation made while normalizing.

Parameters:

data (Any)

dialect: TabularDialect

headers: tuple[str, ...]

rows: tuple[NormalizedRow, ...]

preamble: tuple[NormalizedRow, ...]

summary_rows: tuple[NormalizedRow, ...]

notices: tuple[TabularNotice, ...]

coerce_cell_text(value, *, integral_floats_as_int=False, temporal_as_iso=False)

Render one source cell as the text a reader stores or matches on.

A workbook cell arrives as a typed Python object rather than as text — openpyxl hands back float, datetime and None — so every reader of a spreadsheet needs the same value-to-text step. The base rendering is str(value).strip(), with None becoming the empty string.

Two renderings genuinely differ by consumer, and both are declared rather than assumed:

integral_floats_as_int

A whole number a spreadsheet stores as 3.0 prints as "3". A casilla value is a figure that goes on a return, so a spurious .0 is wrong there; a bank statement’s raw-field archive keeps the source’s own spelling instead.

temporal_as_iso

A date or datetime renders ISO-8601, with a datetime truncated to whole seconds. This pins the archived form of a booking stamp rather than inheriting whatever str() currently prints.

The two axes are disjoint by type, so their order cannot matter.

Return type:

str

Returns:

The cell’s text under the requested renderings.

Parameters:

• value (object)

• integral_floats_as_int (bool)

• temporal_as_iso (bool)

decode_tabular_bytes(source_bytes)

Decode source_bytes to text, reporting when the preferred codec did not fit.

A UTF-8 BOM decides the codec outright: cp1252 and iso-8859-1 decode any byte sequence at all, so a chain that reaches them first would silently mojibake a BOM-prefixed UTF-8 file rather than fail over.

Return type:

tuple[str, str, TabularNotice | None]

Returns:

The decoded text, the codec used, and a notice when the configured preferred codec was not the one that worked.

Raises:

TabularSourceError – No candidate codec decoded the bytes.

Parameters:

source_bytes (bytes)

detect_tabular_delimiter(text, *, quotechar='"')

Return the delimiter that splits text into the most consistent rectangle.

Scored over the whole file rather than a leading sample window, so a metadata preamble a sample-window sniffer would read as the table cannot decide the delimiter for the data below it.

Return type:

str | None

Returns:

The winning member of CANDIDATE_DELIMITERS, or None when no candidate produced a rectangle. A caller that must parse regardless chooses its own fallback.

Parameters:

• text (str)

• quotechar (str)

normalize_tabular_text(text, *, encoding, quotechar='"')

Resolve already-decoded tabular text into a NormalizedTable.

Parameters:

• text (str) – The decoded source.

• encoding (str) – Codec the text was decoded under, recorded on the dialect.

• quotechar (str) – Quoting character to honour.

Return type:

NormalizedTable

Returns:

The normalized table.

Raises:

TabularSourceError – The source carries no delimited rectangle, or no row in it reads as a header.

normalize_tabular_bytes(source_bytes, *, quotechar='"')

Decode and normalize raw tabular bytes into a NormalizedTable.

Raises:

TabularSourceError – The bytes cannot be decoded, carry no delimited rectangle, or carry no header row.

Return type:

NormalizedTable

Parameters:

• source_bytes (bytes)

• quotechar (str)