cadrumo.core.directory_scan module

Canonical directory-scan primitive built on os.scandir.

Every module in the tree that needs “the files under this directory” resolves it here, rather than reaching for pathlib.Path.iterdir(), pathlib.Path.glob(), or pathlib.Path.rglob() directly.

The case for centralising is consistency, not raw speed, and it is worth being exact about that. os.scandir used to be dramatically faster than pathlib.Path.rglob(), because pathlib materialised a Path per entry and re-stat’ed it to decide whether to descend. Python 3.13 rewrote globbing onto glob._Globber, which calls os.scandir itself and reuses the directory entry rather than re-stat’ing – so that gap has largely closed. Measured on the repository’s own tree (4,985 .py files, warm cache, best of ten): a bare scandir walk 0.19s, this primitive 0.23s, sorted(Path.rglob("*.py")) 0.27s. Parity, not an order of magnitude.

What this module still buys is real, and none of it is throughput:

  • One walk with one ordering rule, instead of every caller re-deciding whether to wrap sorted() and what to sort.

  • prune_directories, which Path.rglob cannot express at all. Pruning happens BEFORE descending, so an excluded directory costs one is_dir check rather than a full subtree walk – which is what makes excluding a large data tree cheap rather than merely correct, and is where a genuine order-of-magnitude win is still available.

  • An explicit, tested answer on the three semantics that silently differ between platforms and call styles: sort order, missing directories, and symlinks.

Nothing here is memoised. A cached listing of a directory under construction is simply wrong, and production callers write to the directories they scan; the one place in the tree that can safely cache a walk (the immutable source tree during a test session) owns that cache itself.

Two return shapes exist, and the split is deliberate:

  • scan_directory() materialises and sorts, for the overwhelmingly common caller that wanted a deterministic listing and was wrapping its own sorted(...) around a glob.

  • iter_directory() stays lazy, for the caller that asks any(...) or breaks out of a loop early. Handing those a sorted tuple would walk the whole tree to answer a question the first entry settles.

The default arguments reproduce the pathlib call they replace exactly, so a mechanical migration cannot silently change what a call site sees: scan_directory(root) is root.iterdir(), scan_directory(root, pattern=p) is root.glob(p), and scan_directory(root, pattern=p, recursive=True) is root.rglob(p). Narrowing to files or directories, and pruning, are opt-in.

class DirectoryEntryKind(*values)[source]

Bases: StrEnum

Which directory entries a scan yields.

ALL is the default because it is what the pathlib calls this module replaces actually do: iterdir(), glob("*"), and even glob("*.toml") all yield directories alongside files, and a directory whose name matches the pattern is a real occurrence rather than a hypothetical one. A scan that narrowed to files by default would silently drop those entries at every migrated call site.

FILES

Entries that are not directories.

DIRECTORIES

Entries that are directories.

ALL

Every entry, matching what iterdir, glob and rglob yield.

scan_directory(root, *, pattern=None, recursive=False, select=DirectoryEntryKind.ALL, prune_directories=(), require_root=False)[source]

Return the entries under root, sorted, walked with os.scandir.

The eager shape. Ordering is sorted() over Path objects, which compares on the case-normalised string form – so on Windows B.toml sorts before a.toml, exactly as it would have under sorted(root.glob(...)). Sorting the string forms instead would be case-sensitive and reorder mixed-case trees, which the determinism gates would catch as drift.

Parameters:

  • root (Path) – Directory to scan. Results keep root’s relative or absolute form, as the pathlib calls do; nothing is resolved.

  • pattern (str | None) – Single-segment glob ("*.toml", "test_*.py", "modelo-*.json") matched against each entry’s bare name. None matches every entry. See iter_directory() for the matching semantics and the refused pattern shapes.

  • recursive (bool) – Descend into subdirectories. Descent is unconditional – pattern filters what is yielded, never what is entered, which is how rglob behaves.

  • select (DirectoryEntryKind) – Narrow to files or to directories. Defaults to DirectoryEntryKind.ALL, the pathlib-identical shape.

  • prune_directories (Collection[str]) – Bare directory names never entered and never yielded, checked before descending. This is caller policy, not a default: no name is pruned unless named here, so a scan reproduces its pathlib equivalent until the caller opts out.

  • require_root (bool) – Raise the underlying OSError when root is missing or is not a directory, instead of returning empty.

Return type:

tuple[Path, ...]

Returns:

Sorted paths of every selected entry beneath root.

Raises:

  • CoreValidationError – pattern is empty, spans path segments, or uses **.

  • OSError – root could not be scanned and require_root is set.

iter_directory(root, *, pattern=None, recursive=False, select=DirectoryEntryKind.ALL, prune_directories=(), require_root=False)[source]

Yield the entries under root lazily, walked with os.scandir.

The lazy sibling of scan_directory(), for the caller that stops early: any(iter_directory(root, pattern="*.pdf")) settles on the first match instead of walking the tree to build a listing it discards.

Ordering is a pre-order traversal, with each directory’s own entries yielded in case-normalised name order and a subdirectory’s contents yielded at the point that subdirectory appears. That is deterministic, but it is NOT the ordering scan_directory() returns: a global sort over full paths interleaves depths differently, and no lazy walk can produce it without materialising everything first. A caller that needs the sorted listing wants the eager shape.

Argument validation is eager – a refused pattern raises from this call, not from the first next() – while the walk itself is lazy. Each directory’s entries are read and its handle closed before any is yielded, so a consumer can delete a directory it is iterating without tripping a Windows sharing violation.

Missing directories yield nothing rather than raising, unless require_root is set. Path.glob and Path.rglob silently yield nothing for a missing directory while os.scandir raises FileNotFoundError, and every existing call site was written against the silent behaviour. An OSError encountered below the root – an unreadable subdirectory, a directory removed mid-walk – always skips that subtree, which is what pathlib’s own walk does; only the root’s failure is configurable.

Pattern matching goes through fnmatch.translate() over os.path.normcase()-folded text, which reproduces pathlib.Path.glob() exactly: case-insensitive on Windows, case-sensitive on POSIX, and matching leading-dot names that a shell would hide. fnmatch.fnmatchcase() would instead be case-sensitive everywhere and would quietly stop matching *.toml against a .TOML file that glob finds today.

Multi-segment patterns ("*/*.toml") and ** are refused rather than approximated. Recursion is spelled by recursive, and a pattern silently matched against bare names alone would find nothing at all for "*/*.toml" while looking like it worked.

Symlinks are classified and traversed with follow_symlinks=False, so a symlink loop cannot hang the walk – matching rglob, which stopped following directory symlinks in Python 3.13. One divergence follows and is deliberate: a symlink pointing at a directory is not a directory here, so it appears under DirectoryEntryKind.FILES and not under DirectoryEntryKind.DIRECTORIES, whereas Path.is_dir() resolves the target and would say otherwise. The rule is exact: an entry is a directory precisely when the walk would descend into it.

Parameters:

  • root (Path) – Directory to scan. Results keep root’s relative or absolute form; nothing is resolved.

  • pattern (str | None) – Single-segment glob matched against each entry’s bare name, or None to match every entry.

  • recursive (bool) – Descend into subdirectories.

  • select (DirectoryEntryKind) – Narrow to files or to directories.

  • prune_directories (Collection[str]) – Bare directory names never entered and never yielded.

  • require_root (bool) – Raise the underlying OSError when root is missing or is not a directory, instead of yielding nothing.

Yields:

Paths of every selected entry beneath root.

Raises:

  • CoreValidationError – pattern is empty, spans path segments, or uses **.

  • OSError – root could not be scanned and require_root is set.

Return type:

Iterator[Path]