cadrumo.core.atomic_write module

Shared two-tier atomic-write helper.

Every durable on-disk write in this codebase must never leave a torn or partially-written file behind, whether the process crashes, is killed, or raises mid-write. Before this module, four independent dialects of the same “write a sibling tempfile, then os.replace() it over the target” pattern had accreted across the storage substrate: a standard fsync+replace variant (adapters.persistence.storage.envelope), a hidden-file variant with no fsync, plain-write variants with no fsync at all, and a collision-hardened O_EXCL + mode 0o600 variant reserved for the master-key store. This module collapses all of that onto four named tiers so a new writer picks one deliberately instead of inventing a fifth dialect:

  • Standard tier (atomic_write_bytes(), atomic_write_text()): a tempfile.NamedTemporaryFile() sibling in the target’s own parent directory ({stem}. prefix, .tmp suffix), write, flush, fsync, os.replace(), then a best-effort parent-directory fsync via fsync_parent_dir(). Suitable for ordinary durable application data with a single writer.

  • Hardened tier (atomic_write_hardened_bytes(), atomic_write_hardened_text()): the master-key store’s pattern – O_EXCL plus O_NOINHERIT/O_CLOEXEC where the platform defines them, file mode 0o600, and a collision-hardened {name}.{pid}.{token_hex}.tmp temp name – before the same fsync/replace/parent-fsync sequence. Use this tier for secret-bearing targets or any target more than one process/thread could plausibly write concurrently; the O_EXCL open refuses to silently reuse or truncate an unexpected pre-existing tempfile. A memoryview-and-offset loop completes short os.write() calls and refuses a write that makes no progress.

  • Deferred-publish tier (hardened_staged_publication()): the hardened tier’s staging and publication with the write itself left to the caller, for a producer that cannot hand over a bytes payload – it needs a real file on disk to build incrementally or to read back before the result becomes operator-visible, or it must complete unrelated work between the write and the publication. The staging sibling carries the same unguessable {name}.{pid}.{token_hex}.tmp name, the same O_EXCL reservation and the same 0o600 mode; publication is the same fsync/replace/parent-fsync sequence, requested explicitly so the caller can translate a publication failure into its own domain error. It exists because callers needing that shape were open-coding a predictable {name}.tmp sibling and a bare os.replace() next to the operator’s chosen destination, which is a guessable name, an unsynced publication, and – with a narrow except – a staged file that survives an interrupt.

  • Best-effort tier (atomic_write_best_effort_bytes(), atomic_write_best_effort_text()): the same tempfile-sibling-plus- os.replace() mechanics as the standard tier, but deliberately WITHOUT any fsync call. Reserved for rebuildable derived caches (a compiled- registry pickle, a corpus-text extraction cache, a validation-verdict stamp) where a torn write on crash only costs a recompute on the next process, never data loss – the fsync cost is not worth paying on a cache write. This tier never logs and never swallows: it raises the underlying exception unwrapped, same as the other two tiers, and leaves the try/except/log/swallow policy entirely to the caller, because every existing best-effort caller already wraps its write in its own catch with its own log level and message and duplicating that here would double-log or contradict it.

Every tier guarantees the tempfile is unlinked on ANY failure – a bare try/finally around the whole sequence, not a narrow except OSError – so a KeyboardInterrupt or any other BaseException mid-write cannot leave an orphan tempfile next to the target. No tier wraps or translates the underlying exception: callers see the raw OSError (or whatever the platform raises) so each call site can apply its own domain-specific error class, matching the existing call-site-owns-its-error-type convention this module’s callers already use. Payload content is never logged; only the target path and the exception type are (the best-effort tier logs nothing at all; see above).

The helpers are freestanding primitives at the core layer with no dependency beyond fsync_parent_dir().

atomic_write_bytes(path, data)[source]

Atomically write data to path (standard tier).

Stages a tempfile.NamedTemporaryFile() sibling in path’s parent directory (created if absent), writes and fsyncs it, then replaces path with os.replace() and best-effort fsyncs the parent directory. The tempfile is unlinked on any failure, including a BaseException raised mid-write.

Parameters:

  • path (Path) – Destination file. Parent directory is created if absent.

  • data (bytes) – Full file contents to write.

Raises:

OSError – When staging, writing, or replacing the file fails. The original exception propagates unwrapped; the tempfile is cleaned up first.

Return type:

None

atomic_write_text(path, text, *, encoding='utf-8')[source]

Atomically write text to path (standard tier, text variant).

Encodes text and delegates to atomic_write_bytes(); see its docstring for the write sequence and failure semantics.

Parameters:

  • path (Path) – Destination file. Parent directory is created if absent.

  • text (str) – Full file contents to write.

  • encoding (str) – Text encoding used to produce the bytes payload.

Return type:

None

atomic_write_best_effort_bytes(path, data)[source]

Atomically write data to path (best-effort tier, no fsync).

Stages a tempfile.NamedTemporaryFile() sibling in path’s parent directory (created if absent), writes it, then replaces path with os.replace(). Unlike the standard and hardened tiers, this tier never calls fsync on the tempfile or the parent directory: it exists for rebuildable derived caches where a torn write on crash only costs a recompute, not data loss, so the fsync cost is not worth paying on the write. The tempfile is unlinked on any failure, including a BaseException raised mid-write. This tier never logs; the caller owns its own catch/log/swallow policy (see the module docstring).

Parameters:

  • path (Path) – Destination file. Parent directory is created if absent.

  • data (bytes) – Full file contents to write.

Raises:

OSError – When staging, writing, or replacing the file fails. The original exception propagates unwrapped; the tempfile is cleaned up first.

Return type:

None

atomic_write_best_effort_text(path, text, *, encoding='utf-8')[source]

Atomically write text to path (best-effort tier, text variant).

Encodes text and delegates to atomic_write_best_effort_bytes(); see its docstring for the write sequence and failure semantics.

Parameters:

  • path (Path) – Destination file. Parent directory is created if absent.

  • text (str) – Full file contents to write.

  • encoding (str) – Text encoding used to produce the bytes payload.

Return type:

None

class DurableWriteBatch[source]

Bases: object

Defer the per-file durability sync across many writes, flushing once.

A hardened write costs three syncs — the staged fd, then the parent directory after os.replace(). Measured on this project’s target platform that is ~3.5 ms/file against ~0.6 ms for the same write without them, so a bulk ingest of 20,000 evidence records spends about a minute on durability alone. This batch collapses that to one directory sync per touched directory at commit.

Only for content-addressed or re-derivable payloads. The trade is real: a crash mid-batch can leave a file whose name is present but whose bytes never reached the platter. That is recoverable exactly when the reader can detect it and the caller can reproduce it — a blob named by the SHA-256 of its own contents fails its digest check on read and its source document is still on disk to re-import. It is NOT acceptable for key material, pointers, or session records: a torn key is neither detectable nor re-derivable, and those writers must keep the unbatched path.

Explicit by construction. The batch is passed as an argument rather than bound to ambient context, because a durability policy that travels invisibly would silently weaken whichever unrelated write happened to run inside the scope.

note(path)[source]

Register path as the representative for its parent directory.

Return type:

None

Parameters:

path (Path)

commit()[source]

Sync every directory touched by the batch, then forget them.

Idempotent: the set is drained as it is synced, so an explicit commit() followed by the context manager’s own commit on exit does not sync twice.

Return type:

None

durable_write_batch()[source]

Yield a DurableWriteBatch and commit it on exit.

Commits from a finally, so an exception mid-batch still syncs whatever already landed rather than leaving the completed writes less durable than an unbatched run would have.

Return type:

Generator[DurableWriteBatch]

atomic_write_hardened_bytes(path, data, *, mode=384, batch=None)[source]

Atomically write data to path (hardened tier).

Opens a collision-hardened {name}.{pid}.{token_hex}.tmp sibling with O_EXCL (refusing to reuse or truncate an unexpected pre-existing tempfile) plus O_NOINHERIT/O_CLOEXEC where the platform defines them, at file mode mode. A memoryview-and-offset loop completes short os.write() calls and treats a non-positive write as an error. The staged bytes are fsynced before os.replace(), followed by a best-effort parent-directory fsync. The tempfile is unlinked on any failure, including a BaseException raised mid-write. Use this tier for secret-bearing targets or targets more than one process could plausibly write concurrently.

Parameters:

  • path (Path) – Destination file. Parent directory is created if absent.

  • data (bytes) – Full file contents to write.

  • mode (int) – POSIX file mode for the staged tempfile (and, transitively, the replaced target). Defaults to 0o600.

  • batch (DurableWriteBatch | None) – Optional DurableWriteBatch. When supplied, the write stays atomic but its two fsyncs are deferred to the batch commit. Pass one ONLY for content-addressed or otherwise re-derivable payloads; see DurableWriteBatch for why key material, pointers and session records must not use it.

Raises:

OSError – When staging, writing, or replacing the file fails (including FileExistsError from an O_EXCL collision). The original exception propagates unwrapped; the tempfile is cleaned up first.

Return type:

None

atomic_write_hardened_text(path, text, *, encoding='utf-8', mode=384)[source]

Atomically write text to path (hardened tier, text variant).

Encodes text and delegates to atomic_write_hardened_bytes(); see its docstring for the write sequence and failure semantics.

Parameters:

  • path (Path) – Destination file. Parent directory is created if absent.

  • text (str) – Full file contents to write.

  • encoding (str) – Text encoding used to produce the bytes payload.

  • mode (int) – POSIX file mode for the staged tempfile. Defaults to 0o600.

Return type:

None

class StagedPublication(*, staging_path, target_path)[source]

Bases: object

A reserved hardened staging file awaiting an explicit publication.

Yielded by hardened_staged_publication(). The caller writes the payload at path by whatever means its producer requires, then calls publish() to move those bytes atomically onto the target. Leaving the context without publishing discards the staged file, so an interrupt, a refusal raised between the write and the publication, or an early return can never leave the payload stranded next to the target under a name the caller never told anyone about.

Publication is a method rather than an automatic action on clean exit because the callers that need this tier write sensitive artefacts and must translate a publication OSError into their own typed refusal. A context manager that published on exit would raise from the with statement itself, where the caller can no longer distinguish a publication failure from a failure in its own body.

Parameters:

  • staging_path (Path)

  • target_path (Path)

property path: Path

The reserved staging file the caller writes its payload to.

property target_path: Path

The destination publish() moves the staged bytes onto.

property published: bool

Whether publish() has already moved the staged bytes into place.

publish()[source]

Atomically move the staged bytes onto the target and sync its directory.

Raises:

  • OSError – When the replace or the directory sync fails. The staged file is left in place for the enclosing context manager to discard, so a failed publication never strands the payload.

  • RuntimeError – When called a second time. One staging file publishes once; a second call would replace the target with a path the first call already consumed.

Return type:

None

hardened_staged_publication(target_path, *, mode=384)[source]

Reserve a hardened staging sibling of target_path for a caller-driven write.

The deferred-publish tier. Reserves an unguessable {name}.{pid}.{token_hex}.tmp sibling with the hardened tier’s O_EXCL open at file mode mode, yields it, and discards it on any exit that did not publish – including a BaseException such as the operator interrupting the work, which is precisely the case a narrow except OSError around a hand-rolled staging file misses.

Use this tier when the payload cannot be handed over as bytes: the producer builds the file incrementally, reads it back to verify it before it becomes operator-visible, or must complete unrelated work between the write and the publication. When the payload is in hand, use atomic_write_hardened_bytes() instead – it is the same guarantees in one call.

The reservation matters as much as the name. Opening with O_EXCL refuses to adopt anything already sitting at the staging path, so the name handed to the caller is one this call claimed rather than one it inherited. A caller that writes in place keeps that inode; a caller that stages through its own replace supersedes it, and in both cases the name was never available to anyone else in between.

Parameters:

  • target_path (Path) – Destination the staged bytes are published onto. Its parent directory is created if absent; the target itself is not touched until StagedPublication.publish() runs.

  • mode (int) – POSIX file mode for the reserved staging file (and, transitively, the published target). Defaults to 0o600.

Yields:

A StagedPublication bound to the reserved staging file.

Raises:

OSError – When the parent directory or the staging reservation cannot be created (including FileExistsError from an O_EXCL collision).

Return type:

Generator[StagedPublication]