Skip to content

0056 - Preserve policy-fail rejection across batch resume

  • Status: accepted
  • Date: 2026-07-26
  • Deciders: bioedca
  • PRD anchor: §6, §7.11 (FR-BATCH), §11.2 (over-gate batch policy)
  • Milestone: M9

Context and problem statement

ADR-0030 made provenance-group presence the batch checkpoint: a completed extraction was skipped on resume. Extraction writes its project before the registration gate is applied, however, so a complete but over-gate project failed only on its first policy=fail run. A second run accepted the presence-only checkpoint and allowed correction and idealization to proceed.

How should a completed extraction checkpoint retain its inspectability and remain the single source of truth without losing the fail-policy verdict on resume?

Decision drivers

  • policy=fail must fail the same saved over-gate extraction on every resume.
  • The completed project must remain available for inspection.
  • An explicit recovery path must re-extract and apply the gate to the new result.
  • A stale lock stolen during a long re-extraction must prevent the old owner publishing.
  • Accepted completed checkpoints must keep the normal skip behavior.
  • No second checkpoint file or HDF5 schema addition may be introduced.

Considered options

  1. Keep accepting every completed extraction checkpoint. This preserves ADR-0030 literally but makes policy=fail a first-run-only gate.
  2. Delete or invalidate the project after an over-gate extraction. This preserves failure on resume but discards the inspectable result and weakens atomic extraction.
  3. Persist a second batch verdict. This duplicates state already present in the extraction profile and can drift from it.
  4. Re-evaluate the saved extraction profile when fail policy is active (chosen).

Decision outcome

Chosen option: "Re-evaluate the saved extraction profile when fail policy is active."

/settings/extraction still proves that extraction completed. When policy=fail, the batch runner also reads registration_rms_px and rms_gate from that group's saved profile_json. A finite residual greater than a finite positive saved gate marks the completed checkpoint as policy-rejected:

  • without overwrite, extraction is recorded failed and downstream stages remain blocked;
  • with overwrite, extraction runs again and the fail policy is applied to the new summary;
  • under policy=warn, or when the saved result is not over its gate, the completed checkpoint remains skipped even if overwrite is set.

The overwrite path atomically claims an unowned destination lock before testing whether the HDF5 file exists or running fallible checkpoint probes. This refuses a writer that owns the sidecar before creating the project and closes a file-appearance race. The sidecar claim is paired with a process-local token keyed by the destination's normalized absolute lexical sidecar path; replacing a symlink leaf therefore cannot change the reservation key. The token belongs to the claiming PID and thread. Ordinary recursive acquisition, a leaked token on another thread, and competing same-process Project writers fail fast; unrelated destinations do not share a reservation. The runner then revalidates both ownership tokens around classification and the stage's repeated checkpoint reads. It skips an accepted completed checkpoint under ownership; otherwise it keeps the same lock through extraction, downstream correction and idealization, and the final /settings/batch stamp. extract_movie builds the replacement at a sibling temp path and, immediately before os.replace, calls a publish guard that compares the on-disk lock nonce with this run's acquisition nonce. The batch runner performs the same nonce check before and after every later canonical writer. The correction orchestrator carries the guard into each writer. Photobleach revalidates its mutation boundaries; guarded leakage, gamma, and corrected-FRET stage their complete dependent /molecules update plus /settings/* stamp inside a same-directory sibling project. They publish that pair only through a final nonce check and atomic os.replace, so lock loss cannot leave canonical data without its provenance. Unguarded callers retain their in-place behavior. Guarded idealization likewise builds the complete model in a sibling project before final publication. The final /settings/batch provenance writer also revalidates before opening the canonical project and immediately before replacing its group. Any pre-existing lock fails closed, including one with this process's identity, unless a future API explicitly transfers that writer's critical section. Batch neither refreshes nor releases the caller's lock. A release error for a batch-acquired lock is reported against that movie without aborting the queue. A nonce-checked False release result emits the same warning and preserves the successor. A nested finally clears the process reservation even when sidecar release returns False or raises. If an operator steals the now-stale lock during a long stage, the old run stops that job, suppresses subsequent writes, and never releases the successor's lock. The strict claim uses exclusive create on every bounded retry, even if a losing writer disappears, and it never replaces an intervening successor or honors a steal request.

Missing or malformed legacy profile values do not invent a rejection; those completed checkpoints retain ADR-0030's presence-only skip behavior. The store remains the only checkpoint and the HDF5 schema is unchanged.

This decision supersedes only ADR-0030's unconditional acceptance of a completed extraction checkpoint. Its per-movie isolation, correction and idealization presence checks, sequencing, and sidecar boundary remain accepted.

Consequences

  • Good. A saved over-gate extraction cannot silently pass on the second run.
  • Good. The rejected project remains inspectable and --overwrite --policy fail provides a bounded recovery path.
  • Good. A long-running overwrite cannot publish after its destination lock is stolen.
  • Good. Checkpoint classification cannot collapse a writer-owned HDF5 open failure into an unlocked destructive overwrite, and same-process identity cannot bypass critical-section ownership.
  • Good. A batch reservation isolates only its canonical destination, rejects untransferred recursive/thread access, and cannot survive fork or final cleanup.
  • Good. Accepted completed checkpoints still resume without redundant work.
  • Trade-off. Fail-policy resume depends on the saved extraction profile. Legacy or malformed profiles keep the safe compatibility behavior of skipping rather than guessing a rejection.
  • Enforcement. Batch unit tests cover fail/warn resume, overwrite re-gating, and accepted checkpoint skipping; a CLI end-to-end regression exercises two fail runs over the same on-disk project.

More information

  • ADR-0030 defines the original provenance-derived checkpoint architecture.
  • ADR-0014 defines the registration RMS gate and warn/fail policy.
  • Issue #211 captures the accepted resume behavior and recovery criteria.