bedrock-kv / bedrock / 328efbb5eb46b499a90d7a22c413595246d91647

82%

push

github

Publish local objects atomically, never claim a key with a partial one (#215)

Closes bedrock-ghd.

Readers of the object store are written against S3's contract: an object
is complete or absent, and `put_if_not_exists` claims a key only by
publishing a whole object. The local backend did not honour that —
unlike the S3 backend, which uses one `put_object` with `if_none_match:
"*"`.

`put_if_not_exists` opened the key with `:exclusive` and wrote to it as
a **second step**, so between the two the key existed at zero length. A
crash there claimed the key permanently: every later attempt gets
`:eexist`, and the data-plane callers (`Demux.ShardServer`,
`ChunkWriter`, `Snapshot`) read `{:error, :already_exists}` as success —
so a short object would report success to everyone forever and could
never be rewritten. `olivine/logic.ex` already warned that
put-if-not-exists *"makes the poisoned bundle permanent"*; the backend
could manufacture one on its own.

### Approach
Write to a scratch file in the target's own directory, fsync, publish in
one step — `rename` for `put/4`, `link` for `put_if_not_exists/4`
(rename clobbers, so it cannot express create-only). Failures remove the
scratch and leave the key unclaimed, and write errors are returned
rather than raised — the old `:ok = :file.write(fd, data)` turned ENOSPC
into a MatchError with a poisoned key left behind.

### Two things the audit caught that I'd got wrong
- **The scratch name must be kernel-arbitrated.**
`:erlang.unique_integer/1` is unique only *within* a node, and the
default root is derived from the system tmp dir with nothing
node-specific in it. Two nodes sharing a root could pick the same
scratch name, interleave into one inode, and publish the splice — the
exact failure this change exists to prevent. The auditor reproduced it:
878 spurious `{:error, :not_found}` results from a *PUT* in one paired
run, which would `CaseClauseError` in `Discovery.write_bootstrap/4`.
Scratch files are now creat... (continued)

30 of 39 new or added lines in 2 files covered. (76.92%)

6827 of 8683 relevant lines covered (78.62%)

1028.99 hits per line