decentraland / catalyst-storage / 30272322309

push

github

fix!: refuse partially-read sources, stop treating directories as content, bound the decompress cache (#120)

* fix!: refuse partially-read sources, stop treating directories as content, bound the decompress cache

Findings from a read-only audit of the whole package, each reproduced against the
compiled library before being fixed and pinned by a test.

Silent corruption

- assertStorableStream only rejected a FULLY consumed source. The four flags it
  checked describe a finished stream and all flip a tick after the read that made
  the source unusable, so a caller that hashed, sniffed or measured a body first
  walked through it and the store RESOLVED three different ways: an fs.ReadStream
  after read(4) stored the body minus its first four bytes, a push-based source
  after one iterator.next() stored zero bytes, and a Readable.from whose iterator
  was left suspended never settled at all. Now also rejects readableDidRead, which
  is false for a fresh source and for a live-but-empty one. Exported so a caller
  that must inspect a body can check before storing.

Phantom content paths

- statForRead and existsForInvariant treated "something is here" as "content is
  here". Nested ids are legal, so storeStream('a/b') creates the directory `a` and
  'a.gzip/b' creates a's compressed path: the never-stored id `a` reported exist
  true, a fileInfo sized from the directory, an asStream() that died EISDIR, and a
  delete that rejected FOREVER, poisoning every GC batch containing it. Both now
  require isFile(), which is what allFileIds already assumed.

Resource lifetime

- A store rejecting on id validation never destroyed the caller's stream, leaking
  one descriptor (or one undrained request socket) per rejected call until EMFILE:
  30 per 30 rejected calls, on every backend. runStoreWithSignal now releases it.
- The unencoded asStream()/asRawStream() returned an unlistened source, so the
  documented read/delete race emitted an 'error' nobody handled an... (continued)

751 of 843 branches covered (89.09%)

Branch coverage included in aggregate %.

285 of 296 new or added lines in 11 files covered. (96.28%)

1334 of 1370 relevant lines covered (97.37%)

1277.02 hits per line