decentraland / catalyst-storage / 27240860983

95%

push

github

feat: stream S3 uploads instead of buffering the whole file (#98)

* feat: stream S3 uploads instead of buffering the whole file

storeStream buffered the entire stream into memory to detect the MIME
type before uploading, so a large file was held in RAM in full during
the upload.

Peek only the first bytes (MIME detection needs ~4100) and hand the
reconstructed stream (head + remainder) to the AWS SDK managed upload,
which performs a multipart upload buffering only part-sized chunks.
Peak memory per upload drops from the full file size to a few MB.

Folder-based storage already streams, so only the S3 path is changed.

* test: enable jest ESM so MIME-type detection is covered

file-type v21 is ESM-only and is loaded via a native dynamic import, so
the MIME detection in storeStream silently fell back to octet-stream
under jest (the import throws without --experimental-vm-modules) and was
never exercised by the suite.

Run jest with NODE_OPTIONS=--experimental-vm-modules (via cross-env) so the
real detection runs in tests, and add S3 integration tests asserting that
PNG/JPEG/GLB payloads larger than the detection window are uploaded with
the correct ContentType, while text-based glTF falls back to
application/octet-stream. The suite already runs in-band
(--detectOpenHandles), so the flag applies deterministically.

* refactor: address review feedback on S3 streaming

- Clarify peekHead JSDoc: it reads at least byteCount bytes (chunk-granular,
  may read up to a whole chunk more), not exactly byteCount.
- Release the source stream when an upload fails: peekHead's generator now
  returns the underlying iterator on finalization, and storeStream destroys
  the source on upload error (covering the case where the upload fails before
  consuming the body, so the generator never started). Adds a test asserting
  the source stream is released on upload failure.

101 of 114 branches covered (88.6%)

Branch coverage included in aggregate %.

30 of 30 new or added lines in 1 file covered. (100.0%)

359 of 366 relevant lines covered (98.09%)

123.02 hits per line