archetech / archon / 23326062416

95%

push

github

feat: stream large file uploads/downloads for multi-GB video support (#238)

* feat: stream large file uploads and downloads without memory buffering

Replaces the buffered file upload/download pipeline with an end-to-end
streaming approach that supports multi-GB video files without OOM:

- IPFS clients (Kubo + Helia): add addDataStream/getDataStream
- GatekeeperInterface: expose addDataStream/getDataStream
- Gatekeeper API: POST /ipfs/stream and GET /ipfs/stream/:cid endpoints
  with no body buffering, streaming response, client_max_body_size 0
- GatekeeperClient: addDataStream via axios (Node.js server-side)
- Keymaster: generateFileAssetFromStream, createFileStream, updateFileStream
- Keymaster API: POST /files and PUT /files/:id pipe req stream directly
  to IPFS (removes express.raw() 10MB limit)
- Browser UI (react-wallet + browser-extension): fetch with File as body
  streams upload; download uses direct gatekeeper URL (no JS memory)
- File input now accepts video/* in addition to pdf/doc/txt

Closes #208

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: count bytes during streaming when not provided in options

When the Python SDK (or CLI) calls POST /files without X-Options bytes,
generateFileAssetFromStream now wraps the stream in a counting generator
so the FileAsset records the correct byte count.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: proxy streaming uploads/downloads through drawbridge without buffering

The global express.raw() middleware in drawbridge was buffering the entire
request body before routing, causing PayloadTooLargeError for large files.

- Skip body-parsing for POST /api/v1/ipfs/stream (streams req directly)
- Add POST /ipfs/stream and GET /ipfs/stream/:cid routes to drawbridge
  v1router, proxying to gatekeeper via addDataStream/getDataStream
- Implement GatekeeperClient.getDataStream using axios responseType:stream
- Add /ipfs/stream/:cid to normalizePath for Prometheus metric... (continued)

1775 of 1963 branches covered (90.42%)

Branch coverage included in aggregate %.

32 of 42 new or added lines in 5 files covered. (76.19%)

3783 of 3954 relevant lines covered (95.68%)

1167.09 hits per line