openvax / datacache / 27778178865

79%

push

github

Fix #49: stream downloads to disk + optional progress_callback (#53)

Previously _download read the entire HTTP/FTP response into memory
(response.content / response.read()) before writing it to the temp file,
so downloading a large reference file (e.g. a ~200 MB HPA archive) spiked
RAM by the full file size and gave callers no way to show progress.

Replace _download with _stream_to_file, which streams the response to an
open file handle one chunk at a time (requests stream=True + iter_content
for http, chunked response.read for ftp/file), capping memory at one
chunk. Add an optional progress_callback(bytes_downloaded, total_bytes)
hook and chunk_size, threaded through _download_to_temp_file,
_download_and_decompress_if_necessary and fetch_file, so consumers can
drive a tqdm bar without datacache depending on tqdm.

Adds network-free regression tests using local file:// URLs.

Claude-Session: https://claude.ai/code/session_011bzfZPTzWnhAMVD7msyMg1

20 of 31 new or added lines in 1 file covered. (64.52%)

330 of 450 relevant lines covered (73.33%)

2.2 hits per line