IBM / unitxt / 14254422405

81%

push

github

Add retry policy for huggingface assets downloads (#1711)

* Add retry policy for huggingface assets connection

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Enhance retry mechanism to include FileNotFoundError and set max connection retries in test settings

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Nested Exceptions

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add FileNotFoundError to retry exceptions in exponential backoff decorator

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update default retry setting to use max_connection_retries from settings

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add exponential backoff retry decorator to prepare methods in GraniteDocumentsFormat and HFSystemFormat

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Remove download_config from hf_evaluate_load and hf_load_dataset; set max_connection_retries in test settings

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add unit tests for retry decorator with exponential backoff

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix loader cache size assignment and handle None dataset case in LoadHF class

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add exponential backoff retry to get_splits method and improve error handling for missing datasets

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add error handling for None dataset case in LoadHF class and improve retry logic in utils

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Re-raise ValueError in hf_load_dataset for improved error handling

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update GitHub Actions workflow to include constraints file and adjust package installations

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Remove spacy dependency from pyproject.toml

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Refactor error handling in LoadHF class to raise NotImplementedError for None dataset and im... (continued)

1579 of 1957 branches covered (80.68%)

Branch coverage included in aggregate %.

9878 of 12310 relevant lines covered (80.24%)

0.8 hits per line