IBM / unitxt / 13830772563

81%

push

github

Add inference engine caching (#1645)

* Add inference engine caching

- Caching can be disabled via the `use_cache` attribute of the inference
  engine class (default: `True`).
- Caching is performed in batches, with batch size controlled by the
  `cache_batch_size` attribute (default: `100`).
- The disk cache location is configurable via the
  `inference_engine_cache_path` setting
  (default: `./inference_engine_cache/<inference engine class>`).
- Although the cache is updated after each batch, it operates at the
  granularity of a single instance.
- The cache key includes the instance message and all current inference
  engine attributes.

Signed-off-by: Elad Venezian <eladv@il.ibm.com>

* Add tests to inference engine cache.

Signed-off-by: Elad Venezian <eladv@il.ibm.com>

---------

Signed-off-by: Elad Venezian <eladv@il.ibm.com>

1554 of 1918 branches covered (81.02%)

Branch coverage included in aggregate %.

9720 of 12052 relevant lines covered (80.65%)

0.81 hits per line