1
92%
main: 92%

Ran 15 Sep 2023 04:27PM UTC

Files 260

Run time 14s

Badge

Embed ▾

Committed 15 Sep 2023 04:12PM UTC coverage: 19.735% (-0.001%) from 19.736%

Job # 6200364472.1

Build Type

push

github

Committed by

web-flow

Commit Message

bug: fix the date_fields request bottleneck (#5695)

* bug: fix the date_fields request bottleneck

I encountered a performance issue while attempting to index 1 million vectors. Despite the Weaviate instance having low utilization, the process was estimated to take around 10 hours. 

After some investigation, I identified the bottleneck: _get_date_properties function was being called for every document, consequently a request to the Weaviate client was being sent and awaited for each document.

To address this, I optimized the code by invoking the _get_date_properties function only when there is a schema change. This modification resulted in a notable performance improvement, reducing the indexing time to approximately 90 minutes for the same 1 million vectors.

* bug: fix the date_fields request bottleneck

* fix: executed the pre commit hooks for #9341

Run Details

4765 of 24145 relevant lines covered (19.73%)

0.2 hits per line

deepset-ai / haystack / 6200364472 / 1
92%
main: 92%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job modeling - 6200364472.1

deepset-ai / haystack / 6200364472 / 1 92% main: 92%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job modeling - 6200364472.1

deepset-ai / haystack / 6200364472 / 1
92%
main: 92%

README BADGES
x