6200364472
92%

Ran 15 Sep 2023 04:21PM UTC

Jobs 10

Files 260

Run time 16s

Badge

Embed ▾

Committed 15 Sep 2023 04:12PM UTC coverage: 49.045% (+0.01%) from 49.035%

Build # 6200364472

Build Type

push

github

Committed by

web-flow

Commit Message

bug: fix the date_fields request bottleneck (#5695)

* bug: fix the date_fields request bottleneck

I encountered a performance issue while attempting to index 1 million vectors. Despite the Weaviate instance having low utilization, the process was estimated to take around 10 hours. 

After some investigation, I identified the bottleneck: _get_date_properties function was being called for every document, consequently a request to the Weaviate client was being sent and awaited for each document.

To address this, I optimized the code by invoking the _get_date_properties function only when there is a schema change. This modification resulted in a notable performance improvement, reducing the indexing time to approximately 90 minutes for the same 1 million vectors.

* bug: fix the date_fields request bottleneck

* fix: executed the pre commit hooks for #9341

Run Details

11842 of 24145 relevant lines covered (49.05%)

2.56 hits per line

Jobs

ID	Job ID	Ran	Coverage
1	modeling - 6200364472.1	15 Sep 2023 04:22PM UTC	19.73	GitHub Action Run
2	agents - 6200364472.2	15 Sep 2023 04:22PM UTC	22.04	GitHub Action Run
3	pipelines - 6200364472.3	15 Sep 2023 04:23PM UTC	23.48	GitHub Action Run
4	cli - 6200364472.4	15 Sep 2023 04:21PM UTC	19.41	GitHub Action Run
5	utils - 6200364472.5	15 Sep 2023 04:22PM UTC	19.87	GitHub Action Run
6	document_stores - 6200364472.6	15 Sep 2023 04:22PM UTC	24.87	GitHub Action Run
7	prompt - 6200364472.7	15 Sep 2023 04:22PM UTC	24.21	GitHub Action Run
8	others - 6200364472.8	15 Sep 2023 04:22PM UTC	21.8	GitHub Action Run
9	preview - 6200364472.9	15 Sep 2023 04:25PM UTC	25.37	GitHub Action Run
10	nodes - 6200364472.10	15 Sep 2023 04:25PM UTC	30.67	GitHub Action Run

Source Files on build 6200364472

Detailed source file information is not available for this build.

deepset-ai / haystack / 6200364472
92%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6200364472

deepset-ai / haystack / 6200364472 92%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6200364472

deepset-ai / haystack / 6200364472
92%

README BADGES
x