deepset-ai / haystack

92%

main: 92%

LAST BUILD ON BRANCH improve-flaky-test

branch: improve-flaky-test

CHANGE BRANCH

x

Reset

• improve-flaky-test
• 313-experiment
• LLMMetaDataExtractor-remove-core-integrations-test
• TextFileToDocument-fix-docstring
• add-LLMMetadataExtractor
• add-async-tool-invoker
• add-custom-mime-type-msg
• add-dataframe-to-legacy-fields
• add-default-header-to-azure-embedders
• add-deprecation-warning-NLTKDocumentSplitter
• add-doc-type-router
• add-extended-abbreviations
• add-extra-to-toolcall
• add-image-converters
• add-list-joiner
• add-llm-doc-content-extractor
• add-max-workers-to-agent
• add-messages-to-state
• add-msg-converter
• add-pipeline-tool
• add-recursive-chunking
• add-run-async-huggingface
• add-ser-for-state-dataclass
• add-snapshot-breakpoint-exception
• add-split-by-row-csv-splitter
• add-test-tool-invoker-empty-args
• add-to-dict-bytstream
• add-to-from-dict-streamingchunk
• add-toolset-to-agent
• add-usage-hf-api-chat-streaming
• add-warning-when-pipeline-stops-early
• add_completion_start_time
• add_viz_supercomponent
• adding-SentenceSplitter-init-imports
• agent-as-a-tool
• agent-breakpoint-and-snapshot
• agent-exit-conditions
• agent-exit-conditions-docstring
• agent-refactor
• agent-run-async
• agent-use-deserialization-utility-function
• agent-with-no-tools
• agent_pydoc
• agent_tool_state_validation
• agent_tracing
• agents-warning-no-messages
• allow-python-3.13
• always-run-mypy
• answer-builder-refactoring
• async-HuggingFaceAPI-TextEmbedder
• async-document-writer
• author-as-reviewer
• azure-doc-embedder-inherit
• azure-embeddings-default-headers
• azure-openai-responses
• azure-text-embedder-inherit
• azure-tools
• better-multimodal-experiment
• bilgeyucel-patch-2
• broken-links
• bump-transformers
• change-mistral-to-microsoft
• change-name
• changes-agent-snapshot
• chat-prompt-builder-string-prompt-template
• chatgenerator-protocol
• chatgenerator-protocol-experiments
• chatmessage-deserialization-helpful-err-msg
• chatmessage-fixes
• chatmessage-from-dict-name-meta-not-mandatory
• chatmessage-fromuser-allow-empty-text
• chatmessage-parts-serde
• chatmsg-examples-upd
• chatmsg-from-openai-dict
• chatmsg-serialization-feedback
• check-chatgenerator-tools-support
• chinese-splitter-local
• chore/9854-drop-python39-support
• chore/clean-up-ChatMessage
• chore/clean-up-imports-in-tests
• chore/cleaning-up-tests
• chore/fix-tracker
• chore/pep585-type-hints
• chore/remove-deprecation-warning-evalrunresult
• chore/removing-deprecations-pipeline-draw
• chore/removing-unused-imports-from-tests
• chore/small-fixes-in-sentence-window-retriever
• ci-review-required-jobs
• ci-skip-windows-caching
• clean-pyproject
• clean-warmup
• cleaning-imports
• component-decorator-type-hinting
• component-info-in-streaming-chunk
• component-tool-updates
• component-warmup
• component_tool
• component_tool_state_validation
• conditional_router_pydoc_fix
• configure-openai-client
• configure-openai-client-config-components
• configure-openai-client-generators
• converters_doc
• cr_invalid_template
• csv-cleaner
• curly-brackets-2
• curly-brackets-3
• dalle-fix-max-retries
• dataframe-removal
• david-local-regex-fix
• ddtrace-4-compatibility
• ddtrace-fix
• deprecate-dataframe
• deserialize-callable-handle-class-methods
• detailed-message-show-difference-async-and-sync-args
• dev/DocumentSplitter_update
• dev/issue_9045
• discourage-hfapigenerator-hfinferenceapi
• doc-length-router
• doc-splitter-skip-empty-docs
• doc_snippets_update
• doclengthrouter-fix-example
• docs-component-init-callables
• docs-curly-brackets
• docs-fix-syntax
• docs-haystack-release-automation
• docs-search-feed
• docs/fixing-local-whisper-docstrings
• docs/fixing-python-docstring-automerge-retriever
• docx-links
• docxmetadata-as-dict
• dont-deepcopy-components
• draw-offline
• drop-3.9-check-type-serialization
• drop-python-3.8
• e2e-spacy
• enhance-resume-state
• enhancement/support_multiple_tool_string_outputs
• evalrunresult-deprecation-212
• export-reasoning-content
• extend_answer_builder
• extractor-docstring-edit
• failing_main
• fallback_streaming
• fb_chat
• feat-azure-workloadidentitycredential
• feat/9870
• feat/BrightData_web_search
• feat/EmbeddingBasedDocumentSplitter
• feat/RegexTextExtractor
• feat/add-callable-type-compatibility-clean
• feat/add-checks-component-name
• feat/add-deserialize_text_embedder_inplace
• feat/add-generic-deserialize_inplace
• feat/add-run-async-filter-and-automerging-retriever
• feat/add_async_huggingface_api_text_embedder
• feat/adding-QueryExpander-MultiQueryRetriever
• feat/agent_run_system_prompt
• feat/async-HuggingFaceLocalChatGenerator
• feat/async-InMemoryDocumentStore
• feat/async_pipeline
• feat/auto-merging-retriever
• feat/breakpoints
• feat/brightdata-websearch
• feat/chat_prompt_builder_time
• feat/csv-row-mode
• feat/csv-to-document-row-level-conversion
• feat/datadog-tracer/set-component-name-as-resource-name
• feat/draw-supercomponents-detail
• feat/embedders-protocol
• feat/encode-kwargs-for-sentence-transformers
• feat/fetcher-custom-headers-attrib
• feat/fetcher-custom-headers-clean
• feat/hf-async-stream-handling
• feat/huggingface-api-chat-image-support
• feat/improve-bare-type-handling
• feat/intermediate-results
• feat/last-snapshot-working-for-Agent
• feat/meta-field-sanitize-serialization
• feat/multiple-outputs-conditional-router
• feat/pipeline-checkpoints-crash-resume
• feat/return-last-good-snapshot
• feat/run_async_openai_text_embedder
• feat/save-inputs-outputs-execution
• feat/select_tools_at_runtime
• feat/standardize-streaming-chunk
• feat/streaming_callback_run_param_for_hf_chat_generators
• feat/super-component-common-type
• feat/supercomponent-non-leaf-outputs
• feat/support_sparse_models_in_sentence_transformers
• feat/tool-from-function-with-rest-docstring
• feat/use-importlib-when-deserializing-callable
• feat_bytestream_guess_mime_type
• feature/9701-huggingface-api-reasoning-support
• feature/add-async-openai-document-embedder
• feature/add-warm-up-to-chat-generators
• feature/agent-with-generator_kwargs
• feature/chinese-document-splitter
• feature/csv-splitter
• feature/md-header-splitter
• first_snippets_run
• fix#8759
• fix-agent-snapshot-init
• fix-agent-streaming-cb
• fix-agent-tracing-schema-to-dict
• fix-agents-messages
• fix-asyncpipeline-docstrings
• fix-asyncpipeline-run-error
• fix-azure-tests-from-froks
• fix-bug-in-streaming-openai
• fix-cg-tests-from-forks
• fix-ci-dep-caching
• fix-classifiers-docstrings-errors
• fix-component-tool
• fix-component-tool-deepcopy
• fix-component-tool-invoker
• fix-component-tool-parameters
• fix-component-tool-sede
• fix-conditional-router
• fix-content-tag-name
• fix-convert-streaming-chunks-to-chat-message
• fix-coverage-upload
• fix-datadog-tracer
• fix-dd-trace
• fix-ddtrace-tests
• fix-deepcopy-pipeline-run
• fix-deepset-sync
• fix-deserialization-tool
• fix-deserialize-tools
• fix-deserilization-answer
• fix-doc-preprocessor-sede
• fix-docstring-example-azurechatgen
• fix-docusaurus-pydocs
• fix-embedding-backend-signature
• fix-error-huggingface-async-test
• fix-example
• fix-flaky-hf-test
• fix-hf-api-tests
• fix-hf-token-test
• fix-hfapi-flaky-tool-test
• fix-hfapi-model
• fix-hfhub-embedder
• fix-hflocalchat-msg-conversion
• fix-issue-8719
• fix-jinja2-variable-extraction
• fix-link-tests
• fix-lint-job
• fix-lint-job-mypy
• fix-linting-error
• fix-llm-metadata-extractor
• fix-meta-hf-api-chat
• fix-metafieldgroupingranker
• fix-mypy-1-16
• fix-mypy-jinja2-extension
• fix-openai-embedders
• fix-openai-flaky-test
• fix-openai-responses-streaming-and-reasoning
• fix-openai-tests
• fix-openai-tests-1.99.3
• fix-output-types-decorator-type-hint
• fix-pipeline-dese-error
• fix-preprocessor-example
• fix-print-streaming-chunk
• fix-pytest-warning
• fix-recursive-splitter
• fix-sede-enum
• fix-ser-and-schema-support
• fix-st-embedders-error-msg
• fix-state-validate-schema
• fix-streaming-and-tools
• fix-streaming-callback-type
• fix-tests-on-forks
• fix-too-many-positional-arguments
• fix-toolinvoker-async
• fix-tracing-async-context
• fix-tracing-tool-invoker-async
• fix-transformer-ranker
• fix-trigger-tool-invoker-breakpoint
• fix-type-error-agent
• fix-type-sede
• fix-type-util-tests
• fix-types-pipeline-test-run
• fix-value-error-delta-tool-call
• fix/9127
• fix/FileTypeRouter-handles-non-existent-files-inconsistently
• fix/RegexTextParser
• fix/SentenceSplitter-ReDoS
• fix/SentenceTransformersDiversityRanker-warning
• fix/add-encoding-format-to-openai-embedders
• fix/agent_from_dict_state_schema_none
• fix/azure-embedder-graceful-error-handling
• fix/branchjoiner-docstring
• fix/chat-message-name-field
• fix/cleaning-duplicate-import
• fix/close-openao-asyncio-stream-on-cancellation
• fix/component-decorator-typing
• fix/component-returning-empty-dict
• fix/component_checks
• fix/component_tool_openai_use_strict
• fix/component_type
• fix/core-type-completeness
• fix/deepcopy-documents
• fix/destroy-executor-when-created-inside-class
• fix/detect-CID-characters-PDFMiner
• fix/diversity-ranker-top-k-validation
• fix/docstring
• fix/docstring-dicts-in-ticks
• fix/document-to-dict-field-precedence
• fix/dunder_all
• fix/embedder-deepcopy-documents
• fix/error_msg_on_invalid_type
• fix/eval-cast-to-float
• fix/fully_qualified_callable_serde
• fix/hf-tests
• fix/hfapi-toolcall-args-9298
• fix/httpxClientLimitsParsing
• fix/informative-error-message
• fix/issue-10258-qwen3-streaming-usage
• fix/issue-9580-async-tool-error
• fix/lazy_variadic_pipelines
• fix/llm-metadata-extractor-none-content
• fix/logging_indexerror
• fix/loggingtracer-coerce-tag-values
• fix/many_to_one_socket_connections
• fix/new_ruff_release_format
• fix/pipeline-deserialization-multithreading
• fix/pipeline_logging
• fix/pipeline_run
• fix/pipeline_run_issues
• fix/ranker-score-float
• fix/recall-eval-unique
• fix/recursive-splitter-unique-ids
• fix/set_tag_after_yield
• fix/some-agent-methods-should-be-static
• fix/tool_description_truncation
• fix/toolset-warm-up-individual-tools
• fix/truncate_bytestream_str
• fix/typo-static-methods-recall-evaluator
• fix/typos
• fix/unused-variable
• fix_10108
• fix_agent_pydoc
• fix_component_tool_test
• fix_condition_router_to_dict
• fix_lambda_threshold
• fix_openai_chat_generator_wrapped_streaming
• fixing-HuggingFaceAPIChatGenerator-test_live_run_with_tools-test
• fixing-RecursiveSplitter-pydoc-markdown-rendering
• fixing-splitting-bug-accumulative-count
• from-function-refactoring
• general-cleanup
• guide-migrate-to-haystack
• hf-api-embedders-change-model-test
• hf-embedders-integration-tests-more-lenient
• hf-hub-0.28-types
• hf-tool-definition-0.31.0
• hf_tools
• hf_update
• hfapi-check-choices-none
• hfapi-components-fixes
• hfapi-multimodal-test-change-model
• hfapi-tools
• image-content
• image-content-class-methods
• imp-sentence-trans-embed
• improve-slow-tests
• improve_err_msg_lazy_import
• issue-8048
• jinja2-set-vars-incorrect
• json-converter-update
• jsonschema-dependency
• julian-risch-patch-1
• link_fetcher
• lint-should-run-on-pyproject-changes
• llm-evaluators-chatgenerator-init-param
• llm-me-adopt-cg-protocol
• llm-messages-router
• llm_error_facing
• llmevaluator-chatgenerator
• llmevaluators-rm-deprecated-params
• llmmetadata-extractor-rm-deprecated-params
• main
• make-component-tool-a-dataclass
• make-validate-input-public
• mark-slow-tests
• measure-import-time
• minimal-chat-generator-protocol
• minor-docstring
• more-robust-tests
• move-agent
• move-over-supercomponents
• move-supercomponent
• move-tool
• move-tool-serialization
• multimodal-experiment
• mypy-check-untyped-defs
• mypy-run-read-only
• named-entity-extractor-private-models
• native-python-types
• new-chatmessage
• new-sentence-transformers-ranker
• new_toolset
• nit/typo
• nltk-download-warm-up
• not-async-test
• only-use-openai-function-tool-calls
• opeanai-fix-response-fmt-ser-error
• open_api_chat_message
• openai-embedder-rm-unused-code
• openai-exclude-custom-tcs
• openai-longprobs
• openai-pass-tools-to-client-if-not-null
• openai-relax-flaky-test-async
• openai-responses
• openai-streaming-usage
• openai-structured-outputs
• openai-whisper-incompatible
• openai_parallel_tools
• openaichatgenerator-tools
• openaichatgenerator_run_async
• openapi-llm-optional-dep
• openapi_component
• optimize-import-times
• pass-on-streaming-callback-tool-invoker
• patch-1
• pdfminer-docid
• pdfminer-passage-fix
• pep-604
• pep604
• pin-blis
• pin-ddtrace
• pin-hf-hub
• pin-nltk-version-splitter
• pin-numba
• pin-openai-1.99.2
• pin-openai-httpx-bug
• pin-posthog
• pin-structlog
• pin-transformers-less-4.57
• pipeline-draw-skip-some-integration-tests
• pipeline-draw-timeout
• pipeline-runtime-error-attributes
• poc/resume-state-for-crashes
• prevent-doc-writer-tests-hang
• py-typed
• pydantic-sede
• pypdf-docid
• pypdf-passage-fix
• quantize_sentence_transformers
• readme-clean-up
• reasoning-content
• reasoning-content-streaming-chunk
• recursive-csv-splitter
• recursive-splitter-token
• reexport-missing-symbols
• ref-hf-local
• refactor-agent-breakpoint
• refactor-check-imports
• refactor-deserialize-util
• refactor-hf-api-chat-generator
• refactor-openai-generator
• refactor-pipeline-run-breakpoints
• refactor-save-pipeline-snapshot
• refactor-snapshot
• refactor-tool-invoker
• refactor-typing
• reintroduce-deprecate-tool-deserialization-util
• relax-posthog-pin
• remove-DocumentSplitter-sentence-warning
• remove-NLTKDocumentSplitter-deprecated
• remove-async-executor-toolinvoker
• remove-dataclasses-State
• remove-df-doc-azure-ocr
• remove-gpt-3.5-turbo
• remove-legacy-support-schema-util
• remove-unused-imports
• remove-unused-imports-core-tests
• remove_openai_async_duplicate_test
• removing-invalid-arg-from-docstraing
• removing-unused-fixtures
• rename-docs-folder
• reorganize-testing-for-dataframes
• repair_telemetry_unclosed_files
• replace-tc-model
• replace-use-auth-token
• revert-hf-test-skip
• review-integration-tests
• rexeport-symbols
• rm-deprecated-function-role-in-chatmessage
• rm-deprecation-warning-sentence-window-retriever
• rm-deserialize_tools_inplace
• rm-hf-utils-dead-code
• rm-logger-if-unused
• rm-store-full-path-depr-warning
• rm-type-ignores
• rm-typevar
• rm-unused-files
• router-from-dict
• run-async-azure
• schema-with-message-subclass
• seb-cleanup
• select-streaming-callback-overload
• select_streaming_callback
• self-host-draw
• self-host-draw-from-other-branch
• sentence-transformers-4-type-ignore
• sentence-transformers-local-files-only
• sentence-transformers-ranker
• sentence-window-retriever
• serde_tool_rename
• serialize-pipleine-outputs
• serperdevwebsearch-add-exclude_subdomains-param
• skip-draw-pipeline-int-test
• skip-hf-api-live-integration-tests
• slow-tests
• slow-workflow
• small-fix-set-output-types
• small-st-models-tests
• snapshot_callback
• st-image-doc-embedder
• st-missing-docstrings
• st-similarity-ranker-trust-remote-code
• st_revision
• stability-pipeline-tool-tests
• streaming-in-agent-tools
• streaming_chunk_finish
• stricter-mypy-checks
• subs-for-supercomponents
• suggested-changes-pipeline-crash
• super-component-decorator
• super_component_runtime_input
• supercomponent-async-serialization
• telemetry-rm-heavy-imports
• test-pr
• test-release-notes-check
• test/make-sure-extra-modules-are-not-imported
• testing-only-logging-tracer
• testpr
• tests-type-utils
• timeout-datadog-action
• tool-dataclass
• tool-invoker
• tool-invoker-tools-in-run
• tool-refactor
• tool_invoker_sync_span
• tool_invoker_warmup_fix
• toolinvoker-rm-warning
• toolinvoker-runasync
• tools_streaming_openai
• tools_warmup
• tools_warmup_new
• toolset
• toolset-add-chainable
• toolset_list
• toolset_thomas
• torch-xpu-device
• tracing-avoid-big-payloads
• transformers-4-52
• transformers-4.47.1
• transformers-4.48.3
• transformers-fix-mypy
• try-fix-CI
• try-fork
• try-skipping-slow
• try-slow-tests
• try-slow-workflow
• try-updating-hatch-1.14.1
• try-win-cache
• typing-extensions-license-compliance
• udpate-streaming-callback-type
• unify-DocumentSplitter-NLTKDocumentSplitter
• unify-hatch
• unpin-ruff-update
• unused-imports
• update-agent-docstrings
• update-agent-exit-conditions
• update-async-pipe-docstrings
• update-chatmessage-link-to-docs
• update-convert-streaming-chunk
• update-csv-cleaner
• update-doc-string-openai
• update-docs-branch-joiner
• update-finish-reason-hf-api-chat-gen
• update-list-joiner
• update-metadata-router
• update-openai-response-format
• update-openai-responses-tests
• update-print-streaming-chunk
• update-prompt-metadataa
• update-pydoc-markdown
• update-sentence-window-retriever
• update-st-ranker
• update-store-full-path
• update-streaming-chunk
• update-structured-output
• update-to-chat-completion-chunk-conversion
• update-tool-outputs-to-string
• update-tools-strict-openai
• update-xlsx-converter
• update/in_memory/document_store
• update_cgs
• update_component_error_msg
• update_open_ai
• update_serialization_schema_util
• upgrade_azure
• use-deepcopy-with-exceptions
• use-logging-warning
• use-thread-safe-import
• use-ubuntu-slim
• v1.17.x
• v1.18.x
• v1.19.x
• v1.20.x
• v1.21.x
• v1.22.x
• v1.23.x
• v2.0.x
• v2.1.x
• v2.10.x
• v2.11.x
• v2.12.x
• v2.13.x
• v2.14.x
• v2.15.x
• v2.16.x
• v2.17.x
• v2.18.x
• v2.19.x
• v2.2.x
• v2.20.x
• v2.21.x
• v2.3.x
• v2.4.x
• v2.5.x
• v2.6.x
• v2.7.x
• v2.8.x
• v2.9.x
• variance_fix
• variance_fix_v2
• warm-up-refactor
• warning-prompt-builder
• xlsx-converter

Build # 20592350176

Build Type

Pull #10287

github

Committed by web-flow

Commit Message

Merge b37c8acdf into e86f01614

Pull Request Pull Request #10287: test: Improve flaky test and actually test reasoning

Run Details

14185 of 15375 relevant lines covered (92.26%)

0.92 hits per line