cofacts / rumors-api

83%

master: 82%

LAST BUILD ON BRANCH vector-search

branch: SELECT

CHANGE BRANCH

x

• No branch selected
• 100-update-reply-request
• 110-fix
• 201712-mappings-refactor
• 38-add-opinionated
• 41-create-article
• 41-create-reply
• 41-url-preview
• 52-check-duplicate
• 64-edit-name
• 65-author-articles
• 67-create-article-reason
• 73-74-gamification
• 74-level
• 93-unauthorized-registration
• 95-userlevel-cache
• 99-resolver-docker
• ListArticle-types
• add-article-id
• add-latest-replied-sort-option
• admin-api-moderation
• admin-audit-get
• ai-reply-prompts
• ai-response
• ai-transcript
• ai-transcript-list
• allow-empty-slug
• allow-multiple-origin
• analyticsDataLoader
• anti-seo-m2
• anti-seo-spam
• apply-media-manager
• archive-logging
• article-reply-filter
• articleStats
• articlereply-order
• avatars
• backendUsers
• badge-api
• badge-api-revoke
• block-articlereply-feedbacks
• block-new-articles
• block-user
• block-user-script
• blocked-user
• bugfix-gt-0
• category-mock
• category-review-script2
• cc-by
• change-lint-command
• chore/security-updates-and-bun-migration
• chunked-encoding
• claude/dazzling-ritchie-21ZJO
• cleanup
• cleanuptests
• compress-logs
• contentGroupFlag
• contributions
• contributors
• create-categories-params
• createUsers
• ctxUserId
• dannynash-features/#121
• dateUtil
• delete-script
• disable-log-files
• display-legal
• dotenv
• emptyRows
• feat-remove-reply
• feat/add-claude-code-action
• feat/use-secret-for-project-id
• feature/bff-auth
• feature/env-media-article-support
• feature/upgrade-to-elasticsearch-v9
• feature/user-avatar-resolver
• feature/ydoc
• features/51-highlight
• features/article-with-category
• features/can-edit-field
• features/create-category
• features/delete-reply
• features/image-collection
• features/image-collection-rebase
• features/list-reply
• features/reply-connection-status-filter
• features/time-field-for-replyconnection
• features/update-category-status-n-feedback
• fetchGA
• filter-article-with-feedback
• filter-replyconnection-in-reply
• fix-47
• fix-adm-readme
• fix-arf-multi-filter
• fix-blockuser-script
• fix-build-by-upgrade
• fix-category-id-filter
• fix-ci
• fix-cleanup-url
• fix-cooccurrence
• fix-dev-env
• fix-gen-ai-reply
• fix-get-ydoc
• fix-list-cooccurrence
• fix-media-cooccurrence-error
• fix-mutation
• fix-requestedForReply
• fix-scrapurl
• fix-test
• fix-url-resolver
• fix/ci-duplicate-runs
• fix/gql-doc
• fix/graphiql-cdn-links
• fix/transcript-model-fallback
• flow-tag-importer
• gemini-experiment
• gen-ai-transcript
• genBERTInputArticles
• genai-sdk
• github-actions
• hyperlink-fields
• improve/add-articles-and-replies-time-range-filter
• interfaces-non-null
• issue-77
• johnson/bq-analytics
• langfuse
• langfuse-exp
• legal
• line-bot-cors
• list-api-enhancements
• list-article-reply-feedback
• listReplyRequest
• listairesponses
• listreply-multiple-type
• listreply-selfonly
• llm-adjustment
• llm-transcript
• loadScript
• log-requests
• logging
• master
• maximeri-feat-remove-reply
• mcp
• media-similarity
• media-support
• merged
• moderation-apis
• modify-media
• multi-stage-build
• mutation-adds-status
• mutation-entity
• name-avatar-gen
• new-login-methods
• non-nulls
• normalArticleReplyCount-typo
• nullable-createdat
• ocr-confidence
• openpeeps
• process-image
• profile
• prompt
• pubsub-cmd
• readme_grpc
• record-feedback-target-author
• redirect
• refactor-ai-resp
• refactor-arithmetic-filters
• refactor-uploadmedia
• refactor/rename-rumors-db-test-script
• refs/tags/20230629
• refs/tags/release/20211105
• refs/tags/release/20211126
• refs/tags/release/20211203
• refs/tags/release/20220107
• refs/tags/release/20220131
• refs/tags/release/20220414
• refs/tags/release/20220505
• refs/tags/release/20220701
• refs/tags/release/20220714
• refs/tags/release/20220806
• refs/tags/release/20220807
• refs/tags/release/20220908
• refs/tags/release/20221013
• refs/tags/release/20221213
• refs/tags/release/20221223
• refs/tags/release/20230121
• refs/tags/release/20230203
• refs/tags/release/20230224
• refs/tags/release/20230402
• refs/tags/release/20230406
• refs/tags/release/20230407
• refs/tags/release/20230413
• refs/tags/release/20230520
• refs/tags/release/20230607
• refs/tags/release/20230614
• refs/tags/release/20230629
• refs/tags/release/20230817
• refs/tags/release/20230902
• refs/tags/release/20230913
• refs/tags/release/20231005
• refs/tags/release/20231019
• refs/tags/release/20231027
• refs/tags/release/20240104
• refs/tags/release/20240111
• refs/tags/release/20240205
• refs/tags/release/20240502
• refs/tags/release/20240507
• refs/tags/release/20240630
• refs/tags/release/20240827
• refs/tags/release/20240904
• refs/tags/release/20241007
• refs/tags/release/20241102
• refs/tags/release/20241107
• refs/tags/release/20241112
• refs/tags/release/20241125
• refs/tags/release/20241203
• refs/tags/release/20250121
• refs/tags/release/20250303
• refs/tags/release/20250318
• refs/tags/release/20250702
• refs/tags/release/20260124
• refs/tags/release/20260307
• refs/tags/release/20260328
• refs/tags/release/20260331
• refs/tags/release/20260505
• refs/tags/release/20260717
• related-media-articles
• release/20210218
• release/20210225
• release/20210304
• release/20210311
• release/20210324
• release/20210408
• remove-dreprecation
• renovate/all
• renovate/configure
• renovate/docker-all
• renovate/docker-docker.elastic.co-elasticsearch-elasticsearch-oss-6.x
• renovate/docker-docker.elastic.co-kibana-kibana-oss-6.x
• renovate/elastic-elasticsearch-7.x
• renovate/eslint-7.x
• renovate/googleapis-59.x
• renovate/jest-26.x
• renovate/koa-router-9.x
• renovate/lock-file-maintenance
• renovate/mockdate-3.x
• renovate/pin-docker-digests
• reorder-resolvedUrls
• reply-request-count-filter
• reply-status
• replyrequests
• revert-359-llm-transcript
• review-category
• rumors-db-update
• samesite
• scripts-utils
• search-urls
• similar-replies
• simplify-input-type
• skip-hyperlinks
• sort-analytics
• strict-schema
• stt
• support-cooccurrences
• sync-with-resolver
• timezone
• ts-schema
• ts-scripts
• typescript
• typo
• update-db
• update-prettier
• update-rumors-db
• update-schema
• upgrade-fb
• upgrade-gemini-model-3106540277054212479
• upgrade-graph-api
• upgrade-jest
• upgrade-node-and-add-gen-ai-sdk
• url-cleanup
• url-fill-script
• use-buffer
• user-filter
• userProfile
• vad
• validate-slug
• vector-search
• video-thumbnail
• warn-about-sampled
• wayback-machine
• web-concurrency
• write-ai-transcript
• write-liff-analytics
• year-prompt

Build # 30203190180

Build Type

Pull #388

github

Committed by yutin1987

Commit Message

feat: hybrid BM25 + kNN vector search with Gemini embeddings

Add dense-vector (kNN) retrieval to ListArticles/ListReplies, opt-in via
the `embedding` similarity filter. kNN narrows the candidate set and the
existing BM25 / perceptual-hash scoring ranks the results.

- Embed articles and replies on create; backfill script for old docs.
- Audio/video embed as a single vector capped at 80s (no duration probe).
- Media queries reuse the doc-side embedding by content hash, else fetch
  bytes from the query URL and embed — search never persists to GCS, so
  no orphan files.
- All Gemini calls (embeddings + transcription) use the Developer API and
  feed media via the Files API; Vertex AI is removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #388: vector search

Coverage Stats

1011 of 1294 branches covered (78.13%)

Branch coverage included in aggregate %.

174 of 191 new or added lines in 10 files covered. (91.1%)

8 existing lines in 1 file now uncovered.

1899 of 2202 relevant lines covered (86.24%)

17.2 hits per line