CodeDocs Vault

Agentic Search — hybrid lexical + semantic, fused with RRF

The search MCP tool is how an agent finds things in a knowledge base it didn't write. It is a three-tier ranker — lexical → BM25 → vector — fused with Reciprocal Rank Fusion (RRF) so conceptually-related docs surface even with zero keyword overlap, while exact title/path matches always win.

1. The three tiers

Implemented in packages/core/src/search/workspace-search.ts:

  1. Lexical (:181-206) — bracketed scoring over path/title/name: exact title or name match (700) > path match (650) > startsWith (600/550) > includes (500/450), with a 0.5× penalty for hidden files. This guarantees that typing a doc's title surfaces it first.
  2. BM25 full-text (:405-418) — an Orama index over each WorkspaceSearchDocument (id, kind, path, title, name, pathSegments, content, modifiedTs; built at :142-149), with intent-specific field boosts (full-text intent boosts title 8×; autocomplete boosts title 10×, name 9×).
  3. Vector (:492-592) — semantic similarity over OpenAI embeddings produced by SemanticSearchService (packages/server/src/embeddings/semantic-search-service.ts:129-145): documents are chunked, each chunk embedded, vectors cached under .ok/local/embeddings/ keyed by provider + model + dimensions.

2. RRF fusion

BM25 and vector results are dense-ranked, then combined (:552-591):

rrfScore(id) = 1/(K + bm25_rank) + 1/(K + vector_rank)      # default K = 60

RRF is rank-based, not score-based, so it fuses two incomparable scoring systems (BM25's term scores and cosine similarities) without normalization headaches. Final ordering: lexical matches always top; documents with no lexical match are ranked by max(rrfScore, recency), under result caps (6 body results, 4 path-only, unlimited lexical). Recency as a floor means a brand-new doc isn't buried just because it isn't embedded yet.

3. Configurable, keyring-backed embeddings

The embedder is pluggable (server-factory.ts:333-347): base URL, model, and dimensions are set through the ok embeddings CLI commands, and the API key is read from the OS keyring (08_desktop_runtime.md), not a config file. Because any OpenAI-compatible endpoint works, embeddings can run against a local model — consistent with the OpenCode fully-local story (02_harness_integration.md). Chunk-level caching keyed by (provider, model, dimensions) makes the embedding cost predictable and invalidates correctly when you switch models.

What's worth stealing

  1. RRF to fuse incomparable rankers. When you have a lexical signal and a semantic signal, fuse by rank, not by score — no fragile normalization.
  2. Lexical always wins; recency is a floor. Exact-match-first and new-docs-don't-sink are two cheap rules that make hybrid search feel correct instead of mysterious.
  3. Keyring-backed, pluggable embedder. Secrets in the OS keychain and an OpenAI-compatible base URL means the same search works cloud or fully local.
  4. Chunk-cache keyed by (provider, model, dims). Predictable cost and correct invalidation on model switch — a small detail that matters at scale.