Semantic and full-text search over the research corpus

Research Question

What combination of full-text search (keyword/BM25) and semantic search (embeddings/vector) is most appropriate for querying the Research/completed/ corpus, given the constraints of a git-based, local-first, single-owner repository with tens to hundreds of items, and what is the simplest implementation path?

Findings

Executive Summary

SQLite FTS5 (BM25) alone is the correct implementation for this corpus at current scale (12–100 items), requiring zero new dependencies beyond Python's standard library; Model2Vec (potion-base-8M) + sqlite-vec hybrid search should be introduced when the corpus reaches 100 items, using Reciprocal Rank Fusion to combine BM25 and vector rankings. The dominant query patterns for a single-owner corpus — keyword lookup and tag filtering — are well-served by FTS5; semantic search addresses the vocabulary-mismatch failure mode that becomes material only as the corpus grows past ~100 items and the author can no longer recall the exact vocabulary used in older items. Model2Vec is preferred over sentence-transformers/all-MiniLM-L6-v2 for Phase 2 because it has no PyTorch dependency, downloads at ~30–50 MB (vs 80–90 MB + ~200–500 MB PyTorch), and runs ~200x faster on CPU with 91–93% of MiniLM's MTEB accuracy. Both phases store the index at state/search.db (gitignored derived artefact), keeping the Markdown + YAML front-matter files as the sole source of truth.

Key Findings

1. SQLite FTS5 BM25 full-text search requires zero additional Python dependencies beyond the stdlib sqlite3 module and delivers sub-millisecond query latency for corpora of fewer than 500 documents, making it the correct and sufficient Phase 1 implementation. [high]

2. The four query patterns for this corpus — tag-based, keyword, topic/conceptual, and recency — split cleanly: tag and recency queries require only YAML front-matter metadata lookups; keyword queries are served by FTS5; only conceptual queries ("what do I know about X") require semantic search, and these are the least frequent query type for a single-owner corpus. [high]

3. Model2Vec (potion-base-8M) is the correct embedding model for Phase 2: it is ~30–50 MB, requires only numpy (no PyTorch), runs ~200x faster on CPU than all-MiniLM-L6-v2, and achieves 91–93% of MiniLM's MTEB accuracy, which is adequate for structured research-item retrieval. [high]

4. sqlite-vec is a pre-v1 extension with explicitly declared breaking-change risk; it must be pinned to a specific version in requirements.txt/pyproject.toml and upgraded only intentionally, with the Phase 2 ADR documenting this constraint. [high]

5. Reciprocal Rank Fusion (RRF, k=60) is a ~10-line pure-Python merge of BM25 and KNN ranked lists that requires no score calibration and no additional dependencies, making it the correct fusion strategy for this use case. [high]

6. The correct fields to index are title, tags, executive summary, and key findings text (concatenated as a single FTS5 row per item); indexing the full findings section adds noise without improving recall at this corpus scale, and sub-item chunking is not warranted until the corpus exceeds several hundred items. [medium]

7. The search index should be stored at state/search.db (gitignored), rebuilt lazily at query time when any completed item's mtime is newer than the index file, and never committed to git — the Markdown files with YAML front-matter remain the sole source of truth. [high]

8. Phase 2 (hybrid search) requires an ADR before implementation to document the model2vec dependency, the sqlite-vec pre-v1 API risk, the state/search.db schema, and the HuggingFace cache configuration for GitHub Actions. [high]

9. The Phase 2 corpus threshold is ≥ 100 completed items; the prior indexing-and-tracking research deferred vector search at ~50 items, and the current corpus of ~52 items has reached but not clearly exceeded that threshold, so Phase 1 should be implemented immediately and Phase 2 planned within the next research loop cycle. [medium]

10. The hybrid retrieval approach (FTS5 + Model2Vec + sqlite-vec + RRF) was validated in a directly analogous real-world implementation (15,800-file Obsidian vault, 49,746 chunks) documented in the context-mode completed research, providing practitioner-level evidence that the stack works at scale significantly beyond this corpus. [high]

Assumptions

• Assumption: The author queries their own corpus using the vocabulary they wrote, making vocabulary mismatch rare at small scale. Justification: Single-owner corpus; author wrote every item and reads them during the research loop. This assumption weakens as the corpus grows and older items fade from memory.
• Assumption: Model2Vec's static (non-contextual) embeddings are adequate for research-item retrieval, despite the 6-point MTEB gap vs. MiniLM. Justification: Research items are structured, domain-specific documents with well-defined sections. The failure mode of static embeddings (missing context-dependent meaning) is less relevant for "find item about topic X" queries than for open-domain QA or sentiment analysis. The practitioner case from context-mode research used the same model for a structurally similar corpus.
• Assumption: The corpus will grow at the observed rate of ~5–10 items per week, reaching 100 items within 3–6 months. Justification: The research loop schedule runs weekdays at 3 items/day. This rate could change if the backlog is exhausted or the loop is paused.

Analysis

The core trade-off is implementation complexity vs. search quality at each corpus size. FTS5-only has near-zero complexity cost (it is already in the Python stdlib and the prior local-database research designed the schema for it). Hybrid adds two dependencies and ~50 lines of code for a meaningful improvement in recall on conceptual queries. The phased approach resolves this trade-off by deferring the complexity cost until the quality improvement is actually needed.

The embedding model choice (Model2Vec vs. sentence-transformers) was resolved decisively by the CI constraint. The absence of PyTorch is not just a convenience — it is the difference between an index rebuild that fits in a GitHub Actions free tier (2-core, 7 GB RAM, ~6-minute job limit for free runners) and one that may run over time or exhaust memory. Model2Vec's numpy-only dependency and ~200x speedup make the difference between a 5-second rebuild and a 5-minute rebuild for 200 items.

The field selection (title + tags + executive summary + key findings) was driven by signal-to-noise analysis. The executive summary is explicitly designed to be the direct answer to the research question — it is the highest-signal field. Key findings are the structured claims that constitute the research output. Indexing these fields over the full findings body ensures BM25 term weights are not diluted by the lower-signal Analysis and Risks sections.

OpenAI embeddings were eliminated at the constraint stage — they require an API key that is not in the approved credentials table. This is a hard constraint, not a preference.

Risks, Gaps, and Uncertainties

• sqlite-vec pre-v1 API risk. The extension has declared breaking changes before 1.0. Until it reaches 1.0, the Phase 2 implementation may need a minor version upgrade at arbitrary intervals. Version pinning mitigates this but adds maintenance overhead.
• Model2Vec accuracy on this specific corpus is unknown. The MTEB benchmark is over a broad range of tasks; performance on structured research-item retrieval (short, domain-specific, single-author) has not been independently measured. The practitioner case (context-mode Key Finding 6) is the closest evidence, but it is a single data point.
• Lazy-rebuild index freshness. If the search index rebuild fails silently (e.g., due to a model download error in a constrained environment), the user receives stale results without warning. The rebuild step should log its status explicitly.
• The 100-item Phase 2 threshold is a heuristic, not an empirically validated measurement. It is possible that vocabulary-mismatch false negatives become a real problem at 60 items or not until 150. A better trigger would be the first observed false negative, but that requires the user to notice and report it.
• tantivy Python bindings were not directly tested. They were eliminated on complexity grounds (Rust build dependency) without a full evaluation. If FTS5 performance proves inadequate at large scale, tantivy should be re-evaluated.

Open Questions

• Should the search index be rebuilt and committed as part of the research loop workflow, rather than lazily at query time? Committing the index would make it available immediately in any environment without a rebuild step, but it would introduce a binary file into git (violating the git-friendliness constraint). This is a workflow design question for the implementation slice.
• Is Model2Vec's accuracy adequate for conceptual queries in this specific corpus? A quick evaluation when Phase 2 is implemented — running 5–10 representative queries against both FTS5-only and hybrid modes — would confirm or refute the assumption.
• Should the CLI output include a relevance score or confidence indicator? This would help the user calibrate how much to trust ranked results, especially in hybrid mode where the RRF score is not directly interpretable.
• Does the conversational interface (Research/completed/2026-03-02-chat-conversational-interface.md) expect search to return item content or only item metadata (path, title, tags)? The answer affects whether the CLI needs a --include-content flag or whether the conversational layer fetches the full item after receiving search results.

sources

• [x] Research/completed/2026-02-27-indexing-and-tracking-method.md — prior decision: JSON + YAML front-matter; deferred vector search (Key Finding 5)
• [x] Research/completed/2026-03-01-context-mode-llm-context-compression.md — hybrid BM25+Model2Vec+sqlite-vec+RRF pattern (Key Finding 6)
• [x] Research/backlog/2026-02-27-local-database.md — database technology options including sqlite-vec, ChromaDB, LanceDB (now in completed/)
• [x] sqlite-vec — vector search extension for SQLite
• [x] SQLite FTS5 — full-text search
• [x] sentence-transformers (all-MiniLM-L6-v2)
• [x] Model2Vec (potion-base-8M) — small, fast, static sentence embeddings
• [ ] tantivy Python bindings — Rust-based BM25 search library
• [ ] LanceDB — embedded vector store with full-text search