Research / LanceDB index rebuild speed from git

LanceDB index rebuild speed from git

LanceDB index rebuild speed from git: enabling stateless deployment

2026-03-08 · rag-retrieval tools-infrastructure mlops-deployment cost-performance · medium · source → · wiki →
key claims
  1. LanceDB write operations take under 30ms even at 1000 documents; the embedding model accounts for 99%+ of cold-start rebuild time at every tested corpus size
  2. BAAI/bge-small-en-v1.5 loads from disk cache in 0.68s; the model file is 133 MB, giving a first-download load time of ~2.25s from HuggingFace Hub on a typical connection
  3. BGE-small-en-v1.5 embeds at ~34.5ms/doc for short synthetic texts but ~177ms/doc for real research items; BERT's O(n²) attention cost means document length — not document count — is the primary driver of embedding time
  4. At the current corpus size of 61 items with full Markdown text, cold-start rebuild takes 11.5s with BGE-small, already exceeding the 5–10s target and with no improvement path as the corpus grows
  5. Pre-computed embeddings stored as JSON in git (7.74 KB per 384-dim document) load and write into LanceDB in under 0.2s for 1000 documents, making startup time effectively O(1) with respect to corpus size
  6. JSON storage cost for pre-computed BGE-small embeddings is 7.74 MB for 1000 documents; for a personal research corpus of 200–300 items, this is under 2.5 MB, an acceptable git repository size
  7. all-MiniLM-L6-v2 is approximately 2× faster than BGE-small at embedding (16.4ms/doc vs 34.5ms/doc for short texts) but does not change the fundamental scaling failure: at 500 real research items, MiniLM would still require over 20s to rebuild
  8. GitHub code search is rate-limited to 10 requests per minute, is keyword-only with no semantic understanding, and may return incomplete results for large queries; it is not a viable replacement for vector search and only useful as a supplementary lexical search layer

Research Question

Can the LanceDB index be rebuilt from the .md files in the repo on startup fast enough to enable stateless (per-request) deployment? Measure rebuild time at: current corpus size, 100 files, 500 files, 1000 files. Is the embedding model load time (BAAI/bge-small-en-v1.5) the bottleneck or the LanceDB write operations? Would a lighter embedding model (e.g. MiniLM) or pre-computed embeddings stored in git change the equation?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

The embedding model — not LanceDB — is the bottleneck in cold-start index rebuilds: LanceDB write operations consume under 1% of total startup time at all tested corpus sizes. BAAI/bge-small-en-v1.5 takes 11.5s to rebuild the current 61-item research corpus from Markdown text (10.8s embedding, 0.68s model load), already exceeding the 5–10s target. Pre-computed embeddings stored as JSON in git reduce startup to under 0.2s regardless of corpus size, making stateless deployment viable immediately. Model2Vec (identified in prior semantic-search research as ~200× faster than MiniLM on CPU) would enable per-request rebuild without pre-computed storage and should be evaluated as the production embedding model.

Key Findings

  1. LanceDB write operations take under 30ms even at 1000 documents; the embedding model accounts for 99%+ of cold-start rebuild time at every tested corpus size. [high]

  2. BAAI/bge-small-en-v1.5 loads from disk cache in 0.68s; the model file is 133 MB, giving a first-download load time of ~2.25s from HuggingFace Hub on a typical connection. [high]

  3. BGE-small-en-v1.5 embeds at ~34.5ms/doc for short synthetic texts but ~177ms/doc for real research items; BERT's O(n²) attention cost means document length — not document count — is the primary driver of embedding time. [high]

  4. At the current corpus size of 61 items with full Markdown text, cold-start rebuild takes 11.5s with BGE-small, already exceeding the 5–10s target and with no improvement path as the corpus grows. [high]

  5. Pre-computed embeddings stored as JSON in git (7.74 KB per 384-dim document) load and write into LanceDB in under 0.2s for 1000 documents, making startup time effectively O(1) with respect to corpus size. [high]

  6. JSON storage cost for pre-computed BGE-small embeddings is 7.74 MB for 1000 documents; for a personal research corpus of 200–300 items, this is under 2.5 MB, an acceptable git repository size. [high]

  7. all-MiniLM-L6-v2 is approximately 2× faster than BGE-small at embedding (16.4ms/doc vs 34.5ms/doc for short texts) but does not change the fundamental scaling failure: at 500 real research items, MiniLM would still require over 20s to rebuild. [high]

  8. GitHub code search is rate-limited to 10 requests per minute, is keyword-only with no semantic understanding, and may return incomplete results for large queries; it is not a viable replacement for vector search and only useful as a supplementary lexical search layer. [high]

  9. Model2Vec (potion-base-8M), identified in prior semantic-search research as ~200× faster than MiniLM on CPU with 91–93% of MiniLM's MTEB accuracy and no PyTorch dependency, would reduce rebuild time to under 1s for 500 items — making per-request stateless rebuild viable without pre-computed embedding storage. [medium — requires direct LanceDB benchmark to confirm]

Assumptions

Analysis

The benchmark data makes a clear recommendation. The rebuild-from-text approach fails the target at the current corpus size and degrades further as the corpus grows — there is no configuration of the current stack (batch size, model choice within bge/MiniLM family) that fixes this without switching to a fundamentally faster embedding approach.

Pre-computed embeddings eliminate the bottleneck at the cost of a model-version coupling constraint (all stored embeddings must be regenerated if the model changes). For a single-owner personal project, this cost is low and manageable. The storage overhead (~2.5 MB for 300 items) is negligible in git terms.

Model2Vec represents a possible third path — one that avoids both the storage overhead of pre-computed embeddings and the latency of bge-small/MiniLM. Given that prior research already recommends Model2Vec for Phase 2 of the semantic search system, evaluating it for the LanceDB rebuild path simultaneously would be efficient. The recommendation is to pursue both paths in parallel: implement pre-computed embeddings as the near-term fix (eliminates the latency problem immediately), and benchmark Model2Vec against LanceDB as an input to the production embedding model decision.

Risks, Gaps, and Uncertainties

Open Questions

  1. What is Model2Vec (potion-base-8M) actual rebuild time for the current 61-item research corpus? Should become a backlog item for the Memory-System.
  2. Should embeddings be stored as JSON (human-readable, diffable in GitHub web UI) or numpy binary (5× smaller)? Decision depends on operational tooling preferences.
  3. How should the add_memory write path in mcp_server.py be modified to persist embeddings as a JSON sidecar alongside each .md file? This is the concrete implementation question for W-0015.
  4. What is the RAM footprint of Model2Vec on a 256 MB Fly.io instance during inference?

sources

← newerEmergent Patterns in Software Engineering Prompts and SDLC Guidance
older →iOS Shortcuts + GitHub API

Connected items

Loading…

View full knowledge graph →