Research / Latent Concept Extraction from Confluence

Latent Concept Extraction from Confluence

Latent Concept Extraction from Confluence: Embeddings, Knowledge Graphs, and Epistemic Evaluation

2026-03-16 · rag-retrieval knowledge-graphs knowledge-management ai-architecture · medium · source → · wiki →
key claims
  1. BERTopic's four-stage pipeline (transformer embeddings → UMAP dimensionality reduction → HDBSCAN clustering → c-TF-IDF topic representation) outperforms LDA on topic coherence metrics for longer-form text and does not require pre-specifying the number of topics, making it appropriate for unsupervised topic discovery across Confluence wiki pages without prior domain knowledge of the corpus. Confidence: high
  2. BERTopic and NER (via mREBEL or spaCy) are structurally complementary and should be run in parallel: BERTopic extracts latent topical clusters for document-level metadata tagging; NER extracts explicit named entities and typed relations for knowledge graph population; neither technique alone provides both topic-level and entity-level concept representation. Confidence: high
  3. The MTEB leaderboard is the standard selection criterion for embedding models; `all-mpnet-base-v2` and INSTRUCTOR models from SBERT are strong production baselines for technical organisational prose, and domain adaptation via TSDAE or GPL adds retrieval quality only worth its engineering cost when a held-out Confluence evaluation set confirms measurable degradation. Confidence: high
  4. Confluence's native hierarchical structure (spaces → page trees → section headings) should guide chunking: section-boundary splitting with Confluence hierarchy as parent metadata enables hierarchical retrieval, with factoid queries served at 256–512 token chunks and analytical queries served by merging to 1,024+ token parent chunks, based on NVIDIA's 2024 chunking benchmark results across five datasets. Confidence: medium
  5. HybridRAG (arXiv:2408.04948, 2024) outperforms both VectorRAG and GraphRAG individually on retrieval accuracy and answer quality, confirming that a VDB and knowledge graph are structurally complementary: the VDB handles semantic similarity retrieval; the KG handles precise entity-relationship queries; the hybrid pattern uses vector search to identify knowledge graph entry nodes and graph traversal for relational context. Confidence: high
  6. Weaviate is the optimal single-system store for Confluence-to-KG pipelines because it natively supports built-in hybrid search (BM25 + vector), object classes aligning with KG node types, and embedded vectorisation of graph nodes - eliminating the operational cost of running a separate VDB and a separate graph database concurrently. Confidence: medium
  7. Qdrant achieves 50.3% lower p50 query latency than pgvector at 90% recall (4.74ms vs. 9.54ms) and is the optimal standalone high-performance vector search store; pgvector achieves 11.4× higher throughput at 99% recall (471.57 vs. 41.47 QPS) and is optimal when PostgreSQL is already the organisation's data platform. Confidence: high
  8. mREBEL - a joint NER and relation extraction model trained on Wikipedia–Wikidata aligned pairs - generates entity-relation triples without a predefined ontology and achieves 1.8× the triple coverage of traditional rule-based extraction approaches, enabling incremental knowledge graph construction from Confluence pages as they are indexed rather than requiring an upfront full-corpus batch process. Confidence: medium

Research Question

What are the best approaches for extracting latent concepts from a Confluence wiki, representing them as word embeddings in a vector database (VDB) and as a knowledge graph (KG), and how can the resulting knowledge base be evaluated for truth and utility - and is there a meaningful distinction between the two?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

[inference] BERTopic combined with NER + relation extraction (mREBEL) is the current best-practice approach for latent concept extraction from a Confluence wiki; these techniques are complementary and should be run in parallel, not chosen between. For the embedding layer, SBERT models (all-mpnet-base-v2 or INSTRUCTOR variants) selected via the MTEB leaderboard are the correct starting point, with domain adaptation reserved for confirmed retrieval degradation on Confluence-specific evaluation data. The VDB choice depends on existing infrastructure: Weaviate for combined VDB + KG pipelines, Qdrant for standalone vector search, pgvector if PostgreSQL is already in use. The truth/utility distinction is practically meaningful and maps directly to four required metadata fields - provenance, approval status, recency signal, and superseded-by graph edge - whose absence makes a Confluence RAG system epistemically undefendable in regulated contexts.

Key Findings

  1. BERTopic's four-stage pipeline (transformer embeddings → UMAP dimensionality reduction → HDBSCAN clustering → c-TF-IDF topic representation) outperforms LDA on topic coherence metrics for longer-form text and does not require pre-specifying the number of topics, making it appropriate for unsupervised topic discovery across Confluence wiki pages without prior domain knowledge of the corpus. Confidence: high.

  2. BERTopic and NER (via mREBEL or spaCy) are structurally complementary and should be run in parallel: BERTopic extracts latent topical clusters for document-level metadata tagging; NER extracts explicit named entities and typed relations for knowledge graph population; neither technique alone provides both topic-level and entity-level concept representation. Confidence: high.

  3. The MTEB leaderboard is the standard selection criterion for embedding models; all-mpnet-base-v2 and INSTRUCTOR models from SBERT are strong production baselines for technical organisational prose, and domain adaptation via TSDAE or GPL adds retrieval quality only worth its engineering cost when a held-out Confluence evaluation set confirms measurable degradation. Confidence: high.

  4. Confluence's native hierarchical structure (spaces → page trees → section headings) should guide chunking: section-boundary splitting with Confluence hierarchy as parent metadata enables hierarchical retrieval, with factoid queries served at 256–512 token chunks and analytical queries served by merging to 1,024+ token parent chunks, based on NVIDIA's 2024 chunking benchmark results across five datasets. Confidence: medium.

  5. HybridRAG (arXiv:2408.04948, 2024) outperforms both VectorRAG and GraphRAG individually on retrieval accuracy and answer quality, confirming that a VDB and knowledge graph are structurally complementary: the VDB handles semantic similarity retrieval; the KG handles precise entity-relationship queries; the hybrid pattern uses vector search to identify knowledge graph entry nodes and graph traversal for relational context. Confidence: high.

  6. Weaviate is the optimal single-system store for Confluence-to-KG pipelines because it natively supports built-in hybrid search (BM25 + vector), object classes aligning with KG node types, and embedded vectorisation of graph nodes - eliminating the operational cost of running a separate VDB and a separate graph database concurrently. Confidence: medium.

  7. Qdrant achieves 50.3% lower p50 query latency than pgvector at 90% recall (4.74ms vs. 9.54ms) and is the optimal standalone high-performance vector search store; pgvector achieves 11.4× higher throughput at 99% recall (471.57 vs. 41.47 QPS) and is optimal when PostgreSQL is already the organisation's data platform. Confidence: high.

  8. mREBEL - a joint NER and relation extraction model trained on Wikipedia–Wikidata aligned pairs - generates entity-relation triples without a predefined ontology and achieves 1.8× the triple coverage of traditional rule-based extraction approaches, enabling incremental knowledge graph construction from Confluence pages as they are indexed rather than requiring an upfront full-corpus batch process. Confidence: medium.

  9. The truth/utility distinction in an organisational knowledge base is both philosophically grounded in Goldman's veritistic social epistemology and practically meaningful: a Confluence document can simultaneously be historically accurate, operationally stale, and utility-negative when retrieved in a current compliance query - three states that require distinct metadata fields to distinguish at retrieval time. Confidence: high.

  10. The truth/utility distinction is operationalised through four metadata fields per indexed chunk: provenance (Confluence page ID, author, space, creation date), approval status (approved/team/personal, mapping to high/medium/low confidence), recency signal (days since last edit, staleness flag above a threshold), and superseded-by relationship as a knowledge graph edge pointing to the successor document. Confidence: high (inference grounded in veritistic epistemology + knowledge governance literature).

  11. Knowledge corpus governance - accuracy, currency, and ownership of source documents - is the primary determinant of retrieval quality in any RAG system built over Confluence, independently confirmed by two completed research items in this repository, and cannot be substituted by any combination of BERTopic, MTEB-leading embedding models, or hybrid VDB + KG retrieval architecture. Confidence: high.

  12. Late chunking (Jina AI, 2024) embeds full documents before splitting so each chunk carries context from the entire page, potentially resolving pronoun and reference problems across chunk boundaries in Confluence pages where context is established in the introduction; this technique has not been benchmarked on structured wiki content and should be monitored for future adoption. Confidence: low (novel technique, single source, no Confluence-specific benchmark).

Assumptions

Analysis

The central design tension in a Confluence concept extraction architecture is between completeness and precision. The VDB optimises for completeness through broad semantic retrieval; the KG optimises for precision through typed entity-relationship queries. Neither resolves the epistemics problem alone: the VDB retrieves semantically similar content regardless of currency; the KG models relationships regardless of whether the underlying claims are still accurate.

The epistemic metadata layer - provenance, approval status, recency signal, and superseded-by - is the bridge between the technical retrieval infrastructure and the philosophical distinction between truth and utility. Without this layer, the system cannot distinguish a deprecated policy document from a current one; RAG responses inherit this blindness. With this layer, retrieval can be filtered by approval status and freshness before ranking by semantic similarity.

The practical implication for implementation priority is: build provenance metadata collection into the Confluence ingestion pipeline before the VDB goes into production, not after. Retrofitting provenance metadata to an existing index is more expensive than collecting it at ingestion time.

The BERTopic + NER combination reflects a separation of concerns that matches the downstream consumption pattern: BERTopic topics serve the navigation and discovery use case (what topics does the wiki cover? which pages are about X?); the knowledge graph serves the reasoning use case (what entities are related to X? who owns policy Y?). These are different consumer experiences served by the same ingestion pipeline.

Risks, Gaps, and Uncertainties

Open Questions

  1. At what Confluence wiki scale does the knowledge graph cold-start become a delivery blocker vs. a manageable incremental build? Candidate new backlog item; priority: medium.
  2. What is the minimum viable epistemic metadata schema for a Confluence RAG system to be defensible in a regulated (financial services, healthcare) compliance context? Priority: high if applied in regulated industries - candidate new backlog item.
  3. Does BERTopic's incremental online learning maintain topic coherence as Confluence wikis evolve over months with shifting topic distributions (new products, regulatory changes)? Priority: medium - candidate empirical study.
  4. Can late chunking (Jina AI 2024) be applied to Confluence pages with mixed structured/prose content, and what is its recall improvement on Confluence-specific retrieval benchmarks? Priority: low.

sources

← newerAligned Decision-Making
older →Adam Smith, Organisational Design, Desire Paths, and AI Strategy

Connected items

Loading…

View full knowledge graph →