Context Compression and RAG Techniques for Organisational Knowledge

Research Question

What are the current best practices and bleeding-edge techniques - including Retrieval-Augmented Generation (RAG), context compression, and context architecture - for selectively surfacing the right slice of a large organisational knowledge corpus (regulations, policies, strategy, current state) into a Large Language Model (LLM) context window at decision time?

Findings

(Populated from §6 Synthesis above.)

Executive Summary

[inference] Advanced Retrieval-Augmented Generation (RAG) - combining hybrid search, re-ranking, hierarchical indexing (Recursive Abstractive Processing for Tree-Organized Retrieval (RAPTOR)), and modular pipeline architecture - is the current best practice for surfacing organisational knowledge into a Large Language Model (LLM) context window at decision time. Extending the context window to 1M tokens does not substitute for structured retrieval; [fact] NVIDIA's RULER benchmark (2024) confirms all models degrade significantly on complex reasoning tasks with increasing context length. (Source: arXiv:2404.06654) [inference] LLMLingua-2 (4x compression, 3–6x faster than its predecessor) and LongLLMLingua (question-aware multi-document compression) are the best available tools for reducing retrieved context volume. The primary unsolved challenge is governance of the underlying knowledge corpus: retrieval quality is bounded by source document quality, and no retrieval technique compensates for outdated or contradictory organisational knowledge.

Key Findings

1. Naive RAG fails for hierarchical organisational knowledge because a single cosine-similarity query cannot distinguish regulatory, strategic, and operational knowledge tiers; Advanced RAG with hybrid search and cross-encoder re-ranking is the minimum viable baseline for multi-tier retrieval.

2. Extending the LLM context window to 1M tokens does not eliminate the need for structured retrieval; NVIDIA's RULER benchmark (arXiv:2404.06654, 2024) shows all tested models degrade significantly on complex reasoning and aggregation tasks as context length increases, with effective performance failing at 60–70% of the advertised window.

3. LLMLingua-2 (ACL 2024, arXiv:2403.12968) is a task-agnostic prompt compression method that runs 3–6x faster than the original LLMLingua and accelerates end-to-end latency by 1.6–2.9x at 2–5x compression ratios, making it the most production-ready context compression tool currently available.

4. LongLLMLingua, a question-aware extension of LLMLingua, achieves ~75% accuracy on NaturalQuestions at 3.9x token compression across 20-document retrieval tasks - the most relevant compression technique for surfacing organisational knowledge at query time.

5. RAPTOR (ICLR 2024, arXiv:2401.18059) builds a hierarchical tree of text summaries through recursive clustering and abstractive summarisation, achieving 82.6% accuracy on the QuALITY benchmark with GPT-4 versus 62.3% for prior state-of-the-art, a 20+ percentage point improvement on multi-document reasoning tasks.

6. Modular RAG with tier-specific routing logic is the correct architectural pattern for multi-layer organisational knowledge because it separates query routing (which knowledge tier to query) from retrieval quality (how to search within a tier), eliminating cross-tier noise that degrades Naive RAG responses.

7. Microsoft GraphRAG enables relationship-aware retrieval by building a knowledge graph from unstructured text with hierarchical community summaries, but its indexing cost is approximately 100–1000x higher than vector RAG, making it appropriate only when inter-entity relationships are primary to the decision query.

8. LlamaIndex provides the best out-of-the-box hierarchical retrieval primitives (HierarchicalNodeParser, AutoMergingRetriever, and structured auto-retrieval with metadata filters) for implementing a tiered organisational knowledge architecture, while Haystack offers the strongest production-reliability guarantees for regulated-industry deployments.

9. The RAGAS (RAG Assessment) framework is the dominant open-source evaluation standard, measuring faithfulness, context precision, context recall, and answer relevance, but its correlation with manual human evaluation reaches only ~0.55 harmonic mean - sufficient for continuous monitoring but insufficient as a sole quality gate for compliance-critical decisions.

10. Governance of the source knowledge corpus - specifically, freshness verification, contradiction resolution, and ownership assignment for each document - is the primary determinant of RAG retrieval quality and cannot be substituted by any combination of retrieval or compression techniques.

11. Summarisation-based compression (map-reduce) applied offline at indexing time is appropriate for stable, slow-changing documents such as regulations and strategy papers; token-pruning compression using LLMLingua-2 is more appropriate for dynamic context injected at query time where offline processing is not viable.

12. MemGPT/Letta's three-tier memory model (core memory always in context, recall memory for recent history, archival memory for long-term storage) is the closest operational approximation of a layered organisational context architecture, but it lacks the access control, provenance tracking, and audit trail capabilities required for compliance-sensitive enterprise deployments.

Assumptions

• Assumption: Regulatory and policy documents contain sufficient natural language redundancy for LLMLingua-style compression at 4x with acceptable quality loss. Justification: LLMLingua achieves 20x compression on reasoning benchmarks (GSM8K, BBH) with 1.5% loss; Shannon (1951) established general natural language redundancy; policy text's defined terms, cross-references, and boilerplate represent structured redundancy typically higher than general prose. This is an extrapolation - direct evidence for regulatory text specifically is absent.

• Assumption: The "lost in the middle" performance degradation documented in benchmarks applies to organisational document retrieval in the same way as measured for needle-in-a-haystack and multi-hop reasoning tasks. Justification: The effect has been documented consistently across multiple model families and task types (RULER benchmark, community reports); no counter-evidence found for structured policy document tasks specifically. Applies until domain-specific benchmarks show otherwise.

• Assumption: LlamaIndex hierarchical primitives (HierarchicalNodeParser, AutoMergingRetriever) can be configured to implement a full four-tier regulatory/strategy/policy/operational architecture without fundamental limitation of the framework. Justification: The primitives support arbitrary metadata tagging and routing by metadata label; the tier configuration is a software engineering implementation task, not a research-level gap. Confirmed by documentation review.

Analysis

The central tension in context management for organisational decision support is completeness versus quality. Extending the context window resolves availability but worsens reasoning quality and incurs prohibitive cost at scale. The evidence from the RULER benchmark and production reports is clear: adding more undifferentiated context degrades the model's ability to reason about what is relevant.

Structured retrieval resolves this tension by selecting relevant content before it reaches the model. Advanced RAG techniques (hybrid search, re-ranking) address retrieval precision within a knowledge tier. Modular RAG with routing logic addresses tier selection. RAPTOR addresses cross-level summarisation. Each technique attacks a specific failure mode; they compose rather than substitute.

Context compression (LLMLingua-2, LongLLMLingua) is complementary to structured retrieval, not an alternative. The correct pipeline: (i) route query to relevant tier(s); (ii) retrieve within tier using Advanced RAG; (iii) apply compression to retrieved chunks before injection. This reduces both context noise and token cost.

GraphRAG occupies a genuine niche for relationship-heavy queries - when the question is "which regulation governs which process" rather than "what does this regulation say" - but is not a general-purpose replacement. The indexing cost discourages broad adoption.

The governance gap is the most important finding that the technical literature underweights. Every retrieval system is bounded by source quality. The historical pattern (enterprise search, SharePoint, Confluence) shows that organisations repeatedly invest in retrieval technology while neglecting knowledge governance, producing the same failure mode under each successive technology generation.

Risks, Gaps, and Uncertainties

• No benchmark for RAPTOR on structured policy/regulatory documents. The ICLR 2024 results use literary and general knowledge corpora. Transfer to structured, hierarchically-organised organisational knowledge is an inference.
• RAGAS is an approximation. For compliance-critical deployments, ~0.55 correlation with human evaluation is insufficient as the sole quality gate. Domain-specific human evaluation benchmarks are required.
• GraphRAG cost figure is from a single practitioner source (citing Microsoft Research). The 100–1000x range is plausible but not independently corroborated at the specific multiple.
• LLMLingua compression on regulatory text is unvalidated. The 20x claim applies to reasoning benchmarks; regulatory/policy text may compress more or less favourably. Direct experimentation is required before relying on this in production.
• Governance prerequisite is under-documented. No academic framework provides a structured methodology for assessing knowledge corpus governance readiness for RAG deployment. Practitioners work from ad hoc checklists.
• Multi-tier metadata tagging discipline. A tiered RAG system only provides routing benefits if source documents are accurately and consistently tagged with their knowledge tier. The operational challenge of maintaining this taxonomy at scale is not addressed in the literature.

Open Questions

1. Does explicitly tiered RAG - with regulatory/strategy/policy/operational metadata labels and routing logic - measurably outperform flat vector search over the same corpus on compliance-relevant queries? (→ warrants experimental validation; could become a new backlog item)
2. What provenance and audit trail infrastructure is required alongside a RAG system for retrieved context to be considered auditable evidence in a regulated financial services environment?
3. What compression ratio is achievable on standard regulatory texts (e.g., Basel III, MiFID II, PRA SS10/18) using LLMLingua-2 before meaningful content loss occurs?
4. Can the MemGPT/Letta memory architecture be extended with access control and provenance tracking to serve as an enterprise-grade organisational knowledge agent without replacing the underlying framework?

sources

• [x] "Retrieval-Augmented Generation for Large Language Models: A Survey"
• [x] LLMLingua: Compressing Prompts for Accelerated Inference (correct arXiv ID; item had 2310.06025 which resolves to a different paper)
• [x] MemGPT: Towards LLMs as Operating Systems
• [x] RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
• [x] LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
• [x] LlamaIndex documentation on advanced RAG patterns
• [x] LangChain RAG conceptual overview
• [x] Microsoft GraphRAG
• [x] Advanced RAG Techniques (Weaviate)