Legal Bench RAG

– Hruthika (Junior AI Engineer @ DevDolphins), Ajinkya (AI Lead @ DevDolphins)

We understand that legal document retrieval isn’t just another data problem – it’s a complex undertaking fraught with unique challenges; dense structures, intricate cross-references, and context that often stretches across multiple sections or even entirely separate files. Traditional RAG (Retrieval Augmented Generation) approaches, while powerful, often stumble, inadvertently fragmenting crucial dependencies. This can lead to incomplete or, misleading outputs – a risk simply unacceptable in the high-stakes world of legal analysis. 

That’s why we embarked on a focused journey, experimenting with various RAG strategies to unearth architectures that could truly preserve context, all while meticulously balancing performance, cost, and complexity. It wasn’t about finding a single silver bullet, but rather a robust toolkit. 

To truly put these strategies to the test, we designed a rigorous evaluation framework. We took a corpus of 100 diverse legal documents – like notices, contracts and judgements – and paired them with approximately 200 synthetic queries. These queries, generated using the Ragas framework, were designed to push the boundaries, spanning: 

• Simple fact lookups: ”What is the notice period?” 

• Reasoning tasks: Questions requiring inference across multiple clauses. 

• Multi-context synthesis: Queries demanding information pulled from disparate sections or even different documents. 

To ensure a fair comparison, we built a modular pipeline. This meant consistent ingestion via semantic chunking and an Unstructured parser, robust FAISS vector storage, and a dual-pronged tech stack: one OpenAI-based and another open-source, leveraging Qwen embeddings and Jina rerankers. Our performance benchmarks relied on Ragas metrics, focusing on faithfulness, relevance, and overall answer correctness.

We delved into six distinct RAG architectures: 

• The Baseline: Our starting point used fixed-size 512-character chunks with a sliding overlap, coupled with direct vector similarity search. It proved surprisingly solid for basic queries, achieving 52-53% answer correctness at minimal cost. 

• Summary Indexing: Here, an LLM generated concise summaries of chunks for embedding. This often improved semantic matching but, occasionally, at the cost of obscuring those critical, fine-grained legal details. Still, it offered a decent uplift (+3-6% over baseline). 

• Contextual Embeddings: This involved adding LLM-generated prefixes to chunks – imagine “This is the Indemnity clause from Contract X, Section 5.” This approach injected self-contained semantic richness, particularly shining in multi-document synthesis scenarios. 

• Hybrid Retrieval: We combined the power of contextual embeddings with BM25 lexical search, using an 80/20 semantic weighting via reciprocal rank fusion. This move significantly boosted recall, particularly for those niche, domain-specific legal terms, by an impressive 18-20%. 

• Reranking Variants: Finally, we explored reranking. This involved over-retrieving candidates and then filtering them either with an LLM (like GPT-4o-mini) or specialized neural cross-encoders (such as Jina v3). The latter, in particular, was a game-changer, pushing correctness to 63% with an exceptional 92% context precision. 

Our most advanced setup – a blend of contextual embeddings, hybrid retrieval, and a specialized reranker – delivered the strongest results. It achieved a 63% answer correctness, 92% precision, and 84% recall, truly dominating those complex multi-context questions with a 68% success rate. The baseline, as mentioned, remained remarkably robust for simple factual queries, while the combination of summaries and reranking proved excellent for precise lookups, hitting 72% correctness. 

One standout observation: general-purpose LLMs underperformed as rerankers when compared to models specifically tuned for legal domains. This underscores a crucial point – in legal tech, specialization often trumps generalization. 

Here’s a snapshot of our findings: 

Strategy	Answer correctness	Context precision	Context precision	Relative cost
Baseline	52-53%	70-75%	Simple facts, Budget-sensitive	1x
Summary + Rerank	55-62%	72-83%	Fact lookups	2x
Contextual + Hybrid + Rerank	63%	92%	Complex synthesis	3x

Of course, innovation often comes with a price tag, and understanding the cost tradeoffs was paramount. Our token analysis revealed the baseline’s impressive efficiency, requiring around 3.5 million total tokens for both indexing and 200 queries. In contrast, summary strategies climbed to 7.7 million, and contextual approaches exceeded 10 million tokens due to the upfront LLM preprocessing. 

While these advanced setups certainly amortize their cost over high query volumes, their costs can be triple that of the baseline. This tells us that the baseline remains a strong contender for exploratory searches or high-volume applications where cost sensitivity is key. Ultimately, our decision framework helps map the right choice based on factors like query profile (simple facts versus complex synthesis), desired accuracy thresholds (aiming for 50-60%+), anticipated volume, and the team’s internal expertise.

For a more granular look at our methodology, a detailed introduction to each strategy, comprehensive plots, and token breakdowns, we invite you to explore our Medium articles here and our GitHub repo here. We believe this work lays a solid foundation for more intelligent and reliable legal AI