The six RAFT models plus a three-model ablation, all behind a real retriever: every question first hits a FAISS vector index of the full training corpus (~11M chunks, bge-small embeddings), and the model answers from the retrieved chunks. Same held-out questions and same reference-grounded judge as SLM Arena 21, so the scores are directly comparable: models 22-27 of the family. The bottom row is the control experiment: the closed-book QA-SFT models with retrieved context inlined into their prompt, models that were never taught to read context, so you can watch what RAG does and does not fix.
Loading judged results...
| Model | Bare RAFT /10 | + RAG /10 | QA | Instr | RAFT | Refusal* | Δ |
|---|
* For RAG models the four refusal items are scored against the true answer, since a real retriever may legitimately find the source document; for bare RAFT they were scored against the expected refusal. Same judge, same gold answers everywhere else.
In the 21-model arena, the RAFT models saw an empty context on all 58 closed-book questions; trained to answer only from provided context, they refused, and their scores said more about honesty than about capability. Here the missing piece is installed: a retriever over the corpus the models were trained on.
The corpus (~670k documents, ~9.6B characters of case law, SEC filings, and educational web text) is chunked into ~11 million 1,000-character windows with 150-character overlap. Each chunk is embedded with bge-small-en-v1.5 and stored in a FAISS inner-product index. At question time the query is embedded, the top chunks come back with provenance (source, shard, document, offset), and the RAFT model receives them as its context passages: exactly the input shape it was trained on, golden-plus-distractors. The from-scratch models get 3 chunks trimmed to 800 characters to respect their 1,024-token context; Gemma gets 4 full chunks.
Same 119 held-out items, same blind pointwise judge, same gold answers. Two honest adjustments: RAFT items are judged as plain QA (the model retrieves its own context now, so the curated distractor set no longer describes what it saw), and the four refusal items are scored against the stored true answer, because with a real retriever finding the source document is a legitimate win rather than a failure to refuse.
RAFT training taught these models to extract answers from context and ignore irrelevant passages (they score 8.4-9.1 when the right passage is present). Retrieval's job is only to make that condition true. The leaderboard tab shows exactly how much of the closed-book gap the retriever closes, per model.