RAG Citations & Retrieval Evaluation

A RAG answer can go wrong in two separate places. Retrieval can miss the passage that holds the answer, or generation can ignore, twist or invent beyond the passages it was given. If you only look at final answers, you cannot tell which half failed, so you cannot fix it.

In this lesson you will measure retrieval on its own with two standard numbers, hit rate and MRR, then check the answers: whether every citation points to a real retrieved source, whether quoted text really appears there, and whether each sentence is supported at all.

A labelled retrieval test set

To measure retrieval you need questions with known answers, written down as the ids of the chunks that contain each answer. Build it the same way as any test set (see Evaluating AI Outputs): real questions from users, rephrased versions, questions that use different words from the document, and some questions the documents cannot answer at all.

Two numbers summarise a retriever:

• Hit rate at k (also called recall at k for single-answer questions): the share of questions where a correct chunk appears in the top k results.
• Mean reciprocal rank (MRR): for each question, take 1 divided by the position of the first correct chunk (1 for first place, 0.5 for second, 0.33 for third, 0 if missing), then average. It rewards putting the right chunk near the top.

TESTS = {   # question -> ids of chunks that answer it
    "How many earned leave days do I get?": {"leave-1"},
    "Can I carry casual leave forward?": {"leave-2"},
    "Which train class can an M2 employee book?": {"travel-1"},
    "What is the hotel limit in Pune?": {"travel-3"},
}
RESULTS = {  # what the retriever returned, best first
    "How many earned leave days do I get?": ["leave-1", "leave-2", "travel-1"],
    "Can I carry casual leave forward?": ["leave-1", "leave-2", "leave-3"],
    "Which train class can an M2 employee book?": ["travel-2", "travel-3", "leave-1"],
    "What is the hotel limit in Pune?": ["travel-3", "travel-2", "leave-2"],
}

def evaluate(tests, results, k=3):
    hits, rr_total = 0, 0.0
    for question, relevant in tests.items():
        ranked = results[question][:k]
        rank = next((i for i, cid in enumerate(ranked, start=1) if cid in relevant), None)
        hits += rank is not None
        rr_total += 1 / rank if rank else 0
        print(f"rank {str(rank):<4} {question}")
    print(f"hit rate@{k}: {hits / len(tests):.2f}   MRR: {rr_total / len(tests):.2f}")

evaluate(TESTS, RESULTS)

rank 1    How many earned leave days do I get?
rank 2    Can I carry casual leave forward?
rank None Which train class can an M2 employee book?
rank 1    What is the hotel limit in Pune?
hit rate@3: 0.75   MRR: 0.62

The train-class question failed completely: the right chunk is not in the top three, so no prompt can fix that answer. Now you know where to work. Perhaps "M2" never appears in the travel chunk (it says "below grade M3"), which is a case for better chunk text or keyword search. Run these numbers every time you change chunk size, the embedding model, k or the search method, and keep the change only if they improve.

Citations your code can check

In the RAG lesson the prompt asked the model to cite source ids like [S2]. Citations are only useful if something checks them. Your code can verify three things cheaply:

1. Every cited id is one of the sources actually sent in the prompt.
2. Every sentence with a factual claim has at least one citation.
3. Any text in quotation marks appears word for word in the cited source.

import re

SOURCES = {
    "S1": "Employees get 18 days of earned leave per year. Unused days carry forward up to 45 days.",
    "S2": "Casual leave is 8 days per year and cannot be carried forward.",
}

def check_citations(answer, sources):
    problems = []
    sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]
    for sentence in sentences:
        ids = re.findall(r"\[(S\d+)\]", sentence)
        if not ids:
            problems.append(f"no citation: {sentence!r}")
        for sid in ids:
            if sid not in sources:
                problems.append(f"unknown source {sid}")
        for quote in re.findall(r'"([^"]+)"', sentence):
            if not any(quote in sources.get(sid, "") for sid in ids):
                problems.append(f"quote not found in cited source: {quote!r}")
    return problems or ["all citations check out"]

good = 'You get 18 days of earned leave a year [S1]. Casual leave "cannot be carried forward" [S2].'
bad = ('You get 20 days of earned leave [S1]. Casual leave "can be carried forward" [S2]. '
       'Sick leave is 12 days [S4]. Ask HR for details.')
print(check_citations(good, SOURCES))
for p in check_citations(bad, SOURCES):
    print(p)

['all citations check out']
quote not found in cited source: 'can be carried forward'
unknown source S4
no citation: 'Ask HR for details.'

The checker caught a twisted quote, an invented source and an uncited sentence. Notice what it missed: "20 days" with a valid citation [S1] passed, although the source says 18. Checking that a citation exists is not the same as checking that it supports the claim.

Is each sentence supported?

To catch "20 days [S1]", you need a groundedness (or faithfulness) check: does the cited source actually support the sentence? The strongest method is an AI judge given the sentence and the source, with the care described in LLM Evaluation: Regression Tests & AI Judges. A cheap first filter compares the numbers and important words in a sentence with its source:

import re

def unsupported_numbers(sentence, source):
    sentence_numbers = set(re.findall(r"\d+(?:[.,]\d+)*", re.sub(r"\[S\d+\]", "", sentence)))
    source_numbers = set(re.findall(r"\d+(?:[.,]\d+)*", source))
    return sorted(sentence_numbers - source_numbers)

source = "Employees get 18 days of earned leave per year. Unused days carry forward up to 45 days."
for sentence in ["You get 18 days of earned leave a year [S1].",
                 "You get 20 days of earned leave [S1].",
                 "Up to 45 days can be carried forward, so 63 days in total [S1]."]:
    missing = unsupported_numbers(sentence, source)
    print("OK  " if not missing else f"FLAG {missing}", sentence)

OK   You get 18 days of earned leave a year [S1].
FLAG ['20'] You get 20 days of earned leave [S1].
FLAG ['63'] Up to 45 days can be carried forward, so 63 days in total [S1].

The second sentence is caught, as hoped. The third is more interesting: "63 days" is the model's own sum (18 plus 45), which the source never states, and it may well be wrong, because the 45-day cap could already include the current year's leave. Flagging it is the right decision: a policy bot should quote rules and leave calculations to HR. This number check is crude (it ignores words, so "can" against "cannot" passes), which is why teams add an AI judge for sentences that pass the cheap checks.

Citations built into the API

Some APIs can produce citations for you. With Anthropic's citations feature, you send each source as a document content block with citations enabled, and the reply comes back split into text blocks, where cited blocks carry the exact cited_text and its location in your document. Your code no longer has to trust the model to copy ids and quotes correctly:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

response = client.messages.create(
    model="claude-haiku-5-5",
    max_tokens=1000,
    messages=[{
        "role": "user",
        "content": [
            {"type": "document",
             "source": {"type": "text", "media_type": "text/plain", "data": leave_policy_text},
             "title": "Leave Policy 2026",
             "citations": {"enabled": True}},
            {"type": "text", "text": "Can I carry casual leave forward?"},
        ],
    }],
)
for block in response.content:
    if block.type == "text":
        print(block.text, [c.cited_text for c in (block.citations or [])])

According to Anthropic's documentation, this feature cannot be combined with structured output mode in the same request, so choose one per call. See the citations guide for the exact response fields.

Common mistakes

• Judging RAG only by final answers, so retrieval and generation failures look the same.
• No unanswerable questions in the test set, so you never measure whether the bot says "not found".
• Showing citations without checking them. Unknown ids and invented quotes slip through.
• Treating "has a citation" as "is correct". A real source id can sit next to a wrong number.
• Changing chunking or k without rerunning hit rate and MRR.

Key points

• Evaluate retrieval separately from generation, with a labelled set of questions and correct chunk ids.
• Hit rate at k shows whether the right chunk is found; MRR shows how high it is ranked.
• Check in code that cited ids exist, factual sentences are cited and quotes match the source.
• Groundedness checks compare each claim with its source: cheap number checks first, AI judges for the rest.
• Built-in API citations return the exact cited text and location, so you rely less on the model copying ids.

import re

SOURCES = {
    "S1": "Employees get 18 days of earned leave per year. Unused days carry forward up to 45 days.",
    "S2": "Casual leave is 8 days per year and cannot be carried forward.",
}

def unsupported_numbers(sentence, source):
    sentence_numbers = set(re.findall(r"\d+(?:[.,]\d+)*", re.sub(r"\[S\d+\]", "", sentence)))
    source_numbers = set(re.findall(r"\d+(?:[.,]\d+)*", source))
    return set(sentence_numbers - source_numbers)

def check_citations(answer, sources):
    problems = []
    sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", answer) if s.strip()]
    for sentence in sentences:
        ids = re.findall(r"\[(S\d+)\]", sentence)
        if not ids:
            problems.append(f"no citation: {sentence!r}")
        known = [sid for sid in ids if sid in sources]
        for sid in ids:
            if sid not in sources:
                problems.append(f"unknown source {sid}")
        if known:
            missing = set.intersection(*(unsupported_numbers(sentence, sources[sid]) for sid in known))
            for number in sorted(missing):
                problems.append(f"number not in source: {number} in {sentence!r}")
    return problems or ["all citations check out"]

print(check_citations("You get 20 days of earned leave [S1]. Casual leave is 8 days [S2].", SOURCES))

["number not in source: 20 in 'You get 20 days of earned leave [S1].'"]