Citations and Source Attribution

Goal

Validate that citations refer to sources actually supplied to generation, distinguish citation validity from claim support, and represent claim-to-source attribution explicitly.

A citation makes an answer easier to audit. But a citation marker by itself is not proof. Consider:

Warranty is 24 months [policy-7#03]

If policy-7#03 actually says “Warranty is 18 months,” the citation is syntactically valid and factually contradictory. Citation evaluation therefore needs more than checking that brackets exist.

First check: source identity

For one generation request, record:

supplied chunk IDs:
policy-7#03
faq-2#11

If the model cites:

manual-4#02

that citation is invalid because the source was not supplied. The workflow can reject or flag it deterministically.

Second check: does the cited source support the claim?

Claim:

Battery runtime is ten hours.

Cited source:

Typical battery runtime is approximately 10 hours.

This is supportive. Claim:

Battery replacement is free for ten years.

Same cited source:

Typical battery runtime is approximately 10 hours.

The citation exists but does not support the claim. Support checking may involve:

• exact field matching;
• rule-based checks;
• human evaluation;
• another model/evaluator;
• structured source records.

The method depends on the task.

One answer can contain multiple claims

Answer:

Warranty is 18 months, and battery runtime is ten hours.

One source may support warranty. Another may support battery runtime. Represent attribution at claim level when possible:

[
  {"claim":"Warranty is 18 months","sources":["policy-7#03"]},
  {"claim":"Battery runtime is ten hours","sources":["faq-2#11"]}
]

A single citation at the end of a long paragraph can make support ambiguous.

Citation precision and coverage are different

Two useful questions: Precision-like question When a citation is given, does it actually support the claim? Coverage-like question Do important factual claims have citations? A system can have:

• Precision-like question: when a citation is given, does it actually support the claim?
• Coverage-like question: do the important factual claims have citations at all?

Both are problems.

Source authority matters too

Two retrieved sources can conflict:

old FAQ: warranty 12 months
current policy: warranty 18 months

A citation validator can prove which source was used. It still needs a policy for which source is authoritative or current. Attribution and source selection are related but separate responsibilities.

Citation correctness and citation completeness are separate

Imagine an answer with three factual claims. Two have accurate supporting citations. The third has no citation. Every citation that exists may be correct, yet citation coverage is incomplete.

The opposite can happen too: every claim has a citation ID, but one citation does not actually support the claim. So evaluate at least:

• citation identity validity;
• claim-to-source support;
• coverage of claims that require evidence.

Prefer the narrowest source that actually supports the claim

If a specific chunk contains the exact policy sentence, citing only an entire 200-page handbook makes review harder. Fine-grained citations improve inspectability, provided the chunk retains enough context to interpret the statement correctly. Do not optimize citation precision by making chunks so small that conditions or exceptions disappear. Attribution quality depends on the chunking contract as well as the generated answer.

Predict

A model cites a source that was supplied, but the source does not support the claim. What passed and what failed?

Source-identity validity passed; claim support failedBoth passed because the source ID existsRetrieval failed because any unsupported claim proves the source was not retrieved

Run the local citation validator

Run the passing fixture first:

python labs/notebooks/level-09/l09-10-citation-validator.py

It should report no identity or support failures and full citation coverage.

For each broken fixture below, predict which check should fail before you run it:

python labs/notebooks/level-09/l09-10-citation-validator.py --fixture unsupplied
python labs/notebooks/level-09/l09-10-citation-validator.py --fixture unsupported
python labs/notebooks/level-09/l09-10-citation-validator.py --fixture uncited

All three commands intentionally exit non-zero, but for different reasons: an unknown source ID, a cited source that does not support the claim, and incomplete citation coverage.

Loading lab…

Quick Check

1. What does a valid source ID prove?

That the cited source was part of the allowed supplied evidence setThat the source is the newest authoritative documentA valid source ID proves that claims citing it are supported because the source passed retrieval filters

2. Why use claim-level attribution?

It makes retrieval top-k irrelevantClaim-level attribution is unnecessary when all claims come from one response and share the same retrieved contextDifferent claims in one answer may require different supporting sources

3. What is citation coverage concerned with?

Whether every retrieved chunk receives at least one citationWhether important factual claims have supporting citationsWhether the citations use a consistent format

0 of 3 questions answered.

Explain it back

Create two factual claims with separate source IDs. Then give one example of a citation-ID failure and one example of a support failure.

Key Takeaways

• Citation identity and factual support are separate checks.
• Cited sources should belong to the evidence supplied for that request.
• Claim-level attribution is clearer than one citation for many unrelated claims.
• Citation precision and citation coverage measure different failure modes.
• Source authority/freshness still needs explicit policy.

Next Lesson

Next, evaluate retrieval and grounded answers with separate metrics so one stage cannot hide another stage's failure.

References

• Lewis et al., Retrieval-Augmented Generation.