RAG evaluation: retrieval, grounding and final answer
Evaluate RAG by separating retrieval quality, coverage, source fidelity, generation, refusal and cost.
RAG is a composition: a bad answer can originate in the corpus, index, ranking, context or generator.
Evaluating only the final answer prevents deficit localization and often leads to retraining the wrong component.
Metrics must distinguish retrieval, grounding, generation and behavior when evidence is missing.
What the architecture must make explicit
Evaluating an AI system binds a capability, an exact object, a protocol, observations and a decision rule. An isolated score is not a qualification.
Move from a keyword to an inspectable engineering decision
Contract
Define required behavior, forbidden failures and thresholds before testing.
Separate
Keep search, confirmation, protected data and Fresh Final apart to prevent contamination.
Measure
Record outputs, metrics, failures, latency and measured-subject identity.
Adjudicate
Interpret observations without letting the candidate producer impose the verdict.
A claim is only as strong as its binding
The gate must evaluate the exact output produced by the exact realization. Integrity, scientific validity and authority are related, but never interchangeable.
Questions that change the design
What does rag evaluation: retrieval, grounding and final answer require?
RAG is a composition: a bad answer can originate in the corpus, index, ranking, context or generator. Evaluating an AI system binds a capability, an exact object, a protocol, observations and a decision rule. An isolated score is not a qualification.
Which evidence should be retained?
At minimum: exact realization identity, inputs and outputs, protocol version, measurements, failure cases and the decision rule. Metrics must distinguish retrieval, grounding, generation and behavior when evidence is missing.
Does AMF prescribe a particular model?
No. AMF treats the model, runtime, data, tools and deterministic components as candidate materials. Their retention depends on measured contribution and declared constraints.