Out-of-distribution evaluation: test what the system did not learn
Build OOD evaluation that measures generalization, unknown detection, refusal and collateral regressions.
OOD evaluation covers regions where research assumptions no longer exactly match use.
Reusing OOD cases during optimization consumes their status and turns protected evidence into search data.
OOD sets must stay separated until candidate freeze and every exposure must be recorded.
What the architecture must make explicit
Capability does not grant authority. Critical systems must separate what proposes, verifies, permits and executes.
Move from a keyword to an inspectable engineering decision
Separate
Decouple probabilistic proposal, validation, permission and physical execution.
Bound
Limit tools, data, actions, duration and environments available to each component.
Refuse
Make missing evidence, timeouts and ambiguity explicit states.
Audit
Retain the applied rule, its inputs, version and resulting decision.
A claim is only as strong as its binding
The gate must evaluate the exact output produced by the exact realization. Integrity, scientific validity and authority are related, but never interchangeable.
Questions that change the design
What does out-of-distribution evaluation: test what the system did not learn require?
OOD evaluation covers regions where research assumptions no longer exactly match use. Capability does not grant authority. Critical systems must separate what proposes, verifies, permits and executes.
Which evidence should be retained?
At minimum: exact realization identity, inputs and outputs, protocol version, measurements, failure cases and the decision rule. OOD sets must stay separated until candidate freeze and every exposure must be recorded.
Does AMF prescribe a particular model?
No. AMF treats the model, runtime, data, tools and deterministic components as candidate materials. Their retention depends on measured contribution and declared constraints.