AI system audit: measure before replacing
A full AI system audit method covering baselines, failure maps, dependencies, costs, evidence and transformation decisions.
A serious audit binds every failure to the exact realization that produced it: model, prompt, retrieval, data, rules, runtime and environment.
A global score often hides critical errors, external dependencies and the difference between a model deficit and an architecture deficit.
The audit should produce a traceable RETAIN, REPAIR, REPLACE, CONTINUE SEARCH or REFUSE decision.
What the architecture must make explicit
The engineering object is the complete realization. A model is only one possible component among data, memory, routing, tools, rules, verifiers and runtime.
Move from a keyword to an inspectable engineering decision
Observe
Measure the baseline and bind every behavior to the exact realization that produced it.
Intervene
Test one change at a time to attribute gains and regressions.
Compose
Verify interfaces, shared state and interactions before promotion.
Qualify
Apply hard gates to the frozen realization within a declared scope.
A claim is only as strong as its binding
The gate must evaluate the exact output produced by the exact realization. Integrity, scientific validity and authority are related, but never interchangeable.
Questions that change the design
What does ai system audit: measure before replacing require?
A serious audit binds every failure to the exact realization that produced it: model, prompt, retrieval, data, rules, runtime and environment. The engineering object is the complete realization. A model is only one possible component among data, memory, routing, tools, rules, verifiers and runtime.
Which evidence should be retained?
At minimum: exact realization identity, inputs and outputs, protocol version, measurements, failure cases and the decision rule. The audit should produce a traceable RETAIN, REPAIR, REPLACE, CONTINUE SEARCH or REFUSE decision.
Does AMF prescribe a particular model?
No. AMF treats the model, runtime, data, tools and deterministic components as candidate materials. Their retention depends on measured contribution and declared constraints.