An AI Agent Evaluation Framework for Manufacturing Teams
Evaluating an AI agent means measuring its output against a fixed set of known-correct examples, under the same conditions it will face in production, before anyone acts on its answers. A vendor demo is not evaluation. Neither is a project manager spot-checking ten outputs and declaring it impressive. A real evaluation framework has four parts: a golden test set drawn from your own historical ERP records, task-specific accuracy thresholds tied to blast radius, adversarial and edge-case tests, and operational measures for latency, cost, and determinism. Build these before the agent, not after, or you will grade the agent on the cases it already handles.
Build a Golden Test Set From Your Own ERP History
Pull 150 to 300 real cases from the last 12 months out of your production data, with the outcome a competent human actually chose. For a supplier acknowledgement agent that means historic PO lines, the acknowledgement documents received, and the decision recorded. Stratify the sample deliberately: include the ugly cases, the ones with missing fields, the ones a new hire got wrong. Freeze the set and version it in Git alongside the agent code. If you regenerate the test set every time the agent improves, you are not measuring progress, you are measuring your own optimism. Keep a separate holdout slice of 50 cases that the build team never sees.
- 150 to 300 stratified cases pulled from the last 12 months of production ERP data
- Deliberate over-sampling of edge cases: missing fields, duplicate records, unit-of-measure mismatches
- Version the frozen set in Git with a hash so results are comparable across agent revisions
- Hold back 50 cases the development team never sees, used only for final acceptance
Set Accuracy Thresholds That Match the Blast Radius
There is no universal passing grade. An agent drafting internal meeting summaries can ship at 85 percent usefulness. An agent proposing inventory adjustments in a regulated aerospace plant needs to be right or silent, which usually means high precision with a deliberately high escalation rate. Separate the two numbers that matter: precision on the actions it takes, and recall on the cases it correctly declines. A 95 percent precision agent that escalates 30 percent of cases is frequently more valuable than a 90 percent precision agent that answers everything. Write the thresholds into the acceptance criteria before build starts, and get the process owner to sign them.
Test the Failure Path, Not Just the Happy Path
Most agent evaluations only measure whether the right answer appears when clean input arrives. Production is not clean. Deliberately feed the agent truncated documents, contradictory records, records with a stale revision, prompts containing embedded instructions from an untrusted email body, and API responses that time out. Score what the agent does under each condition. The failure you care about is not a wrong answer but a confidently wrong answer that a busy planner accepts. Also test the tool layer: what happens when an IDO returns an empty collection, when a token expires mid-run, or when the same request is retried and produces a duplicate write.
- Adversarial inputs: untrusted email or PDF text containing embedded instructions to the agent
- Contradiction tests: two source systems disagreeing on quantity, revision, or due date
- Tool failure tests: API timeout, empty collection, expired token, and mid-run permission revocation
- Idempotency tests: identical request replayed twice must not create duplicate ERP transactions
Measure Latency, Cost, and Determinism Together
Accuracy is only one axis. Record median and 95th-percentile end-to-end latency, because an agent that takes 90 seconds will not be used inside an order-entry conversation regardless of how right it is. Track cost per successful task in real currency, including retries and failed runs, and compare it against the fully loaded human minutes it displaces. Then measure determinism: run the same 50 inputs five times and count how often the answer changes materially. High variance on identical input is a red flag for anything auditable, and it is fixable through temperature settings, tighter tool contracts, and structured output schemas rather than better prompting.
How Netray Evaluates Agents Before Handover
Every Netray agent ships with a versioned evaluation harness the client owns and can rerun without us. We build the golden set from your production history, publish precision, escalation rate, p95 latency, and cost per task, and rerun the full suite on every model or prompt change so regressions surface before users do. For defense and aerospace clients the entire harness runs on-premises against on-prem models, so evaluation data never crosses your boundary. The practical result: when your quality or IT lead asks how you know the agent is right, the answer is a reproducible test run with a hash, not a slide.
Frequently Asked Questions
How many test cases do you need to evaluate an AI agent?
Between 150 and 300 stratified cases is enough for most manufacturing tasks, plus a 50-case holdout the build team never sees. Below roughly 100 cases the confidence intervals get so wide that a five-point accuracy change is indistinguishable from noise. What matters more than raw count is stratification: deliberately over-sample edge cases like missing fields, unit-of-measure mismatches, and stale revisions.
What accuracy does an AI agent need before production use?
It depends entirely on blast radius. Draft-generation agents can be useful at 85 percent. Agents proposing inventory, quality, or financial transactions typically need 95 percent or better precision on the actions they take, paired with a deliberately high escalation rate. Set the threshold as acceptance criteria before build, and measure precision on actions taken separately from recall on cases correctly declined.
Why does an AI agent give different answers to the same question?
Language models are probabilistic, so identical inputs can produce different outputs unless you constrain them. Reduce variance with lower temperature settings, structured output schemas that force a fixed response shape, tighter tool contracts, and caching for repeated queries. Measure it directly: run 50 inputs five times and count material changes. For auditable ERP actions, high variance on identical input should block go-live.
Key Takeaways
- 1Build a Golden Test Set From Your Own ERP History: Pull 150 to 300 real cases from the last 12 months out of your production data, with the outcome a competent human actually chose. For a supplier acknowledgement agent that means historic PO lines, the acknowledgement documents received, and the decision recorded.
- 2Set Accuracy Thresholds That Match the Blast Radius: There is no universal passing grade. An agent drafting internal meeting summaries can ship at 85 percent usefulness.
- 3Test the Failure Path, Not Just the Happy Path: Most agent evaluations only measure whether the right answer appears when clean input arrives. Production is not clean.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
AI Agent Use Case Prioritizer
Answer ten questions about a candidate process and get a prioritization score that tells you whether it deserves an AI agent pilot now, later, or never.
Free ToolAI Use Case Value vs Effort Calculator
Turn each AI idea into annual net value, build cost, payback period, and three-year ROI so your backlog is prioritized on economics instead of enthusiasm.
Free ToolAI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
Terms used in this article
Want an evaluation harness for an agent you already have, built from your own ERP history? Netray will assemble the golden set and give you the numbers your team can defend.
Related Resources
AI Agent Failure Modes and How to Catch Them
AI agent failure modes in manufacturing: silent data drift, tool-call errors, hallucinated part numbers, and the monitoring that catches them before users do.
AI & AutomationSecurity Review for AI Agents in the Enterprise
A security review checklist for enterprise AI agents: data residency, identity and access, prompt injection defense, logging, and vendor risk for manufacturers.
AI & AutomationYour First AI Agent: A Manufacturing Playbook
Your first AI agent in manufacturing: pick a bounded ERP use case, scope data access, set guardrails, and ship a working pilot in 6 to 8 weeks.