How to Evaluate AI Vendors at Manufacturing Trade Shows Without Getting Burned
Evaluating AI vendors at manufacturing trade shows requires a structured method: a pre-written question script, a scoring rubric applied identically to every booth, and hard filters for deployment model, ERP integration, and data security. Shows like IMTS, Automate, and FABTECH now feature hundreds of exhibitors claiming AI capability, and booth demos are engineered to impress rather than inform. The difference between a productive show and an expensive one is whether you control the conversation. This guide gives plant managers, IT directors, and ERP administrators a repeatable evaluation framework, the exact questions that separate real products from wrappers, and the red flags that end conversations early.
Before the Show: Build Your Filter, Not a Wish List
Arrive with three documents. First, a one-page problem brief naming your top operational pain points with baseline numbers, for example, quote turnaround averaging four days, 2.1% scrap on a machining cell, or six FTEs on manual order entry. Second, a hard-constraints list that disqualifies vendors instantly: must integrate with SyteLine 10 or your LN version, must support on-premises deployment if you handle ITAR or CUI data, must name reference customers in discrete manufacturing under $500M revenue. Third, a scoring sheet, five criteria on a 1-5 scale, applied to every vendor identically so end-of-day comparisons mean something. Pre-book 20-minute meetings with your shortlist two to three weeks out, because walk-up traffic at major booths gets junior staff reciting scripts.
- Problem brief: three pain points with baseline metrics, one page, shared with vendors
- Hard constraints: ERP version compatibility, on-prem support, reference customers your size
- Uniform scorecard: five criteria scored 1-5 for every vendor, compared nightly
- Pre-booked 20-minute meetings reach product people; walk-ups reach booth staff
The Seven Questions That Expose Weak AI Products
On the floor, run the same script at every booth. Ask what specific data the system requires and in what format, because vague answers about ingesting anything signal an immature product. Ask whether it deploys fully on-premises, and what hardware it needs; a vendor who cannot name GPU requirements has not done real on-prem deployments. Ask which ERP systems they have integrated in production, pushing past logos to specifics like SyteLine IDO endpoints or LN APIs. Ask what happens when the model is wrong, listening for confidence thresholds, human review queues, and audit logs. Ask for two reference customers your size in your vertical. Ask what implementation actually took at those references in weeks and dollars. Ask what the pricing model is at your volume, in writing.
- Data requirements: exact inputs and formats, not marketing claims of ingesting anything
- Deployment: full on-prem support with named GPU requirements, or disqualify for CUI work
- ERP integration proof: production references naming SyteLine, LN, or M3 specifics
- Failure handling: confidence thresholds, human review queues, and audit logs described concretely
Red Flags That Should End the Conversation
Certain answers reliably predict failed implementations. A demo that only runs on the vendor's curated sample data, with refusal or hedging when you ask to see your kind of document processed, means the product breaks on real-world variance. Pricing that cannot be estimated without a discovery engagement usually hides consumption models that explode at production volume. Claims of 99% accuracy without a definition of the measurement set are marketing, not engineering. No security documentation available, no SOC 2 report, no architecture diagram, no answer on data residency, disqualifies a vendor for any defense supplier bound by DFARS 252.204-7012. And a roadmap answer to a current-capability question, we are launching that next quarter, means the feature does not exist. Walk away politely; the show floor has two hundred more booths.
After the Show: Convert Notes Into a Decision in 14 Days
Trade show intelligence decays fast. Within two weeks, consolidate scorecards, rank the top three vendors, and issue each the same structured follow-up: a request for a demo on your data under NDA, two reference calls you schedule yourself, and written answers to your security questionnaire. Insist the proof-of-concept has a defined metric, dataset, and end date, 30-45 days is reasonable, and a pre-agreed price for production conversion so the pilot cannot become an open-ended consulting engagement. Involve your IT security lead and, for defense suppliers, your FSO before signing anything, since AI tools touching CUI must appear in your SSP and data-flow diagrams. Vendors who resist structured evaluation at this stage were selling the demo, not the product.
How Netray Stands Up to This Evaluation
We built Netray to pass exactly this kind of scrutiny, and we encourage you to run the script on us. Deployment: fully on-premises, open-weight models on hardware from a single NVIDIA L40S node to multi-GPU clusters, sized in writing. ERP integration: production experience with Infor SyteLine 9 and 10, CloudSuite Industrial, LN, M3, and Baan via IDO endpoints, ION APIs, and direct integration. Failure handling: human approval gates, confidence-based routing, and complete audit logs on every agent action. Security: architecture documentation and data-flow diagrams built for CMMC 2.0 and ITAR environments, no CUI leaves your network. Timelines and outcomes: first production agent in 60-90 days, with customers reporting 40-60% manual effort reduction. Bring your scorecard; we will fill it in line by line.
Frequently Asked Questions
What questions should I ask AI vendors at a trade show?
Ask seven questions at every booth: what specific data and formats the system requires, whether it deploys fully on-premises and on what GPU hardware, which ERP systems it has integrated in production with specifics like SyteLine IDOs, how it handles model errors including review queues and audit logs, for two reference customers your size, what implementation took at those references in weeks and dollars, and for the pricing model at your volume in writing.
What are red flags when evaluating AI software vendors?
The strongest red flags are demos that only run on curated sample data, pricing that cannot be estimated without a paid discovery engagement, accuracy claims like 99% with no defined measurement set, missing security documentation such as SOC 2 reports or architecture diagrams, and roadmap answers to current-capability questions. For defense suppliers, inability to document data residency and on-prem options disqualifies a vendor under DFARS 252.204-7012 obligations.
How long should an AI vendor proof of concept take?
A well-structured AI proof of concept should run 30-45 days with a defined success metric, a fixed dataset, a named owner on each side, and a pre-agreed price for production conversion. Open-ended pilots without end dates or metrics routinely drift into months of consulting spend. Insist the POC uses your real data under NDA rather than vendor samples, since real-world document and ERP data variance is where weak products fail.
Key Takeaways
- 1Before the Show: Build Your Filter, Not a Wish List: Arrive with three documents. First, a one-page problem brief naming your top operational pain points with baseline numbers, for example, quote turnaround averaging four days, 2.1% scrap on a machining cell, or six FTEs on manual order entry.
- 2The Seven Questions That Expose Weak AI Products: On the floor, run the same script at every booth. Ask what specific data the system requires and in what format, because vague answers about ingesting anything signal an immature product.
- 3Red Flags That Should End the Conversation: Certain answers reliably predict failed implementations. A demo that only runs on the vendor's curated sample data, with refusal or hedging when you ask to see your kind of document processed, means the product breaks on real-world variance.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
ERP Vendor Demo Scorecard
Score each ERP vendor demonstration on the ten factors that predict implementation reality, from scripted scenario coverage to whether integrations and error handling were shown live.
Free ToolAI Agent Use Case Prioritizer
Answer ten questions about a candidate process and get a prioritization score that tells you whether it deserves an AI agent pilot now, later, or never.
Free ToolAI Center of Excellence Setup Checklist
A 32-point checklist for standing up an AI Center of Excellence that delivers working systems instead of slide decks, covering charter, funding, platform, governance, and adoption.
Terms used in this article
Walking a trade show floor this year? Download Netray's AI vendor scorecard, then test it on us first.
Related Resources
Top Manufacturing AI Conferences in 2026: Where to Go and Why
Compare the top manufacturing AI conferences in 2026, including IMTS, Automate, and Hannover Messe. Costs, audiences, and which events fit ERP teams.
AI & AutomationHow the Defense Industry Procures AI: A Practical Guide
A practical guide to how the defense industry procures AI: OTAs, SBIR, CDAO pathways, DFARS and CMMC requirements, and timelines for primes and subs.
AI & AutomationDefense Manufacturing Summits 2026: The Events That Matter
The defense manufacturing summits that matter in 2026: DMC, Sea-Air-Space, AUSA, and CMMC events, with attendee profiles and budget guidance for contractors.