Open-Weight Model Selector: Find Your Right LLM Size Class
This assessment matches your organization to the right open-weight LLM size class - small (4B-8B), mid-size (14B-32B), or frontier (70B+) - for teams planning a self-hosted deployment. Model selection is not about grabbing the biggest weights your hardware can hold; it is a trade among task complexity, latency, concurrency, fine-tuning plans, and operational maturity. Ten questions score those factors and place you in a size class with concrete deployment recommendations, including which specific levers - quantization, fine-tuning, routing - matter most for your band.
1. What GPU budget or capacity can you realistically dedicate to model serving?
2. How complex are the tasks the model must handle well?
Be honest about the hardest 20% of tasks, not the average.
3. What are your latency expectations for a typical response?
4. How many people or processes will use the model in production?
5. Do you plan to fine-tune the model on your own data?
Fine-tuning smaller models often beats prompting larger ones for narrow domains.
6. How will you evaluate whether a model is good enough for your tasks?
7. What is your team's experience operating self-hosted model serving?
8. How long are the contexts the model must handle per request?
9. How sensitive is the data the model will process?
Higher sensitivity strengthens the case for fully self-hosted open weights.
10. What licensing constraints matter for your deployment?
Why size class is the right first decision
The open-weight ecosystem moves fast - specific model names age in months - but size classes are stable planning units because they map directly to hardware, latency, and cost tiers. An 8B model runs on one modest GPU and answers fast; a 32B model needs a server card or two and reasons noticeably better; a 70B+ model demands tensor parallelism across multiple GPUs and delivers near-frontier quality. Choosing class first lets you size hardware, budget, and operations honestly, then pick the best current model within the class at deployment time - which is how Netray runs every selection engagement.
What the questions are really measuring
The assessment weighs four forces that push your size class up or down. Scores are deliberately balanced so that hardware alone cannot carry you into the frontier band - capability without evaluation discipline and operational maturity produces expensive disappointments.
- Demand side: task complexity, context length, and user scale push toward larger classes
- Supply side: GPU budget and serving experience cap what you can operate well
- Quality control: evaluation rigor determines whether any choice can be validated
- Strategic fit: data sensitivity and licensing shape how much self-hosting matters
Interpreting your band - and the common overshoot
The most frequent selection error we see is overshooting: teams deploy a 70B model for workloads a tuned 14B handles indistinguishably, then pay for it in hardware, latency, and operational drag every day after. If you scored mid-band, trust it - a quantized 32B with good retrieval covers the large majority of enterprise document and reasoning tasks in 2026. The reverse error matters too: if agentic workflows and long chains of tool use are genuinely central, undersizing produces failure loops that burn more tokens than a larger model would have. Let your gold-set results, not demos, make the final call.
How Netray runs model selection for clients
Netray turns this band into a named model and a running deployment. We maintain current benchmarks across the Llama, Qwen, Mistral, and leading MoE families, build a gold evaluation set from your real prompts and documents, and test candidates head-to-head on your tasks - including quantized variants, because deployment precision is what you will actually run. The engagement ends with a deployed, monitored serving stack and a written selection rationale your architecture board and security team can audit.
Frequently Asked Questions
Why does the assessment recommend a size class instead of a specific model?
Because specific open-weight leaders change every few months, while the hardware, latency, and cost profile of each size class stays stable. A recommendation naming one model would be stale before your procurement cycle finished. Class-first planning lets you budget hardware and operations now, then select the best current model in that class - validated on your own evaluation set - at deployment time.
Can a fine-tuned small model really beat a larger general model?
For narrow, well-defined tasks, yes - consistently. A 14B model LoRA-tuned on a few thousand quality examples from your domain routinely outperforms a stock 70B on that task, while costing a quarter as much to serve. The larger model retains the edge on open-ended reasoning and unfamiliar tasks. That is why the assessment asks about fine-tuning plans: a real tuning pipeline effectively buys you one size class of headroom.
What about mixture-of-experts (MoE) models - which band are they?
Large MoE models belong in the frontier band operationally: their full parameter count must fit in VRAM even though only a fraction activates per token, so memory needs are frontier-class while per-token compute is closer to mid-size. They are excellent value if you have the VRAM and multi-GPU serving experience. If you scored below the frontier band, dense mid-size models are the simpler, safer operational choice.
Find your size class, then let Netray benchmark specific models against your real tasks before you deploy.
Related Tools
Self-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
On-Prem AIGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
On-Prem AIAir-Gapped AI Readiness Assessment
A 10-question assessment that scores how prepared your organization is to deploy and operate LLMs inside an air-gapped or classified enclave.
Go Deeper
Small Language Models vs LLMs for Enterprise Workloads
Small language models vs LLMs for enterprise workloads: accuracy, cost, GPU footprint, and when a 7B model beats a frontier model for ERP automation.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.