Fine-Tuning LLMs On-Prem with Enterprise Data: When, How, and What It Costs
Fine-tuning an LLM on-prem means adapting an open-weight model - typically Llama 3.x, Qwen 2.5, or Mistral - to your terminology, formats, and tasks by training on your own data, entirely on your own GPUs. It is the right tool for teaching style, structure, and domain vocabulary; it is the wrong tool for injecting facts, which retrieval handles better and more auditably. For manufacturers, the highest-ROI fine-tunes are narrow: quote formatting, ERP transaction extraction, nonconformance classification. This guide covers when to fine-tune versus RAG, LoRA versus full fine-tuning, dataset preparation, hardware budgets, and the evaluation discipline that separates working fine-tunes from expensive failures.
Fine-Tune vs RAG: Decide This First
The most expensive mistake in enterprise AI is fine-tuning to solve a retrieval problem. RAG is the answer when the model needs current facts - item masters, contract clauses, quality procedures - because facts change, retrieval is instantly updatable, and every answer carries citations for audit. Fine-tuning is the answer when the model must reliably produce your formats and vocabulary: quote structures, ECN summaries in your house style, classification against your specific nonconformance codes, or extraction into your exact SyteLine field schema. The practical test: if the failure mode is 'the model does not know X,' use RAG; if it is 'the model knows enough but outputs it wrong,' fine-tune. Most production systems combine both - a LoRA-tuned model for format discipline, RAG for facts.
LoRA and QLoRA: The Default for Enterprise Fine-Tuning
Full fine-tuning updates every weight and needs 8x H100-class hardware for a 70B model - almost never justified. LoRA (low-rank adaptation) trains small adapter matrices at rank 8-64 while freezing base weights, achieving 95%+ of full fine-tune quality on narrow tasks at a fraction of the cost. QLoRA goes further, training adapters over a 4-bit quantized base: a 70B QLoRA run fits on 2x A100/H100 80 GB, and an 8B run fits on a single 24 GB GPU. Adapters are 100-500 MB files you can version, audit, and hot-swap per task in vLLM - one base model can serve a quoting adapter and an extraction adapter simultaneously via multi-LoRA serving.
- LoRA rank 16-32 with alpha 2x rank is a solid starting point for format and classification tasks
- QLoRA fine-tunes Llama 3.1 8B on one 24 GB GPU and 70B on 2x 80 GB GPUs - days, not weeks
- Adapters are 100-500 MB, versionable artifacts - keep them in your artifact registry with training data lineage
- vLLM multi-LoRA serving hosts multiple adapters on one base model with per-request adapter selection
Dataset Preparation: Where Fine-Tunes Are Won or Lost
Quality beats quantity decisively: 500-2,000 excellent examples outperform 50,000 mediocre ones for narrow enterprise tasks. Mine training pairs from systems of record - historical quotes with their source RFQs from SyteLine, approved ECN summaries, dispositioned nonconformance reports with their final codes. Every example should represent output you would be proud to ship, because the model will faithfully reproduce your dataset's median quality, including its errors. Deduplicate near-identical examples, hold out 10-15% as an untouched test set, and scrub rigorously: strip customer identifiers where policy requires, and never include Controlled/ITAR technical data in a training set unless the resulting adapter is itself handled as Controlled - a governance decision to make in writing before training starts.
- Target 500-2,000 curated examples for narrow tasks; curate aggressively rather than accumulating volume
- Source pairs from systems of record: SyteLine quotes, ECN records, dispositioned NCRs with final codes
- Hold out 10-15% as a frozen test set before any training run - never tune against it
- Classify the trained adapter at the level of its most sensitive training data and document that decision
Evaluation and Deployment Discipline
A fine-tune without a pre-committed eval is a demo, not a deliverable. Before training, define task metrics on the held-out set: exact-match or F1 for extraction into ERP fields, accuracy against historical dispositions for classification, and rubric-scored quality (using a judge model plus human spot checks) for generation tasks. Require the tuned model to beat the base-model-plus-good-prompt baseline by a meaningful margin - if careful prompting gets you 92% and tuning gets 94%, the operational cost of maintaining a training pipeline may not pay. Also test for regression on general capability and for catastrophic format drift on out-of-distribution inputs. Retrain on a schedule tied to drift monitoring, not the calendar: when live accuracy drops 2-3 points below the eval baseline, refresh the dataset and retune.
How Netray Runs Fine-Tuning Programs for Manufacturers
Netray runs fine-tuning as an evidence-driven program: we first establish whether RAG plus prompting hits your target (it often does, saving the entire project), then build datasets from your ERP history, run QLoRA training on your hardware or ours, and deliver adapters with a full eval report and data lineage documentation. Typical engagement: 4-6 weeks from data extraction to a deployed adapter, with measured gains like 95%+ field-level extraction accuracy on SyteLine order automation - up from 85-88% with prompting alone - at a training compute cost under $2,000 on client-owned L40S hardware. Every adapter ships with the governance paperwork your CMMC program needs: training data classification, eval results, and version lineage.
Frequently Asked Questions
Should I fine-tune an LLM or use RAG for company data?
Use RAG when the model needs facts - part data, contract terms, procedures - because retrieval is instantly updatable and every answer carries auditable citations. Fine-tune when the model has the knowledge but outputs it wrong: enforcing your quote formats, house writing style, or classification codes. Most production enterprise systems combine both: a LoRA adapter for format discipline plus RAG for current facts. Fine-tuning to inject facts is the most common and most expensive mistake.
How much data do I need to fine-tune an LLM?
For narrow enterprise tasks - classification, extraction, format enforcement - 500 to 2,000 high-quality examples is typically enough with LoRA, and quality matters far more than volume. Examples should come from systems of record, like historical quotes or dispositioned quality records, and represent output you would ship. Hold out 10-15% as a test set before training. Broad behavioral tuning needs tens of thousands of examples, which is rarely justified for a single-company deployment.
What GPUs are needed to fine-tune Llama models on-premises?
With QLoRA, far less than most expect: Llama 3.1 8B fine-tunes on a single 24 GB GPU (an L4 or RTX-class card), and 70B fits on 2x A100 or H100 80 GB. Training runs for narrow tasks complete in hours to a couple of days. Full fine-tuning of a 70B model would need an 8x H100 node and is rarely justified - LoRA reaches 95%+ of the quality at under 10% of the compute cost.
Key Takeaways
- 1Fine-Tune vs RAG: Decide This First: The most expensive mistake in enterprise AI is fine-tuning to solve a retrieval problem. RAG is the answer when the model needs current facts - item masters, contract clauses, quality procedures - because facts change, retrieval is instantly updatable, and every answer carries citations for audit.
- 2LoRA and QLoRA: The Default for Enterprise Fine-Tuning: Full fine-tuning updates every weight and needs 8x H100-class hardware for a 70B model - almost never justified. LoRA (low-rank adaptation) trains small adapter matrices at rank 8-64 while freezing base weights, achieving 95%+ of full fine-tune quality on narrow tasks at a fraction of the cost.
- 3Dataset Preparation: Where Fine-Tunes Are Won or Lost: Quality beats quantity decisively: 500-2,000 excellent examples outperform 50,000 mediocre ones for narrow enterprise tasks. Mine training pairs from systems of record - historical quotes with their source RFQs from SyteLine, approved ECN summaries, dispositioned nonconformance reports with their final codes.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Open-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Free ToolAI Training Data Readiness Assessment
Score your data across volume, machine readability, labeling, lineage, permissions, rights, and refresh so you know what to fix before building an AI system.
Free ToolAI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
Terms used in this article
Before you fine-tune anything, get Netray's RAG-vs-tuning assessment on your actual task data - it regularly saves clients the whole project.
Related Resources
Deploying Llama Models On-Prem for Enterprise
How to deploy Llama models on-prem for enterprise use: hardware sizing, quantization, vLLM serving, licensing, and security for regulated manufacturers.
AI & AutomationChoosing a Vector Database for On-Prem RAG
Compare vector databases for on-prem RAG: Qdrant, Weaviate, Milvus, and pgvector on performance, filtering, security, and ops burden for enterprise AI.
AI & AutomationSmall Language Models vs LLMs for Enterprise Workloads
Small language models vs LLMs for enterprise workloads: accuracy, cost, GPU footprint, and when a 7B model beats a frontier model for ERP automation.