The Data Lakehouse for Manufacturers: When It Makes Sense
A data lakehouse stores data as open table formats such as Delta Lake or Apache Iceberg on cheap object storage, while providing the ACID transactions, schema enforcement, and SQL performance that used to require a warehouse. For manufacturers the appeal is specific: ERP transactions are modest in volume, but machine telemetry, vision inspection images, test data, and quality records are not. A lakehouse lets you keep terabytes of sensor history at object-storage prices and still join it to job and item data in a single SQL query, which a traditional warehouse makes expensive.
Lakehouse Versus Warehouse: Choosing Honestly
If your analytical workload is ERP transactions and a few hundred million rows of finance and order history, a conventional warehouse such as SQL Server, Snowflake, or Postgres is simpler, cheaper to operate, and needs fewer specialist skills. The lakehouse earns its complexity when at least one of these is true: you ingest high-frequency sensor or test data measured in billions of rows, you need to store unstructured artifacts such as inspection images or CT scans alongside structured records, you run machine learning that wants direct file access to training data, or you must retain a decade of raw history for regulatory or warranty analysis. Most mid-size manufacturers start with a warehouse and add lakehouse storage for the machine-data tier.
Medallion Layers Applied to ERP and Machine Data
The bronze, silver, gold pattern maps cleanly onto manufacturing sources. Bronze holds raw appended data exactly as captured: CDC output from SyteLine or LN, OPC UA or MQTT Sparkplug B tag histories, MES event streams, and test result files. Silver cleans and conforms, resolving equipment IDs to an asset registry and item numbers to a golden master. Gold serves business-ready fact and dimension tables that BI tools query directly. Partitioning strategy matters more than in a warehouse: partition sensor tables by date and asset, and keep file sizes in the hundreds of megabytes to avoid the small-file problem that cripples query performance.
- Bronze: raw CDC extracts, MQTT Sparkplug B tag history, MES events, and test files with ingest timestamps
- Silver: conformed asset and item keys, deduplicated, typed, with late-arriving data handled explicitly
- Gold: dimensional models serving BI, partitioned and Z-ordered or clustered on common filter columns
- Compaction and vacuum jobs scheduled nightly to control small files and storage cost
Table Formats, Time Travel, and Governance
Delta Lake and Iceberg both provide ACID commits, schema evolution, and time travel, meaning you can query the table as of a specific timestamp or version. Time travel is unusually valuable in manufacturing: when a customer disputes a shipment or a quality escape triggers an investigation, you can reproduce exactly what the data showed on the day a decision was made. Schema evolution matters because ERP upgrades add columns regularly. Layer a catalog such as Unity Catalog, AWS Glue, or Polaris over the storage to manage table-level and column-level permissions, lineage, and audit, since object storage ACLs alone are far too coarse for regulated data.
Cost, Skills, and Operational Reality
The lakehouse trades storage cost for engineering effort. Object storage at roughly 20 to 25 dollars per terabyte per month makes retaining a decade of sensor data trivial, but compute clusters are the real bill and undisciplined queries can run it up fast. You also need genuine data engineering skills: Spark or an equivalent engine, partition tuning, compaction, and cluster policies. Teams that have only SQL analysts should be honest about this before committing, because an unmanaged lakehouse degrades into a slow, expensive file dump within a year. A practical middle path is to keep the gold layer queryable through a familiar SQL endpoint so analysts never touch Spark, while one platform engineer owns ingestion, compaction, and cluster policy.
- Budget compute separately from storage and set cluster auto-termination at 10 to 20 minutes idle
- Enforce partition filters on large sensor tables so a careless query cannot scan the full history
- Schedule compaction and retention jobs; unmanaged small files degrade query time within months
- Plan for at least one engineer fluent in Spark and table-format tuning, not only SQL
How Netray Builds Manufacturing Lakehouses
Netray designs the split rather than defaulting to one architecture. We typically keep ERP dimensional models in a conventional warehouse where they are cheap and fast, and put machine, test, and image data in a lakehouse joined through conformed asset and item keys. Our AI agents generate the ingestion pipelines, silver-layer conforming logic, and data quality tests from your SyteLine, LN, or M3 schema plus your tag dictionary, and we set cost guardrails from day one. A first production pipeline covering one plant's machine data joined to job history usually lands in eight to twelve weeks, with on-prem deployment available for controlled environments.
Frequently Asked Questions
Do manufacturers actually need a data lakehouse?
Only some do. If your analytics are ERP transactions, finance, and order history, a conventional warehouse is simpler and cheaper. A lakehouse becomes worthwhile when you ingest high-frequency machine telemetry, store unstructured artifacts such as inspection images or test files, run machine learning that needs direct file access, or must retain many years of raw history for warranty and regulatory analysis. Many manufacturers run both.
What is the difference between Delta Lake and Apache Iceberg?
Both are open table formats providing ACID transactions, schema evolution, and time travel over object storage. Delta Lake originated with Databricks and has the deepest integration there; Iceberg has broader multi-engine adoption across Snowflake, Trino, Flink, and Spark, and is often chosen to avoid engine lock-in. For most manufacturing workloads the practical difference is which query engines and catalog you plan to standardize on, not raw capability.
How much does it cost to store machine data in a lakehouse?
Object storage typically runs around 20 to 25 dollars per terabyte per month, so a decade of tag history for a mid-size plant is a modest line item. Compute is the real cost driver and depends on query volume and cluster discipline. Enforce partition filters, set aggressive cluster auto-termination, pre-aggregate common time windows into gold tables, and monitor per-workload spend from the first week.
Key Takeaways
- 1Lakehouse Versus Warehouse: Choosing Honestly: If your analytical workload is ERP transactions and a few hundred million rows of finance and order history, a conventional warehouse such as SQL Server, Snowflake, or Postgres is simpler, cheaper to operate, and needs fewer specialist skills. The lakehouse earns its complexity when at least one of these is true: you ingest high-frequency sensor or test data measured in billions of rows, you need to store unstructured artifacts such as inspection images or CT scans alongside structured records, you run machine learning that wants direct file access to training data, or you must retain a decade of raw history for regulatory or warranty analysis.
- 2Medallion Layers Applied to ERP and Machine Data: The bronze, silver, gold pattern maps cleanly onto manufacturing sources. Bronze holds raw appended data exactly as captured: CDC output from SyteLine or LN, OPC UA or MQTT Sparkplug B tag histories, MES event streams, and test result files.
- 3Table Formats, Time Travel, and Governance: Delta Lake and Iceberg both provide ACID commits, schema evolution, and time travel, meaning you can query the table as of a specific timestamp or version. Time travel is unusually valuable in manufacturing: when a customer disputes a shipment or a quality escape triggers an investigation, you can reproduce exactly what the data showed on the day a decision was made.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Poor Data Quality Cost Calculator
Put a defensible annual dollar figure on bad data using error volume, rework time, downstream escalation multipliers, and revenue leakage.
Free ToolAI Training Data Readiness Assessment
Score your data across volume, machine readability, labeling, lineage, permissions, rights, and refresh so you know what to fix before building an AI system.
Free ToolManufacturing AI Readiness Assessment
Score your manufacturing operation's readiness for AI across data, systems, people, and governance, and get a prioritized roadmap for closing the gaps.
Terms used in this article
Talk to Netray about whether a lakehouse, a warehouse, or a hybrid fits your machine-data and ERP analytics roadmap.
Related Resources
ERP Data Warehouse Architecture
ERP data warehouse architecture: landing, staging, and star schema layers, CDC extraction from SyteLine and Infor LN, plus governance for manufacturers.
AI & AutomationReal-Time Manufacturing Analytics
Real-time manufacturing analytics: streaming ERP and machine data with CDC, Kafka, and materialized views to deliver sub-minute shop floor visibility.
AI & AutomationPredictive Analytics in Manufacturing
Predictive analytics in manufacturing: demand forecasting, predictive maintenance, and quality models built on ERP and machine data, with accuracy benchmarks.