When I run a smart factory pilot under an MBRP program — which I define here as a Model-Based Resource Planning initiative that ties digital models, sensors and control systems to resource planning and execution — the central question I need to answer for stakeholders is simple: did we get a return on the investment? Proving ROI is less about flashy visuals and more about choosing the right mix of operational, financial and adoption metrics, measuring them rigorously, and telling the story with before/after and counterfactual comparisons.
Start with a clear ROI hypothesis
I always begin by writing one crisp ROI hypothesis in plain language. Example: “By deploying model-based scheduling and closed-loop setpoint adjustment on Line A, we will reduce cycle time by 12% and scrap by 20%, yielding an incremental contribution margin of £150k per quarter.” That sentence forces you to pick a target metric, a magnitude, and a monetary framing. If you can’t do that, the program will be hard to prove.
Core operational KPIs to measure
These are the operational levers that usually move when a smart factory pilot succeeds. Measure them continuously and capture a baseline long enough to cover normal variation.
- Overall Equipment Effectiveness (OEE) — availability × performance × quality. Unit: %.
- Cycle time / takt time — average processing time per part. Unit: seconds/minutes.
- Throughput — good parts produced per shift/day. Unit: units/hour.
- First Pass Yield (FPY) / Scrap rate — percent of units passing without rework. Unit: %.
- Downtime and root causes — minutes lost and categorized causes (mechanical, tooling, material, process). Unit: minutes; counts.
- Changeover time — setup and recipe change duration. Unit: minutes.
- Mean time to repair (MTTR) and mean time between failures (MTBF) — for assets under predictive maintenance.
- Energy consumption — kWh per unit or per shift for scope reductions and sustainability impact.
- Inventory metrics — WIP levels and turns, days of supply impacted by planning accuracy.
- On-time delivery / schedule attainment — percent of orders shipped on time.
Financial and commercial KPIs
Operational improvement is only half the story — convert it into cash. Track these financial measures alongside operational KPIs.
- Cost per unit — direct manufacturing cost before and after pilot (materials, labor, energy, scrap allocated).
- Scrap and rework cost — direct cost attributable to defective units.
- Labor productivity — units per operator-hour and labor cost per unit.
- Incremental revenue enabled — if throughput or quality improvement allows new orders or reduced lead times.
- Payback period and Net Present Value (NPV) — include software, hardware, integration, and change-management costs.
- Total cost of ownership (TCO) — 3–5 year view for platform vs point solution.
Data and model performance metrics (specific to MBRP)
Because MBRP ties models to execution, you must measure how well the models and integrations perform:
- Prediction accuracy — for demand, yield, or asset failure. Use mean absolute percentage error (MAPE) or ROC/AUC for classification.
- Prescription adoption rate — percent of automated suggestions that operators or systems accepted.
- Closed-loop effect size — difference in target variable (e.g., cycle time) when model-driven setpoints are active vs inactive.
- False positive / false negative rate — in predictive maintenance alerts, to quantify wasted actions or missed failures.
- Latency & uptime of digital services — time from sensor to decision and service availability (%) — important for production-critical control loops.
Change, adoption and human factors
Even the best tech fails without people. I measure adoption and organizational impact to explain residual performance gaps.
- User adoption — number of users actively using dashboards, issuing overrides, or accepting recommendations.
- Procedure compliance — percent of operations following new digital SOPs or data-entry rules.
- Training hours — hours invested and correlation with operator proficiency and error rates.
- Operator sentiment — simple surveys or Net Promoter Score (NPS) for the toolset to identify resistance or confidence.
How I structure measurement so it’s credible
Stakeholders will ask for rigor. Here’s the protocol I follow:
- Baseline period: collect at least 4–8 weeks of production data that capture normal shifts, product mixes and seasonal effects.
- Control group or A/B testing: where possible, run the pilot on one line and hold another similar line as control to isolate effect.
- Statistical testing: use t-tests or non-parametric tests to show differences are not due to chance. Report confidence intervals on key deltas.
- Time-series adjustment: adjust for upstream material quality changes, late orders, or other confounders.
- Full cost accounting: capture one-time and recurring costs — integration, licensing (e.g., Siemens MES/Opcenter, Rockwell FactoryTalk, PTC ThingWorx, AWS IoT), change management, and cybersecurity controls.
Presenting ROI to different audiences
I tailor the story by audience:
- Operators: show reduced rework, fewer emergency maintenance events, and simplified workflows.
- Plant managers: focus on throughput, OEE, and predictable capacity.
- Finance / execs: show cash impact, payback, and risk-adjusted NPV. Run sensitivity analysis — “what if” scenarios for adoption and worst-case model performance.
- IT and cybersecurity: quantify integration effort, change to architecture, and residual cyber risk mitigations.
Concrete measurement table I’ve used in pilots
| Metric | Definition | Unit | Why it matters |
|---|---|---|---|
| OEE | Availability × Performance × Quality | % | Composite view of equipment effectiveness and capacity impact |
| Cycle time | Average processing time per unit | Seconds/minutes | Directly affects throughput and labor productivity |
| FPY / Scrap rate | % of units passing first inspection | % | Reduces material cost, rework and warranty exposure |
| Prediction accuracy | MAPE / ROC for model outputs | % / score | Shows whether models are reliable enough to act on |
| Payback | Months to recover pilot costs | Months | Common executive threshold for go/no-go |
In practice, a successful MBRP smart factory pilot will link model predictions to measurable operational changes and then to financial outcomes. Avoid the trap of measuring only “engagement” (dashboard hits) or vanity metrics. Instead, show how a change in a model-driven setpoint reduces a failure mode, shortens a setup, or prevents scrap — and then translate that into pounds saved or revenue enabled. Use a control group, capture full costs, and report uncertainty. That is how you make an ROI case that executives, plant teams and finance can all agree on.
