Published 9 April 2026

Running AI Inference at the Edge on Industrial Hardware

Most industrial AI projects do not fail on the model. They fail on the gap between a model that works on a workstation and a system that runs it twenty-four hours a day, in a cabinet, on a production line, for five years.

Closing that gap is a hardware sizing exercise, and it is more tractable than the surrounding discussion usually suggests.

Training and Inference Are Different Workloads

The distinction matters because it determines where each belongs.

Training builds the model. It is enormously compute-intensive, runs on large datasets, tolerates hours or days of latency, and happens occasionally. It belongs in a data centre or the cloud.

Inference runs the finished model against new data. It is far cheaper per operation, must often complete in milliseconds, and happens continuously. It belongs where the data is.

Almost every industrial AI deployment trains centrally and infers at the edge. The edge hardware question is therefore only ever about inference.

Reading the TOPS Figure Honestly

Edge AI hardware is marketed in TOPS — trillions of operations per second. It is a useful shorthand and a poor specification.

The number is usually quoted for INT8 arithmetic. Run the same model at FP16 and effective throughput falls substantially. It is also a peak figure, achieved when the model maps cleanly onto the accelerator's architecture; real models rarely achieve peak utilisation.

More importantly, TOPS says nothing about memory bandwidth, which is frequently the actual bottleneck. A model that does not fit in on-chip memory spends its time waiting for data, and no amount of arithmetic capability helps.

The only figure that means anything is measured throughput for your model, at your input resolution, at the accuracy you need. Everything before that measurement is an estimate.

Where the Inference Runs

There are three practical options in industrial hardware.

CPU only. Modern processors with vector instructions handle small models adequately. Classification on low-resolution images, tabular anomaly detection, audio event detection, simple time-series prediction — all run acceptably on an Intel N100 or J6412 class part. If your workload fits here, this is the cheapest and simplest answer, and it deserves to be tested before anything more elaborate is specified.

Integrated GPU. Intel's integrated graphics support inference through OpenVINO, which is genuinely effective for vision workloads and requires no additional hardware. A 12th-generation Core or Core Ultra part with Iris Xe graphics handles real-time object detection at useful frame rates. Our IBOX-3226 and N3422 fall into this category, and the N3422's Core Ultra platform adds a dedicated NPU alongside the GPU.

Dedicated NPU. A neural processing unit is silicon built specifically for inference, and it delivers far better performance per watt than a CPU or GPU doing the same work. This is what makes continuous inference practical in a fanless, low-power enclosure. The NANO-A352 is built on the Rockchip RK3588 with a 6 TOPS NPU, 8K video decode and a 9–15 V wide-voltage input — an ARM platform designed for exactly this role.

The Workloads That Actually Get Deployed

Edge inference in industry is less exotic than the term suggests. The recurring applications are:

Visual quality inspection. A camera over a line, a model trained on good and defective parts, a reject signal to the controller. The most common industrial AI deployment by a wide margin, and the one with the clearest payback.

Predictive maintenance. Vibration, current and temperature signatures processed locally, with anomalies flagged before a failure. Local processing is essential here because the sample rates are too high to ship raw.

Presence, counting and safety. People counting in retail, occupancy in buildings, detection of a person in a hazardous zone, PPE compliance checks.

Reading things. OCR on labels, licence plate recognition at gates and car parks, barcode reading where a dedicated scanner is impractical.

Process optimisation. Models that adjust setpoints from sensor patterns, keeping a process inside tolerance with less operator intervention.

Sizing Is an Empirical Exercise

The reliable method is unglamorous:

Start with the real model and the real data. Not a benchmark. Your model, your input resolution, your frame rate.
Establish the latency budget. A reject mechanism 300 mm downstream on a line moving at 1 m/s gives you 300 ms, total, including capture and actuation. That is the number to design against.
Test on the CPU first. If it meets the budget, stop. A great many industrial models are small enough.
Try the integrated GPU next. OpenVINO on Intel graphics closes most of the remaining cases without new hardware.
Specify an NPU when the first two fall short, or when the power and thermal budget rules them out.
Measure sustained performance, not peak. A system that hits the target for thirty seconds and then throttles has not met the requirement. Run it hot, for hours, at the ambient temperature of the actual site.
Leave headroom. Models get retrained and grow. A system at 95 % utilisation on day one has no room for the next version.

Two software-side techniques are worth knowing because they often remove the need for bigger hardware entirely: quantisation to INT8 typically delivers a large speed-up for a small accuracy cost, and model selection matters more than people expect — a compact architecture chosen for edge deployment can be several times faster than a general-purpose one at comparable accuracy.

The Unglamorous Requirements Still Apply

An inference node on a production line is an industrial computer first. It needs fanless construction because a fan in a dusty plant is a scheduled failure. It needs to tolerate the cabinet's real ambient temperature while sustaining a continuous compute load — inference is not a bursty workload, and the thermal design has to account for that. It needs industrial-grade storage, because logging inference results continuously wears consumer flash. It needs a watchdog and auto power-on, because nobody is standing next to it. And it needs remote management, because models get updated and the cabinet is not somewhere you want to visit.

Getting the Sizing Right

We supply the platforms this runs on: fanless industrial PCs on Intel Core and Core Ultra processors with integrated graphics capable of real-time vision work, compact ARM systems with dedicated NPU acceleration, and the I/O to connect cameras, sensors and controllers in the same box.

If you have a model and a latency target, send us both. Sizing against a measured workload produces a system that works; sizing against a TOPS figure produces a surprise during commissioning.

From SCADA to the Edge: Where Should Your Control Data Be Processed?

Understanding Edge Computing: The Future of Data Processing

Enhancing Smart City Deployments with Industrial Edge Computing