Edge inference · Quantization · Driver monitoring

INT8 made my models 6 to 11 times slower

I quantized two small vision models to INT8 for an in-cabin driver-monitoring project, expecting them to get faster. They got 3.6 times smaller and between 6 and 11 times slower on CPU. The quantization did what dynamic mode does. My assumption about what it was for did not.

The question

Does INT8 quantization make a small edge CNN faster on CPU, or only smaller?

The setup

Code, benchmark script and saved results: cabinsense.

The result

Model (MobileNetV3-Small, 128×128)fp32 medianINT8 medianSlowdownfp32 sizeINT8 size
Drowsiness classifier2.07 msp95 4.13 ms13.22 msp95 20.78 ms6.4×6.09 MB1.70 MB
Multi-task attributes1.47 msp95 2.20 ms15.71 msp95 23.98 ms10.7×6.11 MB1.71 MB

Slowdown is the ratio of medians. Both models shrank by a factor of 3.6, about 4.4 MB each. The p95 ratios agree: 5.0× and 10.9×.

To see what that means for a frame, add the face detector (YuNet, 3.82 ms median in the same run). Treat that figure as indicative: the benchmark timed the YuNet graph through ONNX Runtime on random input at a fixed shape, while the pipeline runs it through OpenCV's FaceDetectorYN at frame size. With fp32 classifiers the three models sum to about 7.4 ms, roughly 135 frames per second. With INT8 classifiers they sum to about 32.7 ms. A 30 FPS camera gives you 33.3 ms per frame, so the INT8 variants would spend almost the whole budget before landmarks, head pose, gaze or the object detectors run. These are sums of medians measured one model at a time, not an end-to-end pipeline measurement.

Why it got slower

What the evidence establishes. The quantization was dynamic. The file sizes dropped 3.6 times, so the weights really were stored as int8. The fp32 models were already very fast: 1.5 to 2 ms per call. And the INT8 versions were slower on the same CPU provider, in the same script, on random input of the same shape.

What it does not establish. I did not save a profile of either graph, so I cannot tell you which operators got slower, or how many of them were quantized. The gap between 6.4× and 10.7× comes from a single run of two nearly identical graphs, so I would not read anything into it. The finding that holds is roughly an order of magnitude.

The likely explanation, not measured here: in dynamic mode, weights are converted to int8 once, ahead of time, but the scale for each activation is computed at run time, on every call. So each quantized convolution gains extra work around it: quantize the incoming activation, run an integer convolution, rescale the result back to float. MobileNetV3-Small at 128×128 is a stack of many small convolutions, so that fixed per-operator cost is never paid back by faster arithmetic. It becomes the runtime. A second likely factor is that the integer path does not get the fused, tuned fp32 convolution kernels the CPU provider uses for the original graph.

"INT8 is faster" is an intuition from large models, where the arithmetic dominates. At this size the overhead likely dominates. To confirm it, the next run would turn on ONNX Runtime's profiler for both graphs and compare time per operator type.

The same mistake in a different place

The same project produced a second case of trusting an assumption I had not checked. The multi-task model originally had a phone head. On a clip with no phone in it, my write-up records that head's 90th-percentile score as about 0.99. (That figure lives in the report; I did not keep the raw per-frame log.) The lesson I drew at the time: a whole-image classifier has no concept of absence. So phone and drinking moved to a COCO-pretrained YOLOv8n, and smoking, seatbelt and cap to one consolidated YOLOv8n fine-tuned on merged open detection sets.

Rereading the code for this post, I found a blunter reason. The multi-task training script trains on CelebA for glasses and cap only, and masks the phone head out of the loss. The saved evaluation shows zero phone examples. If the checkpoint behind that number matches the current script, the head never saw a phone label. The absence argument still holds for a properly trained head, but my observation cannot support it; what it shows is that an untrained head can sound confident. The config still lists phone among the trained attributes, another stale artifact.

What I would do in production

Keep fp32 for these two models on CPU, which is what the runtime already loads, and treat INT8 as a size option that has to earn its place. The trade-off is concrete: fp32 costs about 8.8 MB more storage across the two models, and in return gives back about 25 ms of every 33 ms frame.

Limits

Code, benchmark script and saved results: github.com/trohit920/cabinsense. Previous post: Zero leaks, two wrong answers.

← All posts