INT8 made my models 6 to 11 times slower
I quantized two small vision models to INT8 for an in-cabin driver-monitoring project, expecting them to get faster. They got 3.6 times smaller and between 6 and 11 times slower on CPU. The quantization did what dynamic mode does. My assumption about what it was for did not.
The question
Does INT8 quantization make a small edge CNN faster on CPU, or only smaller?
The setup
- Project: CabinSense, a local driver and occupant monitoring system that produces eleven signals from one camera (drowsiness, distraction, gaze zone, yawning, phone use, drinking, smoking, seatbelt, glasses, cap and occupant count).
- Models quantized: two classifiers on the same MobileNetV3-Small backbone (
mobilenetv3_small_100from timm), taking a 128×128 RGB face crop: a two-class drowsiness model and a multi-task face-attribute model. Both are exported to ONNX (opset 17) with a fixed input shape of 1×3×128×128. - Quantization: ONNX Runtime's
quantize_dynamicwithweight_type=QInt8and otherwise default settings. Dynamic mode, not static QDQ: no calibration data. - Measurement: ONNX Runtime with
CPUExecutionProvider, the only provider recorded in the results. Random float input at the model's shape, 5 warm-up runs, then 100 timed calls tosession.run. I report the median, the 95th percentile and file size. Default session options, one run per model. - Hardware: one development CPU. The results file does not record the CPU model, thread count or ONNX Runtime version, which I regret now. No GPU was involved.
Code, benchmark script and saved results: cabinsense.
The result
| Model (MobileNetV3-Small, 128×128) | fp32 median | INT8 median | Slowdown | fp32 size | INT8 size |
|---|---|---|---|---|---|
| Drowsiness classifier | 2.07 msp95 4.13 ms | 13.22 msp95 20.78 ms | 6.4× | 6.09 MB | 1.70 MB |
| Multi-task attributes | 1.47 msp95 2.20 ms | 15.71 msp95 23.98 ms | 10.7× | 6.11 MB | 1.71 MB |
Slowdown is the ratio of medians. Both models shrank by a factor of 3.6, about 4.4 MB each. The p95 ratios agree: 5.0× and 10.9×.
To see what that means for a frame, add the face detector (YuNet, 3.82 ms median in the same run). Treat that figure as indicative: the benchmark timed the YuNet graph through ONNX Runtime on random input at a fixed shape, while the pipeline runs it through OpenCV's FaceDetectorYN at frame size. With fp32 classifiers the three models sum to about 7.4 ms, roughly 135 frames per second. With INT8 classifiers they sum to about 32.7 ms. A 30 FPS camera gives you 33.3 ms per frame, so the INT8 variants would spend almost the whole budget before landmarks, head pose, gaze or the object detectors run. These are sums of medians measured one model at a time, not an end-to-end pipeline measurement.
Why it got slower
What the evidence establishes. The quantization was dynamic. The file sizes dropped 3.6 times, so the weights really were stored as int8. The fp32 models were already very fast: 1.5 to 2 ms per call. And the INT8 versions were slower on the same CPU provider, in the same script, on random input of the same shape.
What it does not establish. I did not save a profile of either graph, so I cannot tell you which operators got slower, or how many of them were quantized. The gap between 6.4× and 10.7× comes from a single run of two nearly identical graphs, so I would not read anything into it. The finding that holds is roughly an order of magnitude.
The likely explanation, not measured here: in dynamic mode, weights are converted to int8 once, ahead of time, but the scale for each activation is computed at run time, on every call. So each quantized convolution gains extra work around it: quantize the incoming activation, run an integer convolution, rescale the result back to float. MobileNetV3-Small at 128×128 is a stack of many small convolutions, so that fixed per-operator cost is never paid back by faster arithmetic. It becomes the runtime. A second likely factor is that the integer path does not get the fused, tuned fp32 convolution kernels the CPU provider uses for the original graph.
"INT8 is faster" is an intuition from large models, where the arithmetic dominates. At this size the overhead likely dominates. To confirm it, the next run would turn on ONNX Runtime's profiler for both graphs and compare time per operator type.
The same mistake in a different place
The same project produced a second case of trusting an assumption I had not checked. The multi-task model originally had a phone head. On a clip with no phone in it, my write-up records that head's 90th-percentile score as about 0.99. (That figure lives in the report; I did not keep the raw per-frame log.) The lesson I drew at the time: a whole-image classifier has no concept of absence. So phone and drinking moved to a COCO-pretrained YOLOv8n, and smoking, seatbelt and cap to one consolidated YOLOv8n fine-tuned on merged open detection sets.
Rereading the code for this post, I found a blunter reason. The multi-task training script trains on CelebA for glasses and cap only, and masks the phone head out of the loss. The saved evaluation shows zero phone examples. If the checkpoint behind that number matches the current script, the head never saw a phone label. The absence argument still holds for a properly trained head, but my observation cannot support it; what it shows is that an untrained head can sound confident. The config still lists phone among the trained attributes, another stale artifact.
What I would do in production
Keep fp32 for these two models on CPU, which is what the runtime already loads, and treat INT8 as a size option that has to earn its place. The trade-off is concrete: fp32 costs about 8.8 MB more storage across the two models, and in return gives back about 25 ms of every 33 ms frame.
- Fix the manifest and the docs. The Python runtime loads the fp32 files, but the deployment manifest, the contract a C++ port would follow, still points at the
*.int8.onnxfiles. The quantization module's docstring and the deployment guide still describe dynamic INT8 as faster on CPU. The benchmark contradicts the manifest and the docs. - Try static QDQ quantization next. It computes activation scales ahead of time from a calibration set, which removes the per-call scale computation. It needs a few hundred representative cabin frames, and it is a hypothesis until benchmarked.
- Benchmark on the target, not the development machine. On a GPU or NPU, fp16 through the vendor's execution provider is the more likely win.
- Gate the release on the benchmark. If the variant in the manifest is slower than its fp32 baseline on the target, the build should fail.
Limits
- One run on one unrecorded CPU. 100 timed iterations per model, random input, default threading. The direction is clear; the exact ratios are not portable.
- Accuracy after quantization was not measured. The parity check compares PyTorch with the fp32 ONNX models only. I do not know what INT8 did to accuracy.
- Partial latency coverage. Only YuNet and the two classifiers were benchmarked. The PFLD landmark model is ONNX and required, but it is not in the results; head pose, gaze and both YOLO detectors run in PyTorch and are also excluded. So the 7.4 ms figure covers detection plus the two classifiers, not the full face path.
- The phone number has no saved log. Treat it as an observation from one clip, not a measured false-positive rate. The project's demo uses public dashcam-style footage to exercise the pipeline, not to score it.
Code, benchmark script and saved results: github.com/trohit920/cabinsense. Previous post: Zero leaks, two wrong answers.