When should a model be trusted?
Each post takes one version of that question and answers it with a measured result: the setup, the number, the surprise, and what I would do about it in production.
-
· Career
From dashcam ADAS to LLM evaluation: one question for nine years
Dashcam ADAS, edge vision on factory and railway hardware, LLM agents in manufacturing: three jobs that kept asking when a model's output deserves to be acted on.
-
· Engineering management
Running AI delivery under SAFe: what I'd keep and what I'd drop
SAFe was built for work whose risk is in execution. Model work carries its risk in the unknown. Keep PI planning and system demos; plan experiments as time-boxed questions, not accuracy promises.
-
· Recommenders
Where do recommender wins actually come from?
The retriever did most of the work. The second stage added 12% NDCG@10 for 67x the median latency, and the ranker's top feature appears to be partly an artifact of how I built its training data.
-
· Engineering leadership
Leading an AI-augmented build
Coding agents wrote most of the code for a live 3D browser game. What I owned: the brief, the gates, separate review for every change, and disputes settled by measurement.
-
· Edge inference
INT8 made my models 6 to 11 times slower
Dynamic INT8 made two small driver-monitoring classifiers 3.6 times smaller and much slower on CPU. For backbones this small, quantization speed has to be measured, not assumed.
-
· RAG · Access control
Zero leaks, two wrong answers
Access control kept every confidential passage away from the model. An under-clearance user still got confident, cited, wrong answers, and a similarity gate cannot catch the plausible ones.
-
· LLM evaluation
The free uncertainty signal kept up with the expensive one
A single forward pass matched 10-sample semantic entropy at catching a local LLM's wrong answers. The NLI step meant to help was merging answers that disagree.