Engineering management · Delivery

Running AI delivery under SAFe: what I'd keep and what I'd drop

For three years at OpenSysNet I ran AI delivery inside the Scaled Agile Framework (SAFe): railway pantograph monitoring, sorting-robot vision and industrial time-series models. SAFe was designed for feature delivery, where the work is mostly known and the risk is in execution. Model delivery is the other way round. Here is what I would keep and what I would change if I set up delivery for an AI team again.

The question

Can a planning framework that rewards predictable commitments work for a kind of engineering where you often do not know whether something is possible until you have tried it?

My answer is yes, if you are clear about which parts of the work are predictable and which are not, and you plan them differently.

Two kinds of work in one backlog

An AI team's backlog mixes two kinds of item that look alike on a board:

Most of the friction I would expect between SAFe and an ML team comes from treating the second kind like the first.

What I would keep

What I would change or drop

A definition of done for a model

If I had to keep only one practice, it would be an explicit definition of done that a model must pass before it counts as delivered. Mine would have four gates:

GateQuestion it answers
Held-out metricDoes it meet the agreed threshold on a frozen test set it never trained or tuned on?
Target benchmarkDoes it meet the latency and memory budget on the hardware it ships on, not the development machine?
Failure reviewHas someone looked at the worst errors one by one and accepted them?
Field checkDoes it behave the same on live input from the real site as on the test set?

Each gate maps to a failure I have since measured in my own projects: choosing a signal on test data quietly leaks, an "optimised" model can be slower on the real runtime, and aggregate counts hide the failure that matters.

The trade-off

Planning experiments as time-boxed questions makes the plan less certain on paper. Stakeholders get "we will know by the end of the iteration" instead of "it will be done by the end of the PI". In exchange, the plan stops making promises the team cannot keep, and the risky questions get answered first, when the answer is still cheap to act on.

Limits

Previous post: Where do recommender wins actually come from? Next: From dashcam ADAS to LLM evaluation: one question for nine years.

← All posts