Running AI delivery under SAFe: what I'd keep and what I'd drop
For three years at OpenSysNet I ran AI delivery inside the Scaled Agile Framework (SAFe): railway pantograph monitoring, sorting-robot vision and industrial time-series models. SAFe was designed for feature delivery, where the work is mostly known and the risk is in execution. Model delivery is the other way round. Here is what I would keep and what I would change if I set up delivery for an AI team again.
The question
Can a planning framework that rewards predictable commitments work for a kind of engineering where you often do not know whether something is possible until you have tried it?
My answer is yes, if you are clear about which parts of the work are predictable and which are not, and you plan them differently.
Two kinds of work in one backlog
An AI team's backlog mixes two kinds of item that look alike on a board:
- Engineering work: data pipelines, labelling tools, export to the edge runtime, integration with the camera or PLC, monitoring. This behaves like any other software. It can be sized, committed to and delivered.
- Experiments: "can a detector reach the accuracy we need on this camera, at this frame rate, on this hardware?" Nobody can size that honestly, because the answer is the point of doing it.
Most of the friction I would expect between SAFe and an ML team comes from treating the second kind like the first.
What I would keep
- PI planning, for dependencies. The things that block ML work are rarely inside the team: access to the target hardware, a site visit to collect footage, labelling capacity, a field-test window. A planning event every few months, where those dependencies are put on one board with dates and owners, is the most useful part of the framework for an AI team.
- System demos. A model that only exists as a metric in a notebook is not done. Demoing it running end to end, on the real input and ideally the real hardware, every iteration, surfaces integration problems early, while they are still cheap.
- A shared definition of done. SAFe pushes teams to agree what "done" means. For models, that agreement is worth more than in most software, because "the model works" means very different things to a researcher, an integrator and a customer.
- Inspect and adapt. A regular, structured look back is where a team notices that its evaluation set no longer matches what the field is sending it.
What I would change or drop
- Committing to accuracy targets as if they were features. "Reach a given mAP by the end of the PI" is a guess written as a promise. I would commit to the experiment instead: the question, the data, the time box, and the decision that each possible answer leads to.
- Story points for research. Sizing an experiment produces a number that looks like information and is not. Time boxes are more honest: two weeks to find out, then decide.
- Velocity as the headline metric for an AI team. Velocity measures how much planned work was finished. A team that runs a quick experiment, learns the approach will not work and stops has done something valuable that velocity scores as nothing. I would track decisions made and models that pass the definition of done.
- Heavy ceremony around small teams. A small ML team inside a large train can spend a lot of its time in events designed for many teams. Attend the events where dependencies are decided; delegate or skip the rest.
A definition of done for a model
If I had to keep only one practice, it would be an explicit definition of done that a model must pass before it counts as delivered. Mine would have four gates:
| Gate | Question it answers |
|---|---|
| Held-out metric | Does it meet the agreed threshold on a frozen test set it never trained or tuned on? |
| Target benchmark | Does it meet the latency and memory budget on the hardware it ships on, not the development machine? |
| Failure review | Has someone looked at the worst errors one by one and accepted them? |
| Field check | Does it behave the same on live input from the real site as on the test set? |
Each gate maps to a failure I have since measured in my own projects: choosing a signal on test data quietly leaks, an "optimised" model can be slower on the real runtime, and aggregate counts hide the failure that matters.
The trade-off
Planning experiments as time-boxed questions makes the plan less certain on paper. Stakeholders get "we will know by the end of the iteration" instead of "it will be done by the end of the PI". In exchange, the plan stops making promises the team cannot keep, and the risky questions get answered first, when the answer is still cheap to act on.
Limits
- One company, one team, a few years. This is one manager's view, not a study.
- No internal numbers. I have left out project metrics, client details and anything my former employer would not want public.
- SAFe is configured differently everywhere. Much of what I would drop may already be optional where you work.