Your agent returns HTTP 200 on every call, p99 latency looks fine and the error rate is flat. Meanwhile it has been quietly making up answers for two weeks, ever since someone swapped the model. None of your alerts fired, because none of them were looking at what the agent said.
Grafana Labs wants to close that gap by applying SLOs for AI agents to behaviour, not only uptime. In a blog post dated 24 September, Grafana's Ivana Huckova proposed scoring behaviours such as groundedness and task fulfilment, setting a target for each and treating the rest as an error budget. For SREs who own agents in production, the practical step is to pick a few behaviours, measure each one on its own and alert on slow budget burn.
What is Grafana proposing?
The idea, covered by InfoQ on 16 October, takes the standard SRE toolkit and points it at quality. Each behaviour gets an evaluation. The evaluation produces a pass or fail per conversation. The share of passes becomes the SLI, and the SLI is checked against an SLO.
Grafana's example is 95% of conversations judged fulfilled over a 30-day window. The remaining 5% is the error budget. Grafana's post frames that 5% as something to spend on experiments, such as a new prompt or a different model.
How do you define an SLO for an agent?
Start with the agent's job, not the model. Ask what a bad outcome looks like for the person using it. Then write one SLI per failure type.
Grafana's examples, and the ones InfoQ listed, make a reasonable starting set:
Groundedness. Was the answer based on the context the agent was given?
Fulfilment. Did the agent do what the user asked?
Safety. Was the output toxic, did it leak personal data, did it fall for a prompt injection?
Cost. Did it reach the answer at a sensible token spend?
Latency. The old one still counts.
Keep them separate. Grafana's argument is that a good answer is really several behaviours, and they fail independently. A single blended quality score lets a collapse in groundedness hide behind a strong fulfilment number. You would not average CPU and disk into one health score either.
How do you measure behaviour at all?
Two ways, and you want both. Deterministic checks are cheap and dull: a regex for leaked API keys, a schema check on tool calls, a list of banned terms. Use them wherever a rule can decide.
For the rest, Grafana uses an LLM as a judge. Another model reads a conversation, message or tool call and returns a pass or fail, or a score you turn into one. Grafana's example counts a groundedness score of 4 or more on a 1 to 5 scale as a pass.
That is the catch. The SLO is only as honest as the judge. InfoQ pointed out that a judge that is consistently wrong gives you false precision, a tidy 97% that means nothing. Before you trust the number, check the judge against a sample of human-labelled conversations, and check again whenever you change the judge model. Count the judge's own cost too. Grafana's dashboards track it next to the pass rate.
InfoQ also noted the split between offline evaluations, run against known scenarios before a release, and online evaluations on live traffic. You need the offline set to catch regressions before they ship and the online set to catch drift you did not predict.
What should you alert on?
Not single bad answers. One rude reply is a bug report, not a page. Grafana's post argues that sharp spikes barely touch a 30-day budget, and the real risk is a slow, quiet burn where quality slips a little each day and no single conversation looks alarming.
That is exactly what burn-rate alerting was built for. If you already use multi-window burn-rate alerts for availability, reuse the pattern: a fast window to catch a broken deploy and a slow window to catch drift. Neither Grafana nor InfoQ prescribes thresholds, so start from the ones you trust today and tune.
Where should targets come from?
From history, not ambition. Run evaluations for a few weeks, see where the agent actually sits, set the SLO a little below that and tighten it as the agent improves. Grafana suggests a young internal agent can live with a much looser target than one that faces customers.
Then write the error budget policy, because that is where the value is. When budget is healthy, the team can ship new prompts and models. When it runs low, feature work pauses and quality work takes over. Before any prompt or model change, replay it against a known set of conversations to check it stays in bounds.
The agent will keep returning 200s either way. What matters is whether anyone notices when it stops being useful.
Quick answers
What is an SLO for an AI agent?
It is a target for the share of agent interactions that pass a behaviour check. Grafana's example is 95% of conversations judged fulfilled over 30 days, which leaves a 5% error budget.
How do you measure agent quality for an SLI?
Use deterministic checks such as regex or schema rules wherever a rule can decide, and an LLM judge for the rest. Measure each behaviour separately and check the judge against human-labelled samples.
Should you page on a single bad agent answer?
No. Alert on error budget burn, especially slow burn over days, which shows gradual drift after a model, prompt or retrieval change.
