What Happens When AI Starts Making the Decisions We Used to Make?

I was in a conversation recently about AI agents and orchestration, and we ended up spending quite a bit of time on decision rights. Which decisions could an agent make without involving a person? What guardrails would we need? What would trigger an escalation? And when a human did need to intervene, who should it be—someone in the frontline team, a manager, Risk, Legal, HR, perhaps even an executive? All sensible questions. Although I came away wondering whether we might be starting a little too far downstream.

Before we decide which decisions to delegate to an agent, do we actually understand how those decisions are made today? Not just what the policy says, or what appears in a process map, but what happens when somebody is faced with a real case, incomplete information and a decision that has to be made. In my experience, those can be three rather different things.

There’s the documented process. There’s the process people describe when you ask them how their work is done. And then there’s what they actually do, particularly when circumstances are messy. A policy might say an exception goes to Risk. In practice, somebody might first speak to a colleague they trust, check whether a similar situation has occurred before, and balance a customer relationship against a rule that isn’t quite as clear as it seemed on paper. None of that necessarily means people are doing the wrong thing. It can be where years of practical experience and tacit judgement are hiding.

This opens up a rather interesting design-research challenge. Interviews help, but I’d also want to explore real decisions and critical incidents: tell me about a time the process wasn’t straightforward; what information did you look for; whose advice did you seek; what changed your mind; and what made this case different from the last one? If possible, examine representative cases and their actual decision traces. It’s not about watching someone work for the sake of it. It’s about understanding the reasoning, the exceptions and the context that may never have made it into the official workflow.

Only then would I move into designing the agent’s decision architecture. Which decisions are sufficiently bounded and repeatable to delegate? Under which conditions can an agent proceed? What evidence threshold is acceptable? When does it stop, explain its uncertainty and ask for help? If two agents disagree, or a decision crosses business functions, whose rules take precedence? The human-in-the-loop idea is useful, but it’s only the beginning. There has to be a clear route for authority, accountability and meaningful review.

It connects to the work I’ve been exploring around ontologies and governed meaning. If we want agents to interpret an 'approved customer', an 'exception' or a 'material risk' consistently, we need to be explicit about what those words mean, where the rules apply, and who is allowed to change them. A knowledge graph or meaning layer can help make that context usable across systems and workflows. But we still need people to decide which definitions and behaviours are right for the organisation.

I can see a workshop here beginning with three parallel maps: how we say decisions should be made, how people explain that they make them, and how they are actually made in practice. The discrepancies would be the interesting part. From there, we could design a future-state decision pathway with agent authority, escalation thresholds, human decision owners, safeguards and a sensible way of learning from exceptions over time.

The question I keep coming back to is this: are we teaching AI how our organisation is supposed to make decisions, how it actually makes decisions, or how it should make decisions in the future? I’m increasingly convinced those are three different design briefs. And the third one, rather than being a technology exercise, is fundamentally a leadership decision.