Agentic AI has arrived. Across every session at the AI Agents Conference in Sydney, the same truth kept surfacing: the organisations that will deploy with confidence are not the ones with the biggest ambition. They are the ones with the clearest framework.
I spent two days last week at the AI Agents Conference in Sydney, sitting in sessions that ranged from deep technical architecture to C-suite adoption strategy. I went with Ben, Back on Track Foundation’s General Manager, because AI literacy at every level of leadership matters. And I was fortunate to have Kara Bombell join us, bringing her own sharp perspective to what was a genuinely rigorous two days.
I came away with pages of notes, a LinkedIn network considerably larger than it was on Monday morning, and a clearer sense of exactly where the governance profession needs to move next. Not because governance was absent from the conversation. Quite the opposite. It was the connective tissue running through almost every presentation, the thing speakers kept returning to as the non-negotiable precondition for everything else.
The Architecture Is Moving Fast
For anyone advising boards on AI risk right now, understanding the basic architecture of agentic systems is no longer optional. One of the most useful sessions of the conference walked through the difference between two emerging protocols that are reshaping how AI systems interact with tools and with each other: Anthropic’s Model Context Protocol (MCP) and Google’s Agent-to-Agent Protocol (A2A).
MCP describes a world where a single AI agent connects to multiple tools and APIs through a centralised server architecture. The host application determines which tools to call and when, and the agent’s reach is orchestrated from a central point. A2A describes something structurally different: multiple agents discovering each other and collaborating peer-to-peer, delegating tasks dynamically between specialised systems without a single point of control.
That distinction matters for governance in ways that have not yet been adequately absorbed into most board-level conversations. Centralised architectures, however complex, preserve a degree of auditability. Peer-to-peer multi-agent systems, where agents are discovering each other dynamically and acting on delegated instructions, create accountability gaps that current frameworks were simply not designed to address. Who is responsible when an autonomous agent makes a consequential decision on behalf of your organisation, and neither the decision nor the chain of reasoning was reviewed by a human?
“The companies that fail don’t lack ambition. They lack a framework.”
That line appeared on a slide during one of the adoption strategy sessions and it is, frankly, the most honest summary of the current enterprise AI landscape I have heard in some time. Ambition is not the constraint. The constraint is the absence of structured thinking about what you are deploying, how you will govern it, how you will measure it, and how you will expand or pull back as you learn.
Governance as Foundation, Not Afterthought
What distinguished this conference from some of the more breathless AI events of the past few years was how consistently governance appeared at the centre of the technical conversation rather than appended at the end. The Agentic Reasoning Cycle, presented across multiple sessions as a framework for understanding how autonomous systems function, describes a loop of Perceive, Plan, Act and Reflect. The Reflect phase, covering post-hoc analysis, memory consolidation, and ongoing learning, is where governance either lives or doesn’t. It is where you find out whether the system did what you intended, whether it produced outputs you can stand behind, and whether the decisions it made were ones your organisation is prepared to be accountable for.
The practical adoption framework presented by one speaker was similarly grounded. The argument was structured and sequenced: identify a high-volume, low-risk, rule-heavy process as your first agent deployment. Build the infrastructure before you build the agent, because you cannot bolt autonomous systems onto broken data architecture. Keep humans in the loop, not as a temporary concession but as a deliberate strategy for earning trust incrementally. Measure accuracy, cost per task, time saved and agent reliability, because you cannot govern what you do not measure. Then, once a beachhead is proven, scale to adjacent processes. Multi-agent orchestration comes after single-agent mastery, not before.
That sequence is not just good technology practice. It is good governance practice. It reflects a maturity model that boards and executive teams should be asking their technology leaders to articulate clearly before any significant investment commitment is made.
The Question Boards Need to Be Asking
There is a conversation that needs to happen at board level that most organisations have not yet had, and the conference reinforced why it is becoming urgent. As agentic systems move from recommending actions to executing them, the governance questions shift from ‘what did the AI suggest?’ to ‘what did the AI do, why, and who authorised it?’ Those are fundamentally different accountability questions, and most governance frameworks in use today were designed for the former.
The scalable system architecture discussions were a useful illustration of this. Modular design, cloud readiness, real-time data processing: all legitimate technical considerations. But the governance and compliance layer is not a technical feature to be added at deployment. It is a design requirement that needs to be specified at the architecture stage, before the build begins. Boards that are not asking to see evidence of that layer in the technical design documentation are not yet asking the right questions.
I will be applying what I took from this conference directly to my work with boards and senior leadership teams, and to how Back on Track Foundation continues to build and govern AI within our own operations. A children’s charity working with medically vulnerable students has no margin for AI systems that operate outside clear ethical and governance boundaries. That is not a constraint on innovation. It is the condition under which responsible innovation becomes possible.
The Broader Stakes
What I keep returning to, sitting with two days of content, is how much the governance conversation has matured in twelve months. The question is no longer whether organisations should adopt agentic AI. It is whether they have built the internal capability to do so responsibly. That requires frameworks that are designed for dynamic, multi-agent architectures, not static automation. It requires boards that understand enough about the technology to ask penetrating questions. And it requires leadership that treats governance not as a compliance burden but as the thing that makes confident, sustained deployment possible.
The organisations that get that sequence right will move faster and more safely than those that treat governance as something to retrofit after the fact. That is the central lesson of two days in Sydney, and it is one worth carrying back into every boardroom conversation about AI.
Kylie Dalton is an AI ethics and governance consultant working at board and senior leadership level. She is Chair of the AI Expert Panel at the Governance Institute of Australia and Founder and CEO of Back on Track Foundation. Her work focuses on AI risk in complex public systems, with particular attention to vulnerable populations.
