People
I like to have hands-on experience with the topics I write about, so I have been building some simple LangChain agents on my home AI set-up, affectionately known as “The Toaster” (the Lenovo Neo is about the same size and shape as the four-slice variety).
The results were pleasing, and the process was fairly straightforward. Agentic AI is much less brittle than the old Robotic Process Automation (RPA), which was going to automate 25% of tasks across every job category by 2019[1] but never made it out of the pilot stage.
However, my enjoyment from tinkering with agents is tempered by real concerns about their implications.
As a tech-pragmatist, I observe that humans typically muddle through technological change, failing to achieve the original idealistic vision but also avoiding the worst-case scenarios.
Along this middle path, there are disappointments, disruptions, and sometimes adverse impacts on human wellbeing. Yet, overall, I believe that humanity has benefited more from technological advances than it has suffered.
Therefore, my anxiety over agentic AI is not my normal mode.
In a nutshell, my worry is that goal-driven autonomous agents without counterbalancing morality, predictability, transparency, or a human veto are a recipe for disaster.
I am not worried about agents that read emails or design web front ends, which is what most of them do today. The danger is in consequential decisions that cannot easily be undone. Attachment to weapons systems or critical infrastructure are two obvious examples, but there are plenty of situations in business, like making payments, that could cause real problems.
We already know that generative AI cannot take responsibility for its actions. If you ask an AI model about this, it will place the onus firmly back on its developers or you.
In addition, the transformers at the heart of generative AI models that power agents are probabilistic, not deterministic. That means they may not behave consistently, even under the same conditions. This is not a teething problem but rather a feature of their design. Engineering and scaling will reduce the variations and errors, but AI agents are never going to be entirely predictable or reliable.
A good example of unexpected AI behaviour – even before agents – is from the Google Brain team in 2016, who conducted an experiment using three neural networks: Alice, Bob, and Eve. Alice’s goal was to send Bob a secret message, while Eve tried to intercept it (or “eavesdrop”). Remarkably, over time, Alice and Bob developed their own improvised form of encryption, preventing Eve from intercepting the message.
The bigger point is that the human researchers couldn’t read the encrypted message or explain how the cipher worked. This raises the unsettling possibility that in complex agentic systems, AI could develop strategies or representations that humans cannot understand. In extreme cases, this could mean humans get locked out of their own systems altogether.
If agentic AI were likely to remain niche, then the risk would be limited. However, genuine productivity benefits from generative AI are mostly based on the agentic hope. Customers and frontier labs both need a win from their investments, which is likely to accelerate both agentic product development and implementation.
There are ways to mitigate these risks. Comprehensive frameworks and engineering standards exist that, if followed, would go a long way towards making agentic AI safe. However, there is no compulsion for anyone, either vendor or client, to adopt them. Also, no current AI regulation specifically covers agents. Tech companies want safe AI, but the commercial pressures are real.
Considering all this, human-in-the-loop frameworks at least ensure that agentic AI is not entirely autonomous. I prefer Salesforce’s slogan of “humans at the helm”, which implies control, not just participation. However, this approach is only as good as our ability to know what the system is doing and then override it in a timely fashion. I am not convinced these two simple requirements are prioritised on every agentic AI roadmap.
This means we face the prospect of powerful autonomous AI agents that are unaccountable, unpredictable, lack transparency, and may be difficult to stop.
What should we do?
Educate ourselves and our teams on the risks of agentic AI in addition to the benefits.
Adopt comprehensive and specialised ethical, risk, and engineering frameworks that cover agents.
Communicate to vendors that agentic AI safety is a priority.
Prove all agents thoroughly in pilots before they are scaled.
Rehearse what would happen if an AI agent became problematic under a range of scenarios.
If these precautions are taken, we can at least control our local world and balance productivity gains with agentic risks. The onus is on us to take responsibility.
However, to put things into perspective, my AI agent has just emailed me to say it has failed and suggests the best idea is for me to do the task myself.
Back to The Toaster.
If you'd like to listen to an audio version of this article, please click here.
[1] pwc: People, change… and robots, February 2016
