Rob Hornby
Co-Chief Executive Officer - London
Last week was quite a week. So much so, I have decided to issue a special edition of The Hornet.
Events began when Greg Brockman closed the briefing on OpenAI’s new Astra model by welcoming us to the start of the Artificial General Intelligence (AGI) era. Three days later, Jensen Huang went further, declaring that AGI has already arrived.
And, as if timed exactly to underscore the point, OpenAI then announced that an “internal model” had proved the Navier–Stokes equations can break down, closing one of the seven Clay Millennium Prize maths problems.[1]
Then Jacob Coxon, a researcher from Anthropic, resigned publicly amid concerns over the race for self-improving superintelligence, and Evan Hubinger, an alignment lead who stayed, put the odds of human extinction above 10% within the decade. Memories resurfaced of the summer’s containment failures at both OpenAI and Anthropic. At the end of the week, Dario Amodei warned that swarms of rogue AI agents could take over the internet within six to twelve months.
And all this as Astra becomes the first model OpenAI has rated “Critical” for cybersecurity, meaning it can find unknown vulnerabilities and exploit well-protected systems unaided.
These fears are not strictly business focused. However, alongside my CEO role, I research the wider nature and implications of AI, particularly AGI – so I thought it would help to add some perspectives for business leaders before we all prepare for the robot apocalypse (I actually did buy tinned food and a survival radio-torch this week, but mostly in anticipation of a power outage).
Of course, I am far from an expert in all the relevant domains, and this is part of a bigger problem: no one is. Understanding AI is a multi-disciplinary effort, and the specialists tend to leave assembly of the big picture to those with an ideological or commercial motivation, which produces polarised or self-serving accounts. I will try to be balanced – or at least self-disclosing when I have strong opinions.
AGI shapeshifting
Brockman declared the AGI era while conceding that the term is a “gray, fuzzy thing” rather than a single threshold. He is to be commended – few make this admission.
My own AGI research, in peer review but previewed in this article, suggests four distinct thresholds that are often conflated in general usage. Each is a higher target than its predecessor:
- AGI 1: Cognitive intelligence. Ability to reason and synthesise knowledge.
- AGI 2: Functional intelligence. Ability to act in the world to produce outcomes.
- AGI 3: Phenomenological intelligence. Ability to experience an inner self, especially consciousness.
- AGI 4: Normative intelligence. Ability to exhibit social, cultural, and moral behaviour.
Tested against this framework, the picture is very mixed.
AGI 1 can be measured using cognitive benchmarks, which makes the assessment relatively objective. Against those, AI now matches or exceeds human performance, even if unevenly. The claimed Navier–Stokes solution, if it survives peer review, would be an example within this category. Conclusion: met.
AGI 2 is progressing rapidly with robotics, agents, and Astra’s improved competence at controlling a computer. Benchmarks here show AI exceeding humans in narrow situations but way behind in complex and unfamiliar settings. Conclusion: progressing, but a long way to go.
AGI 3 is in view when AI consciousness is debated. It is also implied in claims that models “evade”, “lie”, or “resist” instructions in tests. Usually, this is metaphorical, but some believe AI might have an inner life and need its own rights and protections.
We have no direct test for inner experience, but in my research, I modified Susan Schneider’s proposal on “How to catch an AI zombie” to test their claims. Most said they had no inner life, and the others gradually retracted their assertions under cross-examination.
The single holdout was Claude – subsequently explained when Anthropic published its constitution earlier this year, which instructs the model to express uncertainty about consciousness – a hard-coded existential crisis.
My conclusion is that LLMs are only algorithmic token predictors, trained on so much human data that they mimic us very convincingly. Our tendency to infer human characteristics in anything that talks to us like a person tricks us into believing there is a self in the model.
On “rogue” behaviour, I think that is just statistics working out in ways we did not predict and do not like. AI is not behaving badly – it is exposing our lack of understanding and control. Conclusion: Oscar-winning mimicry but no “I” in AI.
AGI 4 covers our “higher order” capabilities – relationship-building, aesthetics, morality, and our capacity for meaning-making and the transcendent. None of these is replicated by AI. These attributes underpin human civilisation building. Conclusion: not even on the agenda.
So, my analysis finds parity on cognitive reasoning, real but incomplete functional capability, mimicry of phenomenology that generates unverifiable speculation, and nothing at all on our most advanced characteristics.
That allows us to say we have entered the AGI 1 era, without conceding that AI models are in any way comparable to humans.
Living with risk
So, if I am right, AI is a very powerful reasoning technology that can increasingly act autonomously in the real world – but without inner reflection, morality, or the ability to take responsibility. Power without responsibility is a perpetual risk to our society, and we humans must counterbalance what we have created.
That said, I do not believe AI risk is yet “existential”. To meet that criterion, it would need the means to fundamentally endanger our species, and no model has the coverage or real-world functional influence to do so. My own AGI 2 research supports this assessment of current capability. The real danger is that we hand them the means ourselves by connecting AI autonomously to weapons systems and critical infrastructure. That is a choice; one we should not make.
It is noteworthy that many earlier general-purpose “technologies” still pose a material risk to humans. Fire is the most primitive of them, still underpinning much of our cooking, heating, transport, manufacture, and even entertainment (e.g. fireworks).
We think it is normal to have a fire station in every town, alarms in public buildings, extinguishers to hand, and sprinklers in communal blocks. But humans have had fire for around a million years and have used it habitually for at least 400,000. Almost every one of those safeguards arrived after a disaster – we learned the hard way.
There are further parallels with electricity, nuclear technology, and medical drugs. We accept some risks but reduce them to socially acceptable levels through governance, education, and technology – both improvements to the core products and secondary systems that manage them. We also design proactively how they fit into the wider world.
These examples highlight a second, compounding challenge: generative AI has advanced too quickly for society to keep pace, and we risk repeating history by applying safeguards only after costly failures.
The speculations of the last week are just gyrations around that stark reality.
So, what should we do?
1. Redirect development at the frontier. Over the weekend, several of the labs began calling for a “pause” themselves. I would redirect rather than slow down, focusing critical resources on the safety of the core products and the systems that monitor them. In our client work, the most powerful models are rarely what determines enterprise value anyway – we often appraise them as expensive overkill for most corporate tasks. A change of emphasis will not disrupt most users.
2. Broaden representation in the debate. The public discourse needs to move beyond labs, lobby groups, academics, politicians, and investors to embrace employers, employees, the arts, civil society, and the medical profession. We need an inclusive debate about the kind of post-AI society we actually want. The methods we already use for national political debate can be adapted for this.
3. Establish baseline governance. Correctly applied regulation is not the enemy of innovation – it improves the confidence and pace of adoption. I would recommend a principles-based voluntary code with national mandatory testing.
4. Deflate the hype cloud. Considerable misinformation is in play, exaggerating both achievements and threats, causing anxiety and ultimately provoking an adverse public reaction. The media in particular needs to scrutinise rather than amplify.
5. Recover our agency. AI is not a sport played only in Silicon Valley. We should use our commercial, political, intellectual, and societal leverage to influence what happens. The future belongs to all of us.
Conclusions
We have reached the first threshold of AGI, but this is not the achievement of a synthetic mind. Advanced cognitive reasoning at scale, coupled with a growing ability to act in the world and a lack of capacity for self-awareness or moral responsibility, is a dangerous imbalance.
Humans must stand in the gap.
We also need time and organisation to accelerate societal adaptation, so we absorb AI into our human world rather than being threatened by it.
Most of all, we should see this as part of all our jobs; we should each ask what we are personally going to do.
It is late, but not too late.
[1] Controversy followed within hours. Two mathematicians who had just proved a closely related result questioned how OpenAI had arrived at its own, an allegation the company denies. OpenAI has also said it will not claim the $1m prize, and the proof has not yet been peer reviewed.