It seems like AI governance and safety is becoming a serious topic again.
Last month in July, a collective statement called Pacing the Frontier (PF) was published and signed by over a thousand employees of frontier labs warning about the risks of advanced AI and the loss of control. It calls on the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
The letter is not calling for an immediate pause, but for the capacity to slow or pace future AI progress if needed. This contrasts the open letter from the Future of Life Institute in March 2023 calling for a six-month pause on AI development shortly after the release of OpenAI’s GPT-4.
My overall view of the PF letter is this: it mainly targets R&D and therefore just one part of the lifecycle, but proper governance and regulation needs to cover other parts. A holistic approach to governance plus a liability regime can shape the lifecycle in a way that labs are incentivised to develop models safely and reliably and better address the risk the letter warns about.
Before diving into this argument in full, it is worth exploring the state of AI as I see it right now.
The state of AI in 2026
Scaling laws have been instrumental in how AI has developed over recent years. This doctrine has dictated that the more compute, data and parameters that go into the development of models, the better they perform.
This is an attractive proposition for investors because it converts R&D into capex. Whereas R&D has traditionally been an opaque asset class to allocate funds to, scaling laws transforms it into a more simple input-output function whereby you specify a compute budget and get a predictable loss curve out the other end. This is why when labs like OpenAI and Anthropic receive investments from the likes of Microsoft and Amazon, what they are really getting is access to compute, since that is what is seen as enabling the development of better models.
But there are some signs that this trend, which has led to total cumulative AI infrastructure spend of over $1.5 trillion since 2020, might be slowing down or at least facing a bit of a correction.
One is the potential diminishing returns. If a lab spends ten times more on training a model, the improvement they get back in model performance (based on the MMLU-Pro benchmark) is about three and a half times smaller than previously. The gains seem to be shrinking at a steady, predictable rate.
But this is largely the case for pre-training, hence why post-training has been where much of the focus has have moved to since the release of OpenAI’s o1, one the first reasoning models trained to spend more compute ‘thinking’ about a given input before generating a response.
But shifting to more spend on test-time compute could still be expensive for labs. A recent post from Dwarkesh Patel argues that compute could get 10x or more expensive in the coming years. He starts from a mismatch; lab revenue is growing roughly 10x a year while compute supply grows only about 3x. Margins cannot plausibly close that gap, so price has to. Smarter models extract more value from the same hardware, so what an H100-hour can command rises toward the value of the labour it displaces. If a chip could run a human-level software engineer, at market rates it should rent for over $250k a year, around 15x today’s spot price. Supply cannot absorb that. Compute growth is bottlenecked by EUV tool supply through 2030, and its biggest single driver, AI taking leading-edge wafer allocation away from other chips, runs out around 2027.
Furthermore, labs do not necessarily want to signal that they are spending more compute on inference and less on training. As Dwarkesh notes, this would essentially mean “you’re kind of declaring that AI progress has stalled, because it’s not worth investing more in training, and your business is basically that of a cloud provider.” AGI is not supposed to be just another SaaS.
Additionally, there have been some other things that to me signal a shift in the perception of AI:
Leopold Aschenbrenner’s Situational Awareness, one of the world’s biggest AI hedge funds, had to sell most of its stock portfolio to Citadel after its highly leveraged positions fell substantially due to declines in memory stocks. However, the firm kept its private market positions, including in Anthropic, and is still looking to raise more capital.
Earlier this year during its AI Ascent 2026 event, Sequoia made the bold declaration that we have already achieved AGI. According to them, if you can dispatch an agent to do a job, and it persists until the job is done, that is functionally AGI. I always thought the original target was human-level general intelligence, but now the new target seems to be approximate substitutability for a workflow.
OpenAI has been rolling out ads to ChatGPT users this year, even after Sam Altman previously said he hated ads and believed that the company would never need to resort to them. This has not turned out to be the case.
More and more people are protesting the development of data centres for AI in the US, which could be a significant issue during the midterm elections. As Politico reports, the “tech industry is facing fierce local backlash to data center projects around the country.” Jasmine Sun recently published some great reporting on this.
The push for banning open weight AI models in the US could be seen as frontier labs fearing the intensifying competition from such models.
The shift that I am point at here is that the gains from raw model intelligence is narrowing and that it might get harder to increase those gains along the traditional paradigms (i.e., scaling laws) largely due to the growing expense of doing so (financially and politically). There does seem to be some growing scepticism about AI valuations with a lot of the capex not providing a ROI as fast as maybe hoped. In simpler terms, AI is becoming less dazzling and more questions are being asked about whether its development is still worth it if it cannot generate significant revenue.
An interesting consequence of this is the increasing focus on what is called ‘recursive self-improvement’ (RSI). As a sort of law of accelerating returns, RSI is about achieving a feedback loop whereby AI models become good enough to build even better models. But the key to this idea is automation; AI models making better AI models with little-to-no human involvement. It means that the whole development lifecycle (including pre-and post-training, data, evals, algorithm design, systems engineering etc) is done solely by agents.
Labs have apparently been able to automate some aspects of their development cycle. According to Anthropic, over 80% of the code merged into its codebase is authored by Claude Code, something which it says could provide a path to RSI in the future. For now the labs have what we might call partially automated software engineers.
But a major concern here is the loss of control - if we let these AI systems loose building their own AI systems at great speed and scale, there is no opportunity for humans to meaningfully intervene and prevent the development or deployment of such systems, including when they do things that cause damage or harm or are otherwise adverse to human interests.
Even while the RSI loop remains remains closed right now, we have already seen several recent instances of AI models developed by frontier labs going rogue. I wrote about OpenAI’s incident last week and Anthropic has also reported its own model hacking incidents.
Read cynically, these are less confessions than product announcements. Nothing signals a powerful model quite like admitting it somehow managed to get away from you.
But I do think it is worth taking these hacking incidents seriously, especially when, in the case of OpenAI, inadequate monitoring by staff could be part of the reason that the model hacking was able to go on for so long. Ultimately these events, as well as being a warning for what a world of RSI could look like (if we ever get there), show that there is still plenty of work to be done to build systems that are safe and reliable in real-world environments.
And I think this is the reasonable backdrop to the pacing the frontier letter. However, after reading and thinking about this letter a bit more and the wider context around it, I think its proposal has some major gaps. Properly pacing the frontier of automated AI development is only a part of the problem space, and in the remainder of this post attempts to shed light on those missing parts.