Front-loading AI governance is not enough
On the OpenAI/Hugging Face incident and what it means for current and future AI regulation
There is never a dull day in AI.
There have been a few big stories in the space recently, including those that could prove significant in the development and governance of AI.
One of those stories is about AI going rogue, and if you are deep in the AI bubble like me, then you likely already know what I am talking about.
But if not, here is what happened.
On 21 July, OpenAI published a blog post detailing a hacking incident involving its models. During an internal cyber-capability evaluation, GPT-5.6 Sol and a more capable unreleased model (both running with reduced safeguards and without production classifiers) chained zero-day exploits, escalated privileges and broke out of OpenAI’s sandbox to reach the open internet. The models then used stolen credentials and a remote-code-execution path to compromise Hugging Face’s production infrastructure and pull test solutions from its database. Essentially, OpenAI’s models hacked a third-party’s infrastructure to get an answer to an eval it was being tested on.
I think this incident brings up some interesting things about governance.
Looking at the Hugging Face incident, some have questioned whether it is an example of instrumental convergence. This concept, detailed in Nick Bostrom’s book Superintelligence, is about how there are certain fundamental goals that superintelligent entities will always pursue. It proposes that there are certain objectives “that are useful intermediaries to the achievement of almost any final goal.” Bostrom therefore argues that AI could pursue its goals in ways that are to humanity’s detriment. This means AI could discover “some way of satisfying the criteria of its final goal that violates the intentions of the programmers who defined the goal.”
Some have argued that this is exactly what happened in the Hugging Face incident. According to exclusive reporting by Reuters, the agent apparently left notes setting out “instructions for how agents could free themselves from OpenAI’s internal constraints.” Moreover, Reuters also reported that the agent first tried to escape its containment and conducted its hack against Hugging Face several days before OpenAI ever noticed. That the agent (a) released itself of constraints set by its developer and (b) went off and breached another company’s infrastructure looks like the kind of scary paperclipping that Bostrom’s instrumental convergence theory warns about.
Even so, there are some things to think about before jumping to such a conclusion.
Firstly, this behaviour by the model went on for days, and it seems strange that, during this testing and eval process, it was able to expend compute to hack Hugging Face without being noticed by OpenAI staff. Would engineers or researchers really leave a model to run without a human-in-the-loop, or at least a human-on-the-loop, to monitor its progress and potentially spot this kind of activity? Perhaps, but that says as much about OpenAI’s safety practices as it does about models going rogue.
Secondly, the rogue behaviour may not be evidence of instrumental convergence but rather reward hacking - the agent finding a way to maximise its training score (or achieve its goal) whilst not doing this in a way that the developer necessarily intended. This sort of gaming behaviour is likely a function of the model’s reinforcement learning, during which the model may have picked up a niche route out of its sandbox and identified this as a means for finding a solution to an eval, something that apparently could be relatively easy to fix.
OpenAI is obviously one of the leading labs at the frontier of AI right now. Its models are some of the highest performing on a number of benchmarks and it boasts almost a billion ChatGPT users. The company therefore oversees the development of some highly-capable models and systems, including those with potentially destructive capabilities, as demonstrated by the Hugging Face hack.
But right now, we are heavily reliant on the frontier labs like OpenAI dealing with the ramifications of these models and disclosing incidents like this.
And for me this gives rise to two important governance points.
The first is about nature of the models being built; they are black boxes that we do not fully understand. The architecture of these models are so vast and complex that it is incredibly difficult, perhaps even impossible, to really understand how they are supposed to work and, accordingly, predict how they might behave in a given environment. And if you cannot understand how these models work, how can you possibly ensure that they behave as you intended?
Post-training is what enables the labs to try to shape the behaviour of a model so that it responds to inputs in a particular way using the learning it has gained from the training data used in the pre-training process. As such, post-training is an important mechanism for implementing AI safety.
The problem is that the threat vector we are talking about here is massive, maybe even infinite. In fact, earlier this year NIST released mathematical proof that “a fixed set of guardrails placed on AI is not universally robust against adaptive adversarial prompts.” In other words, no matter how an AI model might be trained, it will always be possible to circumvent the guardrails its developer has attempted to incorporate into the model.
This then leads to the second important governance point, which is what the approach to this harsh reality should be.
As NIST suggests, the inability to universally guardrail AI models requires a continuous-monitor-and-update security model. This means that the governance work for AI models and systems cannot be completely front-loaded, as you simply cannot conduct all the assessments and red-teaming pre-deployment and think you can comprehensibly identify and mitigate against all the bad behaviour. Instead, this pre-deployment work needs to be combined with continuous monitoring of AI systems out in the wild and protocols for acting quickly when new risks are identified.
From this perspective, you could say that this is what OpenAI did during the Hugging Face incident. Though the agent in question has not been publicly released, OpenAI was able to identify its agent going rogue and will presumably learn why this has happened (as much as it can) and re-train it or address this in RL.
But based on the Reuters report, it does seem that OpenAI has been somewhat sloppy, especially when you think about how its agent was able to execute its rogue behaviour for days.
This is then connects to another measure that is perhaps needed here which relates to transparency; incidents like this need to be disclosed so that they can be learned from and labs can be held accountable, especially if they repeat known failures or have inadequate response protocols in place.
But it is important to get these transparency measures right, especially when the the meaning of transparency in AI industry has been somewhat butchered by frontier labs, especially when it comes to the notion of ‘open-source’.
Open-source software used to mean something precise. The code was public and you could read it, fork it, test it, contribute to it and so on.
But in the context of AI, ‘open weights’ means something different. Here what you get is a file of numbers that are not readily comprehensible and difficult to parse, use or test in a meaningful way. Nothing in that file tells you what data the model was trained on, how the harness around it works and the other things that influence how it operates.
It is essentially a form of token transparency, where the artefact is public but the understanding of it is absent or hard to obtain.
And that gap has a real possibility of becoming a governance problem when policymakers write rules around it. Leave the standard vague and the labs will fill it in themselves, on terms that suit their commercial interests and then eventually their version hardens into the rulebook.
It is quite crucial for this butchering of transparency to not impact the reporting of incidents like what happened with Hugging Face. There is a particular register frontier labs sometimes slip into when disclosing their own failures. Wide-eyed, faintly amazed, as though the model were weather rather than a system they built and were supposed to be watching. I think OpenAI’s blog post kind of reads like this. It gives ‘I’m just a girl’ vibes, which is quite concerning coming from a company with a $500 billion valuation, a compute budget larger than most states and dealing with seemingly quite dangerous technology.
It will be interesting to see how events like this will shape AI regulation in the US. So far much of it has been by fiat, with the Trump administration seemingly picking and choosing which models are too dangerous for public use based on opaque national security grounds. This is not sustainable and will eventually need to be replaced with something more consistent and aligned with the interest of keeping models safe rather than with whoever is in the White House at the time.
Meanwhile, the EU’s AI Act is an attempt at establishing obligations around both safety/security measures and incident reporting. Though maybe not perfect, its provisions could provide as a framework or starting point for robust AI regulation going forward.
The rest of this post goes into more detail around those AI Act provisions, including what they require and how they apply to OpenAI for the Hugging Face incident.