Over the past few months, various AI companies disclosed incidents in which their AI agents, operating in supposedly contained environments, escaped those environments and gained unauthorised access to third-party infrastructure.
One clear example concerned OpenAI’s internal model hacking Hugging Face infrastructure. These incidents attracted considerable attention not only because the agents behaved unexpectedly, but because some circumvented the constraints imposed on them.
Some of these incidents map neatly onto the concept of reward hacking. An AI system is, by its very design, optimised to achieve some objective. Reward hacking occurs when the system finds a way to achieve that objective, satisfying the relevant optimisation criterion, in a way that departs from what developers intended – often undercutting the very value that objective was intended to capture. The textbook example is a robot vacuum cleaner that ejects dust just to be able to collect it again. In the Hugging Face incident, the same underlying phenomenon occurred in a more consequential setting: the agents apparently determined that gaining access to Hugging Face’s infrastructure would help them perform better during an evaluation.
On the legal front, these incidents illustrate a conventional legal problem: when AI causes harm, who should bear the costs?
Dr Maarten Herbosch
There are good reasons to take these incidents seriously. Increased capabilities imply increasing autonomy for these systems as they are deployed. This has rightly led many to (re)consider the risk management implemented across AI development lifecycles. On the legal front, these incidents illustrate a conventional legal problem: when AI causes harm, who should bear the costs?
That question is highly relevant – in Europe as elsewhere. What makes it particularly salient in the EU is that the EU has recently modernised its regulatory framework for AI, with the AI Act (Regulation 2024/1689) and the revised Product Liability Directive (Directive 2024/2853). Nevertheless, the European approach to risk and cost allocation in incidents like these remains highly fragmented and nationalised. This, too, matters beyond the EU, as it also shapes the liability exposure of non-EU AI providers.
Upstream Risk
To that end, it is useful to note that the AI Act introduces a – somewhat artificial – distinction between AI systems and AI models. AI models, notably general-purpose models – models that can be deployed across a wide range of tasks, a category that includes many frontier models – are, in that framework, upstream artefacts that can be implemented into more contextualised downstream AI systems. An AI model is thus a form of component of an AI system. The distinction between the two matters, as some risks can be embedded, upstream, in an AI model, making them very difficult or impossible to mitigate downstream. A well-functioning liability regime would thus reach beyond the downstream end of the AI supply chain, so that risks are addressed where they originate.
The Limits of the Revised Product Liability Directive
As part of the response to the rapid proliferation of AI, the revised Product Liability Directive now explicitly includes software and AI systems within its scope. This means that AI system providers – and potentially AI model providers – can be liable on the basis of two well-established liability regimes: national liability and harmonised product liability.
It should be clear, however, that the latter is of very limited help in circumstances such as those illustrated by these incidents. First, interestingly, the revised Product Liability Directive – which, in its recitals, explicitly covers AI systems – is completely silent about AI models. That does not mean that the latter are not captured by that framework – there are good arguments that they are – but it adds unnecessary and unwelcome uncertainty. Moreover, even if we were to accept that models are captured by this instrument, the Directive’s scope contains significant limitations.
One important example concerns the types of damage covered by the Product Liability Directive. The Directive is largely aimed at protecting consumers and other natural persons. It covers death and personal injury and certain property damage, as well as the destruction or corruption of data not used for professional purposes. It does, however, exclude property used exclusively for professional purposes and data if it is also used for professional purposes. Pure economic loss does not independently trigger liability under the Directive.
That limitation is significant in AI contexts. While consumer harms are important, and thus constitute a legitimate target for the Product Liability Directive, these incidents have shown how AI can cause cyber incidents that lead to significant losses for enterprises. An intrusion into a protected network can lead to a significant loss of intellectual property and the corruption or destruction of data. Nevertheless, the harmonised Product Liability Directive would not provide a remedy for such losses.
Beyond Product Liability
While national liability regimes would remain applicable, the answers they provide may diverge. Moreover, on a general level, it is widely argued that these face significant hurdles in dealing with AI – which was among the reasons an AI Liability Directive was considered in the first place. More specifically, claimants face significant evidentiary and causation problems that will likely lead to diverging outcomes across EU jurisdictions.
None of this means that we need an entirely new liability law to deal with AI. I have previously argued the opposite: although AI creates particular challenges – due to its complexity and opacity – this does not fundamentally undermine or displace traditional liability regimes. Existing liability doctrines have dealt with comparable forms of uncertainty before, and many of the needed changes can occur through targeted reforms rather than a complete overhaul. Nevertheless, the rapid adoption of AI tools throughout society, combined with their rapidly advancing capabilities, provides good reasons to consider these questions sooner rather than later.
Several of these considerations were part of the rationale of the AI Liability Directive, which the Commission proposed in response to a request from the European parliament, but which was ultimately withdrawn for various reasons. Many of the criticisms on that proposal were substantive and compelling: the instrument was too heavily oriented towards high-risk AI systems without adequately considering AI models, and it relied more than necessary on the AI Act. Nevertheless, in light of this new-found urgency to tackle the AI liability problem, it is interesting that a small group of MEPs has launched a cross-party call for an AI Liability Act, suggesting that the European Parliament might once again prompt the Commission to revisit AI liability.
Empowering AI Liability in the EU
It seems clear that a supportive instrument could help offset many of the challenges that characterise AI liability today. More specifically, it should, at a minimum, better balance the burden of evidence and address the issues that causation presents. Going further by, for example, setting the actual standards of care – beyond what the AI Act imposes – could be valuable. However, it could also make the proposal more politically contentious, potentially resulting in delays in its adoption that might leave more AI-related harms uncompensated in the meantime.
It is clear that such a proposal, unlike the main focus of the originally proposed AI Liability Directive, should deal with forms of AI other than high-risk AI systems. The classification as high-risk depends mainly on a predefined use-case list and pre-existing product safety legislation, constituting a poor proxy for the system’s ability to cause any harm. Low-risk systems are also capable of causing harm. Importantly, model providers should be made responsible for conduct that influences the downstream supply chain, i.e. the actions of AI systems.
Recent incidents do not show that European liability law has failed.
Dr Maarten Herbosch
The objective, thus, could be comparatively modest and achievable while remaining highly impactful: to make existing liability rules work across the AI supply chain without assuming that AI is legally exceptional. This could go a long way towards creating accountability for inadequate security measures surrounding AI systems and ensuring that AI models are developed responsibly, while protecting AI companies from a plethora of distinct legal regimes and evidentiary standards that might otherwise disincentivise AI development or deployment in the EU.
Some AI systems will cause harm. That is a fact. It is not an argument against developing the technology – as AI may also bring significant advantages to society. It is an argument for actively considering how our society deals with these harms. Under the current framework, many of these harms risk going uncompensated. This is undesirable from the victims’ perspective. But it also means that those developing these models upstream do not experience the incentives that liability law can provide to make their models and systems safer.
Recent incidents do not show that European liability law has failed. They do provide a useful wake-up call, demonstrating its pressure points. The appropriate response should be neither regulatory inaction nor a fundamental overhaul, but rather a more limited effort to ensure that established liability principles can continue to work as AI advances.
This opinion piece reflects the views of the author, and does not necessarily reflect the position of the Oxford Martin School or the University of Oxford. Any errors or omissions are those of the author.