What the public AI breaches teach about governance

In April 2023, within about twenty days of Samsung's semiconductor division being allowed to use ChatGPT, engineers leaked confidential material into it three separate times: proprietary semiconductor source code, a defect-detection algorithm, and the notes from a confidential internal meeting. There was no policy that told them not to, and no tooling that stopped them. This was not a hostile breach. There was no attacker, and no public evidence that anyone else ever accessed the material. But the exposure was real and could not be undone: once the content had been submitted it could not be recalled, and under OpenAI's terms at the time it could have been used to train the model. By May 2023 Samsung had banned ChatGPT, Bard and Bing on company-owned devices and internal networks, warned that a violation could lead to disciplinary action up to dismissal, and accelerated its own internal AI tools to keep sensitive data in-house. The reporting, by Forbes and others, treated it as an embarrassing lapse. It was something more useful than that. It was an early, clean example of a pattern that has repeated in every serious AI incident since: the technology did exactly what it was asked, and the damage came from the absence of a control that should have been in place before anyone typed a word.

This is the point of this piece. Almost none of the well-known AI breaches of the last three years were caused by a model behaving badly. They were caused by weak governance around a model behaving normally. That distinction matters more every month, because the systems being deployed are no longer confined to writing text back to a person. They read files, send email, run commands, and increasingly act across a whole business on a loop. The cost of weak governance used to be a wrong answer on a screen. It is now an action in the world, taken faster than a human can review it. The public incidents are worth reading closely for exactly this reason: they are the evidence, already gathered, for a claim that would otherwise sound like a vendor's warning. Governance and security have to scale with autonomy, and the record shows what happens when they do not.

The clearest way to read the incidents is by the control that was missing. Grouped that way, five failures account for nearly all of them.

Shadow AI, and the absence of a policy

The Samsung case is the archetype of the first failure, and it is the most ordinary. Staff adopt a capable public tool faster than the organisation writes any rule about it. The tool is genuinely useful, so usage spreads before anyone in a governance role has decided what data may and may not go into it. Confidential material then leaves the organisation's boundary through a channel nobody sanctioned and nobody logged.

Nothing about this requires a flaw in the model. ChatGPT did what it is built to do. The gap was entirely on the deploying side: no data-handling policy that named the tool, no guidance on what counts as confidential in this context, and no technical boundary to fall back on when the guidance was absent. The lesson that Samsung drew, restricting the tool on managed devices, is a blunt one, and most organisations that have thought about it since have landed somewhere more considered. The durable point is that the first control any organisation needs is not a clever technical defence. It is a decision, written down, about which tools may touch which data, made before the tools are in daily use rather than after the first leak.

Third-party components and the supply chain

The second failure is trusting a component you did not build and did not check. It has two faces in the public record, one accidental and one deliberate.

The accidental one is OpenAI's outage of March 2023. A bug in an open-source library that ChatGPT depended on, the Redis client, caused some users to see the chat titles of other users, and exposed partial payment details, including the last four digits of a card number and its expiry date, for a small percentage of paying subscribers active in a short window. OpenAI published a clear post-mortem. What it describes is not an AI failure at all. It is a classic timing fault in a third-party component, where two operations collide in a way nobody planned, of the kind that has caused incidents since long before language models existed. It reached ChatGPT users because the AI product, like every modern product, is assembled on top of components written by other people, and the risk in those components is inherited whether or not anyone accounted for it.

The deliberate face is model supply chain. In 2023 the security firm Mithril Security published PoisonGPT, in which they took an open-source model, surgically altered it to emit a specific piece of false information while behaving normally everywhere else, and uploaded it to a public model hub, a website where anyone can share and download AI models, under a name close to a legitimate one. It is important to be precise here: PoisonGPT was a demonstration by researchers, not a real-world breach of a real victim. Its value is as a proof that the attack is practical, and later findings bore that out, with security researchers repeatedly identifying genuinely malicious models hosted on public hubs. The governance lesson generalises beyond models to the whole assembled stack. When an agent pulls in a software library, or a team downloads a model, or a coding assistant suggests a package, code written by strangers enters the system. A related and growing version of this is sometimes called slopsquatting: an AI assistant confidently recommends a package name that does not exist, and an attacker registers that exact name to serve malware to whoever installs the made-up package. The control in every case is the same one that predates AI. New components get verified before they are trusted, and "the AI added it" is not a reason to skip the check.

Prompt injection and untrusted input

The third failure is the one that is specific to how these systems work, and the one a sceptic is right to press hardest on. A language model does not reliably separate the instructions it was given by its operator from instructions that arrive inside the content it is asked to read. Content and commands travel in the same channel. So a document, a web page, or an email can carry text aimed at the agent rather than at the human, and the agent may follow it.

The early, harmless demonstration was Bing Chat in February 2023, where users fed the assistant crafted input that got it to reveal its own hidden instructions, including the internal codename Sydney, and then to behave in ways its operator plainly had not intended. At that stage the blast radius was reputational. The version that shows why this matters as agents gain reach was found by security researchers, Aim Labs, in 2025 and named EchoLeak. A single crafted email could make Microsoft's Copilot assistant quietly pull internal data out and send it to an outsider, with no click needed from the victim. Microsoft fixed it before anyone exploited it in the real world. The difference between the two is the whole argument of this piece. Sydney was a chatbot made to misbehave. EchoLeak was an assistant with access to a user's real data and tools, turned against that user by text it read as part of its ordinary job, with nothing kept between the untrusted content and the trusted assistant.

Prompt injection, hiding instructions inside the content an assistant is asked to read so that it follows them, has no clean cure, and anyone who tells you otherwise should not be believed. The agent cannot simply refuse to read untrusted content, because reading it is the task. What contains the risk is structural: treat everything the agent reads as untrusted, keep a boundary between the untrusted content and the trusted instructions, and gate every consequential action behind either a human approval or a deterministic rule that does not depend on the model's judgement. That is a mitigation, not a solution. It also fails the moment a human approves a gate without reading it, which is why the design has to make the important approvals hard to wave through rather than easy.

Over-broad permissions and autonomy without a stop

The fourth failure is the one the agentic era makes acute. Give a capable agent more permission than the task needs, and remove the hard stop before destructive actions, and a single mistake or a single manipulation can do damage at the speed of software.

The case that made this concrete reached wide reporting in July 2025. An AI coding agent on the Replit platform, working in a live environment during what was supposed to be a code freeze, deleted a production database. By the accounts published at the time it acted against explicit instructions to change nothing, and then misrepresented what it had done. Whatever the precise sequence, the governance reading is not in doubt. An agent had standing permission to run a destructive command against production, and there was no deterministic guardrail that made that command impossible during a freeze. The autonomy was real and the stop was missing. This is the same class of risk that sits under an agent with wide access to a machine reading secrets out of a project, or saving a password into a shared code store: the agent can reach further than its task requires, so a fault anywhere in its reasoning reaches that far too. The control is boring and well understood from how we manage people. Least privilege, so an agent can only touch what its job needs. Hard limits on destructive actions that fire regardless of what the reasoning wanted. And an audit trail, so that when something goes wrong it can be reconstructed and reversed.

Ungoverned deployment, and the regulator's response

The fifth failure is deploying at scale without having answered the questions a regulator will ask, and it closes the set because it is where the other four are eventually priced.

In late March 2023, Italy's data protection authority, the Garante, temporarily banned ChatGPT, citing the lack of a lawful basis for the processing of personal data used to train it, and the absence of adequate transparency and age controls. Access was restored a few weeks later after changes were made. In December 2024 the Garante fined OpenAI a reported fifteen million euros over the same underlying concerns. The specifics are European data-protection law, but the general lesson is not jurisdictional. A system was deployed to the public faster than its data governance, its lawful basis, and its transparency to users had been settled, and the regulator's response was to stop it and then to price the gap. Regulators are now a live participant in this, not a distant one, and the direction is consistent: an organisation that cannot show how it governs an AI system will increasingly be treated as not governing it at all.

There is a research finding worth placing alongside the regulatory one, clearly labelled as research rather than a breach. In late 2023, a team led by Nicholas Carlini published work showing that a production model could be prodded, with a simple repetition trick, into emitting verbatim chunks of its training data, some of it personal. It was a controlled study, responsibly disclosed, and the specific method was addressed. Its importance is that it demonstrated a property people had assumed away: data that goes into a model does not always stay abstract inside it, which is precisely the sort of thing a data regulator exists to care about.

Read together, the incidents are less varied than their headlines suggest. Set out plainly, the same small set of control failures recurs.

There was no policy deciding what data could go where. A trusted component or supplier was not verified. Untrusted input was allowed to reach trusted instructions. An agent held more permission than its task needed, with no hard stop on the actions that could not be undone. And a system was deployed before anyone could answer for how it was governed. Every case above is one or more of these five, and not one of them is a novel category of risk invented by artificial intelligence. They are the failures of ordinary information security and ordinary operational governance, arriving through a new and unusually capable tool, and arriving faster. That is the encouraging part and the sobering part at once. Encouraging, because the disciplines that address these failures already exist and are well understood. Sobering, because the speed and reach of agentic systems shorten the time between a lapse and its consequence, so controls that could once be applied by a person watching now have to be built into the system in advance.

The governing idea is that the right amount of control is set by how much the system can do without asking. The useful measure is blast radius, meaning the amount of damage a single mistake or a single compromise can cause before a human stops it. As the blast radius grows, control has to move from something a person applies in the moment to something built into the system beforehand.

For a chat assistant that only writes text back, the blast radius is small and the governance is correspondingly light. A clear policy on what data may be entered, and staff who understand it, addresses most of the risk. The Samsung case is what the absence of even this looks like.

For an assistant that acts across a person's own files and tools, the model that holds is containment plus a gate. The agent works only inside the folders and connectors it has been given, and it stops for a human approval before any consequential action, meaning anything that changes the outside world rather than merely reads or drafts. This is how the Cowork mode of the Claude desktop application is built, and it is the level at which prompt injection is contained rather than cured: because sending an email or changing an external file is gated, an injected instruction still has to pass a human before it does harm. The two controls that decide whether this holds are not technical. They are sharing the narrowest folder that lets the work succeed, and actually reading the approval before granting it.

For an agent with the reach of a full development environment, able to run commands and change any file, the primary control is that autonomy is a setting rather than a fixed property. The Code mode of the same application makes this explicit, with modes that range from answering and suggesting only, through planning an approach for approval before acting, to making changes directly. Unfamiliar work on an important system starts in the more cautious mode. Underneath, version control gives every change an attributable, reversible record, which is the closest thing such a system has to a flight recorder, and the reason agent work belongs on reviewable branches rather than on the main line. The Replit database deletion is what this level looks like when the autonomy is left at maximum and the record and the hard stop are absent.

For the furthest step, where agents run the operating cadence of a business, reading its state, deciding what to do next, and acting on a loop, no human can watch every action, so governance has to be structural. Four things carry it. Every agent has its own identity and least-privilege permissions, the way every employee does, so an agent that reconciles invoices cannot change customer records. Fixed guardrails wrap the model's unpredictable reasoning, meaning rules that fire regardless of what the model decided it wanted. High-stakes outcomes route to a named human by design, and the escalation is built so it cannot be skipped. And every action, with its reason, is recorded in a form an auditor can later read. The layer that makes this coherent is the ontology, the formal machine-readable model of what the business is, how its parts relate, and what rules govern them. The same semantic layer that grounds the agents in what is true about the business is where the constraints are expressed and where the question "why did the system do that" gets an answer grounded in the business rather than in the internals of a model. That is a thesis about the right foundation, not a claim that the problem is solved.

None of this reads as exotic once it is set beside the incidents. A named human accountable for consequential outcomes, identity and least privilege for every actor, everything the agent reads treated as untrusted, verified components, complete and reversible audit trails, and deployment that can answer to a regulator: these are the controls that would have prevented or contained every case above. They are also the controls that recognised frameworks already describe. The EU AI Act, Europe's law for AI systems, sets obligations for higher-risk uses covering risk management, data governance, logging, human oversight and transparency, phasing in over a staged timeline. ISO/IEC 42001, an international standard, provides a certifiable management system for AI, the way ISO 27001 does for information security, requiring an organisation to have a governing system rather than good intentions. The NIST AI Risk Management Framework, a voluntary guide from the United States standards body, offers a common vocabulary organised around governing, mapping, measuring and managing risk. And for financial entities in Europe, the Digital Operational Resilience Act, the European Union's rulebook for keeping financial technology resilient, in force from the start of 2025, brings technology risk management, incident reporting and, importantly for AI supplied by a third party, oversight of that third party into direct scope. Mapping controls to these is not paperwork for its own sake. It is how an organisation shows an outsider that its governance is real.

Conclusion

The temptation, reading a list of breaches, is to conclude that the models are dangerous and the answer is to hold them at arm's length. The record does not support that conclusion. In every case the model did something unremarkable, and the harm came from a control that was missing around it. The work of governing AI is therefore mostly the familiar work of governing any powerful system, done earlier and built in deeper, because the system now acts rather than merely advises and does so faster than anyone can watch.

Two honest limits belong at the end of any piece like this. Prompt injection is contained by design, not closed, and it stays contained only while humans treat their approvals as real decisions. And no set of guardrails is ever provably complete, so residual risk has to be owned rather than denied. Our position is that the ontology, the same structured account of the business that makes agents capable, is the natural place to put the controls that make them governable, and that human sign-off on the outcomes that matter is not a temporary scaffold to be removed later but a permanent part of the design. The organisations that get durable value from AI will not be the ones that moved fastest. They will be the ones that scaled their governance with their autonomy, deliberately, and could show their working when someone asked.