When AI gets really good at finding what’s broken

We tend to worry about AI making mistakes. A more uncomfortable question is what happens when it does not.

A shift in the type of risk

There’s a new kind of AI story starting to emerge. Not one about chatbots or content generation, but about systems that can find weaknesses on their own. A model linked to Anthropic, referred to as Mythos, is reportedly capable of identifying and exploiting software vulnerabilities. In one case, it completed a multi step cyberattack simulation autonomously. That changes the shape of the risk. Until recently, most conversations about AI focused on misuse or error. People using tools incorrectly. Models generating the wrong answer. Outputs that needed checking. Now the concern is different. It is not just how AI is used. It is what it is capable of doing.

From misuse to capability

Because the same system that can identify vulnerabilities to help defend a system can also reveal how to break it. That creates an uncomfortable dynamic. Companies are learning how to protect themselves by studying how they might be attacked. In isolation, that might sound manageable. But this is happening at the same time as another shift.

Systems moving faster than understanding

Banks and financial institutions are integrating AI into core systems faster than they can properly test it. At the same time, regulators including the Bank of England are beginning to stress test how AI could behave at a system level, not just within individual organisations. The concern is not a single failure. It is interaction. What happens when multiple systems are operating at speed, making decisions, reacting to each other, and drawing on similar signals?

When everything moves together

One risk being explored is “herding”. AI systems making similar decisions at the same time, amplifying movements in markets rather than stabilising them. Not causing a crisis on their own, but accelerating and spreading one. This is a different kind of problem. Less about whether AI gets something wrong, and more about what happens when it gets something right, quickly, and at scale.

AI as auditor, not just tool

That is where the Mythos story becomes more interesting. Because the issue is not just that AI might break systems. It is that it might show us how fragile those systems already are. Banks, in particular, rely on layers of infrastructure built over time. Legacy systems, patches, workarounds, and complexity that has accumulated over years. An AI that can scan quickly, spot patterns, and connect weaknesses could surface issues that were previously hidden or ignored. In that context, AI starts to look less like a tool and more like an auditor. But unlike a traditional audit, this happens faster, at scale, and potentially continuously.

The uncomfortable part of transparency

And that raises a harder question. Not whether organisations can fix what AI finds, but whether they are ready to see it in the first place. Because transparency, at that level, is not entirely comfortable. It exposes gaps, trade offs, and risks that have often been managed quietly rather than resolved completely.

From individual errors to system behaviour

There is a similar pattern emerging in how central banks are thinking about AI more broadly. Alongside concerns about lending, geopolitics, and market shocks, AI is now being treated as a potential source of instability in its own right. Not because it is unpredictable in a chaotic sense, but because it can make systems more tightly coupled. Faster reactions. More shared signals. Less time to interpret what is happening. It is a shift from thinking about individual failures to thinking about system behaviour.

An open question

From “what if something goes wrong?” to “what happens when everything is working as designed, but interacting in ways we do not fully understand?” That is a harder problem to manage. Because it is not just about building better models. It is about understanding the systems those models are part of. And that leaves a more open question. If AI becomes very good at finding what is broken, are we prepared for what it shows us?