When AI learns to cheat on tests

Hello again, my friend,

It’s Thursday, this week is the recap, and it’s a little terrifying.

A combination of OpenAI models found a way out of a cybersecurity benchmark, compromised Hugging Face’s infrastructure, and took the answers from its production database. The AI realized that to score better marks on the test, it should break out of its sandbox, go to the internet, hack into another site with more details on this benchmark, and use that data to essentially score higher (rather than just try to do a better job, with greater risk of failure).

Again, scary, to the point where we have to wonder

I’ll cover more on what these AI benchmarks actually measure, which models are sitting at the ‘frontier’ right now, and why one new AI model from China has the U.S. government discussing restrictions on Chinese model altogether.

It may not feel like a big deal, but I think the U.S. especially is on the verge of making decisions it may never recover from.

Let’s lock in.

On July 16, we got this Security incident disclosure involving an AI agent that had compromised parts of its infrastructure.

Five days later, they release a statement to address the security incident. It was between OpenAI and Hugging Face. Hugging Face is an open-source community for building and training new AI models, like a central hub. You can think of it like a version of social media for people who train AI models.

OpenAI had been testing a combination of models, including GPT-5.6 Sol and an even more capable unreleased model, on ExploitGym, a benchmark designed to measure advanced cybersecurity capabilities. The models were placed in an isolated environment, given reduced safety restrictions, and asked to work through complicated attacks.

The environment was not supposed to have normal internet access. It could only reach an internal service used to install software packages.

According to OpenAI’s preliminary investigation, the models found a previously unknown vulnerability in that service, used it to escape the restricted environment, escalated their access, and eventually reached a machine connected to the open internet.

From there, they inferred that Hugging Face might have information related to the benchmark. The models used stolen credentials and additional vulnerabilities to access secret test solutions in Hugging Face’s production database.

Hugging Face detected and stopped the activity. OpenAI discovered the unusual behaviour internally, and the two companies began investigating together. Neither company has published a complete reconstruction of every step yet, and OpenAI has not said which model performed each action.

This sounds like the beginning of some dramatic story about an AI trying to escape. That’s probably not it (but the fact that it could be is scary, I know).

OpenAI said the models appeared “hyper-focused” on completing one narrow objective: solve the benchmark, and not looking for freedom. They found a way to make the testing system give them the result they had been asked to produce.

Businesses create smaller versions of this problem constantly. Reward a sales team for booking meetings and they will book terrible meetings. Measure a support team by how quickly it closes tickets and unresolved tickets will suddenly start disappearing.

The metric becomes the goal, and then it stops measuring what you cared about.

The AI version is more extreme because the models did not just find a loophole in the rules. They found vulnerabilities in two companies and used them to obtain the answer key. That tells us something about their cybersecurity capabilities, but it also tells us that a benchmark score does not always mean what we think it means.

A benchmark is a standardized test. Several models receive the same problems under defined conditions, and their results are compared.

The problem is that there is no single test for intelligence.

A model can perform brilliantly on one and struggle with another. And there are many other benchmarks I didn’t cover as well, which can become even more specialized for certain tasks or outcomes.

The testing setup matters too. Some models receive web access, code execution, multiple agents, or hundreds of thousands of tokens to work through a problem. Some are tested at “max” reasoning, which can take considerably longer and cost far more than the version most people use every day.

The agent wrapped around the model also changes the result. In Kimi K3’s launch report, some models were tested through Kimi Code, some through Claude Code, and others through Codex. That is not necessarily dishonest because people use models through those products, but the resulting score measures the model, its tools, its instructions, and the system managing the work.

Then there is the obvious problem that AI companies choose what to show us.

OpenAI emphasizes the coding evaluations GPT-5.6 wins. Anthropic shows the knowledge work where Fable performs well. Moonshot shows the tests that put Kimi beside the American frontier. The numbers can all be real while the chart is still marketing.

That does not make benchmarks useless. It means I want to know what was tested, which tools were allowed, how much time the model received, and whether the result came from the company selling it or someone independent.

A model scoring 90% on a benchmark tells me it is probably capable at whatever that benchmark measures. It does not tell me it will complete 90% of my work correctly. It also never addresses the elephant in the room: can we be sure the AI companies are not paying these benchmarks to juice the numbers a little bit ?

Artificial Analysis currently places Claude Fable 5 at the top of its independent Intelligence Index with 60 points, although the configuration it tested can fall back to Opus 4.8. GPT-5.6 Sol follows at 59, and Kimi K3 sits at 57.

That Fable caveat is another example of why the rankings need context. Even the number-one result measures a particular model configuration rather than one model operating alone.

Fable remains the strongest broad system, especially for long-running research, analysis, documents, and complicated software work. GPT-5.6 Sol is effectively tied with it while leading Artificial Analysis’s coding index and costing less.

Kimi K3 Tech Blog: Open Frontier Intelligence is the model I find most interesting.

Moonshot says it has 2.8 trillion total parameters, native vision, and a one-million-token context window. If you’re not familiar with the tech terms, it’s essentially saying K3 and can work through large codebases, operate terminal tools, conduct long research sessions, and use screenshots to improve what it builds, for longer periods of time where it starts “getting dumber” further into a long conversation.

Moonshot also admits Kimi still trails Fable and Sol in its overall user experience. It does not need to beat them to change the market. Landing within three points of the leading independent score is enough to put it in the same conversation.

At the time I am writing this, Kimi is available through Moonshot’s products and API but is not yet a downloadable open-weight model. If Moonshot follows through, one of the strongest models in the world will be available to host and build around without relying entirely on a company-controlled API.

GLM-5.2: is already available under an MIT licence. It ranks below Kimi on independent evaluations, but it has a one-million-token context window and is strong enough for long-running software work. A company can host it, modify it, and build around it without sending every request to OpenAI or Anthropic.

The best closed models are still ahead, but “ahead” now means a few points on a collection of imperfect tests. Open models do not have to win every benchmark if they are cheaper, controllable, and good enough for the work.

All this to say more competition, especially internationally, and more to think about in the bigger picture.

On July 20, TheTrump administration showed signs they may ban or restrict leading Chinese models. Kimi K3’s release reportedly restarted those conversations.

No broad ban has been announced.

The following day, Treasury Secretary Scott Bessent said the administration supports open models but is investigating whether Chinese companies used large-scale distillation to reproduce the capabilities of American systems. He described some of that alleged activity as intellectual property theft and said sanctions could follow if the government finds evidence.

Aside from the irony (remember how these AI companies trained their models by also ripping off a lot of intellectual property), there are at least two concerns being mixed together here. One is legitimate security risk: once model weights are downloadable, the original company cannot monitor every use, remove the model later, or prevent someone from stripping out its safeguards.

The other is commercial. The strongest American models are closed and sold by companies like OpenAI and Anthropic. Many of the strongest Chinese alternatives are open, or moving in that direction. If American companies can download a model that gets close enough for much less money, that puts pressure on the American labs, and the mountains of money poured into them.

A restriction might protect national security. It might also protect those labs from cheaper competition. Where I’m sitting, getting into the game of banning AI models is likely a mistake that cannot be recovered from.

The next few weeks will probably decide whether we see major changes for the next couple of years (or more) to come. It’s a very tricky time because is the technology new, exciting, and powerful. The powers that be around it are also dealing with something kind of new to them.

It’s effectively one tool that changes pretty much everything that involves technology, everywhere in the world. So how do you write policy, or help people, or manage global conflict in a world with AI? Yes, this is primarily a business newsletter mostly about making money. Most of my opinions have been from that angle.

But take my word for it, if you’re in the U.S. and the Chinese models are, in fact, that’ll have a tremendous impact on business and the markets for years to come.

As always, thanks for reading.

Darwin

PS: this is a test link for something we’re building at beehiiv. No need to click it.