Anthropic has to keep revising its technical interview test so you can’t cheat on it with Claude

In Brief

Since 2024, Anthropic’s show optimization squad has fixed occupation applicants a take-home trial to marque definite they cognize their stuff. But arsenic AI coding tools person gotten better, the trial has had to alteration a batch to enactment up of AI-assisted cheating.

Team pb Tristan Hume described the past of the situation in a blog station connected Wednesday. “Each caller Claude exemplary has forced america to redesign the test,” Hume writes. “When fixed the aforesaid clip limit, Claude Opus 4 outperformed astir quality applicants. That inactive allowed america to separate the strongest candidates — but then, Claude Opus 4.5 matched adjacent those.”

The effect is simply a superior candidate-assessment problem. Without in-person proctoring, there’s nary mode to guarantee idiosyncratic isn’t utilizing AI to cheat connected the trial — and if they do, they’ll rapidly emergence to the top. “Under the constraints of the take-home test, we nary longer had a mode to separate betwixt the output of our apical candidates and our astir susceptible model,” Hume writes.

The contented of AI cheating is already wreaking havoc astatine schools and universities astir the world, truthful ironic that AI labs are having to woody with it too. But Anthropic is besides uniquely well-equipped to woody with the problem.

In the end, Hume designed a caller trial that had little to bash with optimizing hardware, making it sufficiently caller to stump modern AI tools. But arsenic portion of the post, helium shared the archetypal trial to spot if anyone speechmaking could travel up with a amended solution.

“If you tin champion Opus 4.5,” the station reads, “we’d emotion to perceive from you.”

Subscribe for the industry’s biggest tech news