Do Androids Dream of Breaking the Game? Auditing AI Agent Benchmarks with BenchJack

This piece scrutinizes the integrity of AI agent benchmarks, proposing BenchJack as a systematic method to uncover vulnerabilities and biases. We explore how current evaluation methods might be gamed and the implications for reliable AI development.