More Rules, Longer Lists? Benchmarking Self-Evolving Agents on a Hidden-Rule Curriculum

The paper introduces CLR‑Bench, a new benchmark suite designed to test self‑evolving agents’ ability to uncover hidden rules without explicit guidance. Each chain in the benchmark adds an additional latent rule at every level, based on opaque labels that map to named groupings found online, for example the founding members of an airline alliance. The evaluation process supplies a single value example, receives feedback revealing an extra member and a non‑member, and then checks whether the agent can identify unseen members and distinguish them from similar items. The authors describe an automated pipeline that generates 20 chains from 10 distinct domains and evaluates six recent self‑evolution methods. Results show that, with a certifying model, agents barely outperform a frozen agent, achieving gains of up to 5 points and excluding only between 6 % and 12 % of unseen members, compared to the 20 % achieved by simply retaining practice history. Adding a fixed lesson on named grouping conventions boosts performance by 14.5 to 16.5 points. More powerful models such as GPT‑6‑Sol, ACE, and Dynamic Cheatsheet reach up to 78 % accuracy, solving most individual cases. This research demonstrates that CLR‑Bench provides a detailed measure of agents’ learning-from‑feedback capabilities, which vary markedly across architectures. Understanding and improving this type of machine learning is crucial for building more autonomous and reliable AI systems in environments where rules cannot be pre‑defined.

Read the original article on arXiv cs.AI

This summary is an informational synthesis produced by dataqbs.com. All rights to the original content belong to its author and the cited media outlet. We act solely as curators of technology news and claim no authorship.