Automated evaluation of language-model (LM) agents is a critical bottleneck in deploying AI systems, as traditional environment-based rewards are often slow, expensive, or unavailable. An LM-based judge is a proxy evaluator that uses a text-based rubric to grade agent performance without needing direct access to the environment. At Groundwork, our analysis shows that existing judging methods, such as G-Eval or weight-tuned models, frequently suffer from 'over-crediting'—incorrectly labeling fluent but unsuccessful trajectories as successes.

The challenge of reward-free evaluation

Reward-free evaluation is the process of assessing an AI agent’s performance using a secondary model rather than an objective, executable ground-truth reward. In many real-world deployment scenarios, waiting for an environment to return a reward—such as a successful database query or a completed purchase—is computationally prohibitive or impossible. Consequently, developers rely on 'judges' to simulate this feedback.

However, these judges are susceptible to linguistic bias. Because LMs are trained to favor fluent, coherent, and professional-sounding text, they often assign high scores to agents that 'sound' correct even when the underlying task logic fails. This phenomenon, known as over-crediting, creates a dangerous discrepancy between the judge's score and the agent's actual capability, leading to the deployment of broken or inefficient systems.

What is RubricForge and how does it work?

RubricForge is a methodology for inducing agent-judging rubrics by evolving text-based criteria against a small, labeled set of ground-truth trajectories. Rather than relying on static, hand-written rubrics or complex model fine-tuning, RubricForge uses reflective evolution to refine scoring criteria until they align with objective environment outcomes.

1. Data collection: Gather a small set of trajectories already labeled with ground-truth environmental rewards.
2. Reflective evolution: Use an optimization loop to generate and refine a text-based rubric that maximizes agreement with the known ground truth.
3. Freezing the rubric: Once the rubric achieves high faithfulness to the ground truth, it is frozen as a human-readable text artifact.
4. Deployment: Apply the frozen rubric in a single model call to evaluate new, unlabeled agent trajectories without requiring further environment access.

Why over-crediting is a critical failure metric

In the context of AI evaluation, not all errors are created equal. A 'false fail'—where a successful agent is incorrectly marked as failing—results only in the minor cost of a manual review or a retry. Conversely, a 'false pass' is a high-stakes failure: it means a broken agent is marked as successful and potentially pushed to production, where it can cause significant downstream errors.

Empirical research indicates that while generic judges might show high aggregate agreement with human scores, their false-pass rates are often unacceptably high. Groundwork’s synthesis of recent benchmarks, including the tau-bench and WebShop datasets, demonstrates that RubricForge reduces the false-pass rate significantly compared to generic G-Eval approaches. Specifically, in trials on the tau-bench dataset, researchers observed a reduction in over-crediting from 0.173 down to 0.115 (arXiv:2608.13564).

Interpreting judge performance: faithfulness vs. agreement

When evaluating a judge, developers often default to 'raw agreement' metrics, which measure how often the judge matches human or environment scores. However, aggregate agreement is a misleading metric for AI safety. A judge could have 90% agreement while still failing to catch the most dangerous errors.

Faithfulness is a more robust metric. It measures the degree to which a judge’s score is grounded in the actual evidence of the trajectory rather than the fluency of the agent's output. By using human-readable, induced rubrics, RubricForge ensures that every verdict is attributable to specific criteria. This transparency allows developers to audit the judge’s logic, ensuring that the scoring process is not just an arbitrary number but a reflection of task-specific success conditions.

How to implement more reliable agent evaluation

To move toward more reliable agent evaluation, you should transition away from generic, one-size-fits-all prompts and toward induced, evidence-based rubrics. This requires a shift in how you treat your evaluation pipeline:

• Audit your existing judge: Test your current evaluation system against a set of known failures to calculate your specific false-pass rate. If your judge is blind to common failure modes, it is likely over-crediting.
• Prioritize transparency: Ensure your rubrics are human-readable. If a judge cannot justify its score based on specific, named criteria, you cannot trust its output in high-stakes environments.
• Use reflective evolution: Where possible, use small, labeled subsets of your data to 'train' your rubric. By aligning your rubric with objective outcomes through iteration, you ensure that the judge prioritizes functional task success over linguistic style.