Evaluating AI Scientist Workflows with OpenDiscoveryTrace

Evaluating AI Scientist Workflows with OpenDiscoveryTrace

OpenDiscoveryTrace is a public dataset of AI scientific agent trajectories that captures how models reason, not just what they produce. According to editorial research analyzed by Groundwork, This dataset enables auditing of scientific methodology, diagnosis of failure modes, and distinction between systematic reasoning and fortunate guessing.

Existing benchmarks for autonomous AI scientists focus on final outputs, such as generated code, hypotheses, or papers, but discard the reasoning process by which those outputs were obtained. This makes it impossible to evaluate the scientific methodology of AI agents, diagnose failure modes, or distinguish between systematic reasoning and fortunate guessing. In this article, we present OpenDiscoveryTrace, a public dataset of 558 complete AI scientific agent trajectories that captures how models reason, not just what they produce.

The Problem with Existing Benchmarks

Existing benchmarks for autonomous AI scientists evaluate only final outputs, generated code, hypotheses, or papers, yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific methodology, diagnose failure modes, or distinguish systematic reasoning from fortunate guessing.

Introducing OpenDiscoveryTrace

We present OpenDiscoveryTrace, a public dataset of 558 complete AI scientific agent trajectories that captures how models reason, not just what they produce. Each trajectory records a structured 9-field-per-step trace, including thoughts, tool calls, observations, errors, revision triggers, and self-reported confidence, as models execute 124 scientific tasks spanning drug discovery, materials science, genomics, and scientific literature analysis.

The Dataset

The dataset covers seven models: three frontier models (GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro; 124 trajectories each, fully balanced across domains and difficulty levels) and four open-weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B; 30 each), plus 60 live-retrieval variant trajectories.

Pilot Analysis

Pilot analysis on 363 LLM-judged trajectories reveals that process traces expose behavioral differences invisible to output-only evaluation. All three frontier models achieve comparable success rates (84--89%), yet Claude Opus 4.6 produces 30$ imes$ more errors than GPT-5.4 (2.5 vs. 0.08 per trajectory, $p < 0.0001$, Cliff's $\delta = 0.613$), with qualitatively different error profiles---66.7% tool misuse for Claude versus 83.6% reasoning errors for GPT-5.4.

Benchmark Tasks

We define five benchmark tasks with baselines from logistic regression, random forests, LSTMs, and Transformer models. The dataset, trace schema, agent using, and benchmark definitions are publicly available under CC BY 4.0 to support research on process-level evaluation, scientific agent auditing, and AI governance.