My intuitions about GPT being slow and token hungry were wrong. I had a feel for how these models worked from daily use, but that feel was built from my own tasks and my own repos. Benchmarks exist because vibes can mislead, but I couldn't find a reliable benchmark that compared execution time, tokens and cost across models.
The SWE-Bench Pro leaderboard reports resolve rates but not cost, tokens, or time. The raw trajectory data is publicly available, and I wanted to check my assumptions against it. These are October 2025 models (GPT-5 and Sonnet 4.5), run under identical conditions on the SWE-Agent scaffold. The coding agent landscape has changed significantly since then. This data is a snapshot, not the current state of things.
I downloaded all 1,460 trajectory files (~23 GB), extracted per-instance cost, token counts, tool execution time, and action breakdowns, then paired them for direct comparison. Both runs use the same scaffold (SWE-Agent v1.1.0), same tools, same prompt, 250-turn limit, no cost limit. GPT-5 runs with reasoning_effort: high. Sonnet 4.5 uses default settings.
The overall patterns hold across repos, but the magnitudes vary. Filter by repo to see how.
See also: Unsubmitted tasks report, covering instances where one or both models failed to submit a patch.