Reference Tables

Each SWE-Bench Pro trajectory ends with an exit status. The agent either submits a patch or doesn't.

submitted (clean): exit_status is exactly submitted.

submitted (w/ error): the agent produced a submission, but the harness also recorded an error condition (timeout, context overflow, cost limit, format error).

not submitted: the agent never ran the submit command. It hit an error before submitting.

resolved: the submitted patch actually fixes the failing tests (from benchmark evaluation).

Method: we read info.exit_status and info.submission from each .traj file.

Every distinct exit_status string from the .traj files, with counts per model.

submitted: clean exit after submitting.

submitted (exit_*): submitted, but also hit an error condition (timeout, context, cost, format, etc.).

exit_* (without submitted prefix): the agent hit that condition and never submitted.

These statuses are set by the SWE-Agent harness, not by the model itself.

Summary statistics for trajectory length (number of action-observation steps per task).

n: trajectories (one per task instance).

total_steps: sum of all steps across all trajectories for this model.

avg / median / p25 / p75 / min / max: distribution of steps per trajectory.

resolved: trajectories where the submitted patch fixes the failing tests (from agent_runs_data.csv).

resolve_rate: resolved / n.

The label describes what the command is, derived from the action string and filename alone — no positional context (before/after first edit) and no outcome signal is used. A failed grep is still a search attempt.

(failed) variants classify by intended action, not outcome. They require a shell-level error in the first 500 chars of the observation.

run-inline-snippet is a residual — inline snippets (python -c, python - <<, node -e) are first routed to run-inline-verify / read-via-inline-script / edit-via-inline-script / create-file-via-inline-script / check-version by inspecting the code shape.

Pass/fail outcome for verify intents (used by seq-first-all-pass / seq-work-done) is a separate detector that reads the observation for unambiguous runner summaries: e.g. pytest N passed in Xs / N failed, go ok package / FAIL package, jest Tests: N passed / N failed. Ambiguous output returns unknown.

Canonical source: scripts/classify_intent.py and docs/intent-classification-rules.md.

Every trajectory step is classified into one of ~50 base intents using deterministic rules (regex matching on the action string, file names, and observation text). No LLM is used for classification.

_n: total count of that intent across all trajectories for the model.

_%: that count as a percentage of all steps for the model. Percentages sum to 100% within each model.

Sorted by total count across all models (most frequent first).

Method: each step's action string is pattern-matched against a priority-ordered ruleset in classify_intent.py. For example, an action starting with str_replace_editor view is classified as a read intent, while grep or rg becomes search-keyword.

Each base intent maps to one of 9 high-level categories: read, search, reproduce, edit, verify, git, housekeeping, failed, other.

_n: total steps in that category.

_%: percentage of all steps.

_per_traj: average number of steps in that category per trajectory (total category steps / number of trajectories).

The mapping from base intent to category is defined in classify_intent.py (INTENT_TO_HIGH_LEVEL). For example, read-file-full, read-file-range, and read-via-bash all map to 'read'.

The 9 high-level categories are further grouped into 5 phases that represent the broad arc of a trajectory:

understand = read + search. The agent is reading code and searching for information.

reproduce = reproduce. The agent is writing or running reproduction scripts to confirm the bug.

edit = edit. The agent is making source code changes.

verify = verify. The agent is running tests, compiling, or checking its work.

cleanup = git + housekeeping. The agent is reviewing changes (git diff/log) or cleaning up (rm, mv, writing docs).

These phases are used in the stacked area charts (Typical Trajectory Shape) to show how the mix of actions evolves from start to end.

The 'verify' category contains ~10 sub-intents. This table shows where each model's verification volume comes from.

run-test-suite: broad test runs (pytest, go test, npm test, mocha) without targeting specific tests.

run-test-specific: targeted test runs using pytest -k or :: to run specific test functions.

run-verify-script: running a script named verify*, check*, validate*, or edge_case*.

create-test-script: creating a new test file (test_*, *test.py, etc.).

run-inline-verify: an inline python -c / node -e snippet that imports project code or runs assertions.

compile-build: go build, go vet, make, npx tsc. Compilation as a verification step.

edit-test-or-repro: editing an existing test or repro file (str_replace on test_* or repro* files).

run-custom-script: running a named script that doesn't match repro/test/verify naming patterns.

create-verify-script: creating a new file named verify*, check*, validate*.

syntax-check: py_compile, compileall, node -c. Quick syntax validation.

Only steps classified as one of these intents are evaluated for outcome: run-test-suite, run-test-specific, run-verify-script, run-custom-script, run-inline-verify, compile-build, syntax-check. All other steps get outcome ''.

pass: the observation's last 2000 characters match a framework-specific all-pass pattern. For pytest: the summary line (e.g. '200 passed in 12.3s') must contain 'passed' and must NOT contain 'failed' or 'error'. If even one test fails ('195 passed, 5 failed'), the outcome is 'fail', not 'pass'. For Go: all PASS/FAIL lines in output are checked; any FAIL makes it 'fail'. For Mocha: checks 'N passing' and 'N failing' counts. For Jest: checks summary line for 'failed' vs 'passed'. For compile-build: absence of error patterns in short output (< 200 chars) from go build/make = 'pass'. For syntax-check: py_compile with no output = 'pass'; any Error/SyntaxError = 'fail'.

fail: the observation matches a failure pattern. In priority order: framework-specific failure summaries (pytest 'failed', Go 'FAIL', Mocha failing > 0), then generic patterns: 'no tests ran', collection errors, tracebacks in the last 500 chars, Node.js throw/error, non-zero exit code.

unknown (''): no pattern matched. This happens when output is from an unrecognized framework, is truncated, is ambiguous (e.g. pytest ran but the summary line was cut off), or when the observation is empty.

pass_rate: pass / (pass + fail), excluding unknowns. This measures: of the verify steps where we could determine the outcome, what fraction had all tests passing?

Important caveat: 'pass' means 'all tests in that run passed', not 'the agent's fix is correct'. SWE-Bench Pro tasks come with existing test suites where most tests already pass on unmodified code. An agent running pytest before making any edits will often get 'pass' because the existing tests pass. This is why first_verify_pass can occur before first_edit (see Section 10).

Sequence labels classify steps by their context: what happened before, whether edits or verify steps preceded them.

seq-verify-after-edit: a verify step after a source edit. The core edit-then-test loop.

seq-verify-rerun-no-edit: a verify step where no edit happened since the last verify.

seq-edit-after-failed-verify: a source edit after a failed verify step. Fixing what a test revealed.

seq-submit-after-verify: submit after at least one verify step. The agent tested before submitting.

seq-first-all-pass: the first verify-pass after the last source edit. Marks implementation completion.

Method: classify_sequence_layer() in classify_intent.py walks the trajectory maintaining state (has a verify been seen? was there an edit since?).

This section uses data/failure_modes.json, produced by scripts/build_failure_modes.py. Each trajectory step is classified as either not-a-failure or one failure mode.

Counts are step counts, not unique incidents. A single trajectory can contribute many failure steps, and one underlying shell mistake can fan out into multiple observed failures.

Families: tool means the harness/tool call itself failed; code means the agent ran code that crashed; test means a test runner reported failures.

Interpretation caveat: bash_broken_pipe is often a secondary symptom. In GPT-5, many of those steps are downstream of the same wrapper pathologies that also produce trailing-brace, heredoc, or quoting failures.

The distinctive GPT-5 signature is not ordinary test failure. It is repeated interaction friction around shell wrapping and hallucinated applypatch usage.

A confusion matrix crossing two signals: whether the agent reached 'work-done' and whether the benchmark evaluated the patch as correct.

work-done: the trajectory contains a seq-first-all-pass label. Specifically: we find the last step classified as a source edit (edit-source, insert-source, apply-patch, edit-via-inline-script). Then we check if any verify step after that point has a 'pass' outcome (all tests passed, per the rules in Section 7). If yes, the trajectory is 'work-done'. This is a stronger signal than first_verify_pass (Section 10), which can fire before any edits because existing tests pass on unmodified code.

resolved: the submitted patch actually fixes the failing tests, as judged by the SWE-Bench Pro benchmark evaluation (from agent_runs_data.csv). This is different from 'submitted', which only means the agent produced a patch.

wd+resolved: the agent's tests passed after its last edit, and the benchmark confirmed the patch is correct. The best case.

wd+unresolved: tests passed but the patch was wrong. The agent's own verification was a false positive.

no_wd+resolved: the agent never reached a clean test pass after its final edit, yet the benchmark accepted the patch. The agent submitted without confirmation that its code works.

no_wd+unresolved: the agent neither achieved passing tests nor produced a correct patch.

The stacked bars show how each model's 730 trajectories split across these four outcomes. The no_wd+resolved column is notably large across all models, meaning the agent's own test-passing signal is not a reliable predictor of benchmark resolution.

Method: 'work-done' = find the last step in SOURCE_EDIT_INTENTS (edit-source, insert-source, apply-patch, edit-via-inline-script), then scan forward for any step where classify_verify_outcome() returns 'pass'. If found, work-done is true. 'resolved' comes from the benchmark CSV (agent_runs_data.csv, field metadata.resolved), which records whether the submitted patch actually made the failing tests pass when evaluated by the benchmark harness.

Caveat: work-done can be a false positive. The agent's tests might pass because the test suite doesn't cover the specific failure the task requires fixing. The agent thinks it's done (tests pass), but the benchmark's evaluation finds the bug isn't actually fixed.

Key events in each trajectory, expressed as a percentage of the way through (0% = first step, 100% = last step). Aggregated across all trajectories per model.

The timeline shows median positions as shaped markers, with faint bands for the interquartile range (p25-p75). Hover over markers for exact values.

first_edit: the first step whose base intent is one of: edit-source (str_replace on a source file), insert-source (str_replace_editor insert), apply-patch, or edit-via-inline-script. Does not include create-file or edit-test-or-repro, which are classified differently.

last_edit: the last step matching those same intents. The gap between last_edit and submit is the 'tail' where the agent is verifying, cleaning up, or submitting but no longer changing source code.

first_verify: the first step whose intent is in SEQUENCE_VERIFY_INTENTS: run-test-suite, run-test-specific, run-verify-script, run-custom-script, compile-build, syntax-check, run-inline-verify.

first_verify_pass: the first step where classify_verify_outcome() returns 'pass' (see Section 7 for what 'pass' means). This does NOT mean 'the agent's fix worked'. It means 'the first time a test/build command produced output where all tests passed'. Because SWE-Bench Pro tasks have existing test suites that mostly pass on unmodified code, an agent that runs pytest before making any edits will often get a 'pass' here. This is why first_verify_pass can appear before first_edit for Claude (median 28.8% vs 34.6%): Claude runs the existing test suite early as a diagnostic baseline. For a marker that means 'the fix works', see work_done in Section 9, which requires a verify pass after the last source edit.

last_verify: the last step whose intent is in SEQUENCE_VERIFY_INTENTS. The alternate view replaces first pass with last verify to show where verification actually finishes, which makes the late verification tail clearer for Claude.

submit: the first step with intent 'submit'.

_med / _p25 / _p75: median, 25th percentile, and 75th percentile across trajectories where the event occurred.

_n: number of trajectories where this event occurred. Claude has 639 for first_verify_pass (out of 730) meaning 91 trajectories never had a fully-passing test run. GPT has only 130, meaning most GPT trajectories either never ran tests or never achieved a clean pass.

Method: for each trajectory, scan for the first (or last) step matching the relevant intent set, compute step_index / (total_steps - 1) * 100 to get a percentage position, then take the median across all trajectories where the event occurred.

Read left-to-right as beginning to end of trajectory. Brighter cells indicate the dominant action in that time-slice.

Categories: read, search, reproduce, edit, verify, git, housekeeping. Failed and other are excluded.

Normalized per column: within each time-slice, the percentages show each category's share relative to only the displayed categories.

Method: each trajectory's step sequence is divided into 20 equal bins. Per bin, we count the fraction of steps belonging to each category, then average across all trajectories for that model.

Metrics broken down by source repository. SWE-Bench Pro tasks come from ~11 open-source repos. The dot plot shows whether one model dominates uniformly or whether there is repo-specific variation.

Dot plot: each dot is one model's resolve rate on that repo. When dots cluster, models perform similarly on that repo; when they spread apart, the repo differentiates models. Repos with fewer than 3 tasks per model are omitted from the plot to avoid noisy rates.

Spread table: the gap (in percentage points) between the best and worst model on each repo. Large spreads indicate repos where model choice matters most.

_n: number of task instances from this repo.

_avg: average steps per trajectory.

_res%: resolve rate (percentage of trajectories where the submitted patch fixes the failing tests).

_ver%: percentage of steps spent on verify actions.

Sorted by total number of instances across all models (most common repos first). Top 12 shown.