Trajectory shapes

Why are we managing our coding agents based on vibes instead of their actual work habits?

Leaderboards tell us how many tasks a model solved but not how. The trajectories that they leave behind tell the story of how, but these aren’t studied as much. The tools a model prefers, and the overall “shape” of its workflow is visible through an empirical lens.

I analyzed the latest SWE-Bench Pro trajectories I could find: runs from October 2025 for Sonnet 4.5 and GPT-5, 730 task trajectories per model. I deterministically classified each step into activities like understand, edit, verify, and cleanup using only tool calls, literal command/filename matches, and regex heuristics, and then computed their share over time. This “trajectory shape” chart I got as a result is very interesting!

Here are the main work habits I see in this chart:

1. Sonnet starts editing early (at 35%), GPT starts editing much later (at 50%)
2. Sonnet is done with the implementation early (at 62%)
3. Sonnet spends a LOT of time verifying after the last source edit
4. Sonnet has to cleanup temporary files, GPT doesn’t have to
5. GPT-5 reads a LOT to front-load context, before it starts editing (at 50%)
6. GPT-5 barely does any verification

If I were to condense the above to “vibes”, I might say “Claude starts editing early and figures it out in the loop. GPT reads first, then goes for the one-shot.” That is also close to what people on my timeline were saying when these runs were current. For example, Eric Provencher wrote:

My take with codex vs claude code

- Use codex if you have a detailed plan and want to walk away for 30 min. If you try and interrupt and iterate codex, you might as well restart from scratch
- Use claude code if youre not 100% sure where you're going and want to iterate

— eric provencher (@pvncher) October 13, 2025

Now I can say the same thing with data, and with a better sense of the nuances. And once we see these habits clearly, we can stop managing agents on vibes and start steering them on evidence.

With newer models and a maintainer

I wanted to replicate this chart for Opus 4.6 and GPT-5.4, but the SWE-Bench Pro trajectories (or any other SWE-agent trajectories) for them are unavailable. I then remembered that Mario Zechner has been publishing his Pi trajectories on Hugging Face, and downloaded them.

Luckily, he works through GitHub issues methodically. Each session is one issue, and one model. It kicks off with the same analysis prompt, waits for his go ahead, there is a varied amount of steering to address it, and then he wraps up explicitly to ship a fix, close the issue, leave a triage comment, and so on. Each one starts with git work (read the PR, read the comments) and ends with git work (push the change, close the issue, or do the final triage), with the source-editing loop in between.

While the issue fixes make for similar trajectories to SWE-Bench Pro, I’m still changing the models, harness, and adding a maintainer, so this is not exactly an apples-to-apples comparison. To keep the trajectory-shape comparison closer to the benchmark, the Pi panels below use only the strict single-model sessions whose final title starts with Issue:. That said, the observations from the previous section still hold. The bars below each Pi panel give a rough sense of where Mario is steering more during the run; taller bars mean more intervention around that part of the trajectory. The labels mark Mario’s go ahead and wrap up moments (hide git).

I think it’s likely that the explicit analysis prompt is pushing the understand phase to be longer. The first edit is pushed from 35% to 47%. After that, there’s a fair amount of steering, QA, critique, and validation, with verify + edit cycles, after which it wraps up. The push to verify in the SWE-agent prompt likely causes the benchmark trajectory to go on longer, and is fairly redundant.

As with Claude, the analysis prompt makes GPT go on even longer, pushing the first edit from 50% to 63%. However, the edit duration has reduced significantly from 40% to 23%. I suspect this is a combination of the model upgrade and Mario’s steering towards “minimal” and “concise” solutions.

Why compare trajectory shapes?

Trajectories are traces of learned workflows. RL shapes both micro preferences, like which tools a model uses, and macro preferences, like the overall workflow it follows from problem statement to verified solution. To study those macro patterns, one trajectory is not enough; you need a corpus. And to compare a corpus, you need normalization. So, each trajectory is mapped onto the same 0–100% start-to-finish frame so that the aggregate shape reflects the learned workflow. For instance, see the difference in trajectory lengths of the SWE-Bench Pro:

Further, tool calls are normalized by structural intent rather than literal tool identity, so that equivalent actions can be compared across harnesses. In this study, I normalised tool calls across SWE-agent and Pi, and the table below documents that taxonomy.

So, if you recognize these patterns in your agents, and they don't fit your task at hand, you could steer them accordingly.

Appendix

Analysis code and data: github.com/nilenso/swe-bench-pro-cost-token-time-analysis.

SWE-Bench Pro

• Charts and takeaways — all 4 models
• Charts and takeaways — Sonnet 4.5 vs GPT-5
• Detailed breakdowns — all 4 models
• Detailed breakdowns — Sonnet 4.5 vs GPT-5

Pi transcripts

• Charts and takeaways — all available Pi models
• Charts and takeaways — Opus 4.5/4.6 vs GPT-5.4
• Detailed breakdowns — all available Pi models
• Detailed breakdowns — Opus 4.5/4.6 vs GPT-5.4

1. GPT-5’s tool failures

I have ignored these failures for the purpose of the charts, but they are significant enough to mention here. GPT-5's tool calls fail in about 20% of the steps throughout the SWE-agent trajectory. Most notably, it expects the apply_patch tool to be present, and keeps calling that. apply_patch is the main editing tool within Codex, and GPT-5’s own system prompts inside Codex had counterweights for this behavior. GPT-5 manages to work through the failures and still achieve a decent resolution rate on the task. However, we do not know what effect those failing tool calls have on the outcome.

Example failures and data: failure modes in the reference appendix.

2. High-level action frequencies

On SWE-Bench Pro, even if we ignore when actions happen and just count what the models spend their steps on, the same signature shows up. GPT-5 spends much more of its trajectory reading, while Sonnet 4.5 spends much more of it verifying. That makes the difference in the trajectory-shape chart feel less like a visual artefact and more like a stable workflow habit.

• To “understand”, GPT reads more, while both models search about the same amount
• Verification leans the other way. Sonnet 4.5 runs ~3 verify steps per edit, GPT-5 runs ~1 verify step per 3 edits
• Git + housekeeping is effectively Sonnet-only (6.2% vs 0.2%). GPT-5’s trajectory has no bookkeeping tail at all.
• Pre-edit information gathering (read + search + reproduce) is 61% of GPT-5’s steps vs 49% of Sonnet’s. GPT-5 front-loads context; Sonnet spreads work more evenly across the trajectory.

• Around 40% of GPT-5 trajectories don’t do ANY verification at all
• Sonnet 4.5 re-runs the same passing tests many times, without any changes in source

5. Mario’s analysis prompt

Every Pi session in the analyzed set kicks off with this prompt. The <issue-number> is filled in per session; Mario steers from there.

Analyze GitHub issue(s): https://github.com/badlogic/pi-mono/issues/<issue-number> you will have to pull down the image and read it as well to understand.

For each issue:

1. Read the issue in full, including all comments and linked issues/PRs.

2. **For bugs**:
   - Ignore any root cause analysis in the issue (likely wrong)
   - Read all related code files in full (no truncation)
   - Trace the code path and identify the actual root cause
   - Propose a fix

3. **For feature requests**:
   - Read all related code files in full (no truncation)
   - Propose the most concise implementation approach
   - List affected files and changes needed

Do NOT implement unless explicitly asked. Analyze and propose only.

6. SWE-agent issue-resolution prompt

For comparison, the SWE-agent setup used in SWE-Bench Pro includes the following issue-resolution scaffold in its default prompt (source):

Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to find and read code relevant to the <pr_description>
2. Create a script to reproduce the error and execute it with `python <filename.py>` using the bash tool, to confirm the error
3. Edit the source code of the repo to resolve the issue
4. Rerun your reproduce script and confirm that the error is fixed!
5. Think about edgecases and make sure your fix handles them as well