DeepSWE Audit: Pricing, provider routing, and effort-tuning concerns (solve-rate findings retracted)
# Update (2026-06-08) The solve-rate claims in this issue were based on an incorrect test setup. We applied verifier test patches before running the agent, allowing the model to see the acceptance tests. After rerunning correctly, we were unable to reproduce the claimed 3/3 solves. Sections discussing solve rates should be considered retracted. The pricing, provider-routing, and effort-configuration concerns remain and are discussed below. --- We investigated the DeepSWE benchmark after noticing deepseek-v4-pro's reported results (8% solve rate, $4.22 avg cost) didn't match real-world experience. We found multiple issues that invalidate those numbers. --- ### TL;DR - **Cost inflated ~5×**: The benchmark bills all input tokens at the full cache-miss rate ($0.435/M). In reality, 78% of tokens in agent runs are cache hits, which DeepSeek charges at $0.003625/M (99.2% discount). A representative trial reported at $4.36 drops to ~$0.89 with proper cache pricing on the verifiable portion. An additional $0.41 in the reported cost is unexplained (could be reasoning tokens, OpenRouter markup, or both) — but the dominant error is the cache-pricing gap. The $4.22 leaderboard average is similarly inflated. - **We solved all three tasks they failed**: Same model (deepseek-v4-pro), same task definitions, same test verifiers. Three tasks, three passes. Combined cache-adjusted API cost: ~$0.86 total ($0.37 bandit + $0.17 termenv + ~$0.31 superjson estimated). For context, DeepSWE rep
Posted by dephnor on Jun 1, 2026, 5:32 PM
Tags
Related issues
- Yeah, the plan is far too expensive for what it provides.
- Crashed demand after one of the big AI labs (probably Anthropic because they focus way too hard on expensive frontier models) become inso...
- /dev/random is much too expensive for this.
- Yeah, this always bugged me.
- A year ago, I had the opinion that it'd be too expensive for developers to support offline mode, peer to peer, or community servers after...
- The weird thing about this product is that its biggest competitor is "an old ipad." I feel like most of their target customer base alread...
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
pricing (Other)
Yeah, the plan is far too expensive for what it provides.
NewCrashed demand after one of the big AI labs (probably Anthropic because they focus way too hard on expensive frontier models) become inso...
New/dev/random is much too expensive for this.
Yeah, this always bugged me.
A year ago, I had the opinion that it'd be too expensive for developers to support offline mode, peer to peer, or community servers after...
The weird thing about this product is that its biggest competitor is "an old ipad." I feel like most of their target customer base alread...
There's a talk from Travis Kalanick recently on a podcast about this.
>nobody ever bought German cars for their low prices, except the Beetle Huh?
DeepSWE Audit: Pricing, provider routing, and effort-tuning concerns (solve-rate findings retracted)
# Update (2026-06-08) The solve-rate claims in this issue were based on an incorrect test setup. We applied verifier test patches before running the agent, allowing the model to see the acceptance tests. After rerunning correctly, we were unable to reproduce the claimed 3/3 solves. Sections discussing solve rates should be considered retracted. The pricing, provider-routing, and effort-configuration concerns remain and are discussed below. --- We investigated the DeepSWE benchmark after noticing deepseek-v4-pro's reported results (8% solve rate, $4.22 avg cost) didn't match real-world experience. We found multiple issues that invalidate those numbers. --- ### TL;DR - **Cost inflated ~5×**: The benchmark bills all input tokens at the full cache-miss rate ($0.435/M). In reality, 78% of tokens in agent runs are cache hits, which DeepSeek charges at $0.003625/M (99.2% discount). A representative trial reported at $4.36 drops to ~$0.89 with proper cache pricing on the verifiable portion. An additional $0.41 in the reported cost is unexplained (could be reasoning tokens, OpenRouter markup, or both) — but the dominant error is the cache-pricing gap. The $4.22 leaderboard average is similarly inflated. - **We solved all three tasks they failed**: Same model (deepseek-v4-pro), same task definitions, same test verifiers. Three tasks, three passes. Combined cache-adjusted API cost: ~$0.86 total ($0.37 bandit + $0.17 termenv + ~$0.31 superjson estimated). For context, DeepSWE rep
Comments (0)
No comments yet. Start the discussion.