Warp Factory Benchmarks | Warp

Benchmark agents on your own code.

Compare models and harnesses across real tasks, then automatically route work to whatever performs best.

Benchmarks

Overall recommendation

Ran by Suraj Gupta
Started 9/2 at 9:45 PM
Completed 9/3 at 1:32 AM
Duration 3h 46m
Total cost $2,130.57

Benchmark run summary:

gpt-5.6-sol (high) had the highest Correctness score at 0.95, followed by glm 5.3 flash at 0.93, the default grok 4.6 (high) at 0.92, claude opus 5 (high) at 0.91, and gemini 3.7 flash at 0.89. gpt-5.6-sol (high) is the highest-quality recommendation, while glm 5.3 flash is both the lowest-credit and most-efficient option. Each non-default configuration differed from the default grok 4.6 (high) by model.

Additional recommendations:

Correctness vs. Average cost

Model Correctness Overall Cost
OpenAI gpt-5.6-sol (high) 0.95 $15.00
Grok grok 4.6 (high) 0.93 $10.00
glm 5.3 flash 0.93 $5.00
Claude claude opus 5 (high) 0.91 TBD
Gemini gemini 3.7 flash 0.90 TBD

Correctness

Benchmarking

Batteries included. No eval stack to stand up.

  1. Historical task replay
    Replay past Factory runs as task sets, starting from their original state.

  2. Custom benchmark task sets
    Curate your own task sets around one repo, one workflow, or one kind of change.

  3. Test different factory configs
    Run identical tasks against different factory versions to see which skills and prompts win.

  4. Model comparison
    Compare frontier and open-weight models directly, on the same real-world tasks.

  5. Harness comparison
    Evaluate agent harnesses side by side, including Warp, Claude Code, and Codex.

  6. Scorers
    Grade runs with LLM-as-a-judge on correctness, quality, efficiency, verbosity, cost, and custom criteria.

  7. Benchmark reports + visualizations
    View performance by task, factory version, and score, with Pareto graphs for cost/quality tradeoffs.

  8. Custom model routing
    Turn results into routing rules that pick the best model for each type of task.