# Benchmark agents on your own code.

Compare models and harnesses across real tasks, then automatically route work to whatever performs best.

## Benchmarks

### Overall recommendation

- **OpenAI**  
  gpt-5.6-sol (high)  
  **Highest quality**  
  Overall 0.95

**Ran by** Suraj Gupta  
**Started** 9/2 at 9:45 PM  
**Completed** 9/3 at 1:32 AM  
**Duration** 3h 46m  
**Total cost** $2,130.57

### Benchmark run summary:

gpt-5.6-sol (high) had the highest Correctness score at 0.95, followed by glm 5.3 flash at 0.93, the default grok 4.6 (high) at 0.92, claude opus 5 (high) at 0.91, and gemini 3.7 flash at 0.89. gpt-5.6-sol (high) is the highest-quality recommendation, while glm 5.3 flash is both the lowest-credit and most-efficient option. Each non-default configuration differed from the default grok 4.6 (high) by model.

#### Additional recommendations:

- **Most efficient:**  
  glm 5.3 flash Overall 0.93

- **Lowest cost option:**  
  glm 5.3 flash $1.99/trial

### Correctness vs. Average cost

| Model                     | Correctness | Overall Cost |
|---------------------------|-------------|--------------|
| OpenAI gpt-5.6-sol (high)  | 0.95        | $15.00       |
| Grok grok 4.6 (high)     | 0.93        | $10.00       |
| glm 5.3 flash            | 0.93        | $5.00        |
| Claude claude opus 5 (high) | 0.91        | TBD          |
| Gemini gemini 3.7 flash   | 0.90        | TBD          |

## Correctness

- Grok grok 4.6 (high)
- Claude claude opus 5 (high)
- glm 5.3 flash
- Gemini gemini 3.7 flash
- OpenAI gpt-5.6-sol (high)

## Benchmarking

### Batteries included. No eval stack to stand up.

1. **Historical task replay**  
   Replay past Factory runs as task sets, starting from their original state.

2. **Custom benchmark task sets**  
   Curate your own task sets around one repo, one workflow, or one kind of change.

3. **Test different factory configs**  
   Run identical tasks against different factory versions to see which skills and prompts win.

4. **Model comparison**  
   Compare frontier and open-weight models directly, on the same real-world tasks.

5. **Harness comparison**  
   Evaluate agent harnesses side by side, including Warp, Claude Code, and Codex.

6. **Scorers**  
   Grade runs with LLM-as-a-judge on correctness, quality, efficiency, verbosity, cost, and custom criteria.

7. **Benchmark reports + visualizations**  
   View performance by task, factory version, and score, with Pareto graphs for cost/quality tradeoffs.

8. **Custom model routing**  
   Turn results into routing rules that pick the best model for each type of task.