# Reducing your team's cost-per-PR with custom benchmarks

Date and time  
Thu, Sep 17, 2026 · 2:00 PM EDT–3:00 PM EDT

You can reduce your cost-per-PR significantly by optimizing model selections. But how do you decide which models to use across different tasks? Public benchmarks like SWEBench? Manual trial-and-error? The Twitter flavor of the day?

From our testing, the best tool isn’t public benchmarks or recommendations; it's testing on your team's own coding tasks.

In this session, I’ll show how to build a benchmark by replaying your team’s past agent runs using [Factory Benchmarks](/content/events/warp.dev/factories/benchmarks/index.html). ​We'll walk through how to find the right sample data, how to score benchmark runs, and how to understand a benchmark report to decide the best model to reduce your team’s cost-per-PR.

We used this approach ourselves to drive down our costs from $80 per PR to $30.

## What you'll learn

- ✓How to build a sample of coding tasks to benchmark from your team's agent runs  
- ✓The metrics worth scoring to benchmark agent quality  
- ✓How to use results to tune model selections by setting defaults or configuring custom model routers  
- ✓Our findings on the best models to use for full-stack programming as part of a software factory