Reducing your team's cost-per-PR with custom benchmarks

Date and time
Thu, Sep 17, 2026 · 2:00 PM EDT–3:00 PM EDT

You can reduce your cost-per-PR significantly by optimizing model selections. But how do you decide which models to use across different tasks? Public benchmarks like SWEBench? Manual trial-and-error? The Twitter flavor of the day?

From our testing, the best tool isn’t public benchmarks or recommendations; it's testing on your team's own coding tasks.

In this session, I’ll show how to build a benchmark by replaying your team’s past agent runs using Factory Benchmarks. ​We'll walk through how to find the right sample data, how to score benchmark runs, and how to understand a benchmark report to decide the best model to reduce your team’s cost-per-PR.

We used this approach ourselves to drive down our costs from $80 per PR to $30.

What you'll learn

• ✓How to build a sample of coding tasks to benchmark from your team's agent runs
• ✓The metrics worth scoring to benchmark agent quality
• ✓How to use results to tune model selections by setting defaults or configuring custom model routers
• ✓Our findings on the best models to use for full-stack programming as part of a software factory