Source: https://openrunners.io/evaluations/

# How a runner vendor gets measured.

*Evaluations*

Scheduled monthly runs are free for vendors. This page says how a run is made, what a vendor can ask for, and where every change is written down.

## Who runs this site

openrunners is maintained by engineers who also build [Sengi](https://sengi.run/), one of the runners listed. The method, the raw runs and the ranking rule are public so anyone can check the numbers, and every change is on the [changelog](https://openrunners.io/changelog/).

## Method

This is the rule for scheduled runs from now on. Every vendor in a run gets:

- **The same two jobs.**
- **The same pinned commit** of the code those jobs run.
- **The same runner size**, compared by vCPU count.
- **The same day.**
- **The same number of runs.** Failed runs are reported, not dropped. Warmup runs are in the raw files and excluded from the medians.
- **All rows together.** When any row is re-measured, all rows are.

Prices are the public list price on the date checked. Each row links to the vendor's own page for it.

## So far

The 11 November 2026 data did not meet that rule. GitHub and Sengi ran at 05:38 UTC and Ubicloud ran at 15:23 UTC. That is the same day, but in separate rounds. GitHub ran at 2 vCPU only. The other listed vendors have not been measured yet. The most recent measured run is dated 2026-11-11. The next measured table has not been published yet. The next full monthly run measures every vendor whose app is installed, all on the same day.

## Scheduled runs

The run happens in the first week of each month. It costs a vendor nothing and covers every listed vendor whose app is installed. Each run is added to the [changelog](https://openrunners.io/changelog/) with a link to its raw file.

## Ask for a re-run

A vendor can ask for a run outside the schedule by emailing [evaluations@openrunners.io](mailto:evaluations@openrunners.io).

- An off-schedule re-run covers the same two jobs and the same sizes for every vendor measured in the latest scheduled run.
- The requesting vendor pays list-price runner minutes for its own rows only. Free tiers are not counted.
- Every other vendor is re-run the same day at our cost.
- The re-run and the reason for it are listed on the [changelog](https://openrunners.io/changelog/).

## Send a correction

A vendor can send a correction to any price or fact at [evaluations@openrunners.io](mailto:evaluations@openrunners.io), with a link to its own page that shows the right figure. Anyone else may write too. We check the vendor's page. If it supports the correction, we update the data and log it on the [changelog](https://openrunners.io/changelog/).

## Data, method and ranking

- Raw data: [benchmark.json](https://openrunners.io/data/benchmark.json), [benchmark.csv](https://openrunners.io/data/benchmark.csv), and the per-run files listed on the [benchmark method](https://openrunners.io/benchmark/#method).
- Changelog as data: [changelog.json](https://openrunners.io/data/changelog.json).
- Failures: every run that failed is in the raw files and counted in the Runs column. An option with more than 2% infrastructure failures carries a flag in its row.
- Ranking rule: rank by measured cost per job at list price, cheapest first. Job time and queue time are shown beside it. Ties go to the shorter queue time. See the [full rule](https://openrunners.io/benchmark/#ranking-rule).
- Results: the [measured ranking](https://openrunners.io/best-github-actions-runners/).


openrunners is maintained by engineers who also build Sengi (https://sengi.run/), one of the runners listed.
