Skip to content

WEB: Update benchmarks page #61289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 19, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 7 additions & 45 deletions web/pandas/community/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ kinds of benchmarks relevant to pandas:

pandas benchmarks are implemented in the [asv_bench](https://github.com/pandas-dev/pandas/tree/main/asv_bench)
directory of our repository. The benchmarks are implemented for the
[airspeed velocity](https://asv.readthedocs.io/en/v0.6.1/) (asv for short) framework.
[airspeed velocity](https://asv.readthedocs.io/en/latest/) (asv for short) framework.

The benchmarks can be run locally by any pandas developer. This can be done
with the `asv run` command, and it can be useful to detect if local changes have
Expand All @@ -22,53 +22,15 @@ More information on running the performance test suite is found
Note that benchmarks are not deterministic, and running in different hardware or
running in the same hardware with different levels of stress have a big impact in
the result. Even running the benchmarks with identical hardware and almost identical
conditions produces significant differences when running the same exact code.
conditions can produce significant differences when running the same exact code.

## pandas benchmarks servers
## Automated benchmark runner

We currently have two physical servers running the benchmarks of pandas for every
(or almost every) commit to the `main` branch. The servers run independently from
each other. The original server has been running for a long time, and it is physically
located with one of the pandas maintainers. The newer server is in a datacenter
kindly sponsored by [OVHCloud](https://www.ovhcloud.com/). More information about
pandas sponsors, and how your company can support the development of pandas is
available at the [pandas sponsors]({{ base_url }}about/sponsors.html) page.
The [asv-runner](https://github.com/pandas-dev/asv-runner/) repository automatically runs the pandas asv benchmark suite
for every (or almost every) commit to the `main` branch. It is run on GitHub actions.
See the linked repository for more details. The results are available at:

Results of the benchmarks are available at:

- GitHub Actions results: [asv](https://pandas-dev.github.io/asv-runner/)
- OVH server: [asv](https://pandas.pydata.org/benchmarks/asv/)

### Original server configuration

The machine can be configured with the Ansible playbook in
[tomaugspurger/asv-runner](https://github.com/tomaugspurger/asv-runner).
The results are published to another GitHub repository,
[tomaugspurger/asv-collection](https://github.com/tomaugspurger/asv-collection).

The benchmarks are scheduled by [Airflow](https://airflow.apache.org/).
It has a dashboard for viewing and debugging the results.
You’ll need to setup an SSH tunnel to view them:

```
ssh -L 8080:localhost:8080 [email protected]
```

### OVH server configuration

The server used to run the benchmarks has been configured to reduce system
noise and maximize the stability of the benchmarks times.

The details on how the server is configured can be found in the
[pandas-benchmarks repository](https://github.com/pandas-dev/pandas-benchmarks).
There is a quick summary here:

- CPU isolation: Avoid user space tasks to execute in the same CPU as benchmarks, possibly interrupting them during the execution (include all virtual CPUs using a physical core)
- NoHZ: Stop the kernel tick that enables context switching in the isolated CPU
- IRQ affinity: Ban benchmarks CPU to avoid many (but not all) kernel interruption in the isolated CPU
- TurboBoost: Disable CPU scaling based on high CPU demand
- P-States: Use "performance" governor to disable P-States and CPU frequency changes based on them
- C-States: Set C-State to 0 and disable changes to avoid slower CPU after system inactivity
https://pandas-dev.github.io/asv-runner/

## Community benchmarks

Expand Down
Loading