Abstract

Tracking the performance of our complex graphics stack is a necessity to avoid unintentional regressions in the performance of the most important games and benchmarks. Such regressions lead to unhappy open source gaming enthusiasts and wasted developer time who need to track down performance regressions sometimes months after they got introduced. Fixing such performance issues may also be a challenge as the code that introduced them may have become a dependency for newer features.

The need for performance tracking is becoming more and more critical as the complexity of our open source drivers increases to reach a performance comparable to their closed-source equivalent. This increased complexity makes it more and more likely for commits to accidentally break the performance of some benchmark/games on some platforms that the developer may not have convenient access to.

In an effort to detect performance regressions before they even hit mainline, automation should be increased. However, when it can take up to an hour to test the performance of one commit on one benchmark, it becomes clear that we will never have the necessary hardware to be able to test all the commits found on the mailing lists and we will have to be smarter than this.

In this presentation, I will describe the different challenges found in benchmarking, some surprising results, some tricks to reduce the variance between runs and what is my current plan for improving our performance QA by automatically tracking the performance, bisecting performance changes and letting everyone know about them by auto answering on the mailing list.

Video

In the press

This presentation was also covered on LWN.