For Developers‎ > ‎

Performance Try Bots

Summary

Telemetry now supports sending a job directly to the performance try bots.

There are python scripts for automating the process of building, and testing commits against tip of tree and comparing performance. The performance try bots have been built on top of the bisect bot architecture. The bot works by syncing to the specified revision, applying your patch, building Chrome, and running the performance test. It then reverts your patch, builds again, and runs the performance test a second time. Results are output on the waterfall, as well as uploaded to cloud storage.

For information about using the performance try bots to perform a bisect, see Bisecting Performance Regressions.

The performance try server is tryserver.chromium.perf.

Supported Platforms

You can list the supported trybots by passing the --browser=list argument to tools/perf/run_benchmark. Here is the current output:

$ tools/perf/run_benchmark --browser=list
Available browsers:
  ...
  trybot-android-gn
  trybot-android-nexus10
  trybot-android-nexus4
  trybot-android-nexus7
  trybot-linux
  trybot-mac
  trybot-mac-10-9
  trybot-win
  trybot-win-8
  trybot-win-xp



Starting a perf try job

From your src/ directory, run tools/perf/run_benchmark, passing in --browser=trybot-you-want. This will take all your command-line arguments, and pass them into the command that runs on the perf trybot. Examples:

# Run sunspider on mac_10_9_perf_bisect
$ tools/perf/run_benchmark --browser=trybot-mac-10-9 sunspider

# Run tab_switching.top_25, repeating 10 times, on win_xp_perf_bisect
tools/perf/run_benchmark tab_switching.typical_25  --pageset-repeat=10 --browser=trybot-win-xp



Submitting jobs with blink patches
The same command works with blink patches. However, you cannot currently submit a tryjob with changes in BOTH the chromium and blink repositories--only the chromium changes will be tried.


Tips about test run time

  • Cycle times for the bots can be vastly different, and cycle times for different tests is also different.
  • If it's possible to reproduce the regression on multiple platforms, keep in mind that linux bisects are usually the fastest, followed by windows, then mac, and finally android.
  • You can expect to wait about 20-30 mins for a linux test, mac and windows may take 2-3x longer than linux, and android can take 3-4x longer.
  • Check the number of pending jobs for the builder by clicking on it from the tryserver.chromium.perf master page. Less pending jobs means yours will run sooner.

Viewing output

You'll get an email when the results come in. In the buildbot output, you'll see a link to HTML results:


When you click the link, you'll get a page with results.

In the "Test" column, you can see all the metrics that were measured. When you click on one, you'll see the values for each run with the patch on the left ("Patch"), and without on the right ("ToT").



Comments