For Developers‎ > ‎

Performance Try Bots

Summary

A python script for automating the process of building, and testing commits against tip of tree and comparing performance. The performance try bots have been built on top of the bisect bot architecture. The bot works by syncing to the specified revision, applying your patch, building chrome, and running the performance test. It then reverts your patch, builds again, and runs the performance test a second time. Results are output on the waterfall, as well as uploaded to cloud storage.

Supported Platforms

 Platform  Builder Name
 Linux linux_perf_bisect
 Windows win_perf_bisect
 Mac mac_perf_bisect
 Android GN android_gn_perf_bisect
 Android Nexus 4 android_nexus4_perf_bisect
 Android Nexus 10
android_nexus10_perf_bisect
 ChromeOS No Plans ATM

Run on Trybot - Waterfall

  • Create new git branch or check out existing branch
  • Edit tools/run-perf-test.cfg (instructions in file) or third_party/WebKit/Tools/run-perf-test.cfg
    • Take care to strip any src/ directories from the head of relative pathnames
    • On desktop, only --browser=release is supported, on android --browser=android-chromium-testshell.
    • You can search through the stdio from performance tests for "run_measurement" to help figure out the command to use
    • Also note that cycle times for the bots can be vastly different. If it's possible to reproduce the regression on multiple platforms, keep in mind that linux bisects are usually the fastest, followed by windows, then mac, and finally android.
      • You can expect to wait about 20-30 mins for a linux test, and several times longer for an android test (this obviously depends greatly on the performance test you're running, how many iterations you used, etc..).
      • A good rule of thumb is that linux is the fastest, mac and windows will take 2-3x longer than linux, and android can take 3-4x longer

To submit to a bot:

    git try --user=<user> -b <bot>

You can also submit jobs with blink patches, but since the bot works by using the config file to pass the parameters of the job you'll need to modify the run-perf-test.cfg file in blink. The file is located in third_party/WebKit/Tools/run-perf-test.cfg.
  • The flow is slightly different for blink. The patch must be uploaded first before a try job can be issued.
    git cl upload --bypass-hooks

    --bypass-hooks is necessary to upload the changes you committed locally to run-perf-test.cfg. But changes to run-perf-test.cfg should never be committed to the project repository.

    git cl try --user=<user> -b <bot>


Comments