Running Chrome Endure tests

TODO: add documentation for basic running of a Chrome Endure test.

Running tests with Web Page Replay

Motivation

Chrome Endure supports recording/replay of webapp interactions through the use of Web Page Replay (WPR).

Running tests with a live site has the following problems. First, the tests are very often flaky. For example, one of the existing Gmail endurance tests previously hit the Gmail server too often and the Google account got temporarily locked down, which leads to a test failure. Also the test results are affected by environmental factors such as an unstable network. Secondly, a live website may change over time without notification. It is hard to diagnose whether a regression is caused by changes in the website or in Chromium itself. To reduce test flakiness and make test results comparable, we integrated Chrome Endure with WPR.

Web Page Replay

WPR is an open source project. With WPR, we can record and replay network requests/responses. It works as an HTTP/HTTPS proxy on the client under test or on a remote machine. You can find more information about WPR at their homepage: http://code.google.com/p/web-page-replay/

Run tests with Web Page Replay

Generally, to run a test against a local WPR server, we need to first make an archive by running the test with WPR in "record" mode. Then, we can run the test again with WPR in "replay" mode using the archive file.  We can reuse the archive file we've made and repeat the test. 

Make a Recording

To make a recording, you will need to run the test with an additional flag, WPR_RECORD. This tells the test to bring up a WPR server in "record" mode in the setup phase of a test.  You should also specify the destination directory into which to store the created WPR archive file using environment variable WPR_ARCHIVE_PATH.

WPR_RECORD=1 WPR_ARCHIVE_PATH=/home/user/wpr_archives \
  TEST_LENGTH=60 PERF_STATS_INTERVAL=20 \
  LOCAL_PERF_DIR=/home/user/www/chrome_graphs python perf_endure.py \
  perf_endure.ChromeEndureGmailTest

After the test finishes, an archive file /home/user/wpr_archives/<TEST_CLASS_NAME>.wpr will be created.
In this case,  /home/user/wpr_archives/ChromeEndureGmailTest.wpr will be created.

Notes:
  1. Naming of the archive file: Note that each archive file is named by the test class name. If you don't run all the tests in one test class(for example perf_endure.ChromeEndureGmailTest) at a time, but run them separately as follows: WPR_RECORD=1 TEST_LENGTH=60 PERF_STATS_INTERVAL=20 \
      LOCAL_PERF_DIR=/home/fdeng/www/chrome_graphs python perf_endure.py \
      perf_endure.ChromeEndureGmailTest.testGmailComposeDiscard
    WPR_RECORD=1 TEST_LENGTH=60 PERF_STATS_INTERVAL=20 \
      LOCAL_PERF_DIR=/home/fdeng/www/chrome_graphs python perf_endure.py \
      perf_endure.ChromeEndureGmailTest.testGmailAlternateThreadlistConversation
    then new content will be appended to the same archive file (which is 
    ChromeEndureGmailTest.wpr in this case).
  2. Remove the existing archive file if you want to make a fresh archive. By default, new content is appended to the existing archive file.
  3. To use a custom archive file, set environment variable WPR_ARCHIVE_PATH if you want to output the archive file to a custom place. For example:
    > WPR_ARCHIVE_PATH=/home/user/archive.wpr WPR_RECORD=1 \
      TEST_LENGTH=60 PERF_STATS_INTERVAL=20 \
      LOCAL_PERF_DIR=/home/user/www/chrome_graphs python perf_endure.py \
      perf_endure.ChromeEndureGmailTest.testGmailComposeDiscard
    After running the above command, an archive file /home/user/archive.wpr will be created.
  4. Length of recording: Usually, you can record a scenario for a short time (e.g 30 seconds) and replay it for a longer time (6 hours or longer). But whether a short recording will work highly depends on the webapp. Using a short recording may cause some features to not work correctly. But in some cases, this might be fine if you don't care about those features very much and want to focus on the performance of Chromium itself.  (TODO: Maybe add some explanation why a shorter recording may work for a longer replay.)

Run Tests in Replay Mode Using a Recording

Run the test with the following command. No extra flags are needed. The test will try to check whether an archive file for a test exists in the default location (WPR_ARCHIVE_PATH can be used if you want to use an alternative archive file). If the archive file exists, the test will automatically run with WPR. If not, it will run with the live site.
  > TEST_LENGTH=60 PERF_STATS_INTERVAL=20 \
    LOCAL_PERF_DIR=/home/user/www/chrome_graphs python perf_endure.py \
    perf_endure.ChromeEndureGmailTest

Notes:
  1. Forcing the test to run against a live site: To force the test run with a live site, set the environment variable  ENDURE_NO_WPR to any value. (Please note that you should not set WPR_RECORD and ENDURE_NO_WPR at the same time.)
    ENDURE_NO_WPR=1 TEST_LENGTH=60 PERF_STATS_INTERVAL=20 \
      LOCAL_PERF_DIR=/home/user/www/chrome_graphs python perf_endure.py \
      perf_endure.ChromeEndureGmailTest

Test result differences introduced by Web Page Reply

Different performance results can show up for test runs with WPR and test runs with a live site. There are several possible causes:
  1. Randomness introduced by the webapp that is not accounted for in WPR (e.g., doing something based on the current window size).
  2. Use a too-short recording (e.g., 2 minutes) to replay for a much longer time (e.g., 6 hours).  The functionality is enabled by "fuzzy request matching" in WPR that uses heuristics to replay server responses even though requests to the server may not exactly match was was specified during recording.
  3. Some webapp features may simply not work well with WPR (e.g., the Gmail prefetcher and chat plugin).
Comments