To run only some of the tests, specify their directories or filenames as arguments to run_webkit_tests.py relative to the layout test directory (src/third_party/WebKit/LayoutTests). For example, to run the fast form tests, use:

Or you could use:

as a shorthand.

Example: To run the layout tests with a debug build of content_shell, but only test the SVG tests and run pixel tests, you would run:

As a final quick-but-less-robust alternative, you can also just use the content_shell executable to run specific tests by using (for Windows):

as in:

but this requires a manual diff against expected results, because the shell doesn't do it for you.

To see a complete list of arguments supported, run: run-webkit-tests --help

Linux Note: We try to match the Windows render tree output exactly by matching font metrics and widget metrics. If there's a difference in the render tree output, we should see if we can avoid rebaselining by improving our font metrics. For additional information on Linux Layout Tests, please see docs/layout_tests_linux.md.

Mac Note: While the tests are running, a bunch of Appearance settings are overridden for you so the right type of scroll bars, colors, etc. are used. Your main display's "Color Profile" is also changed to make sure color correction by ColorSync matches what is expected in the pixel tests. The change is noticeable, how much depends on the normal level of correction for your display. The tests do their best to restore your setting when done, but if you're left in the wrong state, you can manually reset it by going to System Preferences → Displays → Color and selecting the "right" value.

Test Harness Options

This script has a lot of command line flags. You can pass --help to the script to see a full list of options. A few of the most useful options are below:

--debug	Run the debug build of the test shell (default is release). Equivalent to -t Debug.
--nocheck-sys-deps	Don't check system dependencies; this allows faster iteration.
--verbose	Produce more verbose output, including a list of tests that pass.
--no-pixel-tests	Disable the pixel-to-pixel PNG comparisons and image checksums for tests that don't call layoutTestController.dumpAsText()
--reset-results	Write all generated results directly into the given directory, overwriting what's there
--new-baseline	Write all generated results into the most specific platform directory, overwriting what's there. Equivalent to --reset-results --add-platform-expectations
--renderer-startup-dialog	Bring up a modal dialog before running the test, Tseful for attaching a debugger.
--fully-parallel	Run tests in parallel using as many child processes as the system has cores.

Success and Failure

A test succeeds when its output matches the pre-defined expected results. If any tests fail, the test script will place the actual generated results, along with a diff of the actual and expected results, into src/out/{Debug,Release}/layout_test_results/, and by default launch a browser with a summary and link to the results/diffs.

A test that runs but produces the wrong output is marked as "failed", one that causes the test shell to crash is marked as "crashed", and one that takes longer than a certain amount of time to complete is aborted and marked as "timed out". A row of dots in the script's output indicates one or more tests that passed.

Test Expectations

Testing Runtime Flags

Tracking Test Failures

All bugs, associated with layout test failures must have the LayoutTests label. Depending on how much you know about the bug, assign the status accordingly:

• Unconfirmed -- you aren't sure if this is a simple rebaseline, possible duplicate of an existing bug, or a real failure
• Available -- you know the root cause of the issue.
• Assigned or Started -- you will fix this issue.

When creating a new layout test bug, please assign the following labels to it -- having proper label hygiene is good for everyone:

• Type-Bug
• Pri-2 (Pri-1 if it's a crash)
• Area-WebKit
• OS-All (or whichever OS the failure is on)
• LayoutTests
• Mstone-9 (or current milestone)
• Tests-Flaky (if the test is flaky)

Writing Layout Tests

Pixel Tests

Tests that use a HTTP Server

By default, tests are loaded as if via file: URLs. Some web platform features require tests served via HTTP or HTTPS, for example relative paths (src=/foo) or features restricted to secure protocols.

HTTP tests are those tests that are under LayoutTests/http/tests (or virtual variants). Use a locally running HTTP server (Apache) to run. Tests are served off of ports 8000, 8080 for HTTP and 8443 for HTTPS. If you run the tests using run-webkit-tests, the server will be started automatically.To run the server manually to reproduce or debug a failure:

A layout test does not actually draw frames of output until the test exits. If it is required to generate a painted frame, then use window.testRunner.displayAsyncThen, which will run the machinery to put up a frame, then call the passed callback. There is also a library at fast/repaint/resources/text-based-repaint.js to help with writing paint invalidation and repaint tests.

Layout test support for testRunner

Some layout tests rely on the testRunner object to expose configuration for mocking the platform. This is provided in content_shell, here's a UML diagram of testRunner bindings configuring platform implementation:

Writing reliable layout tests

Debugging Layout Tests

1. Take a look at the result. Sometimes tests just need to be rebaselined (see below) to account for changes introduced in your patch.
• Load the test into a trunk Chrome or content_shell build and look at its result. (For tests in the http/ directory, start the http server first. See above. Navigate to http://localhost:8000/ and proceed from there.) The best tests describe what they're looking for, but not all do, and sometimes things they're not explicitly testing are still broken. Compare it to Safari, Firefox, and IE if necessary to see if it's correct. If you're still not sure, find the person who knows the most about it and ask.
• Some tests only work properly in content_shell, not Chrome, because they rely on extra APIs exposed there.
• Some tests only work properly when they're run in the layout-test framework, not when they're loaded into content_shell directly. The test should mention that in its visible text, but not all do. So try that too. See "Running the tests", above.
2. If you think the test is correct, confirm your suspicion by looking at the diffs between the expected result and the actual one.
• Make sure that the diffs reported aren't important. Small differences in spacing or box sizes are often unimportant, especially around fonts and form controls. Differences in wording of JS error messages are also usually acceptable.
• $ ./run_webkit_tests.py path/to/your/test.html --full-results-html will produce a page including links to the expected result, actual result, and diff.
• Add the --sources option to run_webkit_tests.py to see exactly which expected result it's comparing to (a file next to the test, something in platform/mac/, something in platform/chromium-win/, etc.)
• If you're still sure it's correct, rebaseline the test (see below). Otherwise...
3. If you're lucky, your test is one that runs properly when you navigate to it in content_shell normally. In that case, build the Debug content_shell project, fire it up in your favorite debugger, and load the test file either from a file:// URL.
• You'll probably be starting and stopping the content_shell a lot. In VS, to save navigating to the test every time, you can set the URL to your test (file: or http:) as the command argument in the Debugging section of the content_shell project Properties.
• If your test contains a JS call, DOM manipulation, or other distinctive piece of code that you think is failing, search for that in the Chrome solution. That's a good place to put a starting breakpoint to start tracking down the issue.
• Otherwise, you're running in a standard message loop just like in Chrome. If you have no other information, set a breakpoint on page load.
4. If your test only works in full layout-test mode, or if you find it simpler to debug without all the overhead of an interactive session, start the content_shell with the command-line flag --run-layout-test, followed by the URL (file: or http:) to your test. More information about running layout tests in content_shell can be found here.
• In VS, you can do this in the Debugging section of the content_shell project Properties.
• Now you're running with exactly the same API, theme, and other setup that the layout tests use.
• Again, if your test contains a JS call, DOM manipulation, or other distinctive piece of code that you think is failing, search for that in the Chrome solution. That's a good place to put a starting breakpoint to start tracking down the issue.
• If you can't find any better place to set a breakpoint, start at the TestShell::RunFileTest() call in content_shell_main.cc, or at shell->LoadURL() within RunFileTest() in content_shell_win.cc.
5. Debug as usual. Once you've gotten this far, the failing layout test is just a (hopefully) reduced test case that exposes a problem.

To automatically re-baseline tests across all Chromium platforms, using the buildbot results, see the Rebaselining keywords in TestExpectations and Rebaselining Tool. Alternatively, to manually run and test and rebaseline it on your workstation, read on.

By default, text-only tests (ones that call layoutTestController.dumpAsText()) produce only text results. Other tests produce both new text results and new image results (the image baseline comprises two files, -expected.png and -expected.checksum). So you'll need either one or three -expected.* files in your new baseline, depending on whether you have a text-only test or not. If you enable --no-pixel-tests, only new text results will be produced, even for tests that do image comparisons. 

See bugs with the component Blink>Infra for issues related to Blink tools, include the layout test runner.