Blink‎ > ‎

Importing Tests from the W3C (and contributing back)


The W3C provides a large number of conformance tests for different aspects of the Web Platform. Currently the tests are hosted on GitHub, two main repositories, one for the CSS Working Group, and one for pretty much everything else (aka web-platform-tests).

Blink has mirrors (web-platform-tests, csswg-test) of the GitHub repos, and periodically imports a subset of the tests so that they are run as part of the regular LayoutTests testing process (the tests are pulled in via an entry to DEPS to LayoutTests/w3c).

This page documents the current process for modifying the subset that is imported and run, and how we can submit new tests back to the W3C.

For some historical discussion of how things work (written just after the Blink fork from WebKit), see Blink, Testing, and the W3C .

Importing new W3C Tests into Blink

Updating the set of tests run by Blink requires commit access to the Blink mirrors of the W3C repos as well as commit access to both Chromium and Blink (talk to the infrastructure team to get access; you must be a member of the w3c-committers group on gerrit.chromium.org). At the moment the best way to get the set updated is to file a bug w/ the list of tests you'd like updated, and assign it to dpranke@chromium.org. The rest of this section documents the steps that he (or anyone else doing the import) would need to do :).
  1. Update LayoutTests/W3CImportExpectations if you want to change which tests are *not* imported and commit locally.

  2. Run Tools/Scripts/update-w3c-deps --allow-local-blink-commits to import the latest tests from our mirrors of the W3C repos.
      
  3. Run the tests locally against tip-of-tree Blink and/or Chromium's currently pinned version to see if there are new failures that will need to be suppressed. 

  4. If you are satisfied that the latest changes should be rolled in, push your commit back to the mirror on googlesource.com (this is the step that requires the unusual level of access):

        % git push origin/blink

  5. Depending on how the tests run, you can now roll the new revisions into Chromium and Blink. This may take 1, 3, or 5 patches to fully complete:

  6. Ideally, the new w3c revisions introduce no new failing tests and doesn't remove any old tests. In this case, you only need a single CL to update the revisions in the DEPS file.

  7. If the new w3c revisions introduce some new failures, but do not delete any existing failing tests, you need three CLs, and should follow the following process:
    1. Land CL #1 in Chromium with the new DEPS entries and update webkit/tools/layout_tests/test_expectations_w3c.txt with the new expectations.
    2. Merge the new test_expectations entries upstream into Blink's LayoutTests/TestExpectations in CL #2.
    3. Wait for CL #2 to be rolled into Blink.
    4. Land CL #3 in Chromium to remove the entries from webkit/tools/layout_tests/test_expectations_w3c.txt (which should then be empty).

  8. If the new w3c revisions remove some tests that were previously failing in Blink, life gets more complicated because LayoutTests/TestExpectations will contain entries for files that no longer exist (and you'll get warnings, and lint-test-expectations will fail), so we need to deal with them first:

    1. Land CL #1 in Chromium that copies the expectations for the currently-failing-but-will-be-removed tests from LayoutTests/TestExpectations to test_expectations_w3c.txt.
    2. Land CL #2 in Blink that removes the expectations from LayoutTests/TestExpectations
    3. Wait for CL #2 to roll into Chromium.
    4. Land CL #3 in Chromium with the new DEPS entries and an updated test_expectations_w3c.txt that removes the old expectations for the now-deleted tests (and adds any new expectations as needed) 
    5. Merge the new test_expectations upstream into Blink in CL #4
    6. Wait for CL #4 to roll into Chromium.
    7. Land CL #5 in Chromium to remove the entries from test_expectations_w3c.txt 
Obviously, the above is more complicated than we'd like it to be; if we are able to merge the Chromium and Blink repos (which we are currently hoping will happen some time in 2014), then all of these steps collapse to a single CL, and we implement an auto-roll-bot to do this all automatically.

In addition to the above, there are some other bugs remaining that limit which tests we can currently import and run, and require us to have to rewrite some of the tests (as noted above) more than we'd like to have to do. We are currently tracking such bugs in the Chromium issue tracker using the label Cr-Blink-Tools-Test-W3C.

The end goals are 1) to be able to run the W3C tests unmodified locally just as easily as we can run the Blink tests, 2) ensure that we are tracking tip-of-tree in the W3C repos as closely as possible, and 3) run as many of the W3C tests as possible (eliminating old redundant copies of the tests in Blink).

Contributing Blink tests back to the W3C

As described above, the W3C has two main repositories of tests, one for the CSS working group, and one for everything else. The processes for getting new tests committed to each repo are documented here and here.

This section describes how to take tests currently checked into LayoutTests/ and get them added to the W3C repos. If you want to submit tests that are not part of LayoutTests/, you're on your own (but can probably follow the existing W3C documentation).

We currently have Blink specific forks of the W3C repos hosted on GitHub (web-platform-tests and csswg-test), in order to save individual Blink contributors from having to create their own individual forks. For access to the Blink repos, talk to the infrastructure team.

Once you have access, you should push a change to the Blink repo, and then create a Pull request to the W3C repo. Then get one of the W3C committers (TBD: who??) to grant the pull request and land the change. 

If the new tests are currently being run in Blink (i.e., they are checked into LayoutTests somewhere), then in theory that should count as "reviewed" and getting the pull request approved should be straightforward. 

Once the PR has landed in the W3C's repo, you can then refer to the "Importing new W3C Tests into Blink" section above to get the tests pulled back into Blink, at which point you can then finally delete the copies checked in under LayoutTests/.

Comments