Service Worker Testing

Service Worker has multiple kinds of tests: Content Browser Tests, WebKit Unit Tests, Telemetry Performance Tests, and Blink Layout Tests.

Performance Tests

Issue 372759 is tracking the development of Service Worker performance tests.

How to run the performance tests:
  1. Build and install the content_shell_apk target per the Android build instructions.
  2. Build the forwarder2 target.
  3. tools/perf/run_benchmark --browser=android-content-shell [--device=xxxx] service_worker.service_worker
How to update the performance tests:
  1. Check out the performance test sources. Do development, pull request, etc.
  2. Run a local server in that directory, for example: twistd -n web --path . --port 8091 (you must use that port number, it appears in the test Python scripts.)
  3. Build Linux Release Chromium
  4. tools/perf/record_wpr --browser=release --extra-browser-args='--enable-experimental-web-platform-features' service_worker_page_set
    Note: If you change the output directory of build (like 'out_desktop'), you can use '--browser=exact --browser-executable=path/to/your/chrome' instead of '--browser=release'.
  5. Briefly sanity check the tools/perf/page_sets/data/service_worker_nnn.wpr file to see that it doesn't contain requests that should have been handled by a Service Worker (look for GET and browse through the URLs.)
  6. Add the new SHA1 hash file (use git status to find it) and upload for review, commit as usual. Mention the path and commit hash from step 1.

Web Tests

Within Blink Web Tests, Service Worker has two kinds of tests:
  1. Ones that test implementation specific aspects such as not crashing or leaking memory. These often rely on Blink-specific testing harnesses like testRunner or internals. Put these tests in a subdirectory called chromium, for example web_tests/http/tests/serviceworker/chromium. The purpose of these tests is to ensure the stability and security of our implementation.

  2. Ones that test implementation-neutral aspects like whether features conform to the specification. We call these "W3C-style" tests because they use the W3C's test harness and don't have recourse to Blink-specific testing hooks. The purpose of these tests is to ensure the functionality of our implementation, and to characterize the behavior of our implementation when reconciling spec, implementation, and other implementations.

Web Tests Style

Historically, style was not rigorously enforced for Blink Web Tests. Service Worker has decided to implement tests in a consistent style. The style guide we have designed is more heavily influenced by the W3C's test harness than Blink's style guide because we plan to contribute most of our test suite to the W3C. Where this style guide is not explicit, follow the Google JavaScript style guide because it is at least comprehensive.

Use the W3C Test Harness Script Test Template: Minimal for HTML files:

<!DOCTYPE html>
<meta charset="utf-8">
<title>[Title of Test]</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
// Calls to test() or async_test()
</script>

Write variable identifiers and function names in lowercase separated with underscores. Use PascalCase for constructor functions.

Use single quotes for string literals. Reserve double quotes for attribute values in markup. This makes it easy to be consistent in most situations where JavaScript appears in markup and vice-versa.

Always quote attributes. Use double quotes. Exception: Use defaults for brevity, for example write <button disabled> and not <button disabled="disabled">.

Do:

var html_button = '<button onclick="close();">Bye</button>';

Don't:

var htmlButton = "<button onclick=close()>Bye</button>";

Wrapping and Indentation

Indent two spaces.

Do:

for (var w = window; w != window.parent; w = window.parent) {
  Array.prototype.forEach.call(w.frames, function(frame) {
      console.log(
        w.title,
        frame.contentWindow.title);
    });
}

When wrapping function arguments, indent two spaces. Object and function literals are not exempt from this (example 1).

Exceptions. These indent the arguments four spaces:
  • When wrapping actual parameters and there's a chained property access of the return value (example 2), indent the arguments four spaces so that the arguments don't line up with the chained property access.
  • When wrapping formal parameters of function declarations or function literals (example 3) indent the arguments four spaces so that the arguments don't line up with the function body.

Do:

// Example 1

items.forEach(function(item) {
    assert_true(is_expensive(item));
  });


// Example 2

service_worker_unregister_and_register(
    t,
    'resources/a-very-long-path-to-a-worker.js'
    'resources/a-very-long-path-to-a-scope')
  .then(function(registration) {
      // code
    });


// Example 3

function dispatch_keys_for_cache(
    cache_id,
    callbacks,
    request,
    query_params) {
  // code
}

Don't:

items.forEach(function(item) {
  assert_true(is_expensive(item));
});

Note: This is different to the Google JavaScript style guide. It is necessary to have sensible indentation with Promises, which mix functions and long function- or Promise-returning arguments.

Wrap at 80 columns. Exception: Markup with long attribute values.

Break after opening curly braces. Exception: Anonymous functions that are small enough to fit within a line do not need to break after {.

Do:

clients.forEach(function(client) {
    client.postMessage('Happy New Year!);
  });

Don't:

function embiggen(element)
{
  element.baggage = new ArrayBuffer(1000000);
}

When wrapping chained property accesses, break before the period.

Do:

navigator.serviceWorker.register('worker.js')
  .then(function(worker) {
      // party time
    })
  .catch(poop);

Don't:

navigator.serviceWorker.register('worker.js').then(
  function(worker) {
    // party time
  }).
  catch(poop);

Promises

When using Promises, prefer a separate .catch to the two-argument form of .then. Exception: You need to test an expected exception, and using .catch separately would be tedious.

Do:

navigator.serviceWorker.register('worker.js')
  .then(function(worker) {
      // test code
    })
  .then(function(worker) {
      // moar test code
    })
  .catch(unexpected_exception(t));

navigator.serviceWorker.register('http://insecure.example.com/worker.js')
  .then(
    unreachable_code(
      t,
      'registering a script with an insecure origin should throw a ' +
      'SecurityError'),
    expected_exception(t, SecurityError))
  .then(function() {
      // moar test code
    });

Don't:

navigator.serviceWorker.register('worker.js')
  .then(function(worker) {
      // test code
    })
  .then(function() {
      // moar test code
    },
    unexpected_exception(t));

Test Harness specific issues

When writing an async test, use the form that takes a function as the first argument. Capture the test object in the variable 't'. Implement the test in the scope of the anonymous function. This keeps variables touched by async steps of a test isolated which makes the tests more reliable and makes it easier to move tests between files.

Do:

async_test(function(t) {
    // test code
  }, '"ready" resolves to the active Service Worker of a controlled document');

Don't:

var t = async_test(
  '"ready" resolves to the active Service Worker of a controlled document');
// test code

Assertions should have messages. Messages should describe the desired behavior and use the word 'should'; this aids clarity.

Do:

assert_equals(
  navigator.serviceWorker.active,
  null,
  'an uncontrolled document should not have an active Service Worker');

Don't:

assert_equals(
  navigator.serviceWorker.active,
  null,
  'non-null active Service Worker');

To ensure test failures from unhandled exceptions pinpoint the cause of the failure precisely, asynchronous callbacks, such as event handlers and setTimeout callbacks, must use the t.step_func() wrapper. This ensures that exceptions thrown during the execution of the callback cause the test to fail with a useful stack trace, rather than simply timing out.

Exceptions:
  • Using the wrapper is not necessary within Promise chains when there is a terminal catch handler, since exceptions thrown within a Promise callback are converted to rejections of the Promise chain. 
  • The Promise constructor's callback function converts exceptions to rejections; if there is a terminal catch handler the wrapper is not needed. 
  • Other synchronous callbacks, like the comparator function for sort, do not need their own test_step, since the exception is propagated out.
Do:

async_test(function(t) {
    ...
    xhr.onreadystatechange = t.step_func(function() {
        ...
        // If this throws, step_func will fail the test.
        var result = JSON.parse(xhr.responseText);
        assert_equals(result, expected, ...);
        t.done();
      });
  }, 'ServiceWorker onfetch should handle async XHRs');


Don't:

async_test(function(t) {
    ...
    xhr.onreadystatechange = function() {
        ...
        // If this throws, the test will time out.
        var result = JSON.parse(xhr.responseText);
        assert_equals(...);
        t.done();
      };
  }, 'ServiceWorker onfetch should handle async XHRs');

Service Worker specific issues

Register each test with a unique Service Worker scope. (Query strings are convenient for this.) This improves test isolation and stops the test itself from being controlled by a Service Worker. If your test needs to interact with the Service Worker as a client (many do), open an iframe with a URL controlled by the Service Worker.

Prefix Service Worker scopes with "scope/" when appropriate. This helps prevent unintentionally registering a Service Worker that controls resources in the test directory.

Prefix each test by unregistering the Service Worker. If a previous test run failed or was interrupted, it may have left a Service Worker registration in place. Unregistering the existing Service Worker first, if any, improves the reliability of the test.

Clean up resources when the test is done: Unregister the test's Service Worker at the end of the test. Remove any iframes.

Don't reuse a Service Worker registration between tests in the same file. This makes the tests more isolated, making it easier to split and combine tests from different files.
Comments