Isolated Testing for SWEs

Overview

This page explains how to convert a googletest executable into a fully isolated test. This is done by describing runtime dependencies for a test.

For information about the infrastructure itself and the roadmap, see Isolated Testing Infrastructure.

What is the Isolate Project?

Describes the exact list of files needed to run a executable. The isolate project is all the file formats, tools and server code to make this work fast and efficiently. Every test executable list the data files it depends on, which allows the testers bots to only download the files they need instead of the whole Chromium source tree.

What is the Swarming Project?

A task distribution mechanism. The swarming project is all the tools, server and bot code to run a step (like a unit test) on a remote slave and get results back. It has native google-test sharding capability.

What are the advantages?

By reducing the amount of data that we need to transfer to the tester machines, it becomes much easier to increase the number of tester bots that we have using Swarming (https://code.google.com/p/swarming/), which means that the Try Server and Continuous Integration masters' VMs will have shorter cycle times.

How can I run a test isolated locally?

You need to use ninja. Then follow this simple recipe to build browser_tests with ninja in Release:

ninja -C out/Release browser_tests_run
python tools/swarming_client/isolate.py run -s out/Release/browser_tests.isolated

See the Roadmap for what can or can't be done.

Adding a new test

  • For each gyp XXX test executable target, a new XXX_run target is added. For example, base_unittests_run is the target to run base_unittests.
    • src/base/base.gyp defines base_unittests_run.
    • src/base/base_unittests.isolate contains the runtime dependencies of the executable base_unittests.
    • The reason of listing the dependencies in a separate file is that it makes it possible to generate it automatically. See the design document for more information.
  • All the required data files, executables started by the test executable* or source files (!) required by the test to succeed are listed in the .isolate file.
    • For example, browser_tests.exe needs chrome.exe, which needs resources.pak, etc.
    • See src/chrome/browser_tests.isolate for a more complicated example.
    • The command entry is the only one that needs to be written manually.
Expectations of the tests:
  • The test should not do directory walking or otherwise try to guess what should be tested.
  • The test should not edit any input file.
  • The test must eventually not write at all in the current directory. It must only use the temporary directory for temporary files.

Tests we want to see isolated

  1. ash_unittests
  2. content_browsertests
  3. media_unittests
  4. nacl_integration
  5. sync_integration_tests (already isolated but not properly enabled)
  6. telemetry_unittests
  7. ui_unittests

TL;DR

To create a new isolated test, do the following section in order:
  1. Create a "foo_unittests_run GYP target as explained below"
  2. Write a "minimal .isolate file"
  3. Run "fix_test_cases.py"

foo_unittests_run GYP target

foo_unittests_run target is added along side the foo_unittests target in the relevant GYP file. For example, base_unittests_run in base.gyp:
{
  'conditions': [
    ['test_isolation_mode != "noop"', {
      'targets': [
        {
          'target_name': 'base_unittests_run',
          'type': 'none',
          'dependencies': [
            'base_unittests',
          ],
          'includes': [
            '../build/isolate.gypi',
            'base_unittests.isolate',
          ],
          'sources': [
            'base_unittests.isolate',
          ],
        },
      ],
    }],
  ],
}

The target has only one .isolate source and includes src/build/isolate.gypi, which calls tools/swarming_client/isolate.py. The action executed depends on the GYP_DEFINES variable test_isolate_mode . The target imports the .isolate file, which is a dialect of a .gypi file and use the isolate_dependency_XXX variable defined in it to list the build-time dependencies of the runtime dependencies. Reread the last sentence; this means changing a runtime dependency will "re-build" foo_unittests_run. See build/isolate.gypi for the gory details.

.isolate file format

See https://code.google.com/p/swarming/wiki/IsolateDesign#Description for the guts.

The .isolate format is a strict subset of the GYP language and will rejects any file containing variables or data structure it doesn't understand. The .isolate files can be generated with isolate.py itself or with fix_test_cases.py.

Minimal .isolate file


In general, it is preferable to:
  1. When running tests linux, run them under XVFB. Prepend the command list with ['../testing/xvfb.py', '<(PRODUCT_DIR)'] to achieve this.
  2. On other platforms, use the test_env.py wrapper script to setup the environment. Prefix the command list with ['../testing/test_env.py'].
  3. Optionally calling tools/swarming_client/googletest/run_test_cases.py to speed up the unit test execution. It runs tests in parallel, see --help for information.
See src/base/base_unittests.isolate as a good example.

Simplifying .isolate files

By default a directory is only included if all the files in it are needed. If most of the directory is used and you feel that it makes sense to make the full directory available to simplify the .isolate file,  feel free to replace the individual listings with a single directory listing.

isolate.py


isolate.py is run by the foo_unittests_run gyp target and is the front end to do all the commands. It reads a single .isolate file.
  • The logic is implemented in tools/swarming_client/isolate.py. Its action depends on the subcommand.
  • All the XXX_run targets run this script and pass the argument <(test_isolation_mode).
    • test_isolation_mode is a GYP_DEFINES variable that can be overridden, the default is set in build/common.gypi and can vary between noop and check, based on the environment.
    • The GYP_DEFINES variable test_isolation_outdir specifies the location where to archive the data. 
    • Builders have a different value than the developers, so they archive the dependencies on the Isolate Server.
  • Then Swarming picks up the archived files and runs it across a fleet of slaves.
See src/tools/swarming_client/isolate.py --help for the up-to-date behavior description.

Dogfood

Eligibility

Right now, only users @google.com can use the infrastructure. For others, we'll try to make it available to Chromium committers eventually. Note that the whole Swarming infrastructure is open source so if any other company would help to recreate the same infrastructure internally, send us a note at swarming-eng@googlegroups.com

Get the Swarming client code

If you have a chromium checkout, you already have everything you need in src/tools/swarming_client/.

Login on the services

By login first, you have access tokens so that the following commands do not have to constantly prompt for your identity. 

python tools/swarming_client/auth.py login --service https://isolateserver.appspot.com
python tools/swarming_client/auth.py login --service https://chromium-swarm.appspot.com 

If you are running through a text only session on a remote machine, append argument --auth-no-local-webserver

Run the example

This is a good sanity check to ensure that everything works:

python tools/swarming_client/example/3_swarming_simple.py -I https://isolateserver.appspot.com \
    -S https://chromium-swarm.appspot.com

If this doesn't work, see the FAQ before continuing or ping us, we're friendly.

Generate a .isolated file

For example, let's take base_unittests and create out/Release/base_unittests.isolated:

ninja -C out/Release base_unittests_run

Still scratching your head about what base_unittests_run is? Read the top of this page. It compiles base_unittests.isolate into base_unittests.isolated so it can be archived properly.

Trigger the task

Now you built something, time to archives it to the isolate server and request Swarming to run it on your behalf.

python tools/swarming_client/swarming.py run -I https://isolateserver.appspot.com \
    -S https://chromium-swarm.appspot.com --verbose \
    -d os Windows-5.1 out/Release/base_unittests.isolated

First thing it does is to archive the binary. Depending on your connection speed and the size of the executable, it may take up to a minute. Then it triggers the task and wait for results. OS currently available:
That's it. Feel free to contact the team at isolate@chromium.org for any chromium open source specific questions.

Additional Notes

FAQ

First see the non-Chromium specific FAQ at https://code.google.com/p/swarming/wiki/IsolateUserGuide#FAQ.

I run a task on Swarming and it hangs there

It is possible that all the slaves are currently taken. Improving feedback while waiting is planned as issue 83.

What about <insert build flavor here>?

The .isolate format can support it, it is currently a matter of writing the .isolate files to support these conditions, etcSee the Roadmap.

It seems tedious to list each test data file individually, can I just list src/ ?

In theory yes, in practice please don't and keep the list to the strict minimum. The goal is not to run the tests more slowly and having the slaves download 20 gb of data. Reasons includes:
  1. Isolate Server is optimized for < 10000 files scenario. There's a 7ms/file cost per cache hit. So for example, layout tests are currently out of the use case since there's > 80000 files.
  2. It's always possible to go coarser but it's much harder to get back stricter.

Where can I find the .isolated file?

The .isolated files are generated when you build the isolation specific version of a target, e.g. out/Debug or out/Release. The isolation target name is just the normal target name with a _run added to the end.

Why can't I compile with test_isolation_mode != noop?

It's possible your use case (like Chromium for ChromiumOS, android, etc) is not supported yet. See the Roadmap.

Where should I file bugs?

Swarming specific bugs can be filed on https://code.google.com/p/swarming/issues/list. Chromium specific bugs at http://crbug.com.
Comments