The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.

Isolated Testing

Contents

  1. 1 What's happening
  2. 2 Design document
  3. 3 How it works
  4. 4 TL;DR
  5. 5 foo_unittests_run GYP target
  6. 6 .isolate format
    1. 6.1 Minimal .isolate file
  7. 7 isolate.py
    1. 7.1 Background
    2. 7.2 Subcommands
      1. 7.2.1 check
      2. 7.2.2 hashtable
      3. 7.2.3 help
      4. 7.2.4 merge
      5. 7.2.5 read
      6. 7.2.6 remap
      7. 7.2.7 run
      8. 7.2.8 trace
  8. 8 isolate_test_cases.py
  9. 9 fix_test_cases.py
  10. 10 isolate_merge.py
  11. 11 run_isolated.py
  12. 12 FAQ
    1. 12.1 What about disable_nacl=1, component=shared_library, <insert build flavor here>?
    2. 12.2 Roadmap / What's the timeline?
      1. 12.2.1 Summary
    3. 12.3 It seems tedious to list each test data file individually, can I just list src/chrome/test/data/ ?
    4. 12.4 What's the difference between isolate_dependency_tracked and isolate_dependency_untracked?
    5. 12.5 I traced my test executable and some test cases still fail when isolated
    6. 12.6 Where can I find the .isolated file?
    7. 12.7 I have a feature request / This looks awesome, can I contribute?
    8. 12.8 Why not a faulty file system like FUSE?
    9. 12.9 Why can't I compile with test_isolation_mode!=noop?

What's happening

We are in the process of retro-fitting the ability of running tests outside a checkout. This page describes how to describe runtime dependencies for a test and how to generate the configuration.

Design document

It is on its own page. It's an highly recommended reading since it gives background about the rationale why is it done in the first place.

How it works

  • For each gyp XXX test executable target, a new XXX_run target is added. For example, base_unittests_run is the target to run base_unittests.
    • src/base/base.gyp defines base_unittests_run.
    • src/base/base_unittests.isolate contains the runtime dependencies of the executable base_unittests.
    • The reason of listing the dependencies in a separate file is that it makes it possible to generate it automatically. See the design document for more information.
  • All the required data files, executables started by the test executable* or source files (!) required by the test to succeed are listed in the .isolate file.
    • For example, browser_tests.exe needs chrome.exe, which needs resources.pak, etc.
    • See src/chrome/browser_tests.isolate for a more complicated example.
    • The command entry is the only one that needs to be written manually.
Expectations of the tests:
  • The test should not do directory walking or otherwise try to guess what should be tested.
  • The test should not edit any input file.
  • The test must eventually not write at all in the current directory. It must only use the temporary directory for temporary files.

TL;DR

To create a new isolated test, do the following section in order:

foo_unittests_run GYP target

A foo_unittests_run target is added along side the foo_unittests target in the relevant GYP file. For example, base_unittests_run in base.gyp:
{
  'conditions': [
    ['test_isolation_mode != "noop"', {
      'targets': [
        {
          'target_name': 'base_unittests_run',
          'type': 'none',
          'dependencies': [
            'base_unittests',
          ],
          'includes': [
            '../build/isolate.gypi',
            'base_unittests.isolate',
          ],
          'sources': [
            'base_unittests.isolate',
          ],
        },
      ],
    }],
  ],
}

The target has only one .isolate source and includes src/build/isolate.gypi, which calls tools/swarm_client/isolate.py. The action executed depends on the GYP_DEFINES variable test_isolate_mode . The target imports the .isolate file, which is a dialect of a .gypi file and use the isolate_dependency_XXX variable defined in it to list the build-time dependencies of the runtime dependencies. Reread the last sentence; this means changing a runtime dependency will "re-build" foo_unittests_run. See build/isolate.gypi for the gory details.

.isolate format

The .isolate format is a strict subset of the GYP language and will rejects any file containing variables or data structure it doesn't understand. The .isolate files can be generated with isolate.py itself, with isolate_test_cases.py or merged with isolate_merge.py.

See the description of the format in the design document.

Minimal .isolate file

The only item that needs to be written manually is the command variable. A isolate_dependency_tracked is required for the rule defined in isolate.gypi to work.

Here is an example of a minimal .isolate file, assuming the command to run is the same on all OSes, which is rarely the case:
{
  'variables': {
    'command': [
      '<(PRODUCT_DIR)/foo_unittests<(EXECUTABLE_SUFFIX)',
    ],
    'isolate_dependency_tracked': [
      # Make sure the file is mapped.
      '<(PRODUCT_DIR)/foo_unittests<(EXECUTABLE_SUFFIX)',
    ],
  },
}

The EXECUTABLE_SUFFIX gyp variable is automatically set to ".exe" on Windows and empty on the other OSes.

In general, it is preferable to:
  1. When running tests linux, run them under XVFB. Prepend the command list with ['../testing/xvfb.py', '<(PRODUCT_DIR)'] to achieve this.
  2. On other platforms, use the test_env.py wrapper script to setup the environment. Prefix the command list with ['../testing/test_env.py'].
  3. Optionally calling tools/swarm_client/run_test_cases.py to speed up the unit test execution. It runs tests in parallel, see --help for information.
See src/base/base_unittests.isolate as a good example and it is well explained in:

isolate.py

Background

isolate.py is run by the foo_unittests_run gyp target and is the front end to do all the commands. It reads a single .isolate file.
  • The logic is implemented in tools/swarm_client/isolate.py. Its action depends on the subcommand.
    • Instead of putting stale documentation here, please run tools/swarm_client/isolate.py --help to get the up to date help.
  • All the XXX_run targets run this script and pass the argument <(test_isolation_mode).
    • test_isolation_mode is a GYP_DEFINES variable that can be overridden, the default is set in build/common.gypi and can vary between noop and check, based on the setup.
    • The idea is that the builders have a different value than the developers, so they archive the dependencies in a hash table.
  • Arguments:
    • --isolate <.isolate> file containing the command to run and/or its dependencies.
    • --isolated <.isolated> file to write the metadata in json format.
    • --outdir <location>. where to store the dependencies. The option name is illnamed, since it'll usually be an HTTP server.
      • If this is a relative path, it will be based in the current working directory, which is the folder with the .isolate file for the targets.
    • --variable FOO bar which can be specified multiple times to resolve the GYP variables in the .isolate file.
  • Then, Swarm picks up the archived files and runs it across a fleet of slaves in a sharded way.
See src/tools/swarm_client/isolate.py --help for the up-to-date behavior description. The following is an overview.

Note: --isolate is not necessary when --isolated is specified and the .isolated file exists. This is because the .isolated.state file saved beside the .isolated file contains a pointer back to the original .isolate file and persists the variables.

Subcommands

check

Makes sure the dependencies listed in the target exist and dumps the list into the XXX_run.isolated file. It calculates the sha-1 of each dependency.

The XXX_run.isolated file in the output directory lists all the dependencies found that would be mapped.

hashtable

Archives the inputs (dependencies) into a directory (default is hashtable in the same directory as the .isolated file) or at the location specified with --outdir. --outdir can be specified with the GYP_DEFINES test_isolation_outdir. Each file name is the SHA-1 hash of its content. That is, it is a Content-Addressed Datastore. The reverse mapping can be resolved with the information found in XXX_run.isolated. This mode is meant to be run from buildbot builders running the "compile" step.

By setting the environment variable GYP_DEFINES="test_isolation_mode=hastable test_isolation_outdir=https://isolateserver.appspot.com/", the XXX_run targets behavior changes to archive the test executable and its dependencies as part of the build process.
  1. Calculates the SHA-1 of every files in the dependency list.
  2. Depending on --outdir/test_isolation_outdir, it does one of:
    1. Uploads the files to the Isolate AppEngine datastore.
    2. Hardlinks each of the files into an output directory, with the name of the hardlink being its SHA-1.
  3. Writes XXX_run.isolated with a json dictionary containing the filepath->sha1 relationship.

help

Prints information about a specific subcommand.
python tools/swarm_client/isolate.py help trace

merge

Reads and merges the data from a trace generated with the subcommand trace back into the original .isolate. See read.

read

Reads the trace file generated with command trace and outputs it to stdout.

remap

Maps all the inputs into a temporary directory or the directory you specify with --outdir.

By setting the environment variable GYP_DEFINES="test_isolation_mode=remap", the XXX_run targets behavior changes to map the inputs files into a temporary directory as part of the build process. Can be useful for manual testing. Watch out for disk usage since it won't clean up the temporary directories created!

For example;
# Generate out/Release/base_unittests.isolated which contains variables like PRODUCT_DIR:
export GYP_DEFINES="$GYP_DEFINES test_isolation_mode=check"
ninja -C out/Release base_unittests_run
# Rerun it manually to output it at the desired place so you can run the test in an isolated configuration at your leisure.
python tools/swarm_client/isolate.py remap --isolated out/Release/base_unittests.isolated --out $HOME/tmp
cd $HOME/tmp
# See the files mapped there!

run

Runs the test as part of the build process.

By setting the environment variable GYP_DEFINES="test_isolation_mode=run", the XXX_run targets behavior changes to run the test executable as part of the build process. Here's the behavior of isolate.py run:
  1. Creates a temporary directory and link the executable and all its dependencies into the temporary directory, at their respective relative location, like the remap command.
  2. Runs the executable.
  3. Removes the temporary directory.
  4. Returns the result code of the executable run at step 2.
See isolate.py help run for more information. For example;
# Generate out/Release/base_unittests.isolated which contains variables like PRODUCT_DIR:
export GYP_DEFINES="$GYP_DEFINES test_isolation_mode=check"
ninja -C out/Release base_unittests_run
python tools/swarm_client/isolate.py run --isolated out/Release/base_unittests.isolated

trace

Traces the unit test inside the checkout, not isolated. See the tracing tools page for more information specific to the tracing tools.

For example, to run base_unittests;
# Generate out/Release/base_unittests.isolated which contains variables like PRODUCT_DIR:
export GYP_DEFINES="$GYP_DEFINES test_isolation_mode=check"
ninja -C out/Release base_unittests_run

# The --merge flag tells isolate.py to update base/base_unittests.isolate if any file touched by the executable is missing from base_unittests.isolate.
python tools/swarm_client/isolate.py trace --isolated out/Release/base_unittests.isolated --merge

Note that the base_unittests_run gyp target doesn't need its dependencies to be properly configured yet, just the stub (listed in mimimal .isolate above) to start the unit test properly is needed, in particular the command variable.

Note: it generates out/Release/base_unittests.isolated.log that can be also processed directly by tools/swarm_client/trace_inputs.py if desired.

isolate_test_cases.py

This script automates tracing a google-test executable and updating a .isolate file. It takes an .isolated file as input, traces each test case individually and then merges the results back into the .isolate file. This script makes it easy to add dependencies to a new . isolate file.

fix_test_cases.py

It's an does-it-all wrapper script that runs the test cases in an isolated configuration, with isolate.py run, extract the ones that failed, trace them and update the .isolate file. All you need to do is to commit the updated .isolate file!
# Generate out/Release/base_unittests.isolated which contains variables like PRODUCT_DIR:
export GYP_DEFINES="$GYP_DEFINES test_isolation_mode=check"
ninja -C out/Release base_unittests_run
python tools/swarm_client/fix_test_cases.py run --isolated out/Release/base_unittests.isolated

# base/base_unittests.isolate was updated if any file was missing. Simply send a code review.
git add base/base_unittests.isolate
git commit -m "Updated base_unittests.isolate for Windows\nR=joe@chromium.org\nBUG="
git cl upload --send-mail

isolate_merge.py

This scripts takes multiple .isolate files that are OS-specific and merges them to combine the dependencies in an efficient way. For example, if a dependency is needed on all the OS, it will be in the global variables. If the dependency is OS specific, like a dylib for OSX or a .dll for Windows, it will be listed in the variables section of the corresponding conditions. Reading base_unittests.isolate as an example makes this clearer.

run_isolated.py

This script is the complement to isolate.py hashtable. It is meant to be run from the Swarm slaves and is rarely necessary for end-users.

It is documented here for completeness.

The behavior is the corresponding related to builder to be able to run a tests:
  1. Retrieves the .isolated file
  2. Create a temporary directory
  3. For each filepath
    1. Look if the SHA-1 is present in the local cache, if not:
      1. HTTP GET the file
      2. Store it in a local hash table, simply a directory
    2. hardlink the filepath in the temporary directory
  4. Run the test in the temporary directory
  5. Delete the temporary directory, keeping the local cache
The local cache has LRU trimming behavior and maximum cache size.

run_isolated.py also has a download mode that can be very useful for end users. The download options allows users to down the same build that the swarm bots downloaded, this means that if you can't get a test failure that the try bots (running swarm) can (due to a difference in how the tests is built), you could just download the test executable and all the tests data directly from the Isolate server.
This command will download all the required files to the specified directory. No local cache is created and none of the commands are run, this command just downloads the files and leaves them there for you to work with.
When running this command it is highly recommended that you use a .isolated file that has already been uploaded to the Isolate server, otherwise it is very likely that the files listed by your .isolated file will not be found on the Isolate server.
./run_isolated.py -s browser_tests.isolated --download target/directory/

Note: Currently only certain users can access the Isolate server, there are plans to use oauth to allow all chromium developers access but we haven't been able to implement that yet. If you want to access this server, just ping either csharp@ or maruel@.

FAQ

What about disable_nacl=1, component=shared_library, <insert build flavor here>?

The .isolate format can support it but support is not coded yet. So for now only "vanilla build" is supported. There is no support for cross-compiling yet. As such, we do not support ChromeOS, Android, etc.

Roadmap / What's the timeline?

We aim for a partial conversion before the end of the Q1 2013. There's a transition path. For a while, no developer will be required to do anything. As progress is made the more involved your maintenance will be which is very similar to maintaining the .gyp files when you are adding new source files.

At the beginning, only the Try Server will be affected. Some background information first;

A Try Builder is a column on the Try Server waterfall and describes a premade configuration. The most important ones are linux_rel/linux, mac_rel/mac and win_rel/win. Others like linux_chromeos are excluded for now to simplify the explanation but will be supported eventually.

For each for linux_rel|linux, mac_rel|mac and win_rel|win, tests will slowly be moved over to the isolate format, and some tests will get run on the swarm_triggered Try Builder. 
  • linux_rel|linux|mac_rel|mac|win_rel|win
    • These Try Builders compile, optionally archives the tests and optionally run them
      • They archive if a foo_swarm test filter is specified, like git cl try -b linux_rel:browser_tests_swarm
      • They run test specified as usual but if only *_swarm tests are specified, none will be run on the original Try Builder
    • They optionally trigger swarm_triggered
      • Like for archive, the trigger happens if a *_swarm test filter is used. See the Try Server FAQ for more information.
  • swarm_triggered
    • This Try Builder is an interface to Swarm. Swarm takes each of the .isolated and shards it on multiple Swarm slaves simultaneously.
    • The main difference between the normal bots and the Swarm slaves do not have a source checkout. This is the most important difference.
    • Swarm slaves are independent of Try Slaves, they are much weaker and smaller.
    • See the Swarm Design Doc for more explanations.

Summary

Try Builder CompilesChecks out the sources  Multiple test execution
 linux_rel, linux, mac_rel, mac, win_rel, win YesYes (for now)Serially 
 swarm_triggered NoNo (gets everything from isolated)Parallel (on Swarm slaves) 

So the transition path will involve certain tests (starting with base_unittests), always running in isolate mode. The tests will still be run on the same machines, but only have access to the files that are listed in the isolate file. This means that tests run this way will be able to get switched over to run on swarm without any additional work by the developers. 

The swarm bots will probably run on the try server first, offering substantial speed ups.

It seems tedious to list each test data file individually, can I just list src/chrome/test/data/ ?

In theory yes, in practice please don't and keep the list to the strict minimum for now. The goal is not to run the tests more slowly and having the slaves download 20 gb of data partially defeats the purpose. We'll evaluate this but for now please only the strict minimum. Reasons includes:
  1. The AppEngine server and the corresponding client-side code are not optimized for this use case yet and there are known bottleneck that do not scale well with the number of files that are being worked on. For example, when the foo_test_gyp target is being run, there's a 7ms/file cost per cache hit.
  2. The use cases are larger than what is described here, we haven't ruled out reusing this data for other uses cases, for example ChromeOS BVT tests where the costs trade-off are different.
  3. It's always possible to go coarser but it's much harder to get back stricter.
  4. We'll revisit the decision once it'll be required to maintain the file. There is no hurry here. See #3.

What's the difference between isolate_dependency_tracked and isolate_dependency_untracked?

isolate_dependency_tracked are what will be listed as dependencies for the .isolated file. This impose some restrictions:
- Their filename cannot contain whitespace
- It must be a file, not a directory or a symlink
- The file must be present all the time; e.g. it cannot be in an optional gclient dependency

I traced my test executable and some test cases still fail when isolated

Tracing is still not 100% fool proof. It may be possible you have to manually edit the .isolate file. This include a test case looking for the presence of a directory but never opening a file inside or other silly things like that.

Where can I find the .isolated file?

The .isolated files are generated when you build the isolation specific version of a target. The isolation target name is just the normal target name with a _run added to the end. These targets are only visible if you have set the gyp variable test_isolation_mode to something other than noop (which is the default). Most people will just need GYP_DEFINES="test_isolation_mode=remap". With the gyp define properly set, the .isolated files will appear in the output directory after building.

I have a feature request / This looks awesome, can I contribute?

This project is 100% open source. Check out the sources:
git clone https://git.chromium.org/chromium/tools/swarm_client.git
and use git cl for code reviews.

For the AppEngine datastore server,
git svn clone svn://svn.chromium.org/chrome -Ttrunk/tools/isolate_server

Why not a faulty file system like FUSE?

Faulty file systems are inherently slow: every time a file is missing, the whole process hangs, the FUSE adapter downloads the file synchronously, then the process resume. Multiply 3000x; that's what browser_tests lists. With a pre-loaded content-addressed file-system, all the files can be cached safely locally and be downloaded simultaneously. The saving and speed improvement is enormous.

Why can't I compile with test_isolation_mode!=noop?

If you see an error like "LINK : fatal error LNK1201: error writing to program database 'f:\src\chrome3\src\out\Debug\unit_tests.exe.pdb'; check for insufficient disk space, invalid path, or insufficient privilege", check to make sure you aren't using unsupported GYP_DEFINES (such as component=shared_library).

It is unknown why this causes this error, but it seems to.