Isolated Testing for SWEs

Overview

This page explains how to convert a googletest executable into a fully isolated test. This is done by describing runtime dependencies for a test. 

For information about the infrastructure itself and the roadmap, see Isolated Testing Infrastructure.

Note: If you're a Googler who needs to perform a manual test on a platform for which you cannot readily build (e.g. Windows Developer with no development environment on your Mac book), follow these instructions to get the build from a try run.

What's "Isolate"?

Goal: describe the exact list of files needed to run a executable.

The isolate project is all the file formats, tools and server code to make this work fast and efficiently. Every test executable list the data files it depends on, which allows the testers bots to only download the files they need instead of the whole Chromium source tree.

They are described in BUILD.gn files.

What's "Swarming"?

Goal: distribute tasks fast and efficiently in an heterogeneous fleet of bots.

The swarming project is all the tools, server and bot code to run a step (like a unit test) on a remote bot and get results back. It has native google-test sharding capability.

What are the advantages?

By reducing the amount of data that we need to transfer to the tester machines, it becomes much easier to increase the number of tester bots that we have using Swarming (part of LUCI).

Adding a new test

  • Make sure the test exist in a BUILD.gn file.
  • Add missing items to the "data" section until the test runs locally isolated via mb.py run.
  • Add the test to the corresponding json file in //testing/buildbot/.
  • All the required data files, executables started by the test executable* required by the test to succeed are listed as a dependency.
    • For example, browser_tests.exe needs chrome.exe, which needs resources.pak, etc.
Expectations of the tests:
  • Must not do directory walking or otherwise try to guess what should be tested.
  • Must not edit any input file.
  • Must not write at all in the current directory. It must only use the temporary directory for temporary files.

HowTos

Download binaries from a try job

  • Visit the try job build, e.g. https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/1234
  • Search for [trigger] my_test and click Swarming task "shard #0" link
  • Scroll down a bit, follow the instructions in the Reproducing this Task Locally section

Run a test isolated locally

echo gn > out/Release/mb_type  # Must be done once to avoid mb.py from performing a clobber
tools/mb/mb.py run //out/Release base_unittests  # Builds, generates .isolate (via gn desc runtime_deps), and runs (via "isolate.py run")

See the Roadmap for what can or can't be done.

Run a test built locally on Swarming

1. Build & Generate .isolate file:
ninja -C out/Release base_unittests
echo gn > out/Release/mb_type  # Must be done once to avoid mb.py from performing a clobber
tools/mb/mb.py isolate //out/Release base_unittests # Creates out/Release/base_unittests.isolate

2. Compute .isolated file and upload it:
tools/swarming_client/isolate.py archive \
    -I https://isolateserver.appspot.com \
    -i out/Debug/base_unittests.isolate \
    -s out/Debug/base_unittests.isolated \
    --verbose

3. Trigger the task:
tools/swarming_client/swarming.py trigger \
    -S https://chromium-swarm.appspot.com \
    -I https://isolateserver.appspot.com \
    -d os Android \
    -d pool Chrome \
    -s <hash of the command above>

Other values for -d/--dimension os: Mac, Windows, Linux
For other available --dimension values, look at Swarming bots (e.g.: https://build.chromium.org/p/chromium.android/builders/Android%20Swarm%20Builder)

Dogfood

Eligibility

Right now, only users @google.com can use the infrastructure. For others, we'll try to make it available to Chromium committers eventually. Note that the whole Swarming infrastructure is open source so if any other company would help to recreate the same infrastructure internally, send us a note at swarming-eng@googlegroups.com

Get the Swarming client code

If you have a chromium checkout, you already have everything you need in src/tools/swarming_client/.

Login on the services

By login first, you have access tokens so that the following commands do not have to constantly prompt for your identity. 

python tools/swarming_client/auth.py login --service=https://isolateserver.appspot.com
python tools/swarming_client/auth.py login --service=https://chromium-swarm.appspot.com 

If you are running through a text only session on a remote machine, append argument --auth-no-local-webserver

Run the example

This is a good sanity check to ensure that everything works:

python tools/swarming_client/example/3_swarming_trigger_collect.py -I isolateserver.appspot.com \
    -S chromium-swarm.appspot.com

If this doesn't work, see the FAQ before continuing or ping us, we're friendly.

Trigger the task

Now you've built something, time to archive it to the isolate server and request Swarming to run it on your behalf.

python tools/swarming_client/swarming.py run \
    -I isolateserver.appspot.com \
    -S chromium-swarm.appspot.com \
    -d os Windows-7-SP1 \
    -d pool Chrome \
    -d gpu none \
    --verbose \
    out/Release/base_unittests.isolated

First thing it does is to archive the binary. Depending on your connection speed and the size of the executable, it may take up to a minute. Then it triggers the task and wait for results. OS currently available:
  • Windows-7-SP1 (64 bits)
  • Windows-8.1-SP0
  • Windows-2008ServerR2-SP1 (64 bits)
  • Windows-10-10586 (64 bits)
  • Ubuntu-12.04 (64 bits)
  • Ubuntu-14.04 (64 bits)
  • Mac-10.9
  • Mac-10.10
  • Mac-10.11
Visit chromium-swarm.appspot.com/restricted/bots to see all the values available. That's it. Feel free to contact the team at isolate@chromium.org for any chromium open source specific questions.

Note: -d pool Chrome is needed!

Additional Notes

FAQ

I run a task on Swarming and it hangs there

It is possible that all the bots are currently fully utilized.

It seems tedious to list each test data file individually, can I just list src/ ?

In theory yes, in practice please don't and keep the list to the strict minimum. The goal is not to run the tests more slowly and having the slaves download 20 gb of data. Reasons includes:
  1. Isolate Server is optimized for < 50000 files scenario. There's a 2ms/file cost per cache hit. So for example, layout tests are currently out of the use case since there's > 80000 files.
  2. It's always possible to go coarser but it's much harder to get back stricter.

Where can I find the .isolated file?

The .isolated files are generated when you build the isolation specific version of a target, e.g. out/Debug or out/Release. The isolation target name is just the normal target name with a _run added to the end.

I have an error, is it related to .isolate files?

If you have a test that passes locally and fails on some trybots, then it could be related.

This error can be seen when a browser test js file is not found:
  TypeError: Cannot read property 'testCaseBodies' of undefined

Where should I file bugs?

Swarming specific bugs can be filed on github.com/luci/luci-py/issues. Chromium specific bugs at http://crbug.com.
Comments