Chromium OS‎ > ‎Testing Home‎ > ‎

test_that basic usage

test_that basic usage

A replacement for run_remote_tests

(go/cros-test_that)


Basic usage

Sept 9, 2013


As you may know, a new tool test_that is replacing run_remote_tests.sh. Its basic purpose is the same -- to run autotest tests against a machine on your desk. A few initial users have been playing with it, but now it is ready for wider use.  Here are some reasons you might want to switch:

  • CTRL+C kills test_that and all its autoserv children. Orphaned processes are no longer left behind.

  • The --use_emerged flag is eliminated. Tests that require binary autotest dependencies will just work, because test_that always runs from the sysroot location.

  • Running emerge after python-only test changes is no longer necessary. test_that usesautotest_quickmerge to copy your python changes to the sysroot.

  • New features are coming, and run_remote_tests.sh support will be phased out. In the near term,test_that will be able to reimage a device to a desired build (user- or builder-built), and to run tests in the lab.



Example uses (inside the chroot)

Note: unlike in run_remote_tests.sh, tests are generally specified to test_that by the NAME field of their control file. Matching tests by filename is supported using f:[file pattern]


Run the test named dummy_Pass against dut.

$ test_that -b ${board} ${host} dummy_Pass


Run the control file control.suspend in dummy_Pass.

$ test_that -b ${board} ${host} dummy_Pass.suspend


Run the smoke suite against dut.

$ test_that -b ${board} ${host} suite:smoke


Run all tests whose names match the regular expression ^login_.*$.  Note that even though these tests have binary dependencies, there is no longer a need to specify extra flags.

$ test_that -b ${board} ${host} e:login_.*


Run all tests whose control file filename matches the regular expression ^.*control.dummy$

$ test_that -b ${board} ${host} f:.*control.dummy




Running jobs in the lab

Sept 23, 2013

test_that now allows you to run jobs in the test lab. The usage is similar to running tests against a specified host. The keyword :lab: is used as test_that's REMOTE argument, and the -i/--build argument is required:


$ test_that -b lumpy -i ${latest_image} :lab: dummy_Pass


This will kick off a suite in the lab that consists of just 1 job, dummy_Pass, to run in this case on board lumpy. The lab's scheduler will take responsibility for finding a suitable set of hosts, provisioning them to the correct image, and running the tests. test_that will return after the suite finishes running, with a suite run report.


You can specify multiple tests or test-matching expressions in the same way as before:

$ test_that -b lumpy -i ${latest_image} :lab: dummy_Pass dummy_Fail

$ test_that -b lumpy -i ${latest_image} :lab: e:login_.*


Kicking off a run in the lab should be useful whenever you need to run a particular test on a board or image that you do not have readily available locally.For occasional runs of ad-hoc suites in the lab, this will also avoid the need to create a suite control file and wait for it to end up in an image.



A concrete example is to run:

test_that -b peach_pit :lab: suite:pyauto_perf -i 'peach_pit-release/R32-4763.0.0'

That told me that my job ID was 5196037.  I could follow along by going to http://cautotest/afe/#tab_id=view_job&object_id=5195962.


Things to note:





Comments