References and Further Reading
OverviewThe Chrome OS version of Autotest introduces a new type of suite, known as a dynamic suite. Dynamic suites allow for the jobs in a suite to be sharded over a pool of DUTs, and the dynamic suite infrastructure takes care of all of the device imaging and test scheduling/sharding details. Different tests in a suite may require different specific features of a DUT (for instance a certain type of cellular modem, or an attached servo board). These requirements can be specified as test ObjectivesIn this codelab, we will:
Prerequisites
Non Prerequisites
Creating The TestCreate a work branchThis codelab will involve touching or changing code in two git repositories within the ChromiumOS repo. Our suite will be named peaches, so we will start by creating a repo branch named peaches, associated with the two git repos we will be modifying: user@host:~/chromiumos$ repo start peaches src/third_party/autotest/files src/third_party/chromiumos-overlay If this succeeds, then you should be able to see your newly created branch. user@host:~/chromiumos$ repo branch Create a dynamic suiteTest suites are defined by Autotest control files (Made up of Python with some meta variables), similar to the control files used to define tests themselves. The suite control files live in the Chromium OS source tree it the Once you are satisfied, create a new file in this directory named # Copyright 2013 The Chromium OS Authors. All rights reserved. The suite control file’s Create a new test control fileTests can declare themselves to be part of any number of suites. This is done by listing the suite in the test control file’s SUITE variable. To put a test into multiple suites, simply use a comma separated list. In this codelab, we will add two existing control files and two new control files to our suite. Let's start with two new dummy tests control files. Create the file # Copyright 2013 The Chromium OS Authors. All rights reserved. Create a new test control file with DEPENDENCIESIn the same directory, create another file, . . . This label tells the dynamic suite scheduler that this job may only run on DUTs with the bluetooth label. Add two simple existing tests to the suiteLet's also add some existing tests to the peaches suite. For instance, edit the Enumerate the suite’s testsTo verify that we have added these 4 tests to our suite, we can use the user@host:~/chromiumos/src/third_party/autotest/files$ site_utils/suite_enumerator.py peaches -a . Include new test files in autotest-tests ebuildEarlier, we added two new test control files for client-side tests. In order for the new test files to be available to the DUT at test time, they must be included in the appropriate overlay ebuild file. This procedure is explained in more detail in a separate codelab -- writing a client side test (not yet published). Open the file . . . Commit and upload changesWe need to commit our changes to two separate git repositories. The changes that we made to the ebuild are required in order for the suite to run properly on a DUT, so we need to make the Autotest repo changes depend on the ebuild changes. First, from the chromiumos-overlay directory, create a commit with Find the Change-ID for the commit you just created, using Change directory to the Autotest repository, and create another commit, but this time including in your commit message a line Now, upload both your changes to gerrit with Once repo upload has finished its work, you will see links to your two new changes on chromium-review.googlesource.com. Build your changes into a new image using cbuildbotDetermine the Change-ID for your Autotest changes. Then, submit your patch to be built remotely: user@host:~/chromiumos/chromite/bin$ ./cbuildbot --remote -g ***** lumpy-release-tryjob where you have pasted in the Change-ID of the changes to the Autotest repository. The output of this command should give you a buildbot link where you can follow your build progress. The build will take about 6 hours. Run your new suiteOnce the building step in the previous section has concluded, you should receive an email to this effect from cros.tryserver@chromium.org. Follow the link in this email to your build results page, then drill down to the "Report stdio" link, and pull out the build number (which will be a string similar to "trybot-lumpy-release/R26-3556.0.0-b683"). Now, run your suite with the command user@host:~/chromiumos/src/third_party/autotest/files/site_utils$ ./run_suite.py -s peaches -b lumpy -i ***** -p try-bot where ***** is the build number you just extracted. Point your browser at http://cautotest/, and you should soon see your suite job appear in the job list. After the suite job has started to run, it will spawn sub-jobs for all the individual tests. Note that different tests may end up running on different DUTs. Add suite-level dependenciesOne of the tests we created in the codelab, peaches_DummyPass_BT, made use of a DEPENDENCY to require that the test could only run on DUTs with the bluetooth label. In addition to specifying dependencies at the test level, they can also be specified at the suite level. When a suite with suite level dependencies is run, all the jobs kicked off by the suite will have any suite dependencies added in addition to the test level dependencies. To add a suite level dependency, edit the control file for the suite. Add a named argument to the call to dynamic_suite.reimage_and_run, of the form suite_dependencies=’servo’, for example. Now, when the suite is run, all jobs will inherit an additional dependency on servo. The string can contain multiple dependencies as a comma separated list. Suite-level dependencies can be useful when you want to run several closely related suites consisting of the same tests, but with slightly different dependencies. For example, if you want to run a suite focused on network3g connectivity, but separately on devices configured for different cellular carriers. See for instance https://chromium-review.googlesource.com/41260/ Get your suite into the regular rotation with suite_schedulerSuites can be scheduled to run in the test lab automatically, either triggered by build events or at regular timed intervals. To add your suite to the schedule, edit suite_scheduler.ini in the root directory of the Autotest repo. Following in the footsteps of the other suites already in the file, it should be easy to add your suite. To add peaches as a suite that runs nightly, add the following to suite_scheduler.ini . . . The fields above specify when suite runs should be triggered, which suite should be run, which branches should trigger the suite to run, which machine pool the suite should be assigned to, and the number of DUTs that the suite should attempt to use. For more information on what pool to select refere to What pool should I select. If you have added a new suite to suite_scheduler.ini, one for which a suite control file did not exist before, you need to pay attention to the branch_specs attribute. Suite control files are picked up from the build artifacts (unlike other server-side control files). You can either backport your new suite control file to older maintained branches, or avoid scheduling this suite against those branches by using branch_specs to set a cutoff. There are some subtleties in the num parameter, with respect to test dependencies. You must ensure that the num parameter is greater than or equal to the number of unique dependency sets over all the jobs in your suite. So, for instance, if you have a suite (like peaches) that has some jobs with no dependencies, and some jobs with 1 dependency (bluetooth), you must make use num >= 2, otherwise the suite will fail immediately on running. There’s a handy sanity check script to make sure you’re satisfied this: ./site_utils/suite_scheduler/suite_scheduler.py --sanity This sanity check will also run as a pre-submit hook, so even if you forget to run it yourself, you will be warned on repo upload that you have not fulfilled the num criteria. |