the Chromium logo

The Chromium Projects
ChromiumOS > Guides >

Running smoke suite on a VM image

Introduction

The VMTest "smoke suite" is a suite of client-side autotests that performs basic sanity checks on a Chromium OS build.

Each of the tests in the smoke test suite is capable of running in a virtual machine.

This allows the smoke suite to be run as part of the automated build process on a 'builder'.

Smoke Suite tests

The Smoke suite currently consists of the following tests (this list is subject to change):

Running the Smoke Test Suite on a Virtual Machine (VM)

The instructions on this page should be enough to get you going. They install a testable Chromium OS image on a VM and run the smoke suite. However, if you want more details, you can always go to one of the following pages:

Install qemu-kvm

First of all, to run in a vm, you must have vm software installed.

On Ubuntu Lucid, use apt-get to install qemu-kvm:

$ sudo apt-get install qemu-kvm

Build a testable VM image

WARNING: After crbug/710629, 'betty' is the only board regularly run through pre-CQ and CQ VMTest and so is the most likely to work at ToT. 'betty' is based on 'amd64-generic', though, so 'amd64-generic' is likely to also work for most (non-ARC) tests.

Next, build the testable VM image. The following steps assume that:

  1. depot_tools/repo have been installed and configured.
  2. Chromium OS source code has been checked out.
  3. A chroot has been created with cros_sdk --create.
# Build all the source packages, including those required for running autotests
# Optional: "--use-any-chrome" may make your build faster, but will have old
# Chrome.
# NOTE: At the moment, "--with-autotest" is the default, so don't worry if you
# built without it.
(outside) $ cros build-packages --board=${BOARD}

# Build a bootable image
# Optional: Include "--no-enable-rootfs-verification" if you think you might
# need to modify your rootfs.
(outside) $ cros build-image --board=${BOARD} test

# Clone this image, modify it for test, and make image for use in qemu-kvm
# Virtual Machine
# Note: because we use "--test_image", an explicit "modify_image_for_test" is
# not required.
(chroot) $ ./image_to_vm.sh --board=${BOARD} --test_image

The newly created VM image should be in the following path (relative to src/scripts):

../build/images/${BOARD}/latest/chromiumos_qemu_image.bin

Start a VM image

Before launching autotests, its fun to boot up a virtual Chromium OS on your desktop!


(chroot)$ cros_vm --start --board=${BOARD}

Run the autotest tests on the VM image

Autotests can be run against the VM via test_that.

# Run the entire smoke suite
 (chroot)$ test_that --board=${BOARD} localhost:9222 suite:smoke
# Run an individual test from the smoke Suite
(chroot)$ test_that --board=${BOARD} localhost:9222 logging_CrashSender

Don't forget to kill the head-less VM when you are done with it!


(chroot)$ cros_vm --stop

Run the autotest tests on the VM image using cros_run_vm_test

Another script, cros_run_vm_test, can be used to start the VM, run autotest tests on it, and then shut down the VM, all from a single invocation:


# Run the entire smoke suite:
(chroot)$ mkdir results
(chroot)$ cros_run_vm_test --board=${BOARD} --results-dir=results --autotest suite:smoke
# Run an individual test from the suite
(chroot)$ cros_run_vm_test --board=${BOARD} --results-dir=results --autotest platform_OSLimits

Troubleshooting

No space left on device

If you get an error that looks like this:

Unhandled OSError: [Errno 28] No space left on device: '/home/autotest/tmp/_autotmp_Woz_Qyharness-fifo'

...this may be because you have run out of space on your "stateful partition". This can happen if you leave the VM running for a long time and errors fill up the /var/log folder. In my case, this folder contained 250M of data! You should go into /var/log and cleanup stuff.

Use df to determine how much space is available in the stateful partition:


localhost ~ # df /mnt/stateful_partition
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              2064208    450612   1508740  23% /mnt/stateful_partition

stop <daemon_name>.You may be getting lots of errors if you're running a board-specific image with drivers for hardware that is not present on the VM. If this is the case, try stopping the responsible daemon with

Could not set up host forwarding rule 'tcp::9222-:22'

If you get this error when running cros_vm --start, make sure that you don't have a VM already running. This should kill it for you.

cros_vm --stop

Could not initialize KVM, will disable KVM support

This is probably because you don't have the kvm module loaded. The VM will run.....very.......slowly.

open /dev/kvm: No such file or directory
Could not initialize KVM, will disable KVM support
$ modprobe kvm_intel    # if you have an AMD machine, use 'kvm_amd'

Note that this module interferes with VirtualBox so you may need to remove it (rmmod kvm_intel) if you want to use this.

Updating a client autotest

Autotest tests are usually built and compiled on the autotest server machine.

A 'client' test is installed on the client by the autotest infrastructure when the test is run.

The autotest infrastructure is maintained in the chromeos-base/autotest cros_workon ebuild.

Individual tests and test suites are in the chromeos-base/autotest-tests ebuild.

The new way: Use cros_workon with all Python tests

Working on all Python autotests should be simple and fast.

First cros_workon autotest for the board on which you plan to run tests:


(chroot)$ cros_workon start autotest --board=${BOARD}

Now, test_that will detect that you are working on autotests and automatically copy over Python in the autotest source tree to the sysroot.

Run test_that as usual:

(chroot)$ test_that --board=${BOARD} localhost:9222 suite:smoke

The old way: emerge individual autotest tests

If you have a test with binary dependencies, you must rebuild them explicitly:

(chroot)$ emerge-${BUILD} autotest-tests

This can take a very, very long time since it will rebuild ALL tests.

To speed this up, set the TESTS variable on the command line when running emerge to install an individual test:


(chroot)$ TESTS=<TestName> emerge-${BOARD} autotest-tests

Note that doing this will deactivate all other tests! This is a problem for test suites.

To rebuild a whole test suite, specify the suite and all individual tests in the suite. For example, to rebuild smoke suite:


(chroot)$ TESTS='build_RootFilesystemSize desktopui_ChromeFirstRender \
desktopui_FlashSanityCheck desktopui_KillRestart logging_CrashSender \
login_Backdoor login_BadAuthentication login_CryptohomeIncognitoMounted \
login_CryptohomeMounted login_LoginSuccess platform_FilePerms platform_OSLimits' \
emerge-${BOARD} autotest-tests

Then go grab a coffee...

Debugging

You can use chromium os debugging tips on VM as well.