The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.

Android Tests

Concepts

  • Native Unit Tests - Normal Chromium unit tests based on the gtest framework.
  • Instrumentation Tests - Java tests written using the Android Instrumentation Test Framework.
  • Unit Instrumentation Tests - Instrumentation tests that only tests Java code and not native code. Do not require JNI bindings and do not require starting up ContentShell (or another application). These should extend InstrumentationTestCase directly.
  • Integration Instrumentation Tests - Instrumentation tests that tests the entire stack from Java into native code, require JNI bindings to be set up and thus require ContentShell (or another application) to be started. These typically extend more specialized test base classes such as ContentShellTestBase.

Native Unit Tests

These are the common gtest-based C++ framework for tests (such as base_unittests, net_unittests, etc...).
There are some platform-specific requirements for Android:
  • On other platforms, the target test suite is normally an executable (in GYP, this means that gtest_target_type is defined as 'executable').
    On Android, we need to package as library.so (in GYP, gtest_target_type='shared_library') that is then packaged in the APK. The most important reason for this is that only an APK can package .java files that are then accessible for tests; this is used when the platform-specific implementation of a given cross-platform API lives in java);
  • On Android, we have to push and install the APK and the respective data files to the device, then launch it, and observe the output; this is mostly done via build/android/run_tests.py
    check all the options with --help
    • By having test running on the target device, rather than the host machine, we allow "sharding" by using multiple devices connected to the same host.
    • Also, since we can't read directly the stdout of an application, the gtest-based suites in the device redirect its output to a "fifo", which is then read and parsed by the test harness in the host.
  • To bring up a new test suite on android:
    • Make sure the type in gyp is set to gtest_target_type
    • Add a "foo_bar_unittests_apk" target (similar to base_unittests_apk in base.gyp)
    • Add it to the list of supported suites in build/android/run_tests.py _TEST_SUITES
    • Add the necessary files that need to be copied to the device in build/android/single_test_runner.py, GetDataFilesForTestSuite()

Location of Test Code

As a general rule, tests should live next to the production code and test utilities should live in a test folder.

In general, this should look like:
  • foo/android/java/src/bar - production code.
  • foo/android/javatests/src/bar - unit instrumentation tests.
  • foo/test/android/javatests/src/bar/test/util - reusable test utilities used to test the foo production code.
  • native unit tests should live next to the source (same as general Chromium unit test)
Test utilities that are pure Java and do not depend on content or other Chromium code should live in:
  • base/test/android/javatests/src/org/chromium/base/test/util
Test utilities in the content layer that are re-used by chrome should live in:
  • content/public/test/android/javatests/src/org/chromium/content/test/util
The location of integration instrumentation tests depends on what they are testing:
  • If the test is directly testing code in content/public/android/java/src, it should live in content/public/android/javatests/src (although it might require the test base class from content/shell/android/javatests/src).
  • If the test is directly testing code in chrome/android/java/src, it should live in chrome/android/javatests/src (although it might require the test base class from chrome/android/testshell/javatests/src).
  • If the test is a general integration test (i.e. testing general functionality that might not be tied directly to specific code), it should live in the appropriate javatests folder:
    • content/shell/android/javatests/src - Content Shell
    • chrome/android/testshell/javatests/src - Chrome Test Shell
    • android_webview/javatests/src - Android WebView
Please note that this implies that outside of Content Shell, Chrome Test Shell and Android WebView, you should:
  • write unit instrumentation tests to test your Java code.
  • write native tests for everything else.

Guidelines for Writing Tests

TestBase Classes and Test Utilities

  • TestBase classes 
    • should only contain state that all subclasses will need.
    • should be as small and clean as possible.
    • should not include utility methods, instead use utility classes. 
  • Unit instrumentation test classes should extend InstrumentationTestCase and not any other TestBase classes.
  • Utility test classes
    • are the preferred method of sharing functionality between test packages.
    • should only depend on production code or other test utilities.
    • should live in foo/test/android/javatests/src/bar/test/util.

Annotations

Each instrumentation test should have a test size and feature annotation.

Test Size Annotations

  • Always annotate a test with one of the test size annotations:
    • @SmallTest
    • @MediumTest
    • @LargeTest
    • android.test.suitebuilder.annotation.LargeTest/MediumTest/SmallTest 
  • Know the size of your test.
    • In general:
      • unit Instrumentation tests are small.
      • integration Instrumentation tests are medium or large.
  • The timeouts for each test size is:
    • @SmallTest - 1 minute
    • @MediumTest - 3 minutes
    • @LargeTest - 5 minutes

Feature Annotation

  • The feature annotation is used to specify which feature(s) the test is exercising. 
  • The list of features can be found in crbug.
  • To add a feature annotation to your test, add the @Feature annotation and specify the feature(s) that the test is testing. 
    • For example: @Feature({"Android-WebView"})
  • To run the tests for a specific feature, use:
    • python build/android/run_instrumentation_tests.py --test-apk ContentShellTest --sdk --A Feature=Android-WebView,Main

Flaky and Crashing Tests

If a test is flaky or randomly crashes, the test should not run on the build bots.

The guidelines for handling flaky and crashing tests are as follows:
  • If a test is flaky but doesn’t crash, it should be marked flaky using the FlakyTest annotation.
  • If a test is crashing, it should be disabled using the DisabledTest annotation.
  • To mark a test flaky or disabled, do the following
    • create a bug for the test/area owner to fix the test.
    • don't mark a test flaky or disabled without creating a bug!!
    • add the @FlakyTest or @DisabledTest annotation to the test depending on whether the test is flaky or crashing.
    • add the bug number in a comment to the test.
    • please follow this example:
/*
@LargeTest
Bug XX
*/
@FlakyTest 
Replace @FlakyTest with @DisabledTest if disabling a test.
    • if the test has any other annotations, do NOT remove these! 
      • If the annotation is @Smoke, @SmallTest, @MediumTest, @LargeTest or @Feature, keep the annotation in the comment.
      • If the annotation is for example @PhoneOnly, keep the annotation as is.
Comments