Chromium OS‎ > ‎Testing Home‎ > ‎

Power testing

Description

Modern mobile computers are sold with an advertised battery life, ranging from a few to tens of hours. Nonetheless, when these devices are used on a daily basis, many users report the actual battery life doesn’t match up with the advertised numbers. For Chrome OS devices, we wanted to try to report battery life that was as close to what an average user experienced as possible.

The power_LoadTest was created to emulate average user behavior and measure the resultant battery life. The test is implemented as an open source Chrome extension which anyone can install and run.


The power_LoadTest is run as a series of one hour long iterations until battery exhaustion. Within each iteration, a load mix known as “60/20/10/10” is run:
  • [First 60% of the test] Browsing: a new website is loaded every minute.  The web page loaded is scrolled down one page (600px) every 10 second, then scrolled back up at the same rate.
  • [Next 20%] Email: Gmail is loaded in the foreground tab and audio is streamed from a background tab.
  • [Next 10%] Documents: Various Google Docs are loaded.
  • [Final 10%] Video: A full screen 720p YouTube Video is played.

The parameters of the system under test are as follows:
  • Backlight: Set to platform default brightness on battery or 40% if default can not be determined via get_powerd_initial_backlight_level. If platform has keyboard backlight it will be set to 10%.
  • Power management:
    • Dimming, blanking the screen and transitions to suspend-to-RAM state are disabled.
    • Ambient light sensor readings ignored.
  • Battery:
    • Device only powered by battery (no AC power).
    • Battery charged to 100% prior to initializing test. Test continues in 1 hour iterations until battery passes low threshold (currently set at 3%). Initial & remaining battery charge is recorded.
  • USB: No external devices connected
  • Network: Device is associated with a wireless access point via WiFi.
  • Audio: Built-in speakers at 10%, Built-in microphone at 10%

Throughout the duration of the test, there are 5 sites loaded in background tabs. These sites were chosen to represent typical actions of a user on a daily basis:
  • Searching
  • Reading news
  • Checking on finance
  • Shopping
  • Communication

Running


If you are interested in running power_LoadTest on a Chrome OS system, you will need a Chromium OS test image that can be built by following Build your own Chromium image instruction with "./build_image --board=${BOARD} test" command. After the test image is built, you can follow the Installing Chromium OS on your device instruction to install the test image to your device under test (DUT).

UPDATE: starting with R35 pyauto setup below is no longer necessary as power_LoadTest uses telemetry chrome automation framework NOT pyauto.
In addition, power_LoadTest currently relies on the pyauto automation framework.  This dependency is not automatically installed on the test image.  Some insight into the dependency complications can be seen here ( search pyauto ).  To add the pyauto dependency you can run a test that explicitly installs it:

  $ ./run_remote_tests.sh --remote <DUT_ipaddr> ^client/site_tests/desktopui_PyAutoInstall/control$

This may only be necessary for the first test image installed on the device as subsequent updates will maintain these deps from stateful partition.

Since running power_LoadTest requires that the device is disconnected from the wired Ethernet (including USB-Ethernet) as well as from an AC power source, it is trickier to run it compared to running other autotests if you do not have both the build machine and the DUT under a same private WiFi network connected to the Internet.

If your build machine and the DUT are in the same WiFi network, you can run power_LoadTest by running the following command in scripts directory in chroot. (make sure you have battery fully charged with AC and Ethernet disconnected from the DUT before running the test)

  $ test_that <DUT_ipaddr> power_LoadTest

If your build machine is not on the same private WiFi network as the DUT (applicable for most Googlers and any who have the build machine connected to a corporate network), follow the below instruction.
  • Keep the AC charger plugged and the wired Ethernet connected to the same network as your build machine
  • Run the same command as above.  It is expected to fail as the pure purpose is to copy the test suite over to the DUT from the build machine
     $ test_that <DUT_ipaddr> power_LoadTest
  • Disconnect the wired Ethernet
  • On the DUT, enter VT2 by pressing Ctrl + Alt + F2 [-> right arrow key on Chrome keyboard], and login as root (should not require any password on a test image)
    $ cd /usr/local/autotest
    $ # (remove your AC power source right before running the test)
    $ bin/autotest tests/power_LoadTest/control
  • Now you just need to wait until the battery runs out.  (Don't worry, the test result is stored on the DUT)
  • Reconnect the AC power, boot up, and enter VT2 as root
    $ cd /usr/local/autotest/results/default/power_LoadTest/results/

Interpreting Results

When the test completes there will be a keyvals file at power_LoadTest/results/keyval.  The test will publish minutes_battery_life which we use to track platforms battery life.  However that only tells part of the story.  As any other real world test the results need other measurements should be examined to ensure the battery life estimate is genuine.

Keyvals of particular interest beyond minutes_battery_life to judge quality of run are:
  • ext_*_failed_loads
    • Any non-zero values for these keyvals can be problematic.  The keyval itself is a string with counts separated by underscores.  Each count is the failures logged by the particular loop.  For example ‘0_1_2’ would mean no failures for loop0, 1 for loop1 and 2 for loop2.  While failures aren’t good if the subsequent ext_*_successful_loads equals the correct quantity then data may be deemed ok.
  • percent_cpuidle_*
    • These numbers typically stay roughly the same for a particular platform so be cogniscent that they roughly match previously qualified PLT runs or the differences should be investigated.
  • percent_cpufreq_*
    • Similar to cpuidle above this represents the P-state the processor frequency runs in.  These too stay roughly the same and should be compared to previously qualified numbers
  • loop*_system_pwr
    • each loop should be roughly the same average power.  Often the first loop (loop0) consumes more as webpage caching hasn’t occurred yet.
  • *usbidle*
    • should be ~100% unless run is being done with expected USB device that has been attached externally or has had selective suspend disabled for some functional reason.

Conclusion

While the initial version of power_LoadTest seems to emulate well what users experience every day on Chrome OS devices, this test will be constantly improved. As we learn more about how users use Chrome OS devices and how experienced battery life differs from tested battery life, we will use this data to refine the test, potentially changing the load mix or the parameters of the test.

Our goal is to ensure that when you purchase a device, you know - with reasonable certainty - how long that device will last in your daily use.


Subpages (1): PLTP
Comments