Chromium OS‎ > ‎Testing Home‎ > ‎

Autotest Graphics Documentation

Keywords: This testing document describes how to run GLBench, TearTest, Piglit, WebGLConformance and other tests and benchmarks on ChromeOS and ChromiumOS devices.

Source code location

Python sources are under src/third_party/autotest/files/client/site_tests
drwxr-xr-x-graphics_GLAPIChecktree | history
drwxr-xr-x-graphics_GLBenchtree | history
drwxr-xr-x-graphics_O3DSeleniumtree | history
drwxr-xr-x-graphics_SanAngelestree | history
drwxr-xr-x-graphics_TearTesttree | history
drwxr-xr-x-graphics_WebGLConformancetree | history
drwxr-xr-x-graphics_WindowManagerGraphicsCapturetree | history

Binary dependencies are under src/third_party/autotest/files/client/deps
drwxr-xr-x-glbench-imagestree | history
drwxr-xr-x-glbenchtree | history
drwxr-xr-x-piglittree | history

A note on building binaries

The ebuild system knows if a target knows either OpenGL or OpenGL ES, or even both. If only pre-compiled binaries are used there should not be much of a problem with running tests. When editing tests after using the cros_workon script the default system behavior is to recompile the binaries once more without invoking the ebuild system. Unfortunately this means USE="opengles" flags are not honored. Instead the easy way is to specify the proper switch via am environment variable GRAPHICS_BACKEND=OPENGLES./run_remote_tests.sh that gets directly passed to the Makefile. This is somewhat hacky. To use the ebuild system to generate the binaries one has to do a painfully slow USE=opengles emerge-tegra2_seaboard chromeos-base/autotest followed by ./run_remote_tests.sh  --use_emerged [...]. Don't forget the --use_emerged flag as it is not default.

Autotest graphics_GLAPICheck

- two different binaries can be made
-- GRAPHICS_BACKEND=OPENGL or OPENGLES
- runs 25 seconds
- sample results
- for OPENGL creates output below and parses some samples in python to see if they are there and if SUCCEED: run to the end is there

GL_VERSION = 3.2.0 NVIDIA 195.36.24
GL_EXTENSIONS = GL_ARB_color_buffer_float GL_ARB_compatibility GL_ARB_copy_buffer GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_draw_buffers GL_ARB_draw_elements_base_vertex [...] NV-GLX NV-CONTROL Generic
 Event Extension SHAPE MIT-SHM XInputExtension XTEST BIG-REQUESTS SYNC XKEYBOARD XC-MISC SECURITY XINERAMA XFIXES RENDER RANDR XINERAMA Composite DAMAGE
SUCCEED: run to the end

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.71.194 graphics_GLAPICheck --use_emerged
Initiating first contact with remote host
[...]
INFO    : Test results:
---------------------------------------------
graphics_GLAPICheck                     PASS
graphics_GLAPICheck/graphics_GLAPICheck PASS
---------------------------------------------
Total PASS: 2/2 (100%)

No crashes detected during testing.
Elapsed time: 0m25s

Autotest graphics_GLBench

- draws green/ref/purple/fractal rectangles
- runs about 5 minutes

This benchmark executes glbench, a graphics benchmark designed to time how long
various graphic intensive activities take, which includes measuring:
  - fill rate
    - blended
    - opaque
  -Z reject rate
  -triangle rate
    - no cull
    - half cull (half triangles backface culled)
    - full cull (mix of back face and degenerates)
  - blend rate
  - texture fetch
    - nearest
    - bilinear
    - trilinear
  - compute
    - vertex shader
    - pixel shader
    - *fragement shader to test ddx and ddy
  - attribute fetch
  - color depth stencil test
  - *state change
  - texture upload
  - read back

- does MD5 checksums of some but not all images
-- deps/glbench/src/checksums
-- can be re-generated with -save option

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGLES ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.27 graphics_GLBench --use_emerged
[...]
INFO    : Test results:
----------------------------------------------------
graphics_GLBench                               PASS
graphics_GLBench/graphics_GLBench              PASS
  1280x768_fps_compositing                     11.18
  1280x768_fps_no_fill_compositing             41.96
  mpixels_sec_clear_color                      490.74
  mpixels_sec_clear_colordepth                 323.88
  mpixels_sec_clear_colordepthstencil          281.45
  mpixels_sec_clear_depth                      969.71
  mpixels_sec_clear_depthstencil               647.34
  mpixels_sec_fill_solid                       285.1
  mpixels_sec_fill_solid_blended               128.55
  mpixels_sec_fill_solid_depth_neq             973.2
  mpixels_sec_fill_solid_depth_never           1189.08
  mpixels_sec_fill_tex_bilinear                149.56
  mpixels_sec_fill_tex_nearest                 203.7
  mpixels_sec_fill_tex_trilinear_linear_01     164.63
  mpixels_sec_fill_tex_trilinear_linear_04     nan
  mpixels_sec_fill_tex_trilinear_linear_05     nan
  mpixels_sec_fill_tex_trilinear_nearest_05    nan
  mpixels_sec_pixel_read                       6.72
  mpixels_sec_pixel_read_2                     6.72
  mpixels_sec_pixel_read_3                     6.74
  mpixels_sec_varyings_shader_1                270.68
  mpixels_sec_varyings_shader_2                144.33
  mpixels_sec_varyings_shader_4                73.57
  mpixels_sec_varyings_shader_8                24.29
  mpixels_sec_yuv_shader_1                     nan
  mpixels_sec_yuv_shader_2                     47.82
  mpixels_sec_yuv_shader_3                     93.36
  mpixels_sec_yuv_shader_4                     136.15
  mtexel_sec_texture_update_teximage2d_1024    149.85
  mtexel_sec_texture_update_teximage2d_128     80.92
  mtexel_sec_texture_update_teximage2d_1536    146.05
  mtexel_sec_texture_update_teximage2d_2048    149.47
  mtexel_sec_texture_update_teximage2d_256     132.51
  mtexel_sec_texture_update_teximage2d_32      10.29
  mtexel_sec_texture_update_teximage2d_512     142.05
  mtexel_sec_texture_update_teximage2d_768     147.56
  mtexel_sec_texture_update_texsubimage2d_1024 150.94
  mtexel_sec_texture_update_texsubimage2d_128  81.59
  mtexel_sec_texture_update_texsubimage2d_1536 147.2
  mtexel_sec_texture_update_texsubimage2d_2048 149.03
  mtexel_sec_texture_update_texsubimage2d_256  130.59
  mtexel_sec_texture_update_texsubimage2d_32   10.09
  mtexel_sec_texture_update_texsubimage2d_512  113.54
  mtexel_sec_texture_update_texsubimage2d_768  150.0
  mtri_sec_triangle_setup                      30.34
  mtri_sec_triangle_setup_all_culled           49.89
  mtri_sec_triangle_setup_half_culled          44.91
  mvtx_sec_attribute_fetch_shader              149.51
  mvtx_sec_attribute_fetch_shader_2_attr       149.34
  mvtx_sec_attribute_fetch_shader_4_attr       141.36
  mvtx_sec_attribute_fetch_shader_8_attr       106.71
  us_swap_swap                                 19464.19
----------------------------------------------------
Total PASS: 2/2 (100%)

No crashes detected during testing.
Elapsed time: 8m58s
>>> Details stored under /tmp/run_remote_tests.purz

The "nan" in this example are passed because of knownbad output images. For more more details and failures look at
    
(cros-chroot) ihf@ql ~/trunk/src/scripts $ more /tmp/run_remote_tests.purz/graphics_GLBench/graphics_GLBench/summary.txt 

Note on HW Qual failures

Sometimes when site_tests/suite_HWQual/control.auto is run a constraint is violated and failure mtri_sec_triangle_setup >= 10 is thrown. This is due to some x86 boards doing vertex shading in the CPU. This exception is considered harmless until a waiver system is developed.  

Autotest graphics_O3DSelenium

- tries to run O3D tests inside of chrome
- failed to run and is known to be broken

Autotest graphics_Piglit

- tests OpenGL implementation. Does not currently work on OpenGL ES only systems like Tegra2
- about 1000-2000 targeted subtests
- usually a few hundred failures
- mostly care about counts of passes/failures over time like perf test
- runs about 12-18 minutes, but initial copying dep-piglit.tar.bz2 takes 5-10 minutes that seem like a hang. Need to fix this.
- generates html page with detailed results in
  /tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html 
- can compare two test runs against each other via
  ./piglit-summary-html.py summary/compare results/baseline.results results/current.results
- can be build locally from piglit.tar.gz via cmake .;make
- the numbers of passes does fluctuate by about 10 over time, so some intermittency
- about 10 test binaries crash and are blacklisted to not be reported
- sometimes (1 in about 2 weeks of test runs) fbo_depth_sample_compare takes gpu/system with it when it crashes
2011-05-04T11:06:02.995237-07:00 localhost kernel: [   75.994363] fbo-depth-sampl[4250]: segfault at 0 ip (null) sp 7fc7abac error 14 in card0[75f97000+4000]
2011-05-04T11:06:03.040387-07:00 localhost crash_reporter[4251]: Received crash notification for fbo-depth-sample-compare[4250] sig 11 (developer build - not testing - always dumping)
2011-05-04T11:06:03.234936-07:00 localhost crash_reporter[4251]: Stored minidump to /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.dmp
2011-05-04T11:06:03.235566-07:00 localhost crash_reporter[4251]: Leaving core file at /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.core due to developer image
2011-05-04T11:06:17.742377-07:00 localhost kernel: [   90.742080] [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung
2011-05-04T11:06:17.742439-07:00 localhost kernel: [   90.742100] render error detected, EIR: 0x00000000
2011-05-04T11:06:17.742471-07:00 localhost kernel: [   90.742168] [drm:i915_do_wait_request] *ERROR* i915_do_wait_request returns -5 (awaiting 29946 at 29941)
2011-05-04T11:06:18.330692-07:00 localhost crash_reporter[4328]: Received crash notification for Xorg[3498] sig 6 (developer build - not testing - always dumping)

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_Piglit --use_emerged 
[...]
INFO    : Test results:
-------------------------------------
graphics_Piglit                 PASS
graphics_Piglit/graphics_Piglit PASS
  count_subtests_fail           239
  count_subtests_pass           1040
  count_subtests_skip           687
  count_subtests_warn           14
-------------------------------------
Total PASS: 2/2 (100%)

No crashes detected during testing.
Elapsed time: 13m31s

>>> Details stored under /tmp/run_remote_tests.8gjT

Point browser to
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html

Autotest graphics_SanAngeles

- draws a bunch of very coarse 3d models

This test runs the San Angeles Observation GPU benchmark. This benchmark uses
a minimal and portable framework to generate a small demo program. It exercises
basic features of OpenGL like vertex arrays, color arrays, and lighting. It
also uses objects generated using procedural algorithms.

This test is a benchmark. It will fail if it fails to complete.

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_SanAngeles --use_emerged          
[...]
INFO    : Test results:
---------------------------------------------
graphics_SanAngeles                     PASS
graphics_SanAngeles/graphics_SanAngeles PASS
  frames_per_sec_rate_san_angeles       2.1
---------------------------------------------
Total PASS: 2/2 (100%)

No crashes detected during testing.
Elapsed time: 2m17s
>>> Details stored under /tmp/run_remote_tests.vDwN

Autotest graphics_TearTest

- copies glbench as dependency over
- runs deps/glbench/teartest

- This test will fail if there is tearing in the two vertical lines that are
scrolling horizontally.

This is a semi-automated test that displays vertical lines scrolling
horizontally and asks the user if tearing was observed.  Three variants are
available:
  * using uniform update.  This tests that glSwapInterval function performs as
    expected.
  * using full texture update.  This tests that CPU-GPU interaction is properly
    synchronized in the driver.
  * using pixmap to texture extension.  This tests that pixmap to texture
    extension is properly synchronized.

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_TearTest --use_emerged 
[...] - some user interaction
INFO    : Test results:
-----------------------------------------
graphics_TearTest                   PASS
graphics_TearTest/graphics_TearTest PASS
-----------------------------------------
Total PASS: 2/2 (100%)

No crashes detected during testing.
Elapsed time: 1m26s
>>> Details stored under /tmp/run_remote_tests.SHxS

Autotest graphics_WebGLConformance

This test runs the WebGL conformance tests: http://www.khronos.org/webgl/wiki/Testing/Conformance
- currently uses release version 1.0.0
- opens copy of above website in chrome and executes the html page with hundreds of targeted tests
- all suite tests must pass, but there is the ability to waive failures in the python script
- javascript is patched to report individual results
- suite is currently blocked in autotest regression runs by hangs/crashes in browser

Sample output

Doesn't compile any code, so no GRAPHICS_BACKEND needed (but no harm in using it). When the browser doesn't crash/hang it should look like this.
(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.80 graphics_WebGLConformance/control
[...]
INFO    : Test results:
---------------------------------------------------------
graphics_WebGLConformance                           PASS
graphics_WebGLConformance/graphics_WebGLConformance PASS
  count_tests_fail                                  352
  count_tests_pass                                  5064
  count_tests_timeout                               2
  waived_url_000                                    conformance/tex-image-and-sub-image-2d-with-array-buffer-view.html : 192 failures (192 waived)
  waived_url_001                                    conformance/tex-image-with-format-and-type.html : 12 failures (12 waived)
  waived_url_002                                    conformance/texture-npot.html : 12 failures (12 waived)
  waived_url_003                                    conformance/glsl-conformance.html : 1 failures (1 waived)
  waived_url_004                                    conformance/tex-image-and-sub-image-2d-with-image.html : 8 failures (8 waived)
  waived_url_005                                    conformance/copy-tex-image-and-sub-image-2d.html : 34 failures (34 waived)
  waived_url_006                                    conformance/gl-clear.html : 4 failures (4 waived)
  waived_url_007                                    conformance/more/functions/readPixelsBadArgs.html : 1 failures (1 waived)
  waived_url_008                                    conformance/more/conformance/webGLArrays.html : 1 failures (1 waived)
  waived_url_009                                    conformance/gl-teximage.html : 46 failures (46 waived)
  waived_url_010                                    conformance/texture-active-bind.html : 4 failures (4 waived)
  waived_url_011                                    conformance/read-pixels-test.html : 3 failures (3 waived)
  waived_url_012                                    conformance/gl-object-get-calls.html : 2 failures (2 waived)
  waived_url_013                                    conformance/point-size.html : 1 failures (1 waived)
  waived_url_014                                    conformance/texture-formats-test.html : 4 failures (4 waived)
  waived_url_015                                    conformance/texture-complete.html : 1 failures (1 waived)
  waived_url_016                                    conformance/tex-image-and-sub-image-2d-with-video.html : 8 failures (8 waived)
  waived_url_017                                    conformance/context-lost-restored.html : 2 failures (2 waived)
  waived_url_018                                    conformance/tex-image-and-sub-image-2d-with-image-data.html : 16 failures (16 waived)
---------------------------------------------------------
Total PASS: 2/2 (100%)

Crashes detected during testing:
---------------------------------------------------------
chrome sig 6
  graphics_WebGLConformance/graphics_WebGLConformance
---------------------------------------------------------
Total unique crashes: 1
>>> Details stored under /tmp/run_remote_tests.iumX

Autotest graphics_WindowManagerGraphicsCapture

- This test verifies the window manager can capture graphics from applications.
- This test fails if application screen shots cannot capture the screen output.
- draws less than 10 full screen color frames of a dot/sphere
- copies dep-glbench.tar.bz2 as dependency over
- runs deps/glbench/windowmanagertest
- converts/resizes images to 100x100 pixels
- runs perceptualdiff to compare with reference images
- runs about 30 seconds
- fairly stable test

Sample Output

ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_WindowManagerGraphicsCapture --use_emerged
[...]
INFO    : Test results:
---------------------------------------------------------------------------------
graphics_WindowManagerGraphicsCapture                                       PASS
graphics_WindowManagerGraphicsCapture/graphics_WindowManagerGraphicsCapture PASS
---------------------------------------------------------------------------------
Total PASS: 2/2 (100%)

No crashes detected during testing.
Elapsed time: 0m36s
>>> Details stored under /tmp/run_remote_tests.Lb4p



Comments