Autotest Graphics Documentation

Keywords: This testing document describes how to run GLBench, TearTest, Piglit, WebGLConformance and other tests and benchmarks on ChromeOS and ChromiumOS devices.

Source code location

Python sources are under src/third_party/autotest/files/client/site_tests

Binary dependencies are under src/third_party/autotest/files/client/deps

A note on building binaries

The ebuild system knows if a target knows either OpenGL or OpenGL ES, or even both. If only pre-compiled binaries are used there should not be much of a problem with running tests. When editing tests after using the cros_workon script the default system behavior is to recompile the binaries once more without invoking the ebuild system. Unfortunately this means USE="opengles" flags are not honored. Instead the easy way is to specify the proper switch via am environment variable GRAPHICS_BACKEND=OPENGLES./run_remote_tests.sh that gets directly passed to the Makefile. This is somewhat hacky. To use the ebuild system to generate the binaries one has to do a painfully slow USE=opengles emerge-tegra2_seaboard chromeos-base/autotest followed by ./run_remote_tests.sh --use_emerged [...]. Don't forget the --use_emerged flag as it is not default.

Autotest graphics_GLAPICheck

• two different binaries can be made

-- GRAPHICS_BACKEND=OPENGL or OPENGLES

• runs 25 seconds

• sample results

• for OPENGL creates output below and parses some samples in python to see if they are there and if SUCCEED: run to the end is there

GL_VERSION = 3.2.0 NVIDIA 195.36.24

GL_EXTENSIONS = GL_ARB_color_buffer_float GL_ARB_compatibility GL_ARB_copy_buffer GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_draw_buffers GL_ARB_draw_elements_base_vertex [...] NV-GLX NV-CONTROL Generic

Event Extension SHAPE MIT-SHM XInputExtension XTEST BIG-REQUESTS SYNC XKEYBOARD XC-MISC SECURITY XINERAMA XFIXES RENDER RANDR XINERAMA Composite DAMAGE

SUCCEED: run to the end

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.71.194 graphics_GLAPICheck --use_emerged

Initiating first contact with remote host

[...]

INFO : Test results:

graphics_GLAPICheck PASS

graphics_GLAPICheck/graphics_GLAPICheck PASS

Total PASS: 2/2 (100%)

No crashes detected during testing.

Elapsed time: 0m25s

Autotest graphics_GLBench

• draws green/ref/purple/fractal rectangles

• runs about 5 minutes

This benchmark executes glbench, a graphics benchmark designed to time how long

various graphic intensive activities take, which includes measuring:

• fill rate

• blended

• opaque

-Z reject rate

-triangle rate

• no cull

• half cull (half triangles backface culled)

• full cull (mix of back face and degenerates)

• blend rate

• texture fetch

• nearest

• bilinear

• trilinear

• compute

• vertex shader

• pixel shader

• *fragement shader to test ddx and ddy

• attribute fetch

• color depth stencil test

• *state change

• texture upload

• read back

• does MD5 checksums of some but not all images

-- deps/glbench/src/checksums

-- can be re-generated with -save option

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGLES ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.27 graphics_GLBench --use_emerged

[...]

INFO : Test results:

graphics_GLBench PASS

graphics_GLBench/graphics_GLBench PASS

1280x768_fps_compositing 11.18

1280x768_fps_no_fill_compositing 41.96

mpixels_sec_clear_color 490.74

mpixels_sec_clear_colordepth 323.88

mpixels_sec_clear_colordepthstencil 281.45

mpixels_sec_clear_depth 969.71

mpixels_sec_clear_depthstencil 647.34

mpixels_sec_fill_solid 285.1

mpixels_sec_fill_solid_blended 128.55

mpixels_sec_fill_solid_depth_neq 973.2

mpixels_sec_fill_solid_depth_never 1189.08

mpixels_sec_fill_tex_bilinear 149.56

mpixels_sec_fill_tex_nearest 203.7

mpixels_sec_fill_tex_trilinear_linear_01 164.63

mpixels_sec_fill_tex_trilinear_linear_04 nan

mpixels_sec_fill_tex_trilinear_linear_05 nan

mpixels_sec_fill_tex_trilinear_nearest_05 nan

mpixels_sec_pixel_read 6.72

mpixels_sec_pixel_read_2 6.72

mpixels_sec_pixel_read_3 6.74

mpixels_sec_varyings_shader_1 270.68

mpixels_sec_varyings_shader_2 144.33

mpixels_sec_varyings_shader_4 73.57

mpixels_sec_varyings_shader_8 24.29

mpixels_sec_yuv_shader_1 nan

mpixels_sec_yuv_shader_2 47.82

mpixels_sec_yuv_shader_3 93.36

mpixels_sec_yuv_shader_4 136.15

mtexel_sec_texture_update_teximage2d_1024 149.85

mtexel_sec_texture_update_teximage2d_128 80.92

mtexel_sec_texture_update_teximage2d_1536 146.05

mtexel_sec_texture_update_teximage2d_2048 149.47

mtexel_sec_texture_update_teximage2d_256 132.51

mtexel_sec_texture_update_teximage2d_32 10.29

mtexel_sec_texture_update_teximage2d_512 142.05

mtexel_sec_texture_update_teximage2d_768 147.56

mtexel_sec_texture_update_texsubimage2d_1024 150.94

mtexel_sec_texture_update_texsubimage2d_128 81.59

mtexel_sec_texture_update_texsubimage2d_1536 147.2

mtexel_sec_texture_update_texsubimage2d_2048 149.03

mtexel_sec_texture_update_texsubimage2d_256 130.59

mtexel_sec_texture_update_texsubimage2d_32 10.09

mtexel_sec_texture_update_texsubimage2d_512 113.54

mtexel_sec_texture_update_texsubimage2d_768 150.0

mtri_sec_triangle_setup 30.34

mtri_sec_triangle_setup_all_culled 49.89

mtri_sec_triangle_setup_half_culled 44.91

mvtx_sec_attribute_fetch_shader 149.51

mvtx_sec_attribute_fetch_shader_2_attr 149.34

mvtx_sec_attribute_fetch_shader_4_attr 141.36

mvtx_sec_attribute_fetch_shader_8_attr 106.71

us_swap_swap 19464.19

Total PASS: 2/2 (100%)

No crashes detected during testing.

Elapsed time: 8m58s

>>> Details stored under /tmp/run_remote_tests.purz

The "nan" in this example are passed because of knownbad output images. For more more details and failures look at

(cros-chroot) ihf@ql ~/trunk/src/scripts $ more /tmp/run_remote_tests.purz/graphics_GLBench/graphics_GLBench/summary.txt

Note on HW Qual failures

Sometimes when site_tests/suite_HWQual/control.auto is run a constraint is violated and failure mtri_sec_triangle_setup >= 10 is thrown. This is due to some x86 boards doing vertex shading in the CPU. This exception is considered harmless until a waiver system is developed.

Autotest graphics_O3DSelenium

• tries to run O3D tests inside of chrome

• failed to run and is known to be broken

Autotest graphics_Piglit

• tests OpenGL implementation. Does not currently work on OpenGL ES only systems like Tegra2

• about 1000-2000 targeted subtests

• usually a few hundred failures

• mostly care about counts of passes/failures over time like perf test

• runs about 12-18 minutes, but initial copying dep-piglit.tar.bz2 takes 5-10 minutes that seem like a hang. Need to fix this.

• generates html page with detailed results in

/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html

• can compare two test runs against each other via

./piglit-summary-html.py summary/compare results/baseline.results results/current.results

• can be build locally from piglit.tar.gz via cmake .;make

• the numbers of passes does fluctuate by about 10 over time, so some intermittency

• about 10 test binaries crash and are blocklisted to not be reported

• sometimes (1 in about 2 weeks of test runs) fbo_depth_sample_compare takes gpu/system with it when it crashes

2011-05-04T11:06:02.995237-07:00 localhost kernel: [ 75.994363] fbo-depth-sampl[4250]: segfault at 0 ip (null) sp 7fc7abac error 14 in card0[75f97000+4000] 2011-05-04T11:06:03.040387-07:00 localhost crash_reporter[4251]: Received crash notification for fbo-depth-sample-compare[4250] sig 11 (developer build - not testing - always dumping) 2011-05-04T11:06:03.234936-07:00 localhost crash_reporter[4251]: Stored minidump to /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.dmp 2011-05-04T11:06:03.235566-07:00 localhost crash_reporter[4251]: Leaving core file at /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.core due to developer image 2011-05-04T11:06:17.742377-07:00 localhost kernel: [ 90.742080] [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung 2011-05-04T11:06:17.742439-07:00 localhost kernel: [ 90.742100] render error detected, EIR: 0x00000000 2011-05-04T11:06:17.742471-07:00 localhost kernel: [ 90.742168] [drm:i915_do_wait_request] *ERROR* i915_do_wait_request returns -5 (awaiting 29946 at 29941) 2011-05-04T11:06:18.330692-07:00 localhost crash_reporter[4328]: Received crash notification for Xorg[3498] sig 6 (developer build - not testing - always dumping)

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_Piglit --use_emerged

[...]

INFO : Test results:

graphics_Piglit PASS

graphics_Piglit/graphics_Piglit PASS

count_subtests_fail 239

count_subtests_pass 1040

count_subtests_skip 687

count_subtests_warn 14

Total PASS: 2/2 (100%)

No crashes detected during testing.

Elapsed time: 13m31s

>>> Details stored under /tmp/run_remote_tests.8gjT

Point browser to

/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html

Autotest graphics_SanAngeles

• draws a bunch of very coarse 3d models

This test runs the San Angeles Observation GPU benchmark. This benchmark uses

a minimal and portable framework to generate a small demo program. It exercises

basic features of OpenGL like vertex arrays, color arrays, and lighting. It

also uses objects generated using procedural algorithms.

This test is a benchmark. It will fail if it fails to complete.

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_SanAngeles --use_emerged

[...]

INFO : Test results:

graphics_SanAngeles PASS

graphics_SanAngeles/graphics_SanAngeles PASS

frames_per_sec_rate_san_angeles 2.1

Total PASS: 2/2 (100%)

No crashes detected during testing.

Elapsed time: 2m17s

>>> Details stored under /tmp/run_remote_tests.vDwN

Autotest graphics_TearTest

• copies glbench as dependency over

• runs deps/glbench/teartest

• This test will fail if there is tearing in the two vertical lines that are

scrolling horizontally.

This is a semi-automated test that displays vertical lines scrolling

horizontally and asks the user if tearing was observed. Three variants are

available:

* using uniform update. This tests that glSwapInterval function performs as

expected.

* using full texture update. This tests that CPU-GPU interaction is properly

synchronized in the driver.

* using pixmap to texture extension. This tests that pixmap to texture

extension is properly synchronized.

Sample output

(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_TearTest --use_emerged

[...] - some user interaction

INFO : Test results:

graphics_TearTest PASS

graphics_TearTest/graphics_TearTest PASS

Total PASS: 2/2 (100%)

No crashes detected during testing.

Elapsed time: 1m26s

>>> Details stored under /tmp/run_remote_tests.SHxS

Autotest graphics_WebGLConformance

This test runs the WebGL conformance tests: http://www.khronos.org/webgl/wiki/Testing/Conformance

• currently uses release version 1.0.0

• opens copy of above website in chrome and executes the html page with hundreds of targeted tests

• all suite tests must pass, but there is the ability to waive failures in the python script

• javascript is patched to report individual results

• suite is currently blocked in autotest regression runs by hangs/crashes in browser

Sample output

Doesn't compile any code, so no GRAPHICS_BACKEND needed (but no harm in using it). When the browser doesn't crash/hang it should look like this.

(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.80 graphics_WebGLConformance/control

[...]

INFO : Test results:

graphics_WebGLConformance PASS

graphics_WebGLConformance/graphics_WebGLConformance PASS

count_tests_fail 352

count_tests_pass 5064

count_tests_timeout 2

waived_url_000 conformance/tex-image-and-sub-image-2d-with-array-buffer-view.html : 192 failures (192 waived)

waived_url_001 conformance/tex-image-with-format-and-type.html : 12 failures (12 waived)

waived_url_002 conformance/texture-npot.html : 12 failures (12 waived)

waived_url_003 conformance/glsl-conformance.html : 1 failures (1 waived)

waived_url_004 conformance/tex-image-and-sub-image-2d-with-image.html : 8 failures (8 waived)

waived_url_005 conformance/copy-tex-image-and-sub-image-2d.html : 34 failures (34 waived)

waived_url_006 conformance/gl-clear.html : 4 failures (4 waived)

waived_url_007 conformance/more/functions/readPixelsBadArgs.html : 1 failures (1 waived)

waived_url_008 conformance/more/conformance/webGLArrays.html : 1 failures (1 waived)

waived_url_009 conformance/gl-teximage.html : 46 failures (46 waived)

waived_url_010 conformance/texture-active-bind.html : 4 failures (4 waived)

waived_url_011 conformance/read-pixels-test.html : 3 failures (3 waived)

waived_url_012 conformance/gl-object-get-calls.html : 2 failures (2 waived)

waived_url_013 conformance/point-size.html : 1 failures (1 waived)

waived_url_014 conformance/texture-formats-test.html : 4 failures (4 waived)

waived_url_015 conformance/texture-complete.html : 1 failures (1 waived)

waived_url_016 conformance/tex-image-and-sub-image-2d-with-video.html : 8 failures (8 waived)

waived_url_017 conformance/context-lost-restored.html : 2 failures (2 waived)

waived_url_018 conformance/tex-image-and-sub-image-2d-with-image-data.html : 16 failures (16 waived)

Total PASS: 2/2 (100%)

Crashes detected during testing:

chrome sig 6

graphics_WebGLConformance/graphics_WebGLConformance

Total unique crashes: 1

>>> Details stored under /tmp/run_remote_tests.iumX

Autotest graphics_WindowManagerGraphicsCapture

• This test verifies the window manager can capture graphics from applications.

• This test fails if application screen shots cannot capture the screen output.

• draws less than 10 full screen color frames of a dot/sphere

• copies dep-glbench.tar.bz2 as dependency over

• runs deps/glbench/windowmanagertest

• converts/resizes images to 100x100 pixels

• runs perceptualdiff to compare with reference images

• runs about 30 seconds

• fairly stable test

Sample Output

ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_WindowManagerGraphicsCapture --use_emerged

[...]

INFO : Test results:

graphics_WindowManagerGraphicsCapture PASS

graphics_WindowManagerGraphicsCapture/graphics_WindowManagerGraphicsCapture PASS

Total PASS: 2/2 (100%)

No crashes detected during testing.

Elapsed time: 0m36s

>>> Details stored under /tmp/run_remote_tests.Lb4p