Running Chrome Frame Tests

Google Chrome Frame is no longer supported and retired as of February 25, 2014. For guidance on what you need to know as a developer or IT administrator, please read our developer FAQs for Chrome Frame.




This document describes how to build and run Chrome Frame’s set of tests. This can be handy when a try server complains or a builder breaks unexpectedly.

Prerequisites

To build and run Chrome Frame’s tests, you need to be able to build Chromium on Windows following the instructions here:

http://www.chromium.org/developers/how-tos/build-instructions-windows

As of r121750, Chrome Frame projects are no longer included in chrome.sln. In order to build Chrome Frame and its associated tests, the current (as of 16/02/2012) recommended approach is to locally edit src/build/some.gyp as follows:

some.gyp

# Copyright (c) 2011 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
{
  'targets': [
    {
      'target_name': 'some',
      'type': 'none',
      'dependencies': [
        # This file is intended to be locally modified. List the targets you use
        # regularly. The generated some.sln will contains projects for only
        # those targets and the targets they are transitively dependent on. This
        # can result in a solution that loads and unloads faster in Visual
        # Studio.
        #
        # Tip: Create a dummy CL to hold your local edits to this file, so they
        # don't accidentally get added to another CL that you are editing.
        #
        # Example:
        '../chrome_frame/chrome_frame.gyp:*',
        '../chrome/chrome.gyp:chrome',
        '../chrome/installer/mini_installer.gyp:mini_installer',
      ],
    },
  ],
}

Once the file has been edited, you need to run gclient runhooks to regenerate your project files.

Once that has completed, the solution file src/build/some.sln will contain all the projects needed to build Chrome Frame and its tests.

Note for git users: non-checked in locally edited files and git don't play well together. The ever resourceful evanm@ suggested the following work around to keep git happy:

git update-index --assume-unchanged src/build/some.gyp


Description of test executables

chrome_frame_tests.exe

This contains a set of integration tests that exercise the full Chrome Frame stack. This means they run and automate one or more Internet Explorer instances that host Chrome Frame which starts up Chrome. They also start up a local web server to serve requests.

Several of the tests in chrome_frame_tests.exe use accessibility APIs (MSAA) to automate IE, doing things like clicking buttons on download dialogs. As such, they make assumptions about the current focus state of IE dialogs and fail if focus changes unexpectedly or if an interactive user clicks buttons on dialogs unexpectedly. As such, it is recommended that you not interact with the machine while chrome_frame_tests are running.

These tests do not currently run on the try servers.

How to run

  1. Open some.sln and build chrome_frame\chrome_frame_tests
  2. Run chrome_frame_tests.exe from the build output folder as an Admin

Notes

chrome_frame_tests.exe when run from the build folder starts off by registering (machine-wide) the Chrome Frame DLL in the build folder. It needs to be run as admin to do this, and if it fails to register, it will print the message:

[0216/152532:FATAL:test_utils.cc(126)] DLL registration failed (exit code: 0xa00
04005, command: "chrome_frame_tests.exe" --call-registration-entrypoint "E:\src\
chromium\src\build\Debug\npchrome_frame.dll" DllRegisterServer). Make sure you a
re running as Admin.

When this happens, while the tests will continue to run, they may not be testing the Chrome Frame instance you expect them to. To avoid this, always run the tests as admin.

chrome_frame_unittests.exe

This contains the unit tests for Chrome Frame. It can be run stand-alone and runs quite quickly. These tests run on the try servers.

How to run

  1. Open some.sln and build chrome_frame\chrome_frame_unittests
  2. Run chrome_frame_unittests.exe from the build folder.

chrome_frame_reliability_tests.exe

This is the test binary run on chromebot as part of the automated reliability tests. It is little more than a page cycler for IE running chrome frame. You can run it locally if you like, as with chrome_frame_tests.exe it registers CF at startup so needs to be run as admin.

chrome_frame_reliability_tests do not run on the try servers.

How to run

  1. Open some.sln and build chrome_frame\chrome_frame_reliability_tests
  2. Run chrome_frame_reliability_tests.exe from the build fold as admin (see above).

chrome_frame_net_tests.exe

This test is similar to Chrome’s net_unittests, it exercises Chrome-initiated network requests when run through the Chrome Frame network stack, i.e. when run serving network requests via IE over the automation interface, see e.g. (URLRequestAutomationJob). These tests need to be run as admin, since they also register Chrome Frame at startup.

This test works by setting up, in the test process, an environment that is as close as possible to that used by Chrome’s net tests except driven via a fake ExternalTab that emulates Chrome’s startup. Once setup, the tests run in a similar fashion to Chrome’s net tests, except that url requests are handled by Chrome Frame running in an IE instance over the automation interface.

Given that CF’s net tests attempt to start up pieces of Chrome outside of the norm (ie. outside of chrome.exe), it is fairly sensitive to new inter-module dependencies or invalid assumptions about startup order of different Chrome components. Most of the time when these tests break, they are being tickled by bad assumptions about startup order (code that assumes and doesn’t check that some subsystem of Chrome is initialized).

If debugging chrome_frame_net_tests, note that the entry point lives in src\chrome_frame\test\fake_external_tab.cc.

chrome_frame_net_tests run on the try servers.

How to run

  1. Open chrome.sln and build chrome_frame\chrome_frame_net_tests 
  2. Run chrome_frame_net_tests.exe from the build folder as admin (see above).

Notes

NOTE THESE TESTS CURRENTLY DO NOT RUN ON MACHINES WITH IE9 INSTALLED. THIS IS BEING WORKED ON, BUT IN THE MEAN TIME A VM WITH IE8 OR LESS OR A CODE CHANGE TO main() IN fake_external_tab.cc is required.

SSL setup

If you plan to run the Chrome Frame network tests with the host network stack (chrome_frame_net_tests.exe), you need to import the test root CA certificate to your trusted root CA store. 

To do this on Windows XP:
  1. double-click the test root CA certificate file (net/data/ssl/certificates/root_ca_cert.crt in the source tree), this opens the Certificate Windows dialog
  2. click Install Certificate
  3. click Next
  4. select the option 'Place all certificate in the following store'
  5. click Browse.., select 'Trusted Root Certificate Authorities', click OK
  6. click Next and Finish
To do this on Windows Vista or Windows 7:
  1. in the Start menu, type "cmd" in the Start Search box (Note: note the Run... box) and then type ctrl-shift-Enter to run the Command Prompt as Administrator
  2. in the Command Prompt, cd into net\data\ssl\certificates in the source tree, and run the root_ca_cert.crt file
  3. in the "Open File - Security Warning" dialog, click OK
  4. in the Certificate dialog, click Install Certificate
  5. In the Certificate Import Wizard, click Next
  6. select the option "Place all certificates in the following store"
  7. click Browse.., check the "Shown physical stores" box, expand "Trusted Root Certification Authorities", select "Local Computer", click OK
  8. verify that the "Certificate store" field says "Trusted Root Certification Authorities\Local Computer"
  9. click Next and Finish

How to run on a VM / test machine

Before anything else, be aware that running a component=shared_library build on another machine is a tricky affair. If you don't want to worry about getting the right version of the CRT on your test machine (hint: if you don't know what the CRT is, follow the advice to come after the close parens right here:), it's much simpler to use the component=static_library build.

Running some of Chrome Frame's tests from a different location than the build folder in the source tree can be annoying. For example, like Chrome's net tests, CF's net tests makes a number of assumptions about the directory structure it lives in. To ease running on a VM, the indomitable joi@ put together a poor man's trybot script that copies all the needed files to a given directory. Running this script from a mounted share that contains your build output directory on a VM is a common use case.

The script can be found at src/chrome_frame/test/poor_mans_trybot.bat. Sample use case from a virtual PC vm:

Using poor_mans_trybot.bat

net use D: \\tsclient\D\src\chromium
cd D:
src\chrome_frame\test\poor_mans_trybot.bat Debug



In addition, if you build a version of Chrome Frame that dynamically links the MSVCRT (aka the component build) then you may get a SxS error. SxS errors are really annoying to work around. You can work around them by:

  1. creating a folder named  Microsoft.VC90.DebugCRT next to the location of your test executables. If you used the poor man's trybot script from above, this will be c:\trybot\build\Debug. 
  2. Create a file named Microsoft.VC90.DebugCRT.manifest in this directory with the following contents:

Microsoft.VC90.DebugCRT.manifest contents

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
    <noInheritable></noInheritable>
    <assemblyIdentity type="win32" name="Microsoft.VC90.DebugCRT" version="<VERSION>" processorArchitecture="x86"></assemblyIdentity>
    <file name="msvcr90d.dll"></file>
    <file name="msvcp90d.dll"></file>
    <file name="msvcm90d.dll"></file>
</assembly>


where <VERSION> is the version number of the MSVCRT you linked against. Note that this may be different from the one included with your copy of MSVC, as many of these can live on a Windows system. To figure out which version your app expects, you can check the Application log in the Event Viewer (type "Event Viewer in the Win7 start menu) to discover the missing version. Then you have to go and find that version on your system and copy the three DLLs listed above into the Microsoft.VC90.DebugCRT directory.




Comments