For Developers‎ > ‎

Telemetry: Introduction

Telemetry 101

Telemetry is Chrome's performance testing framework. It allows you to perform arbitrary actions on a set of web pages and report metrics about it. The framework abstracts:

  • Launching a browser with arbitrary flags on any platform.
  • Opening a tab and navigating to the page under test.
  • Fetching data via the Inspector timeline and traces.
  • Using Web Page Replay to cache real-world websites so they don’t change when used in benchmarks.

Design Principles

  • Write one performance test that runs on all platforms - Windows, Mac, Linux, Chrome OS, and Android for both Chrome and ContentShell.

  • Runs on browser binaries, without a full Chromium checkout, and without having to build the browser yourself.

  • Use WebPageReplay to get repeatable test results.

  • Clean architecture for writing benchmarks that keeps measurements and use cases separate.

  • Run on non-Chrome browsers for comparative studies (not yet implemented).

Should my test use Telemetry?

It depends, what are you trying to measure? Are you trying to verify correctness?  If yes, Telemetry is not for you, but browser tests are your friend.


Telemetry is designed for performance tests that:

  • Can be written with pure JS/HTML/CSS.

  • Use real world content (rather than synthetic).

  • Could be run on other browsers, not just Chrome.

Code Concepts

Telemetry is a Python wrapper around Chrome's DevTools Remote Debugging Protocol.

The Telemetry framework lives in src/tools/telemetry/ and performance benchmarks live in src/tools/perf/.

Benchmark

Telemetry has an opinionated definition of a benchmark. A benchmark is a wrapper that encapsulates a measurement, a page set, and a set of optional browser options. You can run a benchmark with a short, simple command, and the test configuration is baked into the benchmark to ensure its consistency across machines.

Measurement

A measurement (called PageTest or PageMeasurement in the code) is "something that can be run on a page." Given a page, a measurement evaluates the page’s performance. It can report many results, including numbers, histograms, traces, or pass/fail.

Page Set

A page set is a collection of URLs, which can be live sites, WebPageReplay archives, or local HTML files. Pages also have actions associated with them, such as clicking a button, scrolling the page, or performing a swipe gesture.

Next Steps

Contact Us or Follow Along

If you have questions, please email telemetry@chromium.org.

You can keep up with Telemetry related discussions and code reviews by joining the telemetry group.

Frequently Asked Questions

How do I navigate Chrome to a certain URL?

To visit a URL, all you need to do is add the URL to a page set. For more information on maintaining page sets, see managing page sets.

I get an error when I try to use recorded page sets?

The recordings are not included in the Chromium source tree. If you are a Google partner, run gsutil config to authenticate, then try running the test again. If you don't have gsutil installed on your machine, you can find it in src/tools/telemetry/third_party/gsutil/gsutil.


If you are not a Google partner, you can run on live sites with --use-live-sites or record your own page set archive.


I get mysterious errors about device_forwarder failing?


Your forwarder binary may be outdated. If you have built the forwarder in src/out that one will be used; if there isn't anything there Telemetry will default to downloading a pre-built binary. Try re-building the forwarder, or alternatively wiping the contents of src/out/ and running run_benchmark, which should download the latest binary.


I'm having problems with keychain prompts on Mac. Please help?

Make sure that your keychain is correctly configured.