Web compat analysis tools

When deprecating or changing web-exposed behavior, it's often important to get a clear understanding of the compatibility impact.  We have a variety of tools available to you depending on the scenario. This page is designed to help you choose the appropriate tool (in decreasing order of usage). For questions and discussion, e-mail platform-predictability@chromium.org.

UseCounter

UseCounter is a class in blink which is used to record per-page anonymous aggregated metrics on feature usage, often via the [Measure] idl attribute.  Results are shown on chromestatus.com (with more detailed break-downs available to Google employees via the UMA dashboard).  In the vast majority of cases, compat tradeoffs are made entirely based on UseCounter data.

Pros: 
  • Reflects real Chrome usage - should be the primary source of all compat discussions in blink
  • Has huge coverage - reflects a wide fraction of all usage of Chrome
Cons: 
  • Requires at least a couple weeks to get any useful data (several months to roll out to stable)
  • Can't be used to get specific URLs.
  • Blind to scenarios like where UMA tends to be disabled (eg. enterprises)

Static web search

Often it's useful to find examples of specific coding patterns in order to understand the likely failure modes and formulate migration guidance.  Searching code repositories like GitHub can be useful, as well as technical web search engines like nerdydata.com.

Pros: 
  • Simple, easy to use
  • Results in specific URLs for further analysis
Cons: 
  • Strictly a dumb static search, can't reliably find all uses of an API (especially due to Javascript minifiers which generate code like "a[b]()").

The HTTP Archive

A slightly more advanced form of static web search is to use the HTTP Archive, a database of the top 500k websites, updated by a crawl twice a month.  See HTTP Archive for web compat decision making for details on using it for compat analysis.

Pros: 
  • Can provide an absolute measure of risk ("only 10 sites in the top 500k appear to use this API").
  • Now includes UseCounter data, so can go beyond simple static search to some dynamic behavior
Cons: 
  • Only captures behavior triggered during page load
  • Only reflects the home page of the top 500k sites
  • Analysis is more involved

GitHub and stackoverflow deprecation warning search

Once a deprecation warning has been landed (or made it to stable), it can be extremely informative to search GitHub issues (or other developer help sites like stackoverflow) for discussion of the warning generated on the console (eg. by searching for the chromestatus ID present in the warning).

Pros:
  • Helps identify the real-world pain experienced by developers
  • Provides an avenue for outreach, and can build goodwill
Cons:
  • Long turn-around time
  • Lossy - absence of signal doesn't really indicate an absence of risk

GitHub code search

Most of the popular libraries and frameworks are present on GitHub. Searching for potentially impacted code can be useful in better understanding the risk.

Pros:
  • Provides unobfuscated access to code
  • Provides an avenue to engage with developers to better understand their use cases and their ability to apply mitigations.
Cons:
  • Doesn't provide much signal on magnitude of impact
  • Supports only either simple static searches.

On-demand crawl

Occasionally it's useful to search top sites for a specific behavior (without landing a UseCounter and waiting for the data to show up in HTTP Archive).  For advanced cases like this we can run a custom chromium build on the telemetry cluster to crawl the top 10k (or more) sites and record whatever we like (with a temporary UseCounter).  See Using Cluster Telemetry for UseCounter analysis for details.

Pros: 
  • Can have fast turn-around time (hours)
  • Usually used just for page load, but can be extended to trigger other interactions (scroll, clicking on links, etc.).
Cons: 
  • Limited scope
Comments