For Developers‎ > ‎

Debugging with Crash Keys

Chrome is client-side software, which means that sometimes there are bugs that can occur only on users' machines ("in production") that cannot be reproduced by QA or engineering. When this happens, it's often helpful to gather bug-specific data from production to help pinpoint the cause of the crash. The crash key logging system is a generified method to help do that. 

High-Level Overview

The core of the crash key logging system is in base/debug/crash_logging.h, which allows crash keys to be added in virtually any module. At the most basic level, it is a system that provides a map structure to record key-value pairs. These pairs are uploaded as POST form-multipart data when Breakpad uploads a minidump to the Google crash reporting system. (The data therefore is only accessible to those with access to crash reports internally at Google). Before a key can be used, it must be registered in the list of allowed crash keys, specified at chrome/common/crash_keys.h. Pre-registration is required so that space can be allocated for it prior to any crash happening, since in a crashed process context, dynamic memory allocation routines are not necessarily safe to use. Finally, the platform-specific Breakpad implementations each have a way to record, without allocating, the key and value pairs when they are set using crash_logging.h.

The crash key system is used to report some common pieces of data, as well as things that happen in exceptional cases: the URL of the webpage, command line switches, active extension IDs, GPU vendor information, experiment/variations information, etc. All this metadata is set, stored, and reported using the crash logging system described here.

Getting Started with a Single Key-Value Pair

Imagine you are investigating a crash, and you want to know the value of some variable when the crash occurs; the crash key logging system enables you to do just that.

1. Define and register a new crash key.

In chrome/common/crash_keys.h, define a new extern const char kYourCrashKey[]; and document it, including a link to the bug if it will be a temporary key. Then, in the corresponding crash_keys.cc, declare the value of your constant. This value is the name by which the data will be accessible after it is uploaded. After it has been declared, it needs to be registered with the system in RegisterChromeCrashKeys(). For a single key-value pair, find the C array initialization and add a new pair for { kYourCrashKey, <size> }. Because the crash key system cannot allocate memory (see above), it uses an array of fixed-size C strings called "chunks," into which larger values are split for storage.

There are three size classes for registering keys: kSmallSize, kMediumSize, and kLargeSize. The small size is guaranteed to never be chunked on any platform; the other sizes may be chunked depending on the platform-specific implementation. Keep in mind that crash key storage is limited, so in almost all circumstances, try to use the small size.

2. Set the crash key.

Now that the key has been registered, it can be used in code to gather data from production. The routines are defined in chrome/common/crash_keys.h and the most basic usage is base::debug::SetCrashKeyValue() and base::debug::ClearCrashKey(). They work as one would expect. It is idiomatic to use the key constant defined in crash_keys.h, unless doing so would be a layering violation (e.g. reporting a crash key from content/), since the keys are defined in chrome/.

In addition, there is ScopedCrashKey to allow you to set the key-value pair only for the duration of a function call.

3. Seeing the data.

Using http://go/crash (internal only), find the crash report signature related to your bug, and click on the "N of M" reports link to drill down to report-specific information. From there, select a report and go to the "Fields" tab to view all the crash key-value pairs.

Advanced Topics: Stack Traces

Now imagine a scenario where you have a use-after-free. The crash reports coming in do not indicate where the object being used was initially freed, however, just where it is later being dereferenced. To make debugging easier, it would be nice to have the stack trace of the destructor, and the crash key system works for that, too.

1. Define and register a crash key.

As above, define and register a crash key with the system. If you expect a deep stack trace, you may want to use kMediumSize or kLargeSize to hold the trace.

2. Set the crash key.

To set a stack trace to a crash key, use one of the specialized routines from crash_logging.h. This will typically be done like so:

Usemeafterfree::~Usemeafterfree() {
  base::debug::SetCrashKeyToStackTrace(kYourCrashKey,
                                       base::debug::StackTrace());
}

3. Seeing the data.

Unlike with the previous example, a stack trace will just be a string of hexadecimal addresses. To turn the addresses back into symbols use, http://go/crsym (internal instance of https://github.com/chromium/crsym/). Using the Crash Key input type, give it a crash report ID and the name of your crash key. Crsym will then fetch the symbol data from the internal crash processing backends and return a formatted, symbolized stack trace.
Comments