Chromium‎ > ‎Chromium Security‎ > ‎Security Bugs--‎ > ‎

Using ClusterFuzz

Searching the test case list


The primary purpose of ClusterFuzz is to find bugs by analyzing crashes. The test case list page provides information on recent crashes and allows users to search based on certain criteria.


By default, when typing into the search box ClusterFuzz will filter based on all displayed information about the test case as well as some additional metadata, such as the name of the fuzzer that found the crash. The general formats for searches by a field name are “field:value” or “field=value”. If you wish to search for a string that may contain spaces, enclose the value you want to search for in quotes. A full list of search fields can be found in the advanced options section.


ClusterFuzz will filter the list to only show fields that match all of your criteria. If you wish to list terms that match any of a number of criteria, separate the terms with “or” (e.g. “impacts:stable or impacts:beta”).

List bugs from a particular fuzzer


If you have taken the time to write a fuzzer for ClusterFuzz, you probably want to know when it finds crashes. Luckily, this is very easy. Simply type the name of your fuzzer into the search box, and browse the listed test cases.

List confirmed security bugs


While triaging incoming bugs from ClusterFuzz, it is often useful to search the list for all confirmed (non-flaky) security bugs. To do this, simply enter “confirmed:yes security:yes” into the search box. This works by filtering the list to only display test cases that match both of those criteria. If you wanted to search for all confirmed security bugs found by a fuzzer named “test_fuzzer”, you could search for “confirmed:yes security:yes test_fuzzer”.

List crashes without associated bugs


It can also be useful to display only crashes that do not yet have bugs filed for them. ClusterFuzz supports the keyword “null” for value searches as an alias for the empty string. You can search for test cases that do not have any bugs filed for them by enter “bug_information:null” into the search box (this is equivalent to a search for bug_information:””).

Using detailed view


If you wish to see additional information about all test cases listed, you can click the “Switch to detailed view” link above the search box in the top-right corner of the test case list page. Additional information includes the name of the fuzzer that found the crash, job type, original file name of the test case, and whether or not it is a stable regression. It also provides access to a few options that make it easy to perform bulk operations on test cases, such as redoing jobs and deletion.

Advanced options


The previous examples do not cover all cases. A full list of fields that you can search by are included below.


Search key name

Description

key

Key that was automatically assigned to the test case by ClusterFuzz

crash_type_and_address

Type of crash (e.g. heap buffer overflow) and crash address.

crash_state

Top frames of the crash and free stacks for the issue

confirmed

Whether or not the bug is fully reproducible (“No” for flaky test cases)

security

Whether or not this test case has security implications

impacts

Does this affect current beta and stable builds?

regressed

Range of chromium revisions that this crash was introduced in

fixed

Range of chromium revisions that this crash was fixed in

bug_information

Bug number associated with this crash

source

Fuzzer that identified the test case or uploader’s email address

stable_regression

Whether or not this crash regressed to a stable build

file_path

Full file path of the test case on the bot that discovered it

group_id

ID of the group assigned to this crash on ClusterFuzz

Viewing test case details


Once you have identified a test case that you wish to investigate further, ClusterFuzz will provide you with detailed information on the crash. It also simplifies some common tasks that you may want to perform, such as filing a bug.

Issue tracker options


To open a new bug from the test case details page, simply click on the “file bug” button at the top of the page, fill out any information you can, and submit the form. For security bugs, you will also be prompted to select the severity. For guidance on how to determine the severity of a bug, see the chromium severity guidelines.


Similarly, you can use the “update bug” button to associate a test case with an existing bug. To do this, you only need to provide the bug number that it should be associated with.


Before filing a bug, you should check to see if an issue already exists for this crash. To do this, you can use the “find bug” button. This searches the issue tracker for bugs that may be related to the crash you are viewing. By default, it will only search open bugs. If you wish to search for potentially closed bugs, or want to filter the list in a different way, you can change this option before searching.

Redo analysis for a bug


Sometimes, ClusterFuzz messes up. This is most common with flaky test cases and test cases that take a long time to process. If ClusterFuzz reports information that you suspect is wrong for a particular test case, such as a poorly minimized test case or incorrect regression range, you can redo certain jobs by clicking the “redo” button. Check the boxes for the jobs you wish to redo, and submit the form.

Delete a test case


If for any reason it makes sense to delete a test case, this can be done from the test case details page. Use the “delete” button and follow the prompts to delete the test case if you are sure that it should be deleted.

Code Search integration


For job types that do not use custom builds, frames in the stack trace will link directly to the relevant chromium code in Code Search. This can be useful when analyzing the crash to try to triage it, or for initial investigation if you are assigned a ClusterFuzz bug.

Regression and fixed analysis


Whenever possible, ClusterFuzz will attempt to calculate a range of Chromium revisions where a crash was introduced. This is referred to as the regression range. This is not attempted when a test case is flaky, or when a custom binary is being used by the job type for the test case.


If you click on the link for a range, you will be taken to a page that lists not just the revision range for chrome, but also for dependencies such as Blink, V8, NaCl, and more. For the chromium revision and certain commonly updated dependencies, such as Blink, there will be links directly to the changelist for the revision range in question. For less commonly updated dependences, you may need to manually find the changelist.


In addition to finding ranges for regressions, ClusterFuzz also attempts to identify ranges where crashes are fixed. For any applicable non-flaky tests, ClusterFuzz will attempt to reproduce the crash in an up-to-date build once per day. If a test case is associated with a chromium bug, ClusterFuzz will automatically update the bug with the fixed range once it has been detected. This can be useful for verifying that a bug has been fixed properly.

Additional requirements


Occasionally, ClusterFuzz test cases will have special requirements that must be taken into consideration while trying to reproduce a test case.

HTTP


Test cases that require HTTP must be hosted on an HTTP server to be reproduced.

Interaction gestures


Tests that require interaction gestures will be accompanied by an interaction gesture string. This string was automatically generated by ClusterFuzz and may include various keyboard inputs, mouse movements and clicks, or other gestures. Sometimes, the gesture causing the crash may be obvious, though other times this can be difficult to reproduce these crashes outside of ClusterFuzz.

Uploading test cases


ClusterFuzz does much more than just fuzzing. When crashes are detected, it performs additional tasks such as test case minimization, finding regression ranges, determining whether a bug affects the current stable and beta branches, checking to see if the bug has been fixed, and more. Obviously, all of these things can be useful even for bugs that were not found by fuzzers running in ClusterFuzz. The test case uploads page allows you to upload test cases to ClusterFuzz for analysis.

Uploading a single test case from a file


To begin uploading a test case, navigate to the test uploads tab in ClusterFuzz and select a file to upload. If your test case requires multiple files, you can upload an archive. If you do this, make sure that the file that should be passed to chrome on the command line contains the string “run” or “index” somewhere in its filename. Single file uploads will be passed to chrome regardless of file name.


You will also need to select a job type that is appropriate for your test case. If your test case can be passed to a chromium build directly on the command line, leave the job type set to “linux_asan_chrome_mp” and upload it. If you need to use an official chrome build, usually for testing certain media codecs, you can use the “linux_asan_chrome_media” job type. Most of the other job types have uses for specific areas of the chromium codebase. If you cannot pass your test cases directly to chrome, check the descriptions on the jobs page to see if something suits your needs. If not, please contact the ClusterFuzz maintainers and see the defining job types section.

Uploading multiple test cases at once


Sometimes, it makes sense to upload multiple test cases to ClusterFuzz at once for simple verification and analysis. To do this, bundle the test cases into a zip, tar.gz, or tar.bz2 archive and select the archive as the file to upload. Check the “Multiple test cases” checkbox and continue the upload using the same process used for a single test case. Any other settings, such as job type, will be applied when testing all files in the archive.


Individual reports will be created for each file in the archive. You can view the reports and see which ones reproduced on the test uploads page. Previously unknown test cases that ClusterFuzz could reproduce will also show up in the test case list on the main page.

Uploading a “raw” test case


The “raw” option allows test cases to be uploaded by providing the contents of a file directly instead of uploading a file. This is useful when dealing with reports that have very short reproduction cases, as reporters will usually just paste the contents directly into the report body in these cases.


Using this option also allows you to select a filename for the file that will be created for the test case. Changing the name is not necessary, but it will default to “test.html”. If a certain file extension or name is necessary to run the test case properly, change the filename to something that will work.

Uploading a “URL” test case


Occasionally, it makes sense to have ClusterFuzz create a test case from a URL rather than a file. This can be useful if crash reports are coming in for a particular URL and you want to test it quickly. Simply enter the URL into the text box, and the bots will pull down a copy of the current page contents. This does not always work consistently, so if it fails you may need to take some time to prepare a test case to upload rather than using this option.

Advanced options

Bug information


If you enter a bug number, ClusterFuzz will automatically update that bug once it has finished processing the test case. When ClusterFuzz detects that the issue has been fixed, it will update the bug again with the fixed range.

Timeout


Time in seconds to run this test case before giving up and terminating the binary. This defaults to whatever the specified timeout is for your selected job type. Only change it to a higher value for a flaky test where you think a larger timeout will help. ClusterFuzz tries each test many times before determining that it is unreproducible, so selecting an excessively high value here will usually only result in a long wait before receiving a report.

Gestures


ClusterFuzz generates random user gestures while fuzzing test cases. To be able to reproduce these simulated gestures, it associates a list of them with each test case. If you want to try to reproduce a test case with a ClusterFuzz gesture string, you can provide one when uploading the test case.

HTTP support


This option should be used if the bot should host the test case on a local HTTP server rather than passing it to chrome directly on the command line. This is not usually necessary, but may be required for certain test cases.

High-end bot


A few of the bots that make up ClusterFuzz are more powerful than the others. They are used to perform some specific tasks such as symbolization. Some test cases may not be able to run properly on a lower-end bot. In these cases, you can choose to analyze the test case on a high-end bot. Please make sure that you only do this when it is absolutely necessary. Try running on a lower-end bot first, and reupload with the high-end bot option only if it did not reproduce and you believe that this will make a difference.

Uploading fuzzers


As the name suggests, fuzzing is the primary purpose of ClusterFuzz. For more information on writing a fuzzer and preparing it for use with ClusterFuzz, see the ClusterFuzz fuzzer guide.

Uploading a simple fuzzer


If your fuzzer has been prepared according to the fuzzer guide, the actual upload process is fairly straightforward. From the fuzzers page, you can fill out the fuzzer upload form. Only a name and the archive that makes up your fuzzer are required.


The fuzzer name must be unique. If you give your fuzzer the same name as another fuzzer, it will update the existing fuzzer to a new version. This is handled this way because, occasionally, advanced options must be changed from fuzzer version to version. Many advanced options are not included in the simple “update fuzzer” form in the fuzzer list. In general, it is a good idea to give your fuzzers a simple name prefixed with something that identifies you.


Next, you should select the archive to upload for your fuzzer. ClusterFuzz supports zip, tar.gz, and tar.bz2 uploads. As explained in the fuzzer guide, it is best to include the string “run” somewhere in the name of the file that should actually be executed to run your fuzzer. Doing this will allow ClusterFuzz to detect it automatically. If you have more than one file with “run” in its name, or if for some reason you are unable to name the file to execute your fuzzer in this way, you can specify the name by explicitly entering an “executable path” in the fuzzer upload form.


You will also have to provide job types that should be used for your fuzzer. Job types are defined on the jobs page and control factors such as which platform the test cases should be processed on, which binary should be used for fuzzing, and much more. The most commonly used job type is “linux_asan_chrome_mp”, which simply passes the test cases to an asanified chromium build on a Linux bot.

Updating a fuzzer to a new version


Usually, to update a fuzzer you simply need to select it in the fuzzer list, select “update fuzzer”, and upload the new file. If the update requires a change to one of the advanced options for the fuzzer, upload it as though it were a new fuzzer and use the same name. After you update the fuzzer, you should check that it is running properly by making sure it is still generating the correct number of test cases by viewing test run output in the test case list. Unless you have a good reason to do so, you should not delete your fuzzer. This can cause some issues for existing test cases associated with your fuzzer.

Uploading a data bundle


You can upload a data bundle for a fuzzer from the fuzzer list page. To do this, find the fuzzer in the fuzzer list either by paging through it or searching for the fuzzer name. When you have identified the fuzzer that you want to add a data bundle for, simply click on the update data bundle button, select the archive for the data bundle, and upload it. This does not support any advanced data bundle options. To use these options, the data bundle will have to be uploaded from the data bundles page. See the uploading data bundles section for more information.

Advanced options


Occasionally fuzzers require special configuration. This is most common for fuzzers that test network protocols and require scripts to run not just when fuzzing, but also when Chrome is launched.

Test case timeout


Time, in seconds, to wait before terminating the binary and assuming that there was no crash. This should only be modified if you have a very good reason to do so. Inflating the timeout unnecessarily wastes cycles on the bots.

Test case count


Number of test cases to generate each time the fuzzer runs. In most cases the default is 1000, though this varies by job type.

Executable path


Path within the archive of the binary to run for fuzzing. It must accept the standard command line arguments for ClusterFuzz, as mentioned in the fuzzer writing guide.

Launcher script


Launcher scripts are scripts that are executed instead of directly launching Chrome when processing fuzzed test cases. Arguments passed to the launcher script will make up the command that should be executed for the job type, followed by the resource generated by the fuzzer. See the launcher scripts section of the fuzzer guide for more information.

External contribution


The external contribution checkbox must be checked if you are uploading a fuzzer from an external contributor. This makes it easy to keep track of reports from the fuzzer that may be eligible for rewards under the Chromium Vulnerability Rewards Program.

Uploading data bundles


Many fuzzers rely on some kind of input data to generate test cases. The data associated with a fuzzer is separated into a data bundle so that either one can be updated easily without interfering with the other. It also allows certain data bundles to be shared between multiple fuzzers. For small data sets (less than 100MB), no special configuration is required. The “Cloud Storage” option should be used for larger data bundles.

Uploading a basic data bundle (< 100MB)


Basic data bundles can be uploaded directly from the fuzzer list page as described in the uploading fuzzers section. They can also be uploaded by filling out the form on the data bundle list page. When using the data bundle upload form, the fuzzer name must be specified. If your data bundle is small, do not check the “use Google Cloud Storage” checkbox. For more information, or if you are uploading a large data bundle, see uploading a Cloud Storage data bundle.


The data bundle itself should be a zip, tar.gz, or tar.bz2 archive containing all data used by the fuzzer. The directory that the archive is extracted to will be passed to the fuzzer as the --input_dir command line argument. See the fuzzer documentation for more information.

Uploading a Cloud Storage data bundle (> 100MB)


The process for uploading a Cloud Storage data bundle is essentially the same as uploading a basic data bundle, but the underlying behavior is very different. Disk space on the ClusterFuzz bots is limited, so large data bundles are stored on Cloud Storage and replicated on certain bots in each zone that serve these files using glusterfs. Accessing these files is slower, and unlike the directories for usual data bundles, these are read-only.

Viewing crash statistics


Crash statistics serve multiple purposes in ClusterFuzz. If you do not have a particular fuzzer in mind while viewing the statistics, you can see the most common crash stacks and frequency with which they are being triggered. If you are developing a new fuzzer, you can confirm that it is working properly.

Viewing all crashes


To view a list of all crashes sorted by frequency, simply visit the crash statistics page. From here, you can filter using the list using the search box and order by any of the columns in the table.

Filtering crash statistics


Several options are provided for filtering crash statistics. This can be useful if you only want to see recent crashes, crashes for a particular fuzzer, or have other criteria that you want to filter by.

Fuzzer name


If you only want to see crashes found while running a particular fuzzer, you can filter using the form on the left side of the screen. Choosing a fuzzer and job type will filter to only show crashes found by the selected fuzzer while using the selected job type.

Job type


If you only want to see crashes found while running a particular job type, you can filter using the form on the left side of the screen. Choosing a fuzzer and job type will filter to only show crashes found for the selected fuzzer while using the selected job type.

Security flag


You can also choose to limit the list to crashes with security implications. By default, all crashes are shown.

Last X days


By default, the crash statistics page only shows crashes from the previous day. If you want to search farther back in time, you can change this here.

Defining job types


ClusterFuzz allows job types to be defined easily. A job definition is a collection of environment variables set on the bots running a job. These define which binary to run, how many test cases to generate, and much more.

Defining a job type for an asanified chrome target


Sometimes, managing a custom binary is not ideal. This usually happens if you are testing an area of the code that changes frequently enough that you do not want to manage a custom binary, but also cannot pass test cases to chrome directly. This most commonly comes up when working with IPC handlers.


Often, fuzzer test cases cannot be passed to Chrome directly. Whenever possible, a build target should be added so that ClusterFuzz can pick up changes automatically, and so that it can do additional analysis such as regression or fixed testing.


If you need a special handler that relies on Chromium code, first you will need to write your handler and add its target to build/all.gyp. Once that is done, wait for it to get picked up in a linux symbolized asan release build by the lkgr builder. You can confirm that it will be usable by ClusterFuzz by ensuring that your target was built properly in the most recent revisions here.


Job types for builds from this location should include the following definitions:

RELEASE_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-release-([0-9]+).zip

SYM_RELEASE_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-symbolized-linux-release-([0-9]+).zip

SYM_DEBUG_BUILD_URL_PATTERN = https://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-debug-([0-9]+).zip

REVISION_PATTERN = .*-([0-9]+).zip


STABLE_BUILD_URL_PATTERN = http://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-stable-([0-9.]+).zip

BETA_BUILD_URL_PATTERN = http://commondatastorage.googleapis.com/chromium-browser-asan/asan-linux-beta-([0-9.]+).zip

VERSION_PATTERN = .*-([0-9.]+).zip


These tell ClusterFuzz where to pull its builds from. You will then need to define an APP_NAME, which is the name of the binary within the archive that should be executed, and APP_ARGS, which represents how the command line for running your application should be formed.


APP_NAME = my_target

APP_ARGS = %TESTCASE%


The previous example is the simplest case possible and would launch a command of the form “/path/to/my_target fuzz-mytestcase-1”. You can add as many command line arguments as you need. If your test case is passed as a variable, APP_ARGS can be defined as something like the following:

APP_ARGS = --flag-1 --testcase-file=%TESTCASE%

Defining a job type with a custom binary


Using a custom binary is much simpler than working with an asanified chrome target, but has the disadvantage of needing to be manually updated. When defining the job type, simply upload an archive along with the job definition. This will be the custom binary associated with the job. The name of the binary itself should be specified by the APP_NAME variable, and the format for the command should be specified by the APP_ARGS variable. “CUSTOM_BINARY = True” must also be defined for job types that use custom binaries, but this will be added automatically.


A sample job definition that uses a custom binary follows:


APP_NAME = chrome 

APP_ARGS = --snipped-some-arguments --ppapi-flash-path=%APP_DIR%/libpepflashplayer.so --user-data-dir=%USER_PROFILE_DIR% %TESTCASE%

USER_PROFILE_ARG = --user-data-dir

NET_LOG_ARG = --log-net-log

WINDOW_ARG = --window-size=$WIDTH,$HEIGHT --window-position=$LEFT,$TOP


MAX_FUZZ_THREADS = 4

TEST_TIMEOUT = 18

MULTI_PROCESS = True

CUSTOM_BINARY = True

Advanced options

Variables in job definitions


A job definition is made up of a set of variables that will be defined as environment variables when ClusterFuzz runs. Some commonly used ones are included below, but this is not intended to be a complete list.



Key

Value

Description

APP_NAME

String

Name of the binary to run.

APP_ARGS

String

Format of command line arguments for this binary. See the command format strings table for more information.

CUSTOM_BINARY

True|False

True if a custom binary is used, false otherwise.

MAX_FUZZ_THREADS

Integer

Number of instances of the binary to run in parallel while processing test cases.

MAX_TESTCASES

Integer

Maximum number of test cases that should be processed for this job type. This can be overridden by individual fuzzers.

MIN

Yes|No

Yes if minimization should be attempted for this test case, No if it should not. By default, minimization will be attmpted if this is not defined.

MIN_REVISION

Integer

First chrome revision that should be used for this job type.

NET_LOG_ARG

String

Defining this to --log-net-log will tell ClusterFuzz that the --log-net-log argument should be passed to chrome in order to collect information about which subresources were accessed by a test case. This allows ClusterFuzz to automatically create archives with a test case and all dependencies.

REQUIRED_APP_ARGS

String

Command line arguments that should not attempted to be removed during minimization for a test case.

TEST_TIMEOUT

Integer

Timeout, in seconds, when processing test cases that use this job type.

WINDOW_ARG

String

If defined, the most common value (for chrome) is “--window-size=$WIDTH,$HEIGHT --window-position=$LEFT,$TOP”. This allows ClusterFuzz to attempt to set window position and size for this job type.

Command format strings


You may have noticed that APP_ARGS supports some strings with special meanings, such as %TESTCASE%. A full list of these is included in the following table.


Variable

Description

%TESTCASE%

Absolute path to the test case currently being

%USER_PROFILE_DIR%

Absolute path to the current chrome instance’s profile directory.

%ROOT_DIR%

The directory that ClusterFuzz is installed in.

%APP_DIR%

The directory that the current application (chrome) is installed in.

%TMP_DIR%

A temporary directory cleared after each round of fuzzing.

%SCRIPT_DIR%

The directory that the ClusterFuzz scripts are installed in.

%INPUT_DIR%

Data bundle directory for this fuzzer.

%FUZZER_DIR%

Directory that the current fuzzer is installed in.

Comments