For Developers‎ > ‎Tree Sheriffs‎ > ‎

Sheriff Details: Chromium

Contents

  1. 1 How to join or leave
  2. 2 Supporting docs
  3. 3 How to prepare
  4. 4 Logistics
  5. 5 When should the tree be closed
  6. 6 How to handle failures
  7. 7 Common tasks:
    1. 7.1 Handling the sheriffing bug queue
    2. 7.2 Reverting changes
      1. 7.2.1 "Revert Patchset" button in Rietveld
      2. 7.2.2 Using git
    3. 7.3 Compile failure
    4. 7.4 Handling a failing test
      1. 7.4.1 Check for a bug
      2. 7.4.2 File a bug
      3. 7.4.3 How do I disable a flaky test?
      4. 7.4.4 GTest-based C++ Tests
      5. 7.4.5 Javascript WebUI browser_tests
      6. 7.4.6 Java Tests
      7. 7.4.7 Blink Layout Tests (aka webkit_tests)
      8. 7.4.8 Python infra tests
      9. 7.4.9 Python Telemetry tests (telemetry_perf_unittests or telemetry_unittests)
      10. 7.4.10 WebGL conformance tests or other GPU tests
      11. 7.4.11 Go (golang) tast-tests
      12. 7.4.12 Handling failing perf expectations (like the sizes step)
    5. 7.5 Failures due to DEPS rolls
  8. 8 Tips and Tricks:
    1. 8.1 How to read the tree status at the top of the waterfall
  9. 9 Upcoming sheriffs
We are migrating the wiki into markdown in the repo.
See also Chromium Sheriffing for the latest information.


This page has details to help Chromium sheriffs. For more information and procedures, see Tree Sheriffs and read the Philosophy of Sheriffing.

How to join or leave

To join, leave, or request other membership modifications to the Chromium sheriff rotation, file a bug here.

Supporting docs

Please also read this more up-to-date document in the Chromium tree.

How to prepare

Before you start sheriffing, make sure you have working build environments on all your machines. If you don't have hardware for some platforms, try borrowing it. You may need it to repro breaks locally. Know how to sync to a specific revision (in case you need to bisect to find the CL that introduced the break).

Logistics

  • Be available on Slack.
  • Use bugs to coordinate with the other sheriffs. Any bug with label:Sheriff-Chromium will show up in sheriff-o-matic in the bug queue and is a good way to communicate unfinished issues to the other sheriffs or to the next time zone.
  • Sheriff-o-Matic shows you failures in decreasing order of priority. (Make sure to choose “chromium” from the menu on the top left and be sure to log in or else LINK BUG and other features may silently fail.)
  • Sheriff-o-Matic also allows you to associate a bug you filed with the failure in sheriff-o-matic. Everyone else can see these bug annotations, so you can use this to communicate status of handling a failure.

When should the tree be closed


The tree should only ever be closed (automatically or manually) if something is catastrophically wrong. Here are the common cases:

  • Update and/or compile is broken on any bot due to an infrastructure problem or due to a bad commit. Reopen when the bad commit has been rolled out.
    • Some bots on the waterfalls are not mirrored in the commit queue; failures on those bots should not cause the tree to close.
  • Hundreds of tests are failing.
  • Planned maintenance.

The tree should not be closed for manageable numbers of test failures. The tree should not be throttled ever.


How to handle failures

https://sheriff-o-matic.appspot.com shows you failures in decreasing order of priority: tree closing failures show up at the top, followed by tests that have failed repeatedly, tests that have only failed once, and finally snoozed failures.


It also tries to group failures for you that have the same regression range.


The first three categories are the important ones. Tests that have failed only once are sometimes flakes, but are sometimes also leading indicators of things that will fail repeatedly, so look at them when the first three categories of issues have been dealt with, if you have time.


The culprit finder Findit can auto-revert bad commits once identified, and automatically posts all analysis statuses & identified culprits onto Sheriff-o-Matic UI.


The Flake Portal in Findit is the new entry point for flakiness, especially for flake ranking by negative impact. For the full history of a test, please continue to use the legacy Flakiness Dashboard linked from there.

In all cases, the preferred way of dealing with alerts is to revert the offending patch via Gerrit, even if the failure does not close the tree. Pay attention to auto-revert by Findit on Sheriff-o-Matic as well; if you revert first, Findit will post a confirmation on the culprit's code-review instead. The only effect tree-closingness has on you as a sheriff is that you prioritize fixing tree closers over non-tree-closers. Revert first and contact the author of the patch later. Don't feel bad about reverting. It is easy to re-land as CL and accepted to TBR previous reviewers for code that has not changed.


Disabling tests may need to be done differently, depending on the kind of test; see below for details.


If a test has recently become flaky, spend at least some time trying to diagnose and fix the problem, rather than simply disabling the test. "Some time" is something in the 15-60 minute range, depending on the severity of the problem. Always favor reverting a CL over suppressing test failures. Usually you can get a pretty good guess on what CL might be the culprit in the 15-60 minute window.

When there are no alerts left in sheriff-o-matic, you’re done and the tree should be green. Consider spending your ample free time improving the tooling. sheriff-o-matic is in third_party/WebKit/Tools/GardeningServer. Here are some bugs that need fixing: http://crbug.com?q=label%3aSheriffOMatic.


Note: If an alert shows up, it will frequently disappear after a few minutes. This does not mean that you should spend your sheriff shift waiting for red try runs to turn green. Use the flakiness dashboard to estimate the degree of flakiness if a test has been flaky for some time, disable it and find an owner as described below. Typically you will spend a huge chunk of your time, dealing with flakiness. If you just sit it out, the next sheriff has to deal with it.


More on sheriff-o-matic and what to do if it's down: See sheriff-o-matic.

Common tasks:


Handling the sheriffing bug queue


Bugs are shown in sheriff-o-matic and corresponding to crbug.com/?q=label:Sheriff-Chromium. See Sheriffing Bug Queues for how to handle them.



Reverting changes


  • "Revert Patchset" button in Rietveld


         The button does the following:
    • Creates a new Rietveld issue with the patchset inverted
    • Adds keywords to immediately submit via the CQ (NOTREECHECKS, NOPRESUBMIT, NOTRY, TBR) if CL landed recently (<1 day ago). Otherwise (old CL), only TBR is added, and revert will be tried normally in CQ. You can override that by editing description and adding NOTREECHECKS, NOPRESUBMIT, NOTRY manually.
    • Checks the CQ checkbox (if requested to by the user)
This feature's design document is here. Please file a crbug to rmistry@ for feature requests or bugs.
         If the button does not work because the patch cannot be applied or the CQ is down, then use the options below.

  • Using git

    $ git revert <hash> $ git cl upload $ git cl land

Note:

  • The "Revert Patchset" button updates the original CL saying it is being reverted. If you use Drover or git, please update the original CL. The author must be notified that their CL has been reverted.

  • Drover and "Revert Patchset" button in Rietveld do not work on files larger than ~900KB. If you need to revert a patch that modifies generated_resources.grd, for example, then use git.

Compile failure

REVERT

Waiting for a fix it not a good idea. Just revert until it compiles again.

If it's not clear why compile failed, contact a trooper. Trooper's can also investigate and fix builders that are offline. Note that builders reboot after every build, so being temporarily offline is normal.

Handling a failing test

Check the flakiness dashboard to get a sense of the test's historical failure rate. This can be extremely useful, as it can show that many tests fail on the same run. This may be an indication that the failure is related to a harness shared by many tests, or to the overall test environment.


REVERT if you can figure out which patch caused the failure. Otherwise:

Check for a bug

Before doing anything else, check to see if there's already a bug on file for the specific failure in the failing test or in other tests. There is no need to file a new bug if we already have one for the specific failure. It doesn't help to lump distinct failures into the same bug since their resolutions may be unrelated.

File a bug

More details on how to disable a test and an example bug are on the Handling a failing test sub-page.

File a bug at crbug.com and leave as much context about the problem in the bug report. At least make sure to include the following in the bug report:
  • Copy+paste the relevant parts of the log into the bug report -- it is not sufficient to give a URL to the bot log, as the logs eventually get deleted.
  • Comment detailing the action you took (disabled, on what platform).
  • Indicate if the test is Flaky (intermittently failing), constantly failing, timing out, crashing, etc.
  • Apply the label Test-Disabled.
  • Link to build logs for that test: http://chromium-build-logs.appspot.com/ (but remember to copy+paste the relevant parts of the log)
  • Wherever possible, add any applicable feature components, and use OWNERS or git log for that test file to assign an owner or cc someone who can actively look into it.
  • Check the proper OS boxes. It’s important that the bug be labeled properly so that the right folks see it.
  • Assign the bug. Every bug for a disabled test must be assigned to someone. Look through the test’s history to find the author. If that person no longer seems to be an active contributor to Chromium, pick an OWNER of the test or of the feature being tested.
  • Handy shortlink for filing this type of bug: http://bit.ly/disabled-test-tracker
The document outlining the disabled tests process is here.

How do I disable a flaky test?

File a tracker as outlined in the section just above for "Handling a failing test." In the code, follow the steps as outlined below to disable the test and always include the https://crbug.com/NNNNN link to the tracker you've filed in a code comment on the line above where the test is disabled. Then create a CL, and land it with TBR=<the owner of the test>.

There are many different types of tests and instructions for sheriffs suggest that sheriffs should disable flaky tests. If you’ve encountered a new type of test that you do not know how to disable, please have a look here. If you still don’t find it, ask the authors of the test and help us improve this documentation by adding a section below. We do not yet support automatic one-button disabling of the test, but there are plans to implement this feature in the future. This page will be very useful for that work.

GTest-based C++ Tests

Add prefix DISABLED_ to the name of the crashing/timing out test.


TEST(ExampleTest, CrashingTest) {


becomes


// Crashes on all platforms.  http://crbug.com/1234

TEST(ExampleTest, DISABLED_CrashingTest) {


If the test is crashing/timing out on a proper subset of the major platforms (some, but not all), use an #ifdef to only disable the test for those platforms.


// Crashes on Mac/Win only.  http://crbug.com/2345

#if defined(OS_WIN) || defined(OS_MACOSX)

#define MAYBE_CrashingTest DISABLED_CrashingTest

#else

#define MAYBE_CrashingTest CrashingTest

#endif


TEST(ExampleTest, MAYBE_CrashingTest) {


Further #defines exist for MEMORY_SANITIZER, ADDRESS_SANITIZER and THREAD_SANITIZER.

Notice the use of the MAYBE_ moniker: it's possible that the name of the test is an identifier in the testing code, e.g., the name of the test is the same name as the method being tested in the test.

Javascript WebUI browser_tests


These tests though written in Javascript are actually GTest-based C++ Tests and primarily live under src/chrome/browser/ui/webui/.
Follow the proceduce for regular GTest tests while wrapping C++ lines with GEN('').

Java Tests

If you need to disable an android test in Java, it is a bit different.  First, you import the disabled test module:


   import org.chromium.base.test.util.DisabledTest;


After doing so, add @DisabledTest to the test's annotations.


   @Test

   @Feature({"AndroidWebView", "FindInPage"})

   @DisabledTest


See here for more details.

Blink Layout Tests (aka webkit_tests)

If the test is flaky, please modify TestExpectations file and add new line with the test name and [Pass FailureType] expectation (example). If the test is already present in the file, add new failure type to the list of expectations. Note that this does not disable flaky test, but makes it not fail the build on the specified failure type. The advantage of this approach is that one can disable a test that produces flaky timeouts, but test will still be run and if it produces a crash or incorrect result, it will fail the build.


If you need to completely disable the test, please use [Skip] expectation (example).

More tips for handling layout test failures and other Blink-specific issues can be found on on the Blink gardening page.

Python infra tests

Use unittest.skip decorators on the test method (example).

Python Telemetry tests (telemetry_perf_unittests or telemetry_unittests)

To disable failing tests of telemetry_unittest, use @decorators.Disabled on the test method (example). Note that this requires making changes to https://github.com/catapult-project/catapult

Most failing tests of telemetry_perf_unittest can also be disabled with @decorators.Disabled on the test method. There are two exceptions:
  1. If the name of failing test starts with "benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop/foo:bar" on platform X,  disable the test by adding 
    crbug.com/abcd1234 [ platform X ] system_health.memory_desktop/foo:bar [ Skip ] to the test expectation file.
  2. If the name of the failing test looks something like "benchmarks.benchmark_smoke_unittest.BenchmarkSmokeTest.blink_perf.shadow_dom", disable the test by adding "blink_perf.shadow_dom" to the blacklist.

WebGL conformance tests or other GPU tests

Add lines to src/content/test/gpu/gpu_tests/webgl_conformance_tests.py, pixel_expectations.py, etc. See the GPU Pixel Wrangling instructions for more details. Prefer to mark tests as Flaky rather than Failed, so that at least a little coverage is maintained. Please file a bug about any GPU tests that were disabled due to failures or flakiness. For WebGL tests, use the label Cr-Blink-WebGL; for all others, use Cr-Internals-GPU-Testing.

Go (golang) tast-tests

Note that disabling a tast-test requires a chromiumos checkout (instructions).

Normally, the best way to keep a failing test from breaking a bot is to mark the test as "informational". To do this, add an 'informational' attribute to the 'Attr' list (example).

If for some reason you don't even want the test to run at all (unlikely), add a 'disabled' attribute to the 'Attr' list (example). If the test is parameterized and you only want to disable one case, you can use the 'ExtraAttr' field on the parameterization (example).


Handling failing perf expectations (like the sizes step)


When a step turns red because perf expectations weren't met, use the instructions on the perf sheriffs page to give you information on how to handle it.  It can also help to get in touch with the developer that landed the change along with the current perf sheriff to decide how to proceed. For sizes, the stdio output of the sizes step lists the static initializers found, diffing against a green run can find the culprit of that kind of sizes failure. A CL that increases the number of static initializers should always be reverted.


Failures due to DEPS rolls


If it is a Skia DEPS roll that is causing failures:
  • Go to https://autoroll.skia.org/. Login with google.com account and click the STOP button to pause new rolls.
  • Revert the offending DEPS roll.
  • If an obvious owner cannot be found in the list of CLs, assign to the Skia Sheriff, listed at the top of status.skia.org and as a reviewer on the roll CL.
    • If the Sheriff cannot be assigned, cc them and assign the issue to hcm@.
Catapult, NaCl and WebRTC have similar autorollers at https://catapult-roll.skia.org/, https://nacl-roll.skia.org/ and https://webrtc-chromium-roll.skia.org respectively.

    Tips and Tricks:

    How to read the tree status at the top of the waterfall


    • Chromium / Webkit / Modules rows contain all the bots on the main waterfall.

    • Official and Memory bots are on separate waterfalls, but the view at the top show their status.

    Merging the console view


    If you want to know when revisions have been tested together, open the console view and click the "merge" link at the bottom.


    Other Useful Links


    • Failures-only waterfall. It will show you only the bots a sheriff would need to look at. (A builder is considered failing if the last finished build was not successful, a step in the current build(s) failed, or if the builder is offline.)

    • Console view to make sure we are not too much behind in the testing.

    • Some sheriffs don't look at the waterfall at all, instead they open this console and choose [merge] at the bottom.

    • The Reliability tester. It's very important for Chromium stability.

    • ChromeOS bots. These bots build and run Chrome for ChromeOS on Linux and ChromiumOS respectively and are as important as win/mac/linux bots. If you're not sure how to fix an issue, feel free to contact ChromiumOS sheriffs.

    • Memory waterfall, The bots on the Memory waterfall should be watched by the regular sheriffs. Bugs reported by ASan usually cause memory corruptions in the wild, so do not hesitate to revert or disable the failing test (ASan does not support suppressions). Similarly, MSan and TSan bots find uninitialized memory errors and race conditions, respectively.

    • Note that memory waterfall also contains Chromium OS ASAN bots. See Sheriff FAQ: Chromium OS ASAN bots for more details.

    • There is a #sheriffing channel on https://chromium.slack.com. Join the channel to discuss issues with the other sheriffs and developers.

    Upcoming sheriffs

    The authoritative lists are in the calendars. See how to swap if you can't make it.

    NOTE: If your shift spans a weekend, you aren't expected to sheriff on the weekend (you do have to sheriff on the other days, e.g. Friday and Monday).

    NOTE: You aren't expected to sheriff during holidays in your office. If not all offices in your time zone are on holiday, try to swap with someone in one of those offices.

    NOTE: If you need a substitute, don't send an email to a mailing list and skip your shift if nobody responds. Look at the authoritative calendars, check who has upcoming shifts, check if their calendars seem free during your slot and ask in person, via hangouts or via email (in decreasing order of preference) whether a person could swap with you. This works better than asking a large mailing list.

    Subpages (2): Handling a failing test Memory sheriff
    Comments