1. Read this document.  Oh, I see you are doing it!  Great!
2. Make sure you have permission to submit CLs to the Chromium OS gerrit repositories.
3. Install helpful extensions.

What should I do when I report for duty?

1. Sign on to irc.freenode.net / #chromium-os, and introduce yourself as an on-duty sheriff.
2. Pull up the public and internal (overview) Chrome OS buildbot waterfalls.  If the status is out of date, update it.
3. Take a look at the Test Dashboard for our Build Verification Tests. Try to triage any failures; do the same with autotest failure emails you receive during your shift. See the section titled "How should I handle BVT failures?", below for more details.
4. Read the email from yesterday’s Sheriffs and/or look at the Chromium OS sheriff log, and familiarize yourself with the TreeCloser issues they cite.
5. Proactively ensure that the tree will never be unattended by checking that your co-sheriff will be able to cover you during any meetings you have during the day.  If needed, rope someone else on IRC into covering.
6. Be prepared to get in touch with Chrome sheriffs in case chrome build is misbehaving. See their sheriff names published at http://build.chromium.org/p/chromium/console

What should I do as I prepare to end my shift?

What do the different tree statuses (open, throttled, closed) mean and do?

• Open - ToT is believed to be a healthy, with builds that can make it through the commit queue and canary builders without failing. Commit queue will run unencumbered.
• Throttled - typically the automatic result of a builder failure. ToT is believed to have a bad change, or the build infrastructure is malfunctioning, such that canary or commit queue builders may fail. An already-running CQ run will be allowed to submit changes, if it passes, but further runs will not happen.
• Closed - ToT has a bad change or the build infrastructure is malfunctioning. Changes unrelated to tracing the issue should not be submitted. Commit queue runs will not submit changes.

Ahh, the tree is green.  I can go back to my work, right?

Postmortem?  What is that, and why do we do it?

I am a noob, starting my shadowing shift.  What do I need to do?

How do I find out about build failures?

How do I deal with build failures?

How do I access builder machines?

What should I do if I see a commit-queue run that I know is doomed?

How can I revert a broken commit?

If you've found a commit that broke the build, you can revert it using these steps:

1. Find the Gerrit link for the change in question
• Gerrit lists recently merged changes. View the change in question.
• In some cases (like the PFQ), you can see links to the CL that made it into the build straight from the waterfall. We’re working making this even better.
2. Click the “Revert Change” button and fill in the dialog box.
1. The dialog box includes some stock info, but you should add on to it. e.g. append a sentence such as: This broke the tree (xxx package on xxx bot).
3. You are not done.  This has created an Open change for _you_ go find and submit it.
• Add the author of the change as a reviewer of the revert, but push without an LGTM from the reviewer (just approve yourself and push).

How do I investigate VMTest failures?

There are several common reasons why the VMTests fail. First pull up the stdio link for the VMTest stage, and then check for each of the possibilities below.

Auto-test test failed

Once you've got the VMTest stage's stdio output loaded, search for 'Total PASS'. This will get you to the Autotest test summary. You'll see something like

Total PASS: 29/33 (87%)

Assuming the number is less than 100%, there was a failure in one of the autotests. Scroll backwards from 'Total PASS' to identify the specific test (or tests) that failed. You'll see something like this:

/tmp/cbuildbotXXXXXX/test_harness/all/SimpleTestUpdateAndVerify/<...>/login_CryptohomeMounted            [  FAILED  ]
/tmp/cbuildbotXXXXXX/test_harness/all/SimpleTestUpdateAndVerify/<...>/login_CryptohomeMounted              FAIL: Unhandled JSONInterfaceError: Automation call {'username': 'performancetestaccount@gmail.com', 'password': 'perfsmurf', 'command': 'Login'} received empty response.  Perhaps the browser crashed.

In this case Chrome failed to login for any of the 3 reasons: 1) could not find network, 2) could not get online, 3) could not show webui login prompt. Look for chrome log in /var/log/chrome/chrome, or find someone who works on UI.

(If you're annoyed by the long string before the test name, please consider working on crbug.com/313971, when you're gardening.)

Crash detected

Sometimes, all the tests will pass, but one or more processes crashed during the test. Not all crashes are failures, as some tests are intended to test the crash system. However, if a problematic crash is detected, the VMTest stdio output will have something like this:

Crashes detected during testing:
----------------------------------------------------------
chrome sig 11
  login_CryptohomeMounted

If there is a crash, proceed to the next section, "How to find test results/crash reports"?

ASAN error detected

The x86-generic-asan and amd64-generic-asan builders instrument some programs (e.g. Chrome) with code to detect memory access errors. When an error is detected, ASAN prints error information, and terminates the process. Similarly to crashes, it is possible for all tests to pass even though a process terminated.

If Chrome triggers an ASAN error report, you'll see the message "Asan crash occurred. See asan_logs in Artifacts". As suggested in that message, you should download "asan_logs". See the next section, "How to find test results/crash reports" for details on how to download those logs.

Note: in addition to Chrome, several system daemons (e.g. shill) are built with ASAN instrumentation. However, we don't yet bubble up those errors in the test report. See crbug.com/314678 if you're interested in fixing that.

ssh failed

[ This is being changed significantly as of 12/4/2013. Please ping quiche@ if you need to investigate this kind of failure before the documentation is updated. ] 

The test framework needs to log in to the VM, in order to do things like execute tests, and download log files. Sometimes, this fails. In these cases, we have no logs to work from, so we need the VM disk image instead.

You'll know that you're in this case if you see messages like this:

Connection timed out during banner exchange
Connection timed out during banner exchange
Failed to connect to virtual machine, retrying ... 

When this happens, look in the build report for "vm_image" links. These should be right after the "stdio" link. You'll probably see two vm_image links for each VMTest attempt. You may need to check the timestamp against the stdio logs, to make sure you get the right vm_image. (Note that you see four links in the second attempt, but the first links are just repeats of the images from the first attempt.)

VMTest (attempt 1) stdio vm_image: chromiumos_qemu_image.bin.copy.AXK2wezQQm.20131104-193709.tar vm_image: chromiumos_qemu_image.bin.copy.MJxYAXZLcM.20131104-193155.tar vm_test_results_1.tgz

VMTest (attempt 2) stdio vm_image: chromiumos_qemu_image.bin.copy.AXK2wezQQm.20131104-193709.tar vm_image: chromiumos_qemu_image.bin.copy.LbGGUDpS1C.20131104-194915.tar vm_image: chromiumos_qemu_image.bin.copy.MJxYAXZLcM.20131104-193155.tar vm_image: chromiumos_qemu_image.bin.copy.WCyzOHZOk7.20131104-195455.tar vm_test_results_2.tgz

Click the appropriate link to download the VM image, and then a) uncompress the image, and b) boot the image in kvm. E.g.

$ pbzip2 -d chromiumos_qemu_image.bin.copy.AXK2wezQQm.20131104-193709.tar

$ sudo kvm -m 2G -smp 4 -vga cirrus -net nic,model=virtio  -serial none -net user,hostfwd=tcp::9222-:22 -drive file=chromiumos_qemu_image.bin.copy.AXK2wezQQm,index=0,media=disk,cache=writeback -monitor stdio

(Despite the name, the tarfile is actually just the disk image compressed with bzip2. pbzip2 is a little faster than tar, because pbzip2 will parallelize the decompression.)

At this time, we don't have an easy way to mount the VM filesystem, without booting it. If you're interested in improving that, please see crbug.com/313484.)

How to find test results/crash reports?

The complete results from VMTest runs are available on googlestorage, by clicking the [ Artifacts ] link in-line on the waterfall display in the report section:

 

From there, you should see a file named chrome.*.dmp.txt that contains the crash log. Example

If you see a stack trace here, search for issues with a similar call stack and add the google storage link, or file a new issue.

How do I extract stack traces manually?

Normally, you should never need to extract stack traces manually, because they will be included in the Artifacts link, as described above. However, if you need to, here's how:

1. Download and extract the test_results.tgz file from the artifact (above), and find the breakpad .dmp file.
2. Find the build associated with your crash and download the file debug.tgz
1. Generally the debug.tgz in the artifacts should be sufficient
2. For official builds, see go/chromeos-images
3. TODO: examples of how to find this for cautotest and trybot(?) failures
3. Untar (tar xzf) this in a directory under the chroot, e.g. ~/chromeos/src/scripts/debug
4. From inside the chroot, run the following: minidump_stackwalk [filename].dmp debug/breakpad > stack.txt 2>/dev/null
5. stack.txt should now contain a call stack!

If you successfully retrieve a stack trace, search for issues with a similar call stack and add the google storage link, or file a new issue.

Note that in addition to breakpad dmp files, the test_results.tgz also has raw linux core files. These can be loaded into gdb and can often produce better stack traces than minidump_stackwalk (eg. expanding all inlined frames).

When should I reopen the tree?

• If the build was broken, but a fix has now been checked in, reopen the tree. (There's no need to wait for the tree to cycle green: If the build fails, you can just close the tree again.)
  
• If the buildbot ran into an intermittent issue, but should run cleanly the next time, reopen the tree.

• If the build was broken, and a fix has not been checked in, we should leave the tree closed.
  
• If it's not possible to browse the web with the current build, and a fix has not been checked in, we should leave the tree closed.
  

INFO: Waiting for uploads...

INFO: RunCommand: ['/b/build_internal/scripts/slave-internal/autotest_rpc/autotest_rpc_client.py', 'master2', 'RunSuite', '-i', 'x86-alex-release/R19-2007.0.0-a1-b1822', '-s', 'bvt', '-b', 'x86-alex', '-p', 'bvt']
try_new_image                        [ PASSED ]
build_RootFilesystemSize             [ PASSED ]

Example of fail output

platform_ToolchainOptions            [ FAILED ]
platform_ToolchainOptions              FAIL: The following tests failed. If you expected the test to pass you may have stale binary prebuilts which are causing the failure. Try clearing binary prebuilts and rebuilding by  running: ./setup_board --board=... --force
platform_FilePerms                   [ PASSED ]

How can I pin Chrome to an older version?

• First, send an email to the chromeos-tpms@ internal mailing list letting them know that you are pinning Chrome to an older version.
• Next, determine what git push brought in the broken version of Chrome, using the tools at your disposal: the waterfall, git log, and your brain.  Identify the git hash associated with that push, and the Chrome version that came before that push.  Let's assume that the following is what you were looking for in git log:

• The above is the push from chrome-bot marking chromeos-chrome-14.0.790.0_rc-r1 as stable, moving from the previous version chromeos-chrome-14.0.789.0_rc-r1.  Note the latter version as well as the git hash associated with the above push.

• Create a branch in the ~/chromiumos/src/third_party/chromiumos-overlay project for this change:

• Edit the file at ./profiles/default/linux/package.mask so that the following lines appear (stick it at the end if there isn't an existing one):

• Now you must retrieve the ebuild for this Chrome version from before the offending push:

• Most importantly, you need to comment out the line in profiles/default/linux/package.mask (notice the addition of a single #):