Sheriffs: jrbarnette, namnguyen, hychao

• 391950 - this problem took out daisy 7/7; fixed as of first canary 7/8.  However...
• 383481 - this problem took out the first daisy canary 7/8, when paygen ran AU tests.
• 391952 - this problem has been taking out nyan since 7/3.
• failure on monroe canary - blamed a possibly broken DUT in the test lab

• 390177 - Nyan canaries failing with: platform_DebugDaemonGetPerfData (failing since the 24th).
• 390206 - ivybridge canary failed with: mkfs.vfat: Attempting to create a too large filesystem
• 377104 - autoupdate_Rollback Failure on monroe-release
• 390272 - Chrome PFQ Failing to uprev Chrome (Bad Chrome made it past the Chrome-PFQ)
• 390433 - rambi-c canary Failed Paygen (attempt 2)
• 390451, 390448, 390441, 390436 - [bvt] video_VideoDecodeAccelerator Failure on falco/wolf/enguarde/lumpy-release
• 387152 - video_ChromeHWDecodeUsed failure. This is also a labeling problem. X86 should not execute the test.
• 390693 - Closed the tree because CQ is currently broken. Also refer to http://chromegw/i/chromeos/builders/CQ%20master.

• 389513 - [bvt] logging_UdevCrash Failure
• 389517 - [bvt] login_OwnershipApi Failure
  

• linux_chromiumos dbg failure, touch exploration controller, reverted crrev.com/279638
• crbug.com/388778 - cros_trunk compile failure h264_bitstream_buffer_unittest.cc (Wednesday)
• crbug.com/388778 - cros_trunk compile failure h264_bitstream_buffer_unittest.cc (again on Thursday, vote for crbug.com/388800)

• 387571 - [bvt] video_VideoDecodeAccelerator Failure
• 387525 - ASan broken by Chrome update
• 388114 - lab outage breaking all HWTests
• 388151 - Chrome PFQs broken due to missing quantize_sse4.asm

• 383414 - video_VideoDecodeAccelerator Failure
• 311350 - USB Dongle loses network connectivity
• 387525 - ASan broken by Chrome update
• 387702 - Potentially misconfigured server causing AU test to fail across canary builders
• 339934 - Buildbot failure in DebugSymbols due to tar: debug/bin: file changed as we read it

• 385039 - beltino-a canary, hardware_RamFio aborts
• 386473 - stumpy_moblab canary failed at Paygen with missing images from previous builds. Workaround in CL:204626
• 386419 - Rambi-C canary ... Build Team says it was flake.
• Chrome PFQ failure(daisy nightly) - Caused by CL:278169 (CL:334593004 depends on changes there)
• 386303 - GS upload failures.  Uploads disabled to this bucket as workaround
• 386185 - CL:204164 broke shill build for x86-generic ASAN fixed by CL:204457

• 375721 - graphics_PerfControl Failure
• 383414 - video_VideoDecodeAccelerator failing on falco-chrome-pfq
• 384290 - Incorrect calls to generate debug logs
• 383977 - security_SandboxLinuxUnittests failing on daisy_spring
• 384721 - chromeos-chrome compile failed in PFQ in VideoDecoderShim

2014-06-10 - 2014-06-11

Sheriffs: ???

• 339934 - beaglebone canary failed DebugSymbols: "tar: debug/usr/sbin: file changed as we read it"
• 382975 - Chrome PFQs dying while building mojo with fwrite warning errors
• 383370 - moblab died in UnitTest phase while running autotest-server with Address already in use

Note to next sheriff: Chrome was unpinned late at the request of dharani@, if it still isn't fixed might need to pin back again

2014-06-06 - 2014-06-09

Sheriffs: davidriley, wiley

• jrbarnette reports: the test lab is seeing lots of devices go down after the platform_PowerWash test: turned out to be a result of lab reshuffling and machines not being connected up correctly after
• 381553 - Lab melt-down.  fdeng to the rescue.
• 382605 - telemetry change causing: AttributeError: 'InspectorBackend' object has no attribute 'id'
• 381841 - cryptohome KeysetManagementTest.UpdateKeysetAuthorizedCompatVector failure
• 382605 - Tons of tests using telemetry fail until PFQ picks up newer Chrome rev
  • Telemetry change made which started causing tests to fail with: AttributeError: 'InspectorBackend' object has no attribute 'id'
  • Telemetry change made, then reverted, then corrected
  • Corrected version wasn't getting picked up since PFQ had been failing from ARM bots over the weekend due to crrev.com/275621. crrev.com/275775 fixes this.
  • Pinned Chrome to 37.0.2034.1 to open tree, Chrome gardener will merge fixes to get a new Chrome build

Note to next sheriff: continuous chromium PFQ failures on ARM bots over the weekend like this one should be fixed by crrev.com/275775 - posciak@.

2014-06-05 - 2014-06-06

Sheriffs: djkurtz (non-MTV)

• 379603 - autotests fail with: TypeError: cannot concatenate 'str' and 'int' objects, but this seems to be masking another problem with subprocesses getting nuked b/c they aren't finishing in time.
• Fix in: https://chromium-review.googlesource.com/#/c/202652/
• 380735 - 'x86-generic-asan' fails "security_ASLR" (security_ASLR is not available on 32-bit ASan)
• 381005 - 'sandybridge canary' bvt => flaky(?) PiglitBVT [shaders/glsl-unused-varying] on parrot
• 380376 - 'stumpy_moblab canary' Can not sign moblab image
• 375721 / 381008 - 'rambi-b canary'/glimmer-release - graphics_PerfControl - Unreasonable temperature 0.0C
• 381553 - Lab melt-down.  beeps to the rescue.
  

2014-06-03
Sheriffs: dianders, gwendal, jcliang (non-MTV)

• 380376 - Can not sign moblab image
• 380352 - compilation error in chromeos-chrome: use of undeclared identifier '__sanitizer_sandbox_arguments'
• 379657 - R36 [bvt] login_OwnershipApi Failure across boards
• 379603 - rambi-a, rambi-b: login_Cryptohome Failure (Timed out going through login screen)
• 355603 - rambi-a canary
• 379438 - peach canary cpufreq problems => 379700
• 379426 - rambi-b canary: login_SameSessionTwice
• 380138 - logging_UdevCrash Failure on peach_pi-release - race could affect any device with atmel
• 294221 - daisy canary autoupdate_EndToEndTest.npo_test_delta,Host did not return from reboot
• 380306 - pineview canary security_Firewall_SERVER_JOB Failure

2014-06-02
Sheriffs: dianders, gwendal, cywang (non-MTV)

• 379438 - peach canary cpufreq problems => 379700
• 355603 - rambi-a canary
• 379810 - beaglebone canary failed DebugSymbols: "tar: debug/usr/sbin: file changed as we read it"
• 379871 - beltino-a canary signer
• 379851 - rambi-c canary login_CryptohomeIncognito_SERVER_JOB
• 379657 - R36 [bvt] login_OwnershipApi Failure across the board.

2014-05-29
Sheriffs: benchan, denniskempin, chihchung(TPE)

• Bad ACL on xorg-server prebuilt binpkg fixed; randomly broke some daisy builds in PreCQ and incrementals/PFQ (vapier@)
• platform_HwImageBoardCheck fails on expresso (crbug.com/375737)

2014-05-27 - 2014-05-28
Sheriffs: zeuthen, adlr, chihchung(TPE)

• Chrome PFQ Issue w/ Harfbuzz. CL reverted

2014-05-23 - 2014-05-26
Sheriffs: garnold, sosa, tyanh(TPE)

• 375912 - still failing (5/23); a number of hosts identified as exhibiting the problem were locked for use by hwtests; likely related to crbug.com/361331, fix under review.
• 377070 - failures on rambi-a that looks like kernel crashes due to video driver failures.

2014-05-23
Sheriffs: tyanh(TPE)

• 376715 - Beaglebone canary failing in upload_symbols.py: OSError: [Errno 2] No such file or directory: '/b/cbuild/internal_master/chroot/build/beaglebone/usr/lib/debug/breakpad/mbimcli/5C7E14771137D722C2967C5A2D3B65D40/mbimcli.sym'
• 376629 - Daisy canary failing desktopui_MediaAudioFeedback with cryptohome-path failed: [0522/202013:ERROR:cryptohome.cc(36)] Could not get size of system salt: /home/.shadow/salt: No such file or directory

2014-05-21 - 2014-05-22
Sheriffs: deymo, arakhov, seanpaul [05/21 only], tyanh(TPE)

• 375721 - Rambi-b on DUT 2-2-3-9 reporting invalid temp of 0.0C. Possible DUT or EC problem.
• 376157 - Nyan Canary failing audio_Microphone autotest
• 367278 - cryptohomed killed with SIGPIPE
• 375912 - Daisy Canary HWTests failing with "The name org.chromium.Cryptohome was not provided by any .service files"
• 375886 - Nyan Canary failing HWTest on network_WiFiInvalidParameters
• 375717 - Daisy Canary HWTest failing with "Timed out going through login screen"
• 358737 - Slippy Canary HWTest failing on graphics_GpuReset
• 374641 - Samus & Beltino Canaries failing on Paygen stage with NoSectionError
• 360149 - Beltino Canary also failed HWTest on graphics_GLMark2
• 372024 - Nyan Canary failing HWTest on platform_DebugDaemonGetPerfData
• 375711 - ASAN builders failing to make_chroot
• 375713 - stumpy_moblab Canary failing paygen stage with "Image signing timed out"
• 375737, 375721 - Both rambi Canary builders are failing with a ton of stuff, but don't seem to be green anyways
• 339934 - Buildbot failure in DebugSymbols due to tar: debug/bin: file changed as we read it

2014-05-19 - 2014-05-20
Sheriffs: dgreid, benzh, seanpaul [05/20 only]

• 375275 Corrupted GPT image, all HWTests failing since image doesn't boot
• 374575 ChromeOS EC failing to build for nyan
• 374796 Chrome failing to build in skia/sse code
• 374792 Remote trybots cannot patch in changes
• 367835 AU and HW test experience Suite timed out. failures on slow and numerous DUTs

2014-05-15 - 2014-05-16
Sheriffs: grundler, owenlin, armansito  Gardener: hshi   Build Deputy: davidjames  Lab Sheriff: amilleral

2014-04-21

Sheriffs: wuchengli

• [365195] stumpy_moblab canary unittest failed
• [332665] [bvt] autoupdate_CatchBadSignatures Failure

2014-04-18

Sheriffs: wuchengli, semenzato, benchan

• [359223] [monroe] graphics_SanAngeles suspected to reboot/hang machine
• [365035] ChromeSDK failed on all full builders (because of LD=gold by default and probable gcc incompatibility)
• [364875] beaglebone canary build fails running out of space (FIXED)
• [364887] moblab unit tests failures
• [364818] x86-generic ASAN unittest failing due to compile warnings in metrics_daemon_test.cc

2014-04-17

2014-04-16

2014-04-15

Gardener: derat

• [363884] LKGMSync step failing repeatedly due to bad SHA1 when syncing coreboot
• started new x86-generic nightly chromium PFQ build since last night's run died on some slaves

2014-04-14

2014-04-11

Sheriffs: olofj, adlr, josephshi, ihf

• [362621] Removed pyauto dependencies that broke PFQ.

2014-04-05

Sheriffs: keybuk, bfreed

• [339291] Reverted a set of CLs that caused platform_Powerwash failure.
• [360797] The chumps from 339291 broke incremental builders and required deputy assistance.
• [360898] video_DecodeAccelerator is increasingly unreliable.  Maybe we should make it experimental?

2014-04-04

Sheriffs: keybuk, bfreed

• [360084] rambi-b canary build failed in the Archive stage on loopback mount failure.  Believed transient.
• [360082] Chrome PFQ fails with unknown linker flag (--reduce-memory-overheads), likely because of https://codereview.chromium.org/225093005 last night.

2014-04-02 & 2014-04-03

Sheriffs: pprabhu, dgreid

• [359143] generate_payload failed to unmount a file system, and we tried to rm it later. pprabhu@ forced canaries to restart, since ongoing runs were all going to hit this issue. But it took a while to take this decision.
  This hit us again later in the day, keeping canaries red almost all day. The reason was that a script had to be manually upreved to pull in the revert. See CL.
• [359227] VMTest hung. Root cause unknown.
• [359422] VMTest failed because VM ran out of space when running the tests. We reverted the Cl in the morning. Unfortunately, although all the canaries are back online, we can't get the CQ to pass yet, due to flakes + lots of CLs trying to get in. So, we need to uprev the manifest by allowing a noop CL through a throttled tree.
• [348199] and [353590] daisy_spring-pfq failed because of known GOLO and update engine flakes.
• [359760] beaglebone_servo canary is currently broken. [TODO: If the canary is still red at EOD, revert the CL mentioned in the bug].

2014-03-31

Sheriffs: katierh, dparker

• [358180] Daisy canary builder failure caused by error in a git repro. Existing error turned into a warning due to crbug.com/352692
• [358075] daisy_skate and daisy_spring canary failing due to clustered Chrome builds on one builder. Variants not using pre-built of Chrome from daisy.

2014-03-28

Sheriffs: dbasehore, armansito, sheckylin

• [357364] Tree doesn't close anymore when builds fail. Looks like it's fixed now.
• repeated failures in daisy canary during build packages.

2014-03-27

Sheriffs: dbasehore, armansito, sheckylin

• [353906] Builder out of space
• [357093] x86 generic ASAN fail due to Chrome
• [357202] Pre-CQ timeout.

2014-03-25

Sheriffs: dianders, vbendeb

• [356187] widespread provision failures; waiting for lab sheriff for the most part
• [356198] and [356199] video_VideoDecodeAccelerator - probably a duplicate of [353898].  There's  a possible fix for that but it hasn't gone back to R34 yet.
• ... lab issue is hopefully fixed now ...
• ... David James and crew restarted CQ ...
• ... various things handled by David James ...
• [355843] beltino canary - DebugSymbols hung for 600 seconds
• [348188] slippy canary and daisy canary - Flood of "Too many open files"
• Chrome uprev has failed a few times; Chrome sheriff handling?

2014-03-21 and 2014-03-24

Sheriffs: jrbarnette, tbroch

• [355843] beltino canary: DebugSymbols failed during upload with timeout
• daisy incremental: CQ bug missed bump of chromeos-init for CL:190619 initially (race?) then got it fixed making manual override here unnecessary.
• [353018] sandybridge canary: OSError(16, 'Device or resource busy') ... believed to be not enough loopback devices.
• [354573] A bug in Chrome caused the x86-generic paladin to fail multiple times in VM testing.

2014-03-19 and 2014-03-20

Sheriffs: yjlou, wfrichar, victoryang, hungte (TPE)

• [352994] cros_generate_test_payloads failed to find image folder (race condition)
• [353429] chrome/chromium pfq bots died in build_image due to missing libmojo_system.so
• [353461] failuer in uploading DebugSymbols

2014-03-13 and 2014-03-14

Sheriffs: cywang(TPE), dgarrett, bleung, gwendal

• [348855] amd64-generic-asan: logging_UserCrash timed out (flaky)
• [352093] daisy_spring: HWtest job timeout, but tests are still running
• [350677] x86-generic-full : cryptohome fails to link.
• [352276] platform2-0.0.1-r366 fails on arm-generic full
• [352297] Pre-CQ Failure- Gerrit Code Review requires Java 7
• [348855] amd64-generic-asan: logging_UserCrash timed out
• [352428] x86-generic asan : logging_AsanCrashTelemetry : Unhandled TabCrashException: Handshake Status 500
• [72633] x86-generic incremental: login_OwnershipNotRetaken
• [352520] atom canary: x86-mario: build_image failed (can't read superblock)

2014-03-11 and 2014-03-12

Sheriffs: dlaurie, grundler

• [348188] BVT timeout on panther, daisy
• [351339] Samus canary unittest failure in modemmanager-next
• [351355] Parrot VM has not passed since March 6th, fixed with https://chromium-review.googlesource.com/#/c/189575/
• [351356] x86-mario-canary failed manifestversionedsync since March 10, fixed by removing this canary
• [351719] hwtest timeout on multiple builders (new bug filed as per request)
• [351863] vmtest SimpleTestVerifyAndUpdate failed, unable to download image
• [351876] chromeos-cryptohome unittest failures
• [351870] chromeos-ec unittest failures (succeeds on retry)
• chromium PFQs broken due to bad Chrome, fixed with https://codereview.chromium.org/198103002/

2014-03-05 and 2014-03-06

Sheriffs: miletus, shawnn

• [337490] daisy incremental unmount completed, but returned an error
• [348855] amd64-generic-asain, logging_UserCrash timed out
• [348758] x86-generic-asan failure. not sure how to interpret the failure message.
• [349559] Signer failure on all canaries
• [343442] Wolf Paladin builder wedged
• [349597] chrome-internal-fetch netrc credentials revoked

2014-03-03 and 2014-03-04

Sheriffs: quiche, vpalatin

• [348607] Chrome PFQ failure. later Chrome builds cycled green.
• [345501] platform_FilePerms: jrbarnette@ has CL checked in, but it wasn't picked up by the lab server. lab team will update its server.
• [347932] security_AccountsBaseline (multiple times). cmasone@ investigating.
• [348059] chromiumos sdk builder failing. pinged bug.
• [348758] x86-generic-asan failure. not sure how to interpret the failure message.
• [348799] stumpdgarrett, bleung, gwendaly-paladin, reboot failure in autoupdate_CatchBadSignatures
• [348805] x86-generic, e2fsprogs failed to emerge
• [330670] breakpad unittest failure on amd64-generic
• [348855] amd64-generic-asain, logging_UserCrash timed out (2x)
• [348889] duck canary failure
• [345491] x86-mario canary: GSResponseError 403
• [349073] parrot canary: platform_PowerWash failed
• [337490] daisy incremental unmount completed, but returned an error
• [349187] x86-mario canary failed in ChromeSDK: out of disk?
• [349292] duck canary, GS_ERROR: Attempt to get key

OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)