2014-03-05 and 2014-03-06

Sheriffs: miletus, shawnn

• [337490] daisy incremental unmount completed, but returned an error
• [348855] amd64-generic-asain, logging_UserCrash timed out
• [348758] x86-generic-asan failure. not sure how to interpret the failure message.
• [349559] Signer failure on all canaries
• [343442] Wolf Paladin builder wedged
• [349597] chrome-internal-fetch netrc credentials revoked

2014-03-03 and 2014-03-04

Sheriffs: quiche, vpalatin

• [348607] Chrome PFQ failure. later Chrome builds cycled green.
• [345501] platform_FilePerms: jrbarnette@ has CL checked in, but it wasn't picked up by the lab server. lab team will update its server.
• [347932] security_AccountsBaseline (multiple times). cmasone@ investigating.
• [348059] chromiumos sdk builder failing. pinged bug.
• [348758] x86-generic-asan failure. not sure how to interpret the failure message.
• [348799] stumpy-paladin, reboot failure in autoupdate_CatchBadSignatures
• [348805] x86-generic, e2fsprogs failed to emerge
• [330670] breakpad unittest failure on amd64-generic
• [348855] amd64-generic-asain, logging_UserCrash timed out (2x)
• [348889] duck canary failure
• [345491] x86-mario canary: GSResponseError 403
• [349073] parrot canary: platform_PowerWash failed
• [337490] daisy incremental unmount completed, but returned an error
• [349187] x86-mario canary failed in ChromeSDK: out of disk?
• [349292] duck canary, GS_ERROR: Attempt to get key

2014-02-19 and 2014-02-20

Sheriffs: olofj, pstew

• mario-canary fails in UReadAheadServer.  No logs.
• Chrome uprev failed on thermal, dianders@ to revbump package, but failed again
• [344914] CQ failing due to failure to build hostapd, deemed to be a corrupted tarball in the buildier's cache.
• [345098] New print_repo_status.py factory install script broke the archive process.
• [345210] Rash of signer test failures (alex, slippy, parrot, leon)
• [345479] VMTests fail with 'NoneType' object has no attribute 'Cleanup'
• [345491] GS AccessDenied error while uploading prebuilts for slippy_canary.  Invoking troopers.
• [345476] login_CryptohomeIncognitoTelemetry and ScreenLockerTelemtry [344849] causing chrome uprev issues

2014-02-17 and 2014-02-18

waihong (TPE), bhthompson, marcheu:

• peach_pit canary hwtest flake -> crobug.com/344427
• amd64-generic paladin machine went offline for a while. Contacted the Trooper to fix.
• stumpy canary hwtest flake - happened again, crbug.com/344173

2014-02-13 and 2014-02-14
reinauer, garnold

• beaglebone canary failed on DebugSymbols stage; appears to be a flake (crbug.com/344059).

If you are seeing double-free/heap corruption errors when running gn during ChromeSDK runs (e.g.:

https://uberchromegw.corp.google.com/i/chromeos/builders/x86-alex%20canary/builds/4655/steps/ChromeSDK%20%5Bx86-alex%5D/logs/stdio)

it's probably crbug.com/335587. Please see my explanation in the bug.

- posciak

2014-02-11

benchan, sosa, owenlin:

• canary failures: peach_pit hwtest (crbug.com/336839), parrot autest (crbug.com/342644), stout hwtest (crbug.com/342725).

2014-02-10

dkrahn, adlr, kcwu (TPE):

• Due to DiRT, MTV was offline and internal waterfall was affected ~14:00 - 17:00. Some buildbots were affected as well.
• Another chrome pfq vmtest failure on falco. crbug.com/212879 - these don't actually block uprev, see crbug.com/342425
• Google Storage issues (fiber cut) cause canary failures - https://a.corp.google.com/#102649
• Filed crbug.com/342497 for link canary build error.
• linux_chromeos dbg 2 bot has very long cycle time (~4 hours) so failures may show up late, filed crbug.com/342588

2014-02-07

dkrahn, adlr, kcwu

• mario canary hwtest bvt failure - emailed troppers to escalate. This has been going on for a while now, it seems
• chrome pfq vmtest failure - next build cycled green, so letting this one go
• falco chrome pfq vmtest failure - crbug.com/212879
• mario canary hwtest bvt failure ended up being a combination of crbug.com/339702 and no logs reported - crbug.com/341494
• investigated repeated failures on x86-generic asan builder - filed crbug.com/341922
• daisy_spring starvation in the lab due to crbug.com/339636
• wolf_canary lab flake due to crbug.com/340839

2014-02-06

skuhne, jwerner, thieule:

• Tryserver unavailable ~13:00 - 16:00
• HW lab down 14:39 - 15:46
• crbug.com/341658: dev_install failed due to connection timeout
• crbug.com/337490: amd64-generic-incremental build failure during build image due to unmountable partition. Device busy.
• crbug.com/212879: system sometimes does not come up after reboot in VMTest
• SimpleTestAndVerify fails on x86generic ASAN. Saw that yesterday already, but there were many more problems.. (-> crbug.com/337848)
• PFQ for daisy_spring is still failing in HWTests since there are apparently no machines since 6 days (-> crbug.com/339636)

2014-02-05

mtennant, skuhne, jwerner, thieule (MTV):

• PFQ for falco, lumpy, .. had failures. Might be fluke upon "Failed to run /home/chrome-bot/depot-tools/gclient runhooks". Retriggered / clobbered / but no success (-> crbug.com/341179) PFQ seems still to be broken @5:45pm, but it will cycle tomorrow morning green since Chrome was red as well.
• PFQ x86: lab failure or restarting VM's (-> crbug.com/212879)
• Several x86-generic and amd64-generic failures in SimpleTestUpdateAndVerify (-> crbug.com/212879), VM hung on reboot, rolling back 3.10 kernel switch on generic to fix)
• daisy canary - random tests failed due to crashdumps from (non-fatal) Xorg crashes that took too long to symbolize (client test is marked GOOD but server job times out)
• daisy_spring canary - all tests after a certain point in the suite failed with ABORT (suite hit 2h timeout since not enough lab devices available soon enough to finish on time)
• dev_install test is failing on canaries - crbug.com/341266 (offending change was rolled back with some difficulties problem was identified and will be fixed on reupload)

2014-02-04

posciak (TOK)

• Went over 8MB on kernel image failing the build on bayleybay; see crbug.com/340596
• amd64-generic ASAN failing to clone chromite with RPC errors (http://chromegw/i/chromiumos/builders/amd64-generic%20ASAN/builds/9031)

2014-02-03

dianders, rspangler

• Pre-shift failures:
• x86/amd64 generic ASAN bots seem to have been failing for a long time - TBD
• lumpy canary - http://crbug.com/338622 - login_CryptohomeTelemetry
• daisy spring canary - TBD (seems to have been failing for a long time)
• Chrome PFQ failed to uprev on daisy_spring - Looks similar to daisy spring canary failures?
• Chrome PFQ failed to uprev on falco - TBD
• amd64-generic incremental - build image (filesystem was unmounted, but mount(8) failed: Device or resource busy) => http://crbug.com/337490
• wolf canary - archive build failure - gsutil - ValueError: need more than 2 values to unpack => http://crbug.com/340502

2014-01-31

tbroch, jrbarnette

• crbug.com/339934 race/flake for tar during DebugSymbols ... tar: debug/bin: file changed as we read it
• crbug.com/335587 double-free/corruption errors when running gn during ChromeSDK stage
• crbug.com/339743 [bvt] network_VPNConnect.l2tpipsec_cert Failure
• crbug.com/337490 mount failure during build_image (error status 32)

posciak (TOK)

• See crbug.com/335587 for a probable reason behind occasional double-free/corruption errors when running gn during ChromeSDK stage
• build failures as CQ missed one of the CQ-DEPEND CLs, because it was uploaded as a draft
• login timeouts on link canary in a few bvt tests; suspecting https://codereview.chromium.org/148843002 to have made login last longer/stop working... may need to followed up on if persists;

2014-01-30

tbroch, jrbarnette

• crbug.com/339573 failed to uprev chrome 34.0.1813 due to proto change for LocalExtensionCache::CacheItemInfo::CacheItemInfo
• contacted chrome gardner (harrym) to resolve
• crbug.com/310783 stumpy canary. 
• crbug.com/339135 leon/samus/link/panther canary failures for vm_test fix here.
• crbug.com/338085 radvd ebuild failure ... fixed with clobber build.  Email triage by (jamescook, avi, xiyuan, achuith)

posciak (TOK)

2014-01-24/2014-01-27

vapier

• more fall out from user eclass reverts http://crbug.com/339139
• ReportStage broke most bots CL:184161 & CL:184104
• openssl change broke sdk bot CL:184330

2014-01-24/2014-01-27

katierh, armansito

• Jan 27 was full of clobbering...
• Needed a few reverts due to bad eclass CLs landing - and then lots of clobbering, removing prebuilts, etc to get the tree in a sane shape - crosbug.com/338085
• Jan 24 had a number of failures due to Gaia corp errors...
• spring security_ptraceRestrictions - crosbug.com/333452
• parrot HWTest temporarily disabled on CQ
• Gerrit outtage - crbug.com/337921
• platform_CryptohomeMount flake on daisy crbug.com/337891 
• Failure during build_image (flake?) crbug.com/337881
• butterfly: 328078/337726, samus: 337848, alex: 337800/336767, lumpy: 337817

2014-01-23

derat, wiley, dparker

• crbug.com/337490 failure in build image: mount(8) failed: Device or resource busy
• Timeouts in login_CryptohomeTelemetry. Fixed by this revert.
• PFQ failures with "TEST_NA: Unsatisfiable DEPENDENCIES" caused by a server dying in the lab (per scottz@).
• CQ fails in unittests due to timeout in chromite: crbug.com/337602

2014-01-22

derat, wiley, dparker

• crbug.com/336742: all PFQs failed due to factory-test-init and chromeos-test-init conflict (see 2014-01-20). forced rebuilds
• crbug.com/334958: two CertificateManagerBrowserTest tests failing on "Linux ChromiumOS Tests (dbg)(2)" builder
• BVT failure on pit caused by kernel crashes: crbug.com/336839
• BVT failure on ZGB caused by flake in network_DhcpStaticIP: crbug.com/336767

2014-01-20

2014-01-16 and 2014-01-17

2014-01-10 and 2014-01-13

bleung, dbasehore, spang

• crbug.com/333310: Node.js issue with downloading Chrome. Caused all canary builders and full builders to fail.
• crbug.com/333398: Delay between ebuild commit and uprev commit
• crbug.com/332645: beltino canary failed in archive, might be low memory issue.

2014-01-06 and 2014-01-07

jsalz (TPE), dlaurie, grundler

• crbug.com/332104: Recurring issue in ManifestVersionedSync step on several builders (zgb, falco, peppy, stout) and experimental builders.  This is the top item for tree closure on 1/7.
• crbug.com/332145: Chrome PFQ nightly failing to compile.  Fixed in chrome already, should be good for tomorrow's build.
• crbug.com/327651: autopdate_EndToEndTest failure
• crbug.com/329248: "Update failed" in VMTest
• daisy_incremental out of space
• crbug.com/327388: experimental_platform_RebootAfterUpdate suspected of putting machines in Repair Failed state.
• crbug.com/331176: login_CryptohomeIncognitoTelemetry suspected of putting machines in Repair Failed state.
• crbug.com/328360: audiovideo_VDA failed
• crbug.com/331318: "Suite prep" failure - probable bvt timeout [update: fixed on parrot_ivb]
• crbug.com/331754: platform_FilePerms: "/dev/pts" is missing options "set(['mode=620', 'gid=5'])"
• crbug.com/324907: GerritHelperTest unit test failure
• crbug.com/331756: UploadPrebuilts fails with CommandException: Invalid canned ACL

2014-01-02 and 2014-01-03

djkurtz (TPE)

• crbug.com/329777 - autoupdate_CatchBadSignatures hash failure
• crbug.com/317309 - "daisy canary" - autoupdate_Rollback failed failed to find a job_repo_url for the given host
• crbug.com/331318 - "parrot canary" - time out during bvt 

2013-12-25 and 2013-12-26

hungte (TPE)

• crbug.com/330553 - Mario BSP build failure in incremental builds
• crbug.com/330704 - EC unit test flaky (next few builds all pass)
• crbug.com/330670 - Breakpad test failure

2013-12-23 and 2013-12-24

jcliang (TPE)

• crbug.com/325610 - Reproduced on Slippy

2013-12-19 and 2013-12-20

cywang (TPE), gabeblack, zork

• crbug.com/322436 - Reproduced on Lumpy
• crbug.com/329777 - autoupdate_CatchBadSignatures failed
• crbug.com/329814 - (experimental bvt test failure in udev)
• crbug.com/330119 - bvt test failure by connection issue between buildbot and test machine.

2013-12-17 and 2013-12-18

waihong (TPE), shawnn, charliemooney

• crbug.com/329145 - chromiumos-status.appspot.com is down: OverQuoteError: The API call datastore_v3.RunQuery() required more quota than is available.
• crbug.com/328589 - Butterfly provisioning failing due to 64-bit AU
• crbug.com/329260 - UploadSymbols network flake
• crosbug.com/p/24530 - ARM generic builder failing
• crbug.com/329370 - Falco provisioning failure
• crbug.com/322436 - Re-occurrence on Lumpy
• crbug.com/329567 - Lumpy network_DefaultProfileCreation HWTest failure
• crbug.com/328731 - Daisy recurring logging_CrashSender issue
• Various crbugs: Butterfly R33 provisioning failure, fallout from crbug.com/328589 not making R33 branch point

2013-12-09 and 2013-12-10

sabercrombie, milleral, miletus, mtennant (Chrome OS build deputy), rginda (Chrome gardner)

• crbug.com/327005 - chromeos-base/telemetry failed on chrome_pfq, multiple platforms
• crbug.com/327007 - Parrot canary failed because of chromeos-chrome build failed

2013-12-5 and 2013-12-6

2013-12-3 and 2013-12-4

2013-12-2 (and 11-29 - holiday)

OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)