Sheriff Log: Chromium OS

2014-09-01 - 2014-09-02

Sheriffs: tyanh

  • 409774: Canary master keeps getting timed out from daisy-release-group
  • Waiting for this CL to be in to fix canary master on spitting TypeError

2014-08-29

Sheriffs: jsalz, arakhov, benzh, jamescook (gardener)

  • 409346: amd64-generic incremental UnitTest failed
  • 409337: daisy canary failed paygen
  • 409312: falco nightly chrome PFQ failed HWTest
  • 409145: unable to find the server at www.googleapis.com
  • 408999: update_stateful fails on nyan
  • 407486CIDB connection failures
  • 408835: Google storage flakiness when uploading image.zip / prebuilts

2014-08-28

Sheriffs: dianders, sosa, henryhsu, jamescook (gardener)

  • 408436 x86 ASAN build broken, clang-only
  • 393738 Chrome uprev failure due to Git on borg git push failure
  • 408818 asan amd64 is leaky (shill)

2014-08-27

Sheriffs: waihong

  • 408005 rambi-b canary: failed to update DB: Data too long
  • 407877 again: cros flash failure
  • 408028 autotest unittest flake in autotest_lib.site_utils.log_socket_server_unittest
  • 408106 asan amd64 bot is leaky (debugd)
  • CL:214696 & CL:214676CL:214677 arm-generic kernel build broken in thermal layers

2014-08-26

Sheriffs: tyanh

  • 407486 daisy incremental: failed to access ICDB (so is x86-generic ASAN)
  • 357939 Call timed out: [au] platform_Powerwash_SERVER_JOB Failure on kip-release/R39-6202.0.0
  • 407480 Call timed out: [au] autoupdate_Rollback_SERVER_JOB Failure on kip-release/R39-6202.0.0
  • 407481 Connection reset by peer: [bvt-inline] security_Minijail0_SERVER_JOB Failure on quawks-release/R39-6202.0.0

2014-08-25 - 2014-08-26

Sheriffs: jwerner

  • 407879 x86-alex-paladin: HWTest infrastructure issues (cautotest-cq Internal Server Error)
  • 407877 rambi-c-release-group: VMTest: cros flash test failed
  • 396100 ManifestVersionedSync: Failed command "repo --time sync -n", and further up: fatal: cdb...822^0: not a valid SHA1
  • 404858 VMTest (login_GuestAndActualSession and others): No space left on device
  • 390433 some BuildAndArchiveFactoryImages problem (but looks like Paygen: "PushImage results not available" at first glance)
  • 407358 LockNotAcquired in gslock (Paygen)
  • 383453 security_NetworkListeners: Baseline mismatch, no idea what that means
  • CL:214001 broke chromite unittests... please trybot before you chump!

2014-08-22 - 2014-08-25

Sheriffs: seanpaul

  • 407106 slippy canary failing paygen (different from duck)
  • 406970 duck/daisy-skate/peach/nyan/stumpy canary failing paygen
  • 406971 sdk bot failing chrome sync

2014-08-20 - 2014-08-21

Sheriffs: waihong, henryhsu
  • CL:212994 - fix Chrome builds on x86-generic asan bots
  • 405609 Daisy chromium bot fails to build xorg-server-1.12.4-r39
  • 405662 daisy spring timed out in paygen
  • 405708 daisy full building chrome failed: transition from svn to git, will be good in the next build
  • 393749 again: Pre-CQ builder fails to push references
  • 406232 asan test failure w/shill

2014-08-18 - 2014-08-19

Sheriffs: vapier, wiley

  • GS network problems affecting all boards
  • 404410 - nyan fails building packages in Chrome
  • 236772 - nyan flake building Chrome (internal gold linker error)
  • 404215 - lumpy incremental fails with message about can't find cros_sdk
  • CL:213000 - Reverted, missing package kills tree
  • 405046 - freon builders broken by ozone change (405047 to help prevent future breakage)

2014-08-15 - 2014-08-17

Sheriffs: wiley, victoryang

  • 402335 - Reverted a flaky chromeos-ec test
  • b/17046642 Flicker in the bug tracker caused a bunch of 502s
  • 404074 - Closure in the morning due to overly slow HW tests
  • 404018 - gizmo paladin failing update engine unittests

2014-08-13 - 2014-08-14

Sheriffs: garnold, wfrichar, reveman

  • 403474 - SignerTest failed for candy (new board, not configured)
  • Reverted CL:170880 which caused a paygen failure (multiple closures)
  • 403555 - sdk bot fails due to librsvg->gtk+ dep
  • 402000 - canary failure due to lab network flake; lab sheriff (fdeng) notified.

2014-08-11 - 2014-08-12

Sheriffs: pprabhu, adlr,posciak

  • 402451 - PFQ failed due to telemetry bug in chrome
  • 402335 - EC unittest added recently failed.
  • 402645 - GoB manifest push failure -- flake, not hitting it any more (b/16958652).
  • 402691 - Full builders raced against CQ uprev? (Fixed itself in the next run)
  • 402793 - x86-generic asan failures
  • 403026 - Chromeos-sdk builder failure

2014-08-08 - 2014-08-11

Sheriffs: posciak, deymo, zeuthen

  • 402335 - link ec unittests failed
  • 402373 - Chrome PFQ uprev failing on importing gtest_output_formatter

2014-08-07

Sheriffs: deymo, zeuthen, kinaba

  • 401738 - autoupdate_EndToEndTest fails on R36
  • 401642 - File collision between opengles-headers and mesa on arm

2014-08-06

Sheriffs: ihf (gardener), armansito, kinaba


2014-08-04 - 2014-08-05

Sheriffs: derat (gardener), vapier (silently watching)

  • 400277 - daisy_spring PFQ failed with "Not enough DUTs to run this suite"
  • 400292 - Chromium OS (x86) Asan builder update_scripts fails with WebRTC checksum mismatch
  • 400484 - chromeos-chrome build fails with unrecognized -Wheader-hygiene and -Wstring-conversion flags
  • CL:211060 - jimtcl upgrade reverted to unbreak openocd-0.6.1 builds
  • 400917 - chromeos-chrome build fails when trying to copy output files
  • 394497 - PFQ failed due to missing libppGoogleNaClPluginChrome.so; chromite DEPS roll is in CQ

2014-07-31 - 2014-08-02

    Sheriffs: avakulenko, vbendeb, jchuang, vapier (silently watching)

    • cbuildbot config has been changed [398079] and waterfall restarted. This broke a bunch of builders which had checkouts to previous locations. davidjames@ fixed the configs...
    • 399285 - link_freon canary broken. Reverted CL:209574
    • "stumpy_moblab canary" repo sync fai - retrying
    • 399359 - Chrome PFQ AFDO failure (impacts Revision 286922, 286923)
    • SDK bot broke due to lddtree upgrade (CL:210352); landed CL:210358 to fix
    • 399512 - canary master timeout too low
    • erikchen@ reverted webrtc roll for Chrome PFQ failing (CL:435823002)
    • 399579 - debugd RandomSelector.SetOddsFromFileTest unittest flake
    • 399905 - PreCQ hitting traceback while parsing results
    • 399918 - Chrome PFQ wrongly publishing updates
    • 399920 / 399921 - Bad ACLs on rambi/link_freon binpkg breaking builds

    2014-07-29 - 2014-07-30

      Sheriffs: deanliao avakulenko vbendeb, flackr (gardener)

      • Pre-CQ launcher was inadvertently broken by clearing the history of the trybot repository and .trybot_minimal_format_version file was also deleted which was required for PCQL operation.
      • 7/30, 14:30 PST, Tree reopened, another instance of 394760
      • Tree reopen for known issue 383481
      • Duplicate issue 394760
      • 398947 - Compile failure on cros builder
      • 398505 - Passing builds on group builders marked as failures, preventing LKGM upgrade.
      • parrot replaced x86-alex as critical builder for chromeos.chrome waterfall

      2014-07-18 - 2014-07-21

        Sheriffs: dlaurie, stevefung, wuchengli, bleung

        • 394965 - telemetry_UnitTests was removed from autotest-chrome ebuild to let chrome PFQ succeed.
        • 395151 - GoogleKeyboardInput-xkb change added new invalid symlink, was chumped into the tree.  Reverted by sheriff, lots of subsequent builders failed.
        • failure in beaglebone canary during sync - likely due to http://crbug.com/375945.  CleanUp stage of next forced build cleared kernel 3.10 git folders, allowing it to continue syncing.
        • video_ChromeHWDecodeUsed test failures, not always the same failure mode but happening to several canary builders

        2014-07-16 - 2014-07-17

          Sheriffs: dparker, furquan, wuchengli

          • Tree down for maintenance 8:30 - 10ish (US-Pacific)...restarted.
          • 390719 - Tree throttled 12-2PM due to missing build configuration for public gizmo-full builder. Fall-out from earlier maintenance.  
          • 394623, 394625 - [bvt] audio_Microphone Failure
          • 394626 - [bvt] kernel_PerfEventRename Failure on nyan_big
          • 394001 - Multiple network problem in build infrastructure
          • 311350 - Lab machine failed to come back up on the network during platform_Powerwash. 200th occurrence of this bug :( 
          • 394965 - daisy-full failed at BuildPackages with telemetry_UnitTests import errors. Note that chromium and autotest changes were out of sync
                         There is expected to be a reciprocal error after chromium revs...and the revert will need reverting.

          2014-07-15 - 2014-07-16

            Sheriffs: itspeter(TPE)

            • 394241 - canary bot failed at cros_generate_update_payload step (still investigating)

            2014-07-14 - 2014-07-15

            Sheriffs: puthik, charliemooney, itspeter(TPE)

            • 394166 - update_engine fail in x86-generic ASAN
            • 394153 - Failed download in PayGen step aborts build
            • 394036 - security_ASLR failed on amd64-generic ASAN
            • 394001 - Builders unable to access chromite.git
            • 393749 - Pre-CQ builder fails to push references
            • 393818 - Storm paladin failed after merge chromeos-init to platform2
            • 393647 - kernel 3.14 build fail on arm-generic
            • 393749 - Pre-CQ builder fails to push references

            2014-07-11 - 2014-07-14

            Sheriffs: cychiang(TPE)

            • 355287 - chrome pfq graphics_GLMark2 bvt test failed. Flaky and failed the 3rd times this week.
            • 383481 - autoupdate_EndToEnd flaky on daisy_spring.

            2014-07-09 - 2014-07-10

            Sheriffs: skuhne (gardener), itspeter(TPE)

            • 392712 - file size mismatch of chromeos-chrome-amd64-38.0.2085.1.afdo.bz2, gabeblack@ and itspeter@ think this might be a transient issue on gsutil.
            • 392686 - HWTest failed. sbasi@ feels it is just flaky.
            • 392770 - chrome_pfq is failing because compiling error on chromeos-chrome-38.0.2086.2_rc-r1
            • 329912 - PDFBrowserTest.FindAndCopy flaky test on multiple bots
            • 392961 - cros-trunk and cros-stable have apparently never worked in the past 1000 builds

            2014-07-08 - 2014-07-09

            Sheriffs: jrbarnette, namnguyen, hychao

            • 391950 - this problem took out daisy 7/7; fixed as of first canary 7/8.  However...
            • 383481 - this problem took out the first daisy canary 7/8, when paygen ran AU tests.
            • 391952 - this problem has been taking out nyan since 7/3.
            • failure on monroe canary - blamed a possibly broken DUT in the test lab

              2014-07-03 - 2014-07-04

              Sheriffs: spang

              • 390177 - Nyan canaries failing with: platform_DebugDaemonGetPerfData (failing since the 24th).
              • 391306 - Chromite bug broke versioned manifest sync everywhere (fixed).
              • 391395 - [bvt] graphics_PiglitBVT Failure on quawks-release

              2014-06-30 - 2014-07-01

              Sheriffs: bfreed, olofj, josephsih
              • 390177 - Nyan canaries failing with: platform_DebugDaemonGetPerfData (failing since the 24th).
              • 390206 - ivybridge canary failed with: mkfs.vfat: Attempting to create a too large filesystem
              • 377104 - autoupdate_Rollback Failure on monroe-release
              • 390272 - Chrome PFQ Failing to uprev Chrome (Bad Chrome made it past the Chrome-PFQ)
              • 390433 - rambi-c canary Failed Paygen (attempt 2)
              • 390451, 390448, 390441, 390436 - [bvt] video_VideoDecodeAccelerator Failure on falco/wolf/enguarde/lumpy-release
              • 387152 - video_ChromeHWDecodeUsed failure. This is also a labeling problem. X86 should not execute the test.
              • 390693 - Closed the tree because CQ is currently broken. Also refer to http://chromegw/i/chromeos/builders/CQ%20master.

              2014-06-26 - 2014-06-27

              Sheriffs: zqiu, katierh, sbasi,
              • 389513 - [bvt] logging_UdevCrash Failure
              • 389517 - [bvt] login_OwnershipApi Failure

              2014-06-24 - 2014-06-25

              Sheriffs: jamescook (gardener)

                2014-06-23 - 2014-06-24

                Sheriffs: thieule, dhendrix, sheckylin
                • 387571 - [bvt] video_VideoDecodeAccelerator Failure
                • 387525 - ASan broken by Chrome update
                • 388114 - lab outage breaking all HWTests
                • 388151 - Chrome PFQs broken due to missing quantize_sse4.asm

                2014-06-20 - 2014-06-23

                Sheriffs: thieule, dhendrix, henryhsu
                • 383414 - video_VideoDecodeAccelerator Failure
                • 311350 - USB Dongle loses network connectivity
                • 387525 - ASan broken by Chrome update
                • 387702 - Potentially misconfigured server causing AU test to fail across canary builders
                • 339934 - Buildbot failure in DebugSymbols due to tar: debug/bin: file changed as we read it

                2014-06-18 - 2014-06-19

                Sheriffs: itspeter, sbasi, tbroch, tbarzic (gardener)

                2014-06-16 - 2014-06-17

                Sheriffs: marcheu, snanda, itspeter, tbarzic (gardener)

                • 383481 - daisy canary autoupdate_EndToEndTest.npo_test_delta,Host did not return from reboot
                • 385684 - remote trybots failing w/missing pbzip2 util
                • 385039 - hardware_RamFio aborts
                • 385465 - BVT times out on all paladins
                • 385571 - video_ChromeHWDecodeUsed and several others tests failed due to Chrome_IOThread segfault
                • 385819 - master-paladin encountered infra failures

                2014-06-12 - 2014-06-15

                Sheriffs: zork, yjlou, hungte (TPE), derat (gardener)
                • 375721 - graphics_PerfControl Failure
                • 383414video_VideoDecodeAccelerator failing on falco-chrome-pfq
                • 384290 - Incorrect calls to generate debug logs
                • 383977security_SandboxLinuxUnittests failing on daisy_spring
                • 384721 - chromeos-chrome compile failed in PFQ in VideoDecoderShim

                2014-06-10 - 2014-06-11

                Sheriffs: ???

                • 339934 - beaglebone canary failed DebugSymbols: "tar: debug/usr/sbin: file changed as we read it"
                • 382975 - Chrome PFQs dying while building mojo with fwrite warning errors
                • 383370 - moblab died in UnitTest phase while running autotest-server with Address already in use

                Note to next sheriff: Chrome was unpinned late at the request of dharani@, if it still isn't fixed might need to pin back again

                2014-06-06 - 2014-06-09

                Sheriffs: davidriley, wiley

                • jrbarnette reports: the test lab is seeing lots of devices go down after the platform_PowerWash test: turned out to be a result of lab reshuffling and machines not being connected up correctly after
                • 381553 - Lab melt-down.  fdeng to the rescue.
                • 382605 - telemetry change causing: AttributeError: 'InspectorBackend' object has no attribute 'id'
                • 381841 - cryptohome KeysetManagementTest.UpdateKeysetAuthorizedCompatVector failure
                • 382605 - Tons of tests using telemetry fail until PFQ picks up newer Chrome rev
                  • Telemetry change made which started causing tests to fail with: AttributeError: 'InspectorBackend' object has no attribute 'id'
                  • Telemetry change made, then reverted, then corrected
                  • Corrected version wasn't getting picked up since PFQ had been failing from ARM bots over the weekend due to crrev.com/275621. crrev.com/275775 fixes this.
                  • Pinned Chrome to 37.0.2034.1 to open tree, Chrome gardener will merge fixes to get a new Chrome build

                Note to next sheriff: continuous chromium PFQ failures on ARM bots over the weekend like this one should be fixed by crrev.com/275775 - posciak@.

                2014-06-05 - 2014-06-06

                Sheriffs: djkurtz (non-MTV)

                • 379603 - autotests fail with: TypeError: cannot concatenate 'str' and 'int' objects, but this seems to be masking another problem with subprocesses getting nuked b/c they aren't finishing in time.
                • 380735 - 'x86-generic-asan' fails "security_ASLR" (security_ASLR is not available on 32-bit ASan)
                • 381005 - 'sandybridge canary' bvt => flaky(?) PiglitBVT [shaders/glsl-unused-varying] on parrot
                • 380376 - 'stumpy_moblab canary' Can not sign moblab image
                • 375721 / 381008 - 'rambi-b canary'/glimmer-release - graphics_PerfControl - Unreasonable temperature 0.0C
                • 381553 - Lab melt-down.  beeps to the rescue.

                2014-06-03
                Sheriffs: dianders, gwendal, jcliang (non-MTV)

                • 380376 - Can not sign moblab image
                • 380352 - compilation error in chromeos-chrome: use of undeclared identifier '__sanitizer_sandbox_arguments'
                • 379657 - R36 [bvt] login_OwnershipApi Failure across boards
                • 379603 - rambi-a, rambi-b: login_Cryptohome Failure (Timed out going through login screen)
                • 355603 - rambi-a canary
                • 379438 - peach canary cpufreq problems => 379700
                • 379426 - rambi-b canary: login_SameSessionTwice
                • 380138 - logging_UdevCrash Failure on peach_pi-release - race could affect any device with atmel
                • 294221 - daisy canary autoupdate_EndToEndTest.npo_test_delta,Host did not return from reboot
                • 380306 - pineview canary security_Firewall_SERVER_JOB Failure

                2014-06-02
                Sheriffs: dianders, gwendal, cywang (non-MTV)

                • 379438 - peach canary cpufreq problems => 379700
                • 355603 - rambi-a canary
                • 379810 - beaglebone canary failed DebugSymbols: "tar: debug/usr/sbin: file changed as we read it"
                • 379871 - beltino-a canary signer
                • 379851 - rambi-c canary login_CryptohomeIncognito_SERVER_JOB
                • 379657 - R36 [bvt] login_OwnershipApi Failure across the board.

                2014-05-29
                Sheriffs: benchan, denniskempin, chihchung(TPE)

                • Bad ACL on xorg-server prebuilt binpkg fixed; randomly broke some daisy builds in PreCQ and incrementals/PFQ (vapier@)
                • platform_HwImageBoardCheck fails on expresso (crbug.com/375737)

                2014-05-27 - 2014-05-28
                Sheriffs: zeuthen, adlr, chihchung(TPE)

                2014-05-23 - 2014-05-26
                Sheriffs: garnold, sosa, tyanh(TPE)

                • 375912 - still failing (5/23); a number of hosts identified as exhibiting the problem were locked for use by hwtests; likely related to crbug.com/361331, fix under review.
                • 377070 - failures on rambi-a that looks like kernel crashes due to video driver failures.

                2014-05-23
                Sheriffs: tyanh(TPE)

                • 376715 - Beaglebone canary failing in upload_symbols.py: OSError: [Errno 2] No such file or directory: '/b/cbuild/internal_master/chroot/build/beaglebone/usr/lib/debug/breakpad/mbimcli/5C7E14771137D722C2967C5A2D3B65D40/mbimcli.sym'
                • 376629 - Daisy canary failing desktopui_MediaAudioFeedback with cryptohome-path failed: [0522/202013:ERROR:cryptohome.cc(36)] Could not get size of system salt: /home/.shadow/salt: No such file or directory

                2014-05-21 - 2014-05-22
                Sheriffs: deymo, arakhov, seanpaul [05/21 only], tyanh(TPE)

                • 375721 - Rambi-b on DUT 2-2-3-9 reporting invalid temp of 0.0C. Possible DUT or EC problem.
                • 376157 - Nyan Canary failing audio_Microphone autotest
                • 367278 - cryptohomed killed with SIGPIPE
                • 375912 - Daisy Canary HWTests failing with "The name org.chromium.Cryptohome was not provided by any .service files"
                • 375886 - Nyan Canary failing HWTest on network_WiFiInvalidParameters
                • 375717 - Daisy Canary HWTest failing with "Timed out going through login screen"
                • 358737 - Slippy Canary HWTest failing on graphics_GpuReset
                • 374641 - Samus & Beltino Canaries failing on Paygen stage with NoSectionError
                • 360149 - Beltino Canary also failed HWTest on graphics_GLMark2
                • 372024 - Nyan Canary failing HWTest on platform_DebugDaemonGetPerfData
                • 375711 - ASAN builders failing to make_chroot
                • 375713 - stumpy_moblab Canary failing paygen stage with "Image signing timed out"
                • 375737375721 - Both rambi Canary builders are failing with a ton of stuff, but don't seem to be green anyways
                • 339934 - Buildbot failure in DebugSymbols due to tar: debug/bin: file changed as we read it

                2014-05-19 - 2014-05-20
                Sheriffs: dgreid, benzh, seanpaul [05/20 only]

                • 375275 Corrupted GPT image, all HWTests failing since image doesn't boot
                • 374575 ChromeOS EC failing to build for nyan
                • 374796 Chrome failing to build in skia/sse code
                • 374792 Remote trybots cannot patch in changes
                • 367835 AU and HW test experience Suite timed out. failures on slow and numerous DUTs

                2014-05-15 - 2014-05-16
                Sheriffs: grundler, owenlin, armansito  Gardener: hshi   Build Deputy: davidjames  Lab Sheriff: amilleral

                • Unused Variable failure: kernel-next failed to build for 1 of 8 builds in that log. Even on the retry. LOCKDEP=y in base.config so this should fail for all builds. Temporary fix: crosbug.com/p/8952, crosreview.com/200327
                • ABORT Status failures: all canaries suffered timeouts likely due to graphics_PiglitBVT being added to both BVT and QAtest...and getting run twice for each build exceeding Canary test timeout. :( FIX: revert https://chromium-review.googlesource.com/#/c/199382/, bump timeouts: crosreview.com/200198
                • top people are working on fixing 374228 and 374288 (additional factors to ABORT timeouts).
                • daisy full and arm-generic-full first ones to fail in ChromeSDK: 373956 - fixed with CL 200130
                • all PFQ failed in ChromeSDK with same issue. restarted them all.
                • amd64-generic failed due to git push failures
                • 28176  daisy,sandybridge, etc canaries failing video tests 
                • 372991 Tests ABORT status - likely still related to 373022 (DHCP server not running)
                • 28868 video Bay trail graphics failures
                • 355780 video_VimeoVideo has been failing since March :( - added owenlin/yuli/posciak to the bug
                2014-05-13 - 2014-05-14
                Sheriffs: jwerner, pstew Gardener: harrym, Build Deputy: yjhong, Lab Sheriff: dshi
                • 372906 Unit test failures in chromite holding back chromium PFQ
                • 372991 Tests aborting.  Turns out this was a lab DHCP server issue
                • 373022 DHCP server in the lab failed to start up
                • 372572373083 More chromite unittest breakage, continuing overnight
                • 28868 Bay trail graphics failures in BVT
                • 373368 transient issue with Pre-CQ (HTTP 404)
                2014-05-08 - 2014-05-09
                Sheriffs: bleung | ChromeOS: djkurtz (non-PST) | Gardener: skuhne
                • 371244 on rambi-c/beltino.  signerresults failed with ParallelAttributeError: No such board-specific parallel run attribute 'instruction_urls_per_channel'
                  • Fixed by yjhong: https://chromium-review.googlesource.com/198785
                • 366141 ALL canaries timing out during bvt
                • 371324 security_SandboxStatus failing with Unhandled TabCrashException on peach_pit
                • PFQ failed to uprev. -> 371877 fixed script which should cycle green again
                • 370416 caused failures with speech recognition. Fix got merged into branch 1916

                2014-05-06 - 2014-05-07
                Sheriffs: grt, dalecurtis, andresantoso | ChromeOS: zqiu, denniskempin, fjhenigman (non-PST) | Gardener: girard
                • PFQ failure caused by 281689. Got fixed, but takes a while to get picked up by the PFQ bots.
                • 370829 provision_AutoUpdate fails in BVT.
                • 370803 autoupdate_Rollback fails in BVT.
                2014-04-30 - 2014-05-01
                Sheriffs: ..., ..., posciak (non-PST), Gardener: rginda
                • 366981 network_Ping fails pinging one of the hosts (default gateway?)
                • More of 355843, (DebugSymbols 600sec timeout)
                2014-04-24 - 2014-04-25
                Sheriffs: jwerner, pstew Gardener: bshe, Build Deputy: yjhong, Lab Sheriff: beeps
                • 366158 video_VideoEncodeAccelerator failure now breaking BVT as well
                • 366537 login_MultipleSessions fails in BVT
                • 365973365976366283366292366460366465366552366588 video_ChromeHWDecodeUsed causing a spate of failures
                • 346868 login_LoginSuccess makes a re-appearance in the PFQ uprev failures, but this is likely due to BrowserConnectionGoneException
                • 366577366581 graphics_SanAngeles failures popping back up
                • 366592 login_LogoutProcessCleanup also failing too, so there's clearly something systematic (in telemetry?) causing BrowserConnectionGoneException failures.
                • 366593 (login_OwnershipTaken), 
                • 356020 - Reset failing in BVT sporadically since Mar 24.  Odd that it hasn't been assigned to anyone yet for triage.
                • 366823 - daisy_spring DUTs are all non-functioning.
                • 366988 parallel_emerge failed with IOError: [Errno 9] Bad file descriptor
                • CL:196992 to fix sdk bot failure in strongswan and ipsec user/group
                • 367086: [bvt] login_OwnershipNotRetaken Failure
                • 367174: Tests aborting on HW tests
                2014-04-23
                Sheriffs: vapier Gardener: jamescook
                • 365507 Login screen failures due to multiple blink regressions
                • 365982 security_Firewall failed
                • 366067 graphics_GpuReset hung the kernel
                • 366142 Assertion about window opacity/visibility
                • 366141 Suite aborts from a timeout without any other failures (causes PFQ failures, due to lab overload)
                • 366158 video_VideoEncodeAccelerator failure breaking Chrome PFQ
                • CL:196473 and CL:196481 asan bots failed in unittest due to bug in bootstat unittest
                        2014-04-22
                        Sheriffs: vapier Gardener: jamescook

                                2014-04-21

                                Sheriffs: wuchengli

                                • [365195stumpy_moblab canary unittest failed
                                • [332665] [bvt] autoupdate_CatchBadSignatures Failure

                                2014-04-18

                                Sheriffs: wuchengli, semenzato, benchan

                                • [359223] [monroe] graphics_SanAngeles suspected to reboot/hang machine
                                • [365035] ChromeSDK failed on all full builders (because of LD=gold by default and probable gcc incompatibility)
                                • [364875] beaglebone canary build fails running out of space (FIXED)
                                • [364887] moblab unit tests failures
                                • [364818] x86-generic ASAN unittest failing due to compile warnings in metrics_daemon_test.cc

                                2014-04-17

                                Sheriffs: milleral, thieule

                                • [364617] Beaglebone servo image is too big
                                • [356020] [bvt] reset Failure on falco-chrome-pfq/R35-5684.0.0-rc4
                                • [355843] TreeCloser: build failure in DebugSymbols 600sec timeout
                                • [364669] Daisy skate build is failing on serial-tty
                                • [359223] [monroe] graphics_SanAngeles suspected to reboot/hang machine
                                • [358737] [bvt] graphics_GpuReset Failure on stout-release/R36-5718.0.0

                                2014-04-16

                                Sheriffs: milleral, thieule

                                2014-04-15

                                Gardener: derat

                                • [363884] LKGMSync step failing repeatedly due to bad SHA1 when syncing coreboot
                                • started new x86-generic nightly chromium PFQ build since last night's run died on some slaves

                                2014-04-14

                                Sheriffs: jchuang, reinauer, Gardener: derat

                                • [363339] stumpy moblab failure 
                                • [363294] sandybridge-canary failed.
                                • [362999] Failed cbuildbot failed hwtest [bvt] [parrot_ivb]. Transient.
                                • [358737, 356020] Transient HWtest fail on falco and wolf (both issues have been auto filed many times)
                                • [363015] Failed cbuildbot failed debugsymbols [x86-mario]
                                • [363167] HWTest step timed out on daisy_spring and falco PFQ

                                2014-04-11

                                Sheriffs: olofj, adlr, josephshi, ihf

                                • [362621] Removed pyauto dependencies that broke PFQ.

                                2014-04-05

                                Sheriffs: keybuk, bfreed

                                • [339291] Reverted a set of CLs that caused platform_Powerwash failure.
                                • [360797] The chumps from 339291 broke incremental builders and required deputy assistance.
                                • [360898] video_DecodeAccelerator is increasingly unreliable.  Maybe we should make it experimental?

                                2014-04-04

                                Sheriffs: keybuk, bfreed

                                • [360084] rambi-b canary build failed in the Archive stage on loopback mount failure.  Believed transient.
                                • [360082] Chrome PFQ fails with unknown linker flag (--reduce-memory-overheads), likely because of https://codereview.chromium.org/225093005 last night.

                                2014-04-02 & 2014-04-03

                                Sheriffs: pprabhu, dgreid

                                • [359143] generate_payload failed to unmount a file system, and we tried to rm it later. pprabhu@ forced canaries to restart, since ongoing runs were all going to hit this issue. But it took a while to take this decision.
                                  This hit us again later in the day, keeping canaries red almost all day. The reason was that a script had to be manually upreved to pull in the revert. See CL.
                                • [359227] VMTest hung. Root cause unknown.
                                • [359422] VMTest failed because VM ran out of space when running the tests. We reverted the Cl in the morning. Unfortunately, although all the canaries are back online, we can't get the CQ to pass yet, due to flakes + lots of CLs trying to get in. So, we need to uprev the manifest by allowing a noop CL through a throttled tree.
                                • [348199] and [353590] daisy_spring-pfq failed because of known GOLO and update engine flakes.
                                • [359760] beaglebone_servo canary is currently broken. [TODO: If the canary is still red at EOD, revert the CL mentioned in the bug].

                                2014-03-31

                                  Sheriffs: katierh, dparker

                                  • [358180] Daisy canary builder failure caused by error in a git repro. Existing error turned into a warning due to crbug.com/352692
                                  • [358075] daisy_skate and daisy_spring canary failing due to clustered Chrome builds on one builder. Variants not using pre-built of Chrome from daisy.

                                  2014-03-28

                                  Sheriffs: dbasehore, armansito, sheckylin

                                  • [357364] Tree doesn't close anymore when builds fail. Looks like it's fixed now.
                                  • repeated failures in daisy canary during build packages.

                                  2014-03-27

                                  Sheriffs: dbasehore, armansito, sheckylin

                                  • [353906] Builder out of space
                                  • [357093] x86 generic ASAN fail due to Chrome
                                  • [357202] Pre-CQ timeout.

                                  2014-03-25

                                  Sheriffs: dianders, vbendeb

                                  • [356187] widespread provision failures; waiting for lab sheriff for the most part
                                  • [356198] and [356199] video_VideoDecodeAccelerator - probably a duplicate of [353898].  There's  a possible fix for that but it hasn't gone back to R34 yet.
                                  • ... lab issue is hopefully fixed now ...
                                  • ... David James and crew restarted CQ ...
                                  • ... various things handled by David James ...
                                  • [355843] beltino canary - DebugSymbols hung for 600 seconds
                                  • [348188] slippy canary and daisy canary - Flood of "Too many open files"
                                  • Chrome uprev has failed a few times; Chrome sheriff handling?

                                  2014-03-21 and 2014-03-24

                                  Sheriffs: jrbarnette, tbroch

                                  • [355843] beltino canary: DebugSymbols failed during upload with timeout
                                  • daisy incremental: CQ bug missed bump of chromeos-init for CL:190619 initially (race?) then got it fixed making manual override here unnecessary.
                                  • [353018] sandybridge canary: OSError(16, 'Device or resource busy') ... believed to be not enough loopback devices.
                                  • [354573] A bug in Chrome caused the x86-generic paladin to fail multiple times in VM testing.

                                  2014-03-19 and 2014-03-20

                                  Sheriffs: snanda

                                  • [344506]: peppy canary failed to reboot due to ASIX USB issue.
                                  • [352276]: falco canary platform2 build failure.  http://crosreview.com/190820 is the fix but still waiting to be blessed by CQ.
                                  • [354496]: monroe paladin misbehaved for a while.
                                  • [354262]: sandybridge build failed.  DUT was down?
                                  • [311350]: platform_Powerwash Failure on daisy_spring-release. USB dongle flakiness?

                                  2014-03-17 and 2014-03-18

                                  Sheriffs: yjlou, wfrichar, victoryang, hungte (TPE)

                                  • [352994] cros_generate_test_payloads failed to find image folder (race condition)
                                  • [353429] chrome/chromium pfq bots died in build_image due to missing libmojo_system.so
                                  • [353461] failuer in uploading DebugSymbols

                                  2014-03-13 and 2014-03-14

                                  Sheriffs: cywang(TPE), dgarrett, bleung, gwendal

                                  • [348855] amd64-generic-asan: logging_UserCrash timed out (flaky)
                                  • [352093] daisy_spring: HWtest job timeout, but tests are still running
                                  • [350677] x86-generic-full : cryptohome fails to link.
                                  • [352276] platform2-0.0.1-r366 fails on arm-generic full
                                  • [352297] Pre-CQ Failure- Gerrit Code Review requires Java 7
                                  • [348855] amd64-generic-asan: logging_UserCrash timed out
                                  • [352428] x86-generic asan : logging_AsanCrashTelemetry : Unhandled TabCrashException: Handshake Status 500
                                  • [72633] x86-generic incremental: login_OwnershipNotRetaken
                                  • [352520] atom canary: x86-mario: build_image failed (can't read superblock)

                                  2014-03-11 and 2014-03-12

                                  Sheriffs: dlaurie, grundler

                                  2014-03-05 and 2014-03-06

                                  Sheriffs: miletus, shawnn

                                  • [337490] daisy incremental unmount completed, but returned an error
                                  • [348855] amd64-generic-asain, logging_UserCrash timed out
                                  • [348758] x86-generic-asan failure. not sure how to interpret the failure message.
                                  • [349559] Signer failure on all canaries
                                  • [343442] Wolf Paladin builder wedged
                                  • [349597] chrome-internal-fetch netrc credentials revoked

                                  2014-03-03 and 2014-03-04

                                  Sheriffs: quiche, vpalatin

                                  • [348607] Chrome PFQ failure. later Chrome builds cycled green.
                                  • [345501] platform_FilePerms: jrbarnette@ has CL checked in, but it wasn't picked up by the lab server. lab team will update its server.
                                  • [347932] security_AccountsBaseline (multiple times). cmasone@ investigating.
                                  • [348059] chromiumos sdk builder failing. pinged bug.
                                  • [348758] x86-generic-asan failure. not sure how to interpret the failure message.
                                  • [348799] stumpdgarrett, bleung, gwendaly-paladin, reboot failure in autoupdate_CatchBadSignatures
                                  • [348805] x86-generic, e2fsprogs failed to emerge
                                  • [330670] breakpad unittest failure on amd64-generic
                                  • [348855] amd64-generic-asain, logging_UserCrash timed out (2x)
                                  • [348889] duck canary failure
                                  • [345491] x86-mario canary: GSResponseError 403
                                  • [349073] parrot canary: platform_PowerWash failed
                                  • [337490] daisy incremental unmount completed, but returned an error
                                  • [349187] x86-mario canary failed in ChromeSDK: out of disk?
                                  • [349292] duck canary, GS_ERROR: Attempt to get key


                                  OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)

                                  Comments