Sheriff Log: Chromium OS

2015-03-06
Sheriffs: benchan, tbroch
Gardener: tengs


2015-03-05
Sheriffs: benchan, tbroch
Gardener: tengs
  • 464407 Your "Oauth 2.0 User Account" credentials are invalid .... Failure: Invalid response 302..

2015-03-04
Sheriffs: dhendrix, waihong
Gardener: derat

2015-03-03
Sheriffs: dhendrix, waihong
Gardener: derat
  • 462842 Chromium OS (amd64) Asan bot failing unittests due to webserver leaks
  • 463493 Flaky failure in breakpad's linux_client_unittest on X86 (chromium)
  • 463532 browser_tests failing on cros trunk in WebViewTest.FileSystemAPIRequestFromWorkerDeny
  • 464053 Beaglebone kernel too big, causing build failures
  • 463411 pool: bvt, board: leon in a critical state. (Some DUTs had their USB ethernet dongles swapped and were down for a bit)
2015-03-02
Sheriffs: dgreid, tbroch
Gardener: derat
  • 461406 Chromium OS (x86) Asan bot still failing
  • 458775 peach_pit nightly chrome PFQ failed with too few available DUTs
  • 463213 X86 (chromium), Daisy (chromium), and AMD64 (chromium) failing on chromeos-chrome with "ValueError: invalid literal for int() with base 10"
2015-02-27
Sheriffs: dgreid, tbroch
2015-02-25
Sheriff: dianders, pstew, jchuang
  • 462240 [storm-release canary] storm-release: The BuildPackages stage failed: Packages failed in ./build_packages
  • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
  • 461841 rambi-c-release-group: timed out

2015-02-25
Sheriff: dianders, pstew, tyanh
  • 461184 [canary] HWTest did not complete due to infrastructure issues again in canary 670; also "peach-release-group: timed out"
  • 461841 [canary] sandybridge-release-group: The Paygen [butterfly] stage failed: <class 'chromite.lib.timeout_util.TimeoutError'>: Timeout occurred- waited 13800 seconds
  • 461893 [canary] rambi-a-release-group: The HWTest [expresso] [bvt-inline] stage failed: ** Suite timed out before completion **
  • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
2015-02-24
Sheriff: amstan, gwendal, tyanh
  • 461184 [canary, chrome pfq] HWTest did not complete due to infrastructure issues (HWTest Lab closed?)
  • 461188 [pineview canary] Operation timed out at the end of a build; transient, subsequent build succeeded.
  • 438908 Image signing timed out across many platforms on canary; transient, subsequent builds succeeded.
  • 461378 chrome compilation error prevents some build to complete in CQ.
  • 461415 critical fix in chrome was only present on chrome ToT.

2015-02-23
Sheriff: amstan, gwendal  Gardener: flackr
  • 460815 Cros SDK broken in .bashrc_profile
  • 460693 Chrome PFQ failing with exception: global name 'AccessDeniedException' is not defined in File "src/build/download_sdk_extras.py", line 71
  • 460951 ChromeMetricsServiceAccessorTest.MetricsReportingEnabled, ExternalCacheTest.Basic, ExternalCacheTest.PreserveInstalled, DeviceLocalAccountExternalPolicyLoaderTest.ForceInstallListSet failing on cros_trunk
  • 458122 Reopened FileSystemProviderApiTest.BigFile failing on cros_trunk as test is failing 98% of runs on cros_trunk
  • 461021 MetricsServicesManagerTest.GetRecordingLevelCoarse faililng on cros_trunk.
  • 461046 MetricsServicesManagerTest.GetRecordingLevelFine faililng on cros_trunk.
    2015-02-18
    Sheriff: posciak, namnguyen, snada
    • Note to PST sheriffs: lots of infrastructure failures (I think) over the last few days, which I don't really understand. I think we need help from Infra team before we can reopen the tree for good.
    • 419904 IndexError: list index out of range on moblab_RunSuite 
    • 458613 Pre-CQ Launcher failures
    • 224 Moblob failures
    • 459679: Moblab blocks canaries
    • b/19426205: Missing commit info

    2015-02-13
    Gardener: jonross
    • Reverted a change that was failing a compile on Chrome OS.
    • 458567 Disable flaky InProcessAccessibilityBrowserTest.VerifyAccessibilityPass
    • 458549 Disable flaky TextInput_TextInputStateChangedTest.SwitchingAllTextInputTest
    • 458526 Chromium OS Waterfall bots falling on LKGMSync
    • 458918 amd64 asan failing shill unittests due to leaks
    2015-02-12
    Gardener: jonross
    • 458341 Disable flaky LoginPromptBrowserTest.LoginInterstitialShouldReplaceExistingInterstitial
    • 458333 Disable flaky AutofillDialogControllerSecurityTest.DoesntWorkOnBrokenHttps
    • Reverted patch that broke Chrome OS LoginUITests
    • 458154 Chrome PFQ peach_pit. HWTest failure, RPC Connection Timeout.
    • 458122 cros_trunk FileSystemProviderApiTest.BigFile failure. Disabling the test, passing bug to owners.
    • 457993 The pool is in a critical condition and cannot complete build verification tests in a timely manner.
    2015-02-10
    Sheriffs: jwerner, victoryang, hungte
    Gardener: girard
    • reverted https://codereview.chromium.org/899973006  - suspect it caused an x86 ASAN failure 
    • 456993 Chrome PFQ failure
    2015-02-03
    Sheriffs: dbasehore, katierh
    Gardener: ihf
    • 453090 Pre-CQ failure
    • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
    • SSL connection flake for build
    • 455728: ASAN unittest failure in permission broker
    • 456501: canaries dying during ChromeSDK due to missing gbm.h header
    • 456491: chrome pfq dying during BuildPackages due to dpkg-architecture errors
    • 456829: arm-generic_freon chrome pfq failing with conflicting minigbm/mesa depends
    2015-02-02
    Sheriffs: djkurtz
    Gardener: 
    • 448208 - pool: bvt, board: daisy_spring in a critical state
    • 454561 - pool: bvt, board: expresso in a critical state
    • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
    2015-01-30
    Sheriffs: vbendeb, armansito, wuchengli
    Gardener: jamescook
    • 401341: update_engine UnitTest failures in P2PManagerTest.ShareFile, out of disk space on /tmp
    2015-01-29
    Sheriffs: wuchengli
    Gardener: jamescook
    • 452349: Canary Chrome failures because of mixed Freon / non-Freon
    • 36103: storm-release: BuildPackages failed in chromeos-base/ap-daemons
    • 453201: [bvt-inline] provision Failure on zako-release/R42-6735.0.0
    • 428058: [bvt-inline] security_NetworkListeners Failure on daisy_spring-chrome-pfq/R40-6412.0.0-rc2
    • 446221: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures -> disabled
    2015-01-28
    Sheriffs: sonnyrao, arakhov, vapier
    Gardener: jamescook
    • 452911: Chrome PFQ failing due to ozone/evdev/input_controller_evdev.cc warnings -> reverted, asked chromeos-tpms to bump PFQ
    • 450335: [bvt-cq] video_VideoSanity Failure on daisy_skate-chrome-pfq -> flaky test -> disabled
    • 446221: cros_trunk: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures on official builders
    • 452623: cros_trunk: WebRtcSimulcastBrowserTest.TestVgaReturnsTwoSimulcastStreams browser_tests failures -> disabled
    • 453090: pre-cq failing with commit KeyError
    • 453208: cidb connection failed with buildStageTable key error
      2015-01-27
      Sheriffs: zeuthen, shawnn, vapier
      Gardener: jamescook
      • 452497: canaries all dying in chrome with /home/chrome-bot/depot_tools/external_bin errors
      • 452534: pre-cq bots timing out due to most slaves offline
      • 450278: Chromium OS Asan bots failing in logging_AsanCrash, telemetry exception problem
      • 451603: Chromium OS (amd64) Asan: security_SandboxLinuxUnittests failing
      • 449103: cros_trunk: WebInputEventAuraTest.TestMakeWebKeyboardEventWindowsKeyCode fails under ThreadSanitizer
      • 371290: cros_trunk: ICOImageDecoderTest.Decoding content_unittest fails on 8010 Mac, Linux32, Linux64 bots
      • 452647: cros_trunk builder failures: base_unittests: runtest.py test.exe no such option --parallel
      • 452706: syncing bluez repo broke with upstream ref errors
      2015-01-23 - 2015-01-26
      Sheriffs: zeuthen, shawnn, reveman
      Gardener: tbarzic
      • 452073: Beltino-B builder unable to build chrome from source.
      • 452070: Missing prebuilts for nyan_freon.
      • 452329: Chrome PFQ uprev failure.
      2015-01-20 - 2015-01-21
      Sheriffs: garnold, avakulenko, itspeter
      Gardener: xiyuan, zork
      • 445705: peach-pit ethernet issues cause update signals to not be received, failing autoupdate_EndToEnd.
      • 450244: paygen timing out waiting on rambi-c-canary, waiting for DUTs.
      • 450407: A CL in chryptohome seems to cause a unit test to fail. Reverted.
      • 450771: Chrome PFQ is broken on MIPS platform. Related to this CL.
      2015-01-14 - 2015-01-15
      Sheriffs:  wfrichar, adlr, kpschoedel
      Gardener: skuhne
        • Network issues: https://b2.corp.google.com/u/0/issues/19028546
        • veyron-pinky-nightly-chrome-pfq is red, looking at log seems a flake, rebuild
        • Reverted https://codereview.chromium.org/857613002/ since it broke many builders and updated the PFQ build to get the PFQ to uprev.

        2015-01-12 - 2015-01-13
        Sheriffs:  bfreed, bsimonnet, rongchang
        Gardener: achuith
        • crbug.com/447821: Scheduled Lab shutdown on Jan 9 is complete.  Let's see if the tree comes back up.
        • crbug.com/448079: Chrome failed in the PFQ: git error.  Should not close our tree on PFQ failure.
        • crbug.com/446889: Tree throttled due to video_ChromeHWDecodeUsed Failure.
        • crbug.com/448244: beltino-freon full release failed to build binutils and chrome.  First build, so might be just plain broken.
        • crbug.com/448414: Canary timeouts in report stage, but jrbarnette did some additional cleanup as well.

            2015-01-08 - 2015-01-09
            Sheriffs:  quiche, bhthompson, rongchang
            Gardener: achuith
            • crbug.com/447324: rojen and tkensinger replaced old winky duts with new ones

                2015-01-06 - 2015-01-07
                Sheriffs:  bhthompson, quiche, sheckylin, rongchang
                Gardener: 

                  2015-01-06 - 2015-01-07
                  Sheriffs:  grundler, jrbarnette, owenlin
                  Gardener: oshima
                  • 445705: AU and Paygen failures on peach_pit
                    • Closed 446463 as a duplicate.
                    • Two tickets filed:  b/18918701 b/18936609
                    • No root cause:  expect more peach_pit failures to follow
                  • CanaryCompletion timeouts caused by master restart (yjhong/cmasone)
                  • winky DUTs in lab *locked* by rojen - caused winky paygen test failures
                    • The DUTs were locked in order to replace them with MP hardware.
                  • 322072: peach-canary, nyan-canary and winky timed out in paygen test
                  • 446177: intermittent login test failures on x86, especially VM tests.
                  • 446463: AU test failure on peach_pit.
                  • 446885: security_OpenFDs failing in vmtests on asan bots
                  • CL:239300: sync errors due to glibc upstream/ refs changing from a file to a dir

                  2015-1-2 - 2015-1-5
                  Sheriffs: benchan, namnguyen, dhendrix  Gardener:
                  • 445068: logging_CrashServices found to be bricking DUTs, temporarily disabled
                  • 286343: git push failures: missing permissions
                  2014-Q3 ENTRIES MOVED TO  2014-Q3-Archive
                  OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)