Sheriff Log: Chromium OS

2015-04-17
Sheriffs: semenzato, quiche, mcchou (shadow)
  • 477747 ImageTest failure on canries due to /usr/include/tiffio.hxx, /usr/libexec/perf-core/tests/attr.py, and /lib64/libthread_db-1.0.so
  • 477883 kayle-paladin failed due to chromeos-initramfs
  • 477888 CQ failure due to VM test timeout in buffet_InvalidCredentials
  • 477889 CQ failure due to VM test timeout in buffet_Registration
  • 477941 link pre-cq failure (with no CLs): build_image failed due to no space on loop device
  • 473970 canary failure due to autoupdate_EndToEndTest
  • 474831 moblab_runsuite failure: /root/.boto did does not exist
2015-04-16
Sheriffs: semenzato, quiche, mcchou (shadow)
  • 477703 make_factory_toolkit.sh broke Archive step
  • 477712 enabling frame pointers broke optimized webrtc code in chrome
  • 477739 samus-release failed in suite prep
2015-04-15
Sheriffs: avakulenko, jrbarnette, fjhenigman
  • 476434 Network flakiness causes a dozen canaries go red.
    • As of this evening, the problem is still causing intermittent failures across the board.
  • 477352 Chrome fails to start on jerry and mighty
    • Early morning: Chrome has been pinned to 44.0.2368.0 until the problem is fixed.
    • Mid-day: Testing showed the problem is on the Chrome OS side; reverted this CL.
    • Afternoon: Chrome is unpinned.
    • Evening: Waiting for the CL to make it through a canary, so that the veyron builders will show green.
2015-04-14
Gardener: jonross
Sheriffs: avakulenko, jrbarnette, kpschoedel
  • 438729 OutOfProcessPPAPITest.MediaStreamAudioTrack flaky on CrOS. Taking out the X server, causing other tests to fail.
  • 469119 TouchExplorationTest.RewritesEventsWhenOn is flaky
  • 476934 Chrome PFQ bots failing with config failures, blocking uprev.
  • 475923 TouchExplorationTest.SplitTapExplore flaking 50% of the time.
  • 476550 Paygen signing failure on multiple boards (including veyron)

2015-04-10 and  2015-04-13
Gardener: jonross
Sheriffs: kpschoedel, denniskempin, bleung
  • 476607 Peach_pit Chrome PFQ failng HWTest step due to Infra issue.
  • 476577 KioskUpdateTests are flaky on Linux Chromium OS.
  • 469119 Flaky TouchExplorationTest.RewritesEventsWhenOn is causing failures on CrOS trunk
  • 476338 PFQ Failure on lumpy. Provision failure tracking SSH timeout.
  • 476434 Tree is throttled, devservers are causing a P0 issues causing AU testing to flake heavily for the morning PST
2015-04-08
Gardener: zork
  • 475170: Autotest security_OpenFDs failing.

2015-04-06 and 2015-04-07
Gardener: xiyuan
Sheriffs: deymo, shawnn, vapier, deanliao
  • 474227: Lack of speedy DUTs in lab
  • 465862: desktopui_ScreenLocker failing on asan bots
  • br:764: buffet autotests flaking on amd64-generic-full
  • 474497: Canaries dead in paygen
2015-04-02 and 2015-04-03
Gardener:
Sheriffs: wfrichar, adlr, itspeter (on holiday), vapier
  • cbuildbot gsutil errors, solved by fixing permissions manually (by system services team) and https://chromium-review.googlesource.com/#/c/263633/1
  • Unittest break in germ, quickl fixed by Jorge
  • repo upload failing on kernel repo: b/20062832b/19932429
  • CL:263970 gnutls pulled due to new pyshark code (CL:262457)
  • 473738: sumo board failing to build adhd
  • 473742: rush_ryu recovery kernel failing
  • br:344: kernel git checkouts failing on gizmo/project-sdk bots
  • CL:*211751: nyan_freon failing signer tests
  • 473721: paygen ran out of space on swanky
  • 473899: paygen not finding images
  • 473900: mips qemu fc-cache call crashing

2015-03-31 and 2015-04-01
Gardener: 
Sheriffs: bfreed, bhthompson, josephsih
  • Throttled tree in the morning, most canaries red.  471656: Paygen: failed to set up loop device
  • 472378: x86 asan builder failing unittests due to leak san not being supported
  • 472658: Paygen fails: Permission denied: '/dev/loop27p4'
    • The reverts above get the canaries past "failed to set up lop device", but many now fail with "Permission denied: '/dev/loop27p4'".
  • 449738: "Internal server error" should be treated as infrastructure failure so it doesn't reject CLs
    • "CQ encountered a critical failure" email.
    • Pointer to https://uberchromegw.corp.google.com/i/chromeos/builders/CQ%20master/builds/5150 shows most failures due to "FAILED RPC CALL: get_jobs" in HWTest.
  • 347423: Gerrit failed to submit change.
    • Sounds like a transient failure.  Should just try it again.
    • 13:57:51: WARNING: Change 263241 was submitted to gerrit without errors, but gerrit is reporting it with status "NEW" (expected "MERGED").
    • 13:57:51: ERROR: Most likely gerrit was unable to merge change 263241.
  • 472858: VmTest fails when dbus "Did not receive a reply"

2015-03-30 and 2015-03-31
Gardener: 
Sheriffs: josephsih
  • 471656: Paygen: failed to set up loop device: No such file or directoryIs. This is a new issue causing the failure of 20 canary builders.
  • 466777: HW_Test error: this issues still exists.
  • 469259: image_to_vm.sh fails: this issue still exists.

2015-03-26 and 2015-03-27
Gardener: skuhne
Sheriffs: puthik, furquan
  • PFQ failed to uprev. Caused by 991533002, reverted (Chrome and ChromeOS PFQ).
  • 469566: pool: bvt, board: veyron_jerry in a critical state
  • 466777: pool: bvt, board: veyron_speedy in a critical state
  • 469259: image_to_vm.sh fails with "partx: /dev/loop0: error deleting partition ..."
  • 471032: PFQ still fails - now: The binary size of the image exceeds the limits. (e.g. Daisy-freon, Daisy-skate)
  • 449766PFQ still fails - now: [bvt-inline] login_LogoutProcessCleanup Failure on Falco

2015-03-24 and 2015-03-25
Gardener: jamescook
Sheriffs: charliemooney, kcwu
  • 463530461087  Chrome uprev failure on tricky, cryptohome not mounted
  • 466719 lumpy ssh timeout resulting in provisioning failures
  • 470172 MasterUploadPrebuilts failure in Update PFQ config dump
  • 470118 Amd64-generic failing vmtest "buffet_Registration" with urlopen "Connection Refused"
  • 470237: WallpaperManagerBrowserTest.DevLaunchApp failing on cros_trunk official builders
  • 470130: Chromium OS (x86) Asan always failing with MySQL connection error
  • 470381: [bvt-cq] graphics_SanAngeles Failure on tricky-chrome-pfq/R43-6910.0.0-rc5 - wflinfo / waffle problem, test disabled
  • 1035763003: Revert of Test Accelerators In Interactive UI Tests - cros_trunk official builder was failing
  • 448247: [bvt-inline] provision Failure on candy-release/R42-6683.0.0 - update_engine failed
  • 470701: Flaky BVT security_Firewall failure, "Mismatched firewall rules"
2015-03-18 and 2015-03-19
Gardener: abodenha
Sheriffs: stevefung
  • 468340, 468770: PFQ and canaries are a mess due to infra issues
  • 465862: amd64 ASAN builds are failing
  • 468394: autoupdate_EndToEndTest.paygen_au_stable_test_delta failures
  • 466777: not enough duts
  • 467975: Image Signing Timeouts
  • 463805:  autoserv timeouts
2015-03-17
Sheriffs: littlecvr,stevefung

  • 467975: Image Signing Timeouts
  • 260602: Fix auron_paine build break

2015-03-16
Sheriffs: littlecvr,dlaurie,zqiu

  • 466972: insufficient DUTs for butterfly
  • 466777insufficient DUTs for veyron_speedy
  • 465230: wolf: login_OwnershipTaken_SERVER_JOB Failure
  • 433970: wolf: login_LogoutProcessCleanup_SERVER_JOB Failure
  • 419772: gnawty: security_ProfilePermissions_SERVER_JOB Failure
  • 411608: gnawty: security_NetworkListeners_SERVER_JOB Failure
  • 434148: gnawty: login_MultiUserPolicy_SERVER_JOB Failure


2015-03-13
Sheriffs: tyanh,dlaurie,zqiu

  • 426164: [au] autoupdate_EndToEndTest.npo_test_delta Failure on nyan_blaze-release/R38-6158.71.0
  • 466919[paygen_au_canary] autoupdate_EndToEndTest.paygen_au_canary_test_full Failure on quawks-release/R43-6872.0.0
  • 8 issues on [bvt-inline] login_* Failure on wolf-release/R43-6872.0.0
    • 434202login_RetrieveActiveSessions_SERVER_JOB Failure
    • 434182login_SameSessionTwice_SERVER_JOB Failure
    • 434178login_OwnershipNotRetaken_SERVER_JOB Failure
    • 434185login_Cryptohome_SERVER_JOB Failure
    • 419772security_ProfilePermissions_SERVER_JOB Failure
    • 434148login_MultiUserPolicy_SERVER_JOB Failure
    • 403701login_MultipleSessions_SERVER_JOB Failure
    • 434195login_GuestAndActualSession_SERVER_JOB Failure
  • b/19729024: incorrect DHCP config change was pushed breaking DHCP in the lab
  • br/590: amd64 asan failing unittests due to leaks related to protobuf

2015-03-12
Sheriffs: tyanh

  • 462734: [sanity] provision Failure on lumpy-release/R43-6869.0.0

2015-03-10
Sheriffs: chirantan, thieule
Gardener: oshima

  • 465752: [bvt-inline] login_RetrieveActiveSessions Failure on tricky
  • 465963: daisy canary: The AUTest [daisy_spring] [au] stage failed: ** HWTest did not complete due to infrastructure issues (code 3)
  • 464171: Multiple canary's are failing due to kernel size limits
  • 465877: [bvt-inline] login_OwnershipTaken Failure on x86-mario-release/R43-6865.0.0
  • 464751: [au] provision Failure
2015-03-09
Sheriffs: chirantan, thieule
Gardener: achuith

  • 464938: Test lab performance issue
  • 465596update_engine is failing on all canary builders
2015-03-06
Sheriffs: benchan, tbroch
Gardener: tengs


2015-03-05
Sheriffs: benchan, tbroch
Gardener: tengs
  • 464407 Your "Oauth 2.0 User Account" credentials are invalid .... Failure: Invalid response 302..

2015-03-04
Sheriffs: dhendrix, waihong
Gardener: derat

2015-03-03
Sheriffs: dhendrix, waihong
Gardener: derat
  • 462842 Chromium OS (amd64) Asan bot failing unittests due to webserver leaks
  • 463493 Flaky failure in breakpad's linux_client_unittest on X86 (chromium)
  • 463532 browser_tests failing on cros trunk in WebViewTest.FileSystemAPIRequestFromWorkerDeny
  • 464053 Beaglebone kernel too big, causing build failures
  • 463411 pool: bvt, board: leon in a critical state. (Some DUTs had their USB ethernet dongles swapped and were down for a bit)
2015-03-02
Sheriffs: dgreid, tbroch
Gardener: derat
  • 461406 Chromium OS (x86) Asan bot still failing
  • 458775 peach_pit nightly chrome PFQ failed with too few available DUTs
  • 463213 X86 (chromium), Daisy (chromium), and AMD64 (chromium) failing on chromeos-chrome with "ValueError: invalid literal for int() with base 10"
2015-02-27
Sheriffs: dgreid, tbroch
2015-02-25
Sheriff: dianders, pstew, jchuang
  • 462240 [storm-release canary] storm-release: The BuildPackages stage failed: Packages failed in ./build_packages
  • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
  • 461841 rambi-c-release-group: timed out

2015-02-25
Sheriff: dianders, pstew, tyanh
  • 461184 [canary] HWTest did not complete due to infrastructure issues again in canary 670; also "peach-release-group: timed out"
  • 461841 [canary] sandybridge-release-group: The Paygen [butterfly] stage failed: <class 'chromite.lib.timeout_util.TimeoutError'>: Timeout occurred- waited 13800 seconds
  • 461893 [canary] rambi-a-release-group: The HWTest [expresso] [bvt-inline] stage failed: ** Suite timed out before completion **
  • 460174 [canary] peach-release-group: The HWTest [peach_pit] [sanity] stage failed: ** HWTest did not complete due to infrastructure issues (code 3) **
2015-02-24
Sheriff: amstan, gwendal, tyanh
  • 461184 [canary, chrome pfq] HWTest did not complete due to infrastructure issues (HWTest Lab closed?)
  • 461188 [pineview canary] Operation timed out at the end of a build; transient, subsequent build succeeded.
  • 438908 Image signing timed out across many platforms on canary; transient, subsequent builds succeeded.
  • 461378 chrome compilation error prevents some build to complete in CQ.
  • 461415 critical fix in chrome was only present on chrome ToT.

2015-02-23
Sheriff: amstan, gwendal  Gardener: flackr
  • 460815 Cros SDK broken in .bashrc_profile
  • 460693 Chrome PFQ failing with exception: global name 'AccessDeniedException' is not defined in File "src/build/download_sdk_extras.py", line 71
  • 460951 ChromeMetricsServiceAccessorTest.MetricsReportingEnabled, ExternalCacheTest.Basic, ExternalCacheTest.PreserveInstalled, DeviceLocalAccountExternalPolicyLoaderTest.ForceInstallListSet failing on cros_trunk
  • 458122 Reopened FileSystemProviderApiTest.BigFile failing on cros_trunk as test is failing 98% of runs on cros_trunk
  • 461021 MetricsServicesManagerTest.GetRecordingLevelCoarse faililng on cros_trunk.
  • 461046 MetricsServicesManagerTest.GetRecordingLevelFine faililng on cros_trunk.
    2015-02-18
    Sheriff: posciak, namnguyen, snada
    • Note to PST sheriffs: lots of infrastructure failures (I think) over the last few days, which I don't really understand. I think we need help from Infra team before we can reopen the tree for good.
    • 419904 IndexError: list index out of range on moblab_RunSuite 
    • 458613 Pre-CQ Launcher failures
    • 224 Moblob failures
    • 459679: Moblab blocks canaries
    • b/19426205: Missing commit info

    2015-02-13
    Gardener: jonross
    • Reverted a change that was failing a compile on Chrome OS.
    • 458567 Disable flaky InProcessAccessibilityBrowserTest.VerifyAccessibilityPass
    • 458549 Disable flaky TextInput_TextInputStateChangedTest.SwitchingAllTextInputTest
    • 458526 Chromium OS Waterfall bots falling on LKGMSync
    • 458918 amd64 asan failing shill unittests due to leaks
    2015-02-12
    Gardener: jonross
    • 458341 Disable flaky LoginPromptBrowserTest.LoginInterstitialShouldReplaceExistingInterstitial
    • 458333 Disable flaky AutofillDialogControllerSecurityTest.DoesntWorkOnBrokenHttps
    • Reverted patch that broke Chrome OS LoginUITests
    • 458154 Chrome PFQ peach_pit. HWTest failure, RPC Connection Timeout.
    • 458122 cros_trunk FileSystemProviderApiTest.BigFile failure. Disabling the test, passing bug to owners.
    • 457993 The pool is in a critical condition and cannot complete build verification tests in a timely manner.
    2015-02-10
    Sheriffs: jwerner, victoryang, hungte
    Gardener: girard
    • reverted https://codereview.chromium.org/899973006  - suspect it caused an x86 ASAN failure 
    • 456993 Chrome PFQ failure
    2015-02-03
    Sheriffs: dbasehore, katierh
    Gardener: ihf
    • 453090 Pre-CQ failure
    • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
    • SSL connection flake for build
    • 455728: ASAN unittest failure in permission broker
    • 456501: canaries dying during ChromeSDK due to missing gbm.h header
    • 456491: chrome pfq dying during BuildPackages due to dpkg-architecture errors
    • 456829: arm-generic_freon chrome pfq failing with conflicting minigbm/mesa depends
    2015-02-02
    Sheriffs: djkurtz
    Gardener: 
    • 448208 - pool: bvt, board: daisy_spring in a critical state
    • 454561 - pool: bvt, board: expresso in a critical state
    • 454657 - Canary master Build #601 failed HWTest on rambi-[a,b,c]-release-group
    2015-01-30
    Sheriffs: vbendeb, armansito, wuchengli
    Gardener: jamescook
    • 401341: update_engine UnitTest failures in P2PManagerTest.ShareFile, out of disk space on /tmp
    2015-01-29
    Sheriffs: wuchengli
    Gardener: jamescook
    • 452349: Canary Chrome failures because of mixed Freon / non-Freon
    • 36103: storm-release: BuildPackages failed in chromeos-base/ap-daemons
    • 453201: [bvt-inline] provision Failure on zako-release/R42-6735.0.0
    • 428058: [bvt-inline] security_NetworkListeners Failure on daisy_spring-chrome-pfq/R40-6412.0.0-rc2
    • 446221: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures -> disabled
    2015-01-28
    Sheriffs: sonnyrao, arakhov, vapier
    Gardener: jamescook
    • 452911: Chrome PFQ failing due to ozone/evdev/input_controller_evdev.cc warnings -> reverted, asked chromeos-tpms to bump PFQ
    • 450335: [bvt-cq] video_VideoSanity Failure on daisy_skate-chrome-pfq -> flaky test -> disabled
    • 446221: cros_trunk: PDFBrowserTest.Basic & PDFBrowserTest.Scroll failures on official builders
    • 452623: cros_trunk: WebRtcSimulcastBrowserTest.TestVgaReturnsTwoSimulcastStreams browser_tests failures -> disabled
    • 453090: pre-cq failing with commit KeyError
    • 453208: cidb connection failed with buildStageTable key error
      2015-01-27
      Sheriffs: zeuthen, shawnn, vapier
      Gardener: jamescook
      • 452497: canaries all dying in chrome with /home/chrome-bot/depot_tools/external_bin errors
      • 452534: pre-cq bots timing out due to most slaves offline
      • 450278: Chromium OS Asan bots failing in logging_AsanCrash, telemetry exception problem
      • 451603: Chromium OS (amd64) Asan: security_SandboxLinuxUnittests failing
      • 449103: cros_trunk: WebInputEventAuraTest.TestMakeWebKeyboardEventWindowsKeyCode fails under ThreadSanitizer
      • 371290: cros_trunk: ICOImageDecoderTest.Decoding content_unittest fails on 8010 Mac, Linux32, Linux64 bots
      • 452647: cros_trunk builder failures: base_unittests: runtest.py test.exe no such option --parallel
      • 452706: syncing bluez repo broke with upstream ref errors
      2015-01-23 - 2015-01-26
      Sheriffs: zeuthen, shawnn, reveman
      Gardener: tbarzic
      • 452073: Beltino-B builder unable to build chrome from source.
      • 452070: Missing prebuilts for nyan_freon.
      • 452329: Chrome PFQ uprev failure.
      2015-01-20 - 2015-01-21
      Sheriffs: garnold, avakulenko, itspeter
      Gardener: xiyuan, zork
      • 445705: peach-pit ethernet issues cause update signals to not be received, failing autoupdate_EndToEnd.
      • 450244: paygen timing out waiting on rambi-c-canary, waiting for DUTs.
      • 450407: A CL in chryptohome seems to cause a unit test to fail. Reverted.
      • 450771: Chrome PFQ is broken on MIPS platform. Related to this CL.
      2015-01-14 - 2015-01-15
      Sheriffs:  wfrichar, adlr, kpschoedel
      Gardener: skuhne
        • Network issues: https://b2.corp.google.com/u/0/issues/19028546
        • veyron-pinky-nightly-chrome-pfq is red, looking at log seems a flake, rebuild
        • Reverted https://codereview.chromium.org/857613002/ since it broke many builders and updated the PFQ build to get the PFQ to uprev.

        2015-01-12 - 2015-01-13
        Sheriffs:  bfreed, bsimonnet, rongchang
        Gardener: achuith
        • crbug.com/447821: Scheduled Lab shutdown on Jan 9 is complete.  Let's see if the tree comes back up.
        • crbug.com/448079: Chrome failed in the PFQ: git error.  Should not close our tree on PFQ failure.
        • crbug.com/446889: Tree throttled due to video_ChromeHWDecodeUsed Failure.
        • crbug.com/448244: beltino-freon full release failed to build binutils and chrome.  First build, so might be just plain broken.
        • crbug.com/448414: Canary timeouts in report stage, but jrbarnette did some additional cleanup as well.

            2015-01-08 - 2015-01-09
            Sheriffs:  quiche, bhthompson, rongchang
            Gardener: achuith
            • crbug.com/447324: rojen and tkensinger replaced old winky duts with new ones

                2015-01-06 - 2015-01-07
                Sheriffs:  bhthompson, quiche, sheckylin, rongchang
                Gardener: 

                  2015-01-06 - 2015-01-07
                  Sheriffs:  grundler, jrbarnette, owenlin
                  Gardener: oshima
                  • 445705: AU and Paygen failures on peach_pit
                    • Closed 446463 as a duplicate.
                    • Two tickets filed:  b/18918701 b/18936609
                    • No root cause:  expect more peach_pit failures to follow
                  • CanaryCompletion timeouts caused by master restart (yjhong/cmasone)
                  • winky DUTs in lab *locked* by rojen - caused winky paygen test failures
                    • The DUTs were locked in order to replace them with MP hardware.
                  • 322072: peach-canary, nyan-canary and winky timed out in paygen test
                  • 446177: intermittent login test failures on x86, especially VM tests.
                  • 446463: AU test failure on peach_pit.
                  • 446885: security_OpenFDs failing in vmtests on asan bots
                  • CL:239300: sync errors due to glibc upstream/ refs changing from a file to a dir

                  2015-1-2 - 2015-1-5
                  Sheriffs: benchan, namnguyen, dhendrix  Gardener:
                  • 445068: logging_CrashServices found to be bricking DUTs, temporarily disabled
                  • 286343: git push failures: missing permissions
                  2014-Q3 ENTRIES MOVED TO  2014-Q3-Archive
                  OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)