Sheriff Log: Chromium OS

2014-11-21

Sheriffs: dianders, bleung, chihchung, Gardener: skuhne

  • 435564 ASAN security_Firewall failures
  • Falco failed PFQ 2 times for different reasons, GPU reset failure, network listener [assuming flaky since benchmarks run - restarted]
  • 435615 Rush_ryu PFQ build broken
  • 435640 ChromiumOS TryServer not executing non-pre-cq jobs
  • 418539 [bvt-inline] security_NetworkListeners Failure on daisy_skate-release/R39-6315.0.0 [only this time on falco]

    2014-11-20

    Sheriffs: semenzato, arakhov, chihchung.  Gardener: skuhne

    • 434498 CROS trunk builder fails since ui_unittests binary is renamed to ui_base_unittests
    • 435322 Precise64 Trunk builder is failing in net_unittests: HTTPSOCSPTest.RevokedStapled
    • 435362 elliptic curve error while uploading prebuilts
    • 435360 Chromium OS (x86) Perf failed the past 1000 builds
    • 434939 chromiumos sdk build kernel failure
    • CL:231230 chromite lint unittest

    2014-11-18 - 2014-11-19

    Sheriffs: semenzato, arakhov.  Gardener: mukai

    • 434498 CROS trunk builder fails since ui_unittests binary is renamed to ui_base_unittests
    • browser_tests get flaky on CROS trunk, probably due to the file access failures to /tmp.  Asked chrome-troopers to reboot the builder.
    • 434738434871434875 extremely flaky hwtest on x86-zgb-paladin
    • 434958: bad file descriptor when unpacking ebuilds
    • 434939: 3.14 kernel fails to build on arm

    2014-11-14 - 2014-11-17

    Sheriffs: ???

    • CL:230011 new gcc triggered signed warnings in buffet
    • 433628 link_freon paygen failed
    • CL:230001 sdk refactoring triggered existing bug in boards that did not declare arm-none-eabi dependency
    • CL:*184305 new gcc triggered warning about unused func in ap-daemons
    • missing mipsel/aarch64 toolchain prebuilts -> waited for SDK to finish and generate them

    2014-11-12 - 2014-11-13

    Sheriffs: vapier, victoryang, avakulenko

    • 432666 V4L2 change broke 32 bit 3.14 kernels (x86-generic/arm-generic/etc...)
    • 432705 Lumpy Chrome PFQ lagging behind in hwtest
    • 432793 x86-generic-incremental died looking for libjpeg-turbo license
    • 432929 pineview canary red due to CL 227543

    2014-11-10 - 2014-11-11

    Sheriffs: djkurtz

    • crbug.com/431622 canary builder failures / timeouts throttling tree
    • crbug.com/426672 GS paygen lockfile error (repeat)
    • crbug.com/427187 pool: bvt, board: daisy_spring in a critical state.
    • crbug.com/420132 flaky chromite unittests timeout when machine is loaded
    • crbug.com/427469 pool: bvt, board: monroe in a critical state.
    • crbug.com/432020 many canaries => DebugSymbols upload -> gs flake => "WARNING: could not upload: *.sym: HTTP 403: Forbidden" => ProcessExitTimeout => hung for 600 seconds

    2014-11-07 - 2014-11-10

    Sheriffs: pprabhu, shawnn, posciak

    2014-11-04 - 2014-11-05

    Sheriffs: dbasehore, denniskempin, fjhenigman, Gardener: jamescook, Lab: dshi

    • crbug.com/430182 daisy BuildPackage flake due to desktopui_CameraApp autotest
    • http://b2/18249316 git 502 errors to https://chromium.googlesource.com/, backend load issue
    • crbug.com/418918 Infrastructure issues (HWTest failed due to #DUTs not sufficient)

    2014-10-30 - 2014-10-31

    Sheriffs: wfrichar, waihong

    2014-10-28 - 2014-10-29

    Sheriffs: zeuthen, adlr, seanpaul

    2014-10-24 - 2014-10-27

    Sheriffs: deymo, garnold, rongchang

    2014-10-22 - 2014-10-23

    Sheriffs: bfreed, bsimonnet

    2014-10-20 - 2014-10-21

    Sheriffs: jrbarnette, quiche

    2014-10-13 - 2014-10-14

    Sheriffs: cychiang

    • 422188: rambi series failed to boot started from 10/10  http://chromegw/i/chromeos/builders/rambi-a%20canary/builds/617
      • Check on clapper: 6348 can boot, 6353 can not boot. blamelist:  http://chromeos-images/diff/report?from=6348.0.0&to=6353.0.0
      • Check on squawks: 6349 can boot, 6351 can not boot. blamelist:  http://chromeos-images/diff/report?from=6349.0.0&to=6351.0.0
      • Found CL https://chromium-review.googlesource.com/#/c/222290/.
    • 358933: nyan canary https://uberchromegw.corp.google.com/i/chromeos/builders/nyan%20canary/builds/1063
    • 422702, 422703, 422704, 422705, 422760, 422801:  telemetry failed to do Oobe.loginForTesting on lumpy. Passed in this build http://chromegw/i/chromeos/builders/sandybridge%20canary/builds/672 .
    • 409332: provision_AutoUpdate sometimts fails on engurade and gnawty.
    • 424900: sdk bot fails w/missing telemetry imports
    Gardener: stevenjb
    • 421453 continues to cause flakiness on http://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%281%29
    • 423032 filed: HistoryWebUIIDNTest may be flakey
    • 422703: Various telemetry tests flaking with "Cannot set property 'disabled' of null" - causing some flakiness, being investigated.

    2014-10-10 - 2014-10-13

    Sheriffs: puthik, dlaurie

    • Duck paladin broken, CL reverted.
    • 422406: bvt does not have enougn nyan_blaze machine
    • 420344: HW Test flake - desktopui_Screenlocker
    • 422700: HW Test flake - security_SandboxedServices
    • 422703: HW Test flake - telemetry

    2014-10-09 - 2014-10-10

    Gardener: michaelpg

    • 422102: Chrome crashes on start-up, fixed 10/10 by: https://codereview.chromium.org/642153003
    • Some PFQs failing autotests using 40.0.2184.0, suspect fixed by https://codereview.chromium.org/641693008. Verify tomorrow.

    2014-10-08 - 2014-10-09

    Sheriffs: grundler, stevefung, spang; Gardener: michaelpg
    • b/17880749: GoB outage happened again. Tree closed most of 10-08. http://go/omg/121. davidjames deployed rtc bomb. :)
    • git and gerrit service restored by late afternoon. CQ still closed as of 6pm or so.
    • crbug.com/421943: AUTest bug is fixed 
    • PFQ autotests were broken, fixed by: https://chromium-review.googlesource.com/#/c/222713/

    2014-10-07 - 2014-10-08

    Sheriffs: wuchengli, chirantan, furquan, spang
    • b/17880749: GoB outage
    • 421245: DB connection error (too many connections)
    • 358933umount leaves mountpoint marked as 'busy'

    2014-10-03 - 2014-10-06

    Sheriffs: wuchengli
    • 358933umount leaves mountpoint marked as 'busy'
    • 418358Paygen failed with TypeError: __str__ returned non-string (type list)
    • 419581emerge: there are no ebuilds to satisfy chromeos-test-root for beaglebone_servo
    • 418918HWTest failed due to #DUTs not sufficient
    • 420344: desktopui_ScreenLocker Failure due to Bad password bubble did not show.

    2014-10-02 - 2014-10-03

    Sheriffs: namnguyen, charliemooney, rongchang, Gardener: flackr

    • 419659: CheckCriticalProcesses failed on ASAN bots. Tests were moved out of "smoke" suite.
    • 419752: chromeos-base/platform2 fails configure first time looking for ModemManager
    • 401258: Seeing daisy fail on building glbench before opengles-headers.
    • 419964419965: pre-cq p2p & modem-manager unittest flakes
    • 420080: nyan_kitty failing signing
    • 420066: Chrome PFQ failure

    2014-10-01 - 2014-10-02

    Gardener: flackr

    • 419393: net-misc/modemmanager-next-1.5.0-r244 tests flaky on Chromium OS (amd64) Asan

    2014-9-29 - 2014-9-30

    Sheriffs: vpalatin, alecaberg, josephsih, Gardener: derat

      • 41846423 VM Tests failed on multiple platforms due to VM-only Chrome startup failures caused by removal of --disable-vaapi-accelerated-video-encode.
      • 418850: VMTests failed: failed to start/stop VM
      • 418650: image_to_vm.sh flakes 
      • 418928: HWTest failed due to infrastructure issues: fail to untar test_suites.tar.bz2
      • 418918: HWTest failed due to number of available DUTs is less than the minimum value 4
      • 418921: SignerTest failed: security_test_image failed
      • 418994: Daisy (chromium) build warnings about missing GLES2 headers when compiling autotest-deps-glbench
      • 418998: Nightly PFQ bots failed to build libxkbcommon due to GitHub throwing 500s
      • 419390: google-breakpad unittests hung on x86-generic-asan 

      2014-9-23 - 2014-9-24

      Sheriffs: sbasi, thieule, sheckylin

      • 417355: Builder should retry MySQL connection in case of intermittent network issues
      • 417302: login_OwnershipNotRetaken still failing
      • 401615: canaries often take more than 8 hours to run
      • 418296: harfbuzz build sometimes flakes due to missing cairo
      • 418358 / p/32327: storm canary paygen stage failing

      2014-9-22 - 2014-9-23

      Sheriffs: dhendrix, sheckylin

      • 416755paygen: PayloadTestError: cannot find source payload
      • 413682: Residual paygen issues on nyan_big and daisy builders
      • 416204: PFQ failed to uprev due to missing omxtypes.h causing Chrome to fail to build
      • 417050: old platform2 pulled in by various packages, but newer version needed by shill (shouldn't be a problem, clobber builder if it fails due to this)
      • 417094: openvpn fails to build on mipsel-o32-generic (should be fixed when crosreview.com/219635 lands)

      2014-9-18

      Sheriffs: dgreid, puneetster, Gardening: skuhne

      Fixed:

      • 415281: Falco failing "[bvt-inline] login_OwnershipNotRetaken Failure on falco-chrome-pfq/R39-6276.0.0-rc3". Found culprit after bisect.
      • CL:215220 reverted via CL:218925 to fix arm-generic/chromiumos-sdk kernel builds

      2014-09-16 - 2014-09-17, Gardening: skuhne

      Sheriffs: zqui, gwendal

      Pending issues:

      • 415281: Need to rollback to a stable chromeos-chrome: cl/217936 in progress.
      • 414345: transient authentication issue prevent Chrome PFQ to run. 
      • 415281: Falco failing "[bvt-inline] login_OwnershipNotRetaken Failure on falco-chrome-pfq/R39-6276.0.0-rc3". Created a falco with the latest image and a TOT Chrome on top of it - but the problem was not showing up when running locally.
      Fixed issues:
      • 414322 has been identified, tree is reopened.
      • A trailing slash caused the build to fail with a style warning. (!?!? Why do our checks differ between systems?) CL 552103003 takes care of that.
      For the experimental builds:
      • 408263: gizmo paladin is broken, gizmo canary is fine.
      • 389617: rush needs a new tool chain in place

      2014-09-12 - 2014-09-15

      Sheriffs: dkrahn, tbroch

      Decision was made to leave the tree throttled until 414322 is fixed.

      • 414128: kernel build failure
      • 411693: image_to_vm failure
      • 414322: paygen/autest bug ... device fails to run new image delivered (split out of 370302)
      • 370302: paygen/autest bug caused by devserver availability issue still happening -- apparently not fixed

      2014-09-09 - 2014-09-10

      Sheriffs: semenzato, snanda, amstan, etc. (add your names) djkurtz for one day

      Situation is relatively calm as of 5pm.  Tree has been open a few hours.  Yesterday and this morning the tree was closing every hour or so.

      • 370302: paygen/autest bug happened a lot, but is now believed to be fixed by removing an errant devserver
      • 413014 and 355843 also apparently stopped hitting

      2014-09-08 - 2014-09-09

      Sheriffs: davidriley, achaulk, djkurtz

        • 370302: devserver/paygen failures manifesting as different autoupdate_EndToEndTest failures
        • 410482412563: b-lab network saturation causing issues
        • 412564: cascading pre-CQ failures trying to get tryjobs

        2014-09-03 - 2014-09-04

        Sheriffs: cywang

          • 410716: master canary is waiting for results of all other release builds which are not started yet

          2014-09-02 - 2014-09-03

          Sheriffs: pstew, marcheu

          2014-09-01 - 2014-09-02

          Sheriffs: tyanh

          • 409774: Canary master keeps getting timed out from daisy-release-group
          • Waiting for this CL to be in to fix canary master on spitting TypeError

          OLDER ENTRIES MOVED TO THE ARCHIVE so this page doesn't take forever to load.  See Sheriff Log: Chromium OS (ARCHIVE!)

          Comments