Please update go/cros-sheriff-playbook when you find a build/infra failure and can map it to what action the sheriff should take for it.
2/25-3/4 Sheriffs: kirtika, shapiroc, pmarko (non-PST) Ongoing issues:
10/15-10/19 Sheriffs: seobrien, amstan, josephsih Ongoing issues: - crbug.com/895203: caroline-arcnext-paladin: Not enough DUTs (fixed)
- crbug.com/895215: A few platforms: HWTest failed: ui.ChromeLogin (fixed)
- crbug.com/895221: guado: HWTest failed: video_WebRtcMainFeedSwitching (kerl@ had a fix. Eventually, should remove bluestreak-pre-cq from HWTest)
- crbug.com/895241: amd64-generic-paladin: TastVMTest failed: platform.CheckProcesses (assigned to semenzato@)
- crbug.com/895247: Many (66) builders: failed with _ShutDownException (closed)
- crbug.com/894820: amd64-generic-paladin UploadTestArtifacts Failures (still seeing this issue frequently)
- crbug.com/896158: hana-paladin: Not enough DUTs (assigned to akeshet@)
- crbug.com/896528: Multiple paladin builds failing in provision_AutoUpdate.double (closed)
10/08-10/12 Sheriffs: saklein, yueherngl, tbroch Ongoing issues:
8/20-8/24 Sheriffs: tfiga, bleung, akhouderchah Ongoing issues: - crbug.com/875710 - bob-paladin: login_RetrieveActiveSessions: fails with "Can't create session"
- crbug.com/873687 - Bob-paladin: logging_CrashSender fails in policy signature verification
- crbug.com/876186 - cyan-paladin:6819 failed: cheets_Midis: midis didn't restart.
- (Resolved) crbug.com/876188 - edgar-paladin:3720 failed: Failed to start ARC: failed waiting for ro.arc.boot_completed=1
- crbug.com/876196 - wizpig-paladin:3846 failed: login_LoginSuccess: Timed out waiting for condition: Session stopped.
- crbug.com/876197 - wizpig-paladin:3846 failed: login_GuestAndActualSession: Unhandled DBusException: org.freedesktop.DBus.Error.NoReply
- crbug.com/876236 - master-paladin:19462 failed: Cannot find overlay: [...]/overlay-grunt-ndktranslation-private
- crbug.com/876238 - tatl-paladin:4295 failed: InvalidLayout: Unknown items in layout recovery
- (Resolved) crbug.com/876630 - shard heartbeat queries timing out all over the fleet
- crbug.com/876634 - nyan/peach/veyron/elm/hana: multiple graphics_/login_ HWTest failures due to missing python modules
- crbug.com/876988 - winky-paladin:5905 failed: security_StatefulPermissions: Unexpected files/perms in stateful
- (Resolved) b/113088029 - Boot time regression of ARC++ in VMTests?
- crbug.com/877342 - master-paladin: SQL error in CommitQueueHandleChanges if all builds are cancelled?
- (Resolved) crbug.com/877372 - logs.chromium.org returns 503 Over Quota
- crbug.com/877390 - edgar: All DUTs in pool:cq are "Repair Failed"
- crbug.com/877394 - wolf-paladin: Provision failed on all DUTs
8/13-8/19 Sheriffs: moragues,jwerner Ongoing issues: - 871513: Multiple release/paladin failing with " [Test-Logs]: Suite job: FAIL: retry exception (function="_call()"), timeout = 180s"
- crbug.com/874705 - Unnecessary kevin-paladin & kevin-arcnext-paladin failures due to the share of DUTs
- EXPERIMENTAL=coral-paladin crbug.com/866543 process leak caused : NotEnoughDutsError: Not enough DUTs for board: coral, pool: cq; required: 4, found: 3
- 873822: Kernel warning spam on RK3399 devices causes disks to fill up, leading to all sorts of random failures. Affects mostly Bob since Kevin has larger disks.
- EXPERIMENTAL=bob-paladin crbug.com/873687 Bob-paladin: logging_CrashSender fails in policy signature verification
8/6-8/12 Sheriffs: jkwang, kirtika, stimim Ongoing issues: - 871513: Multiple release/paladin failing with " [Test-Logs]: Suite job: FAIL: retry exception (function="_call()"), timeout = 180s"
7-30-8/3 Sheriffs: teravest, evgreen,
Ongoing issues: - 868331: dragonegg paladin: Detected file collision - tpm_version
- 867319: Not enough DUTs for board: daisy_spring, pool: bvt; required: 4, found: 3 (also b/111838841)
- veyron_rialto: Known issue, not enough DUTs
- falco_li: Known failures, not enough DUTs
- 870787: samus-release: key_verify failed for server_host_key
- 870511: squawks-release failures: graphics_idle: Short blanking not added
- Untriaged: Some (many?) release builders are failing with [Test-Logs]: Suite job: FAIL: retry exception (function="_call()"), timeout = 180s
- b:112082609: kefka-release: Device not coming back after reboot
- 869952: security_SandboxLinuxUnittests: test does not exist
- 870005: nami-release failed cheets_CameraOrientation
7/23-7/27 Sheriffs: sjg, shurst, wuchengli Ongoing issues: - 867319: Not enough DUTs for board: daisy_spring, pool: bvt; required: 4, found: 3 (also b/111838841)
- 867320: newbie-release: Unreasonably low CPU frequency in cheets_SettingsBridge (now replaced by crbug.com/867649 to get rid of newbie!)
- 867341: kefka-release: flaky "Host did not return from reboot"
- 867357: master-paladin: PublishUprevChanges failed
- 868331: dragonegg paladin: Detected file collision - tpm_version
Resolved issues: - 865064 - Lotsa dut pools be unhappy, causing problems for veyron_rialto, banjo, etc.
- 867317: Not enough DUTs for board: falco_li, pool: bvt; required: 4, found: 2
- 867318: veyron_jaq-nyc-android-pfq failing in UprevAndroid
- 867360: daisy-incremental: Resolving storage.googleapis.com... failed
- 867326: Not enough DUTs for board: veyron_rialto, pool: bvt; required: 1, found: 0
4/23-4/27 Sheriffs: jrio, gwendal, yhong Ongoing issues: - b/78599178: Git on Borg (GoB) issue, git fetch fails.
- 837388: cros-full servers overloaded, leading into HW Test suite timeout.
4/16-4/20 Sheriffs:nshai, lepton, hychaoOngoing issues: - 831624: eve-arcnext-chrome-pfq is failing in the PFQ due to libassistant compilation failure
- 832880: rmi4update kills chrome on nautilus
- 831732: scarlet-release: Not enough DUTS available - need to remove temp. experimental from chromiumos-status when fixed!
- 827347: security_SandboxedServices test failure on stout due to root run dbus-monitor in rf-led-handler.conf (EOL device)
- 795128: evemu-device did not die resulting in security_SandboxedServices failing
- 796254: Archive stage fails in build_image
- 833478: some autotest_lib.site_utils.lxc.* unittests are flaky.
- 547055: tar: file changed as we read it.
Resolved issues:- 833886: Data base error cause most builder error.
- 833504: reef release broken.
- 833456: Chrome crashes when running in mash on device.
- 833499: nami release broken.
- 830321: 67: Caroline and Terra builds are RED; Chrome crashes at boot
- 829780: fizz R66 hwtest failures (waiting for devices in lab to get new base image flashed) - need to remove temp. experimental from chromiumos-status when fixed
- 831391: tatl-paladin:3250 failed: cannot find wayland-scanner
- 832280: android-container-nyc failing BuildPackages on an unrelated CL
4/9-4/14 Sheriffs:pberny, dianwa, sheckylinOngoing issues: - 831624: eve-arcnext-chrome-pfq is failing in the PFQ due to libassistant compilation failure
- 830321: 67: Caroline and Terra builds are RED; Chrome crashes at boot
- 829780: fizz R66 hwtest failures (waiting for devices in lab to get new base image flashed) - need to remove temp. experimental from chromiumos-status when fixed!
- 831391: tatl-paladin:3250 failed: cannot find wayland-scanner
- 832880: rmi4update kills chrome on nautilus
- 831732: scarlet-release: Not enough DUTS available - need to remove temp. experimental from chromiumos-status when fixed!
- 831624: eve-arcnext-chrome-pfq is failing in the PFQ due to libassistant compilation failure
- 832280: android-container-nyc failing BuildPackages on an unrelated CL
- 827347: security_SandboxedServices test failure on stout due to root run dbus-monitor in rf-led-handler.conf (EOL device)
Resolved issues:- 830865: PFQ builds failing - SyncChrome failure: ParseDepsFile:Found recursedeps
4/2-4/6 sheriff: smbarber, zhengpan, mnissler Ongoing issues: - 828371: Not enough DUTs for board: eve
- 828054: sporadic build_image failures
- 829055: whirlwind build flakiness
- 829289: CanaryCompletion failing on multiple builders
- 824808: intermittent failures due to Permission denied: '/proc/sys/fs/binfmt_misc/qemu-arm'
- 829780: fizz R66 hwtest failures
3/19-3/23 sheriff: eugenegonzalez, shawnn, akahuang
3/5-3/12 sheriffs: mgild, nvaccaro, itspeter Ongoing issues: - 813916: "Not enough DUTs for board: expresso"
- 815308: coral-release fails HWTests; probably compound flakiness from having to pass on 13 different boards in the lab. Unsure what we can do about this.
- Investigate on master-paladin failure. It will fail with CL:*581566, ask the author to take a look.
- Update: owner submit again after other dependent CLs merged. It is merged as well now.
- 819576: moblab-generic-vm-paladin seems flaky and failing mater-paladin
- 819695: Merge 819576 to this one as this is actively investigating.
- moblab-paladin flake: ERROR: Unhandled UpstartServiceNotRunning: Upstart service moblab-gsoffloader-init not in running state.
2/26-3/2 sheriffs: ejcaruso, matthewmwang, wnhuang Ongoing issues: - 789058: CreateTarball race in DebugSymbols step of several release builders
- 813916: "Not enough DUTs for board: expresso"
- 814343: daisy_skate-release: NotEnoughDutsError
- 814347: falco_li-release: NotEnoughDutsError
- 814352: nautilus-release fails on cheets test ( adb is not ready in 60 seconds.)
- 814500: scarlet-release: NotEnoughDutsError
- 815308: coral-release fails HWTests; probably compound flakiness from having to pass on 13 different boards in the lab. Unsure what we can do about this.
- 817074: chromeos2-row8-rack2-host5 is in a bad state, causing setzer-release failures
- 817437: cros-beefy280-c2 is offline
- 817948: soraka usb/ethernet instability, likely product issue
Resolved issues: - 814345: falco_release: NotEnoughDutsError
- 815250: newbie-release fails Signing stage and no email notification is generated
- 816563: ultima-release: NotEnoughDutsError
- 816584: enguarde-release: NotEnoughDutsError
- 816983: guado-release failed from one DUT having issues updating
- 816986: reef-release won't get past CleanUp stage after build crash
- 817022: veyron_tiger-release is not running
- 817063: poppy DUTs in verify-repair loop
- 817126: chromeos2-row3-rack1-host13 is in a bad state, causing soraka-release failures
- 817478: 2-3-1-9 is down, causing soraka-release failures
- 817925: fizz EC fails to sign
2/19-2/23 sheriffs: norvez, waihong, rongchang Ongoing issues: - 814340: platform_addPrinter is flaky (low frequency), sometimes hits the paladins
- 812425: quipper build seems to be failing unittests occasionally
- 814500: scarlet-release: NotEnoughDutsError
- 814343: daisy_skate-release: NotEnoughDutsError
- 814345: falco_release: NotEnoughDutsError
- 814347: falco_li-release: NotEnoughDutsError
- 813916: "Not enough DUTs for board: expresso"
- 814352: nautilus-release fails on cheets test ( adb is not ready in 60 seconds.)
- 815250: newbie-release fails Signing stage
- 815308: coral-release fails HWTests - DUTs look really unstable
- 798618: Intermittent paladin failures with cheets_StartAndroid.stress
- 813791: Changes to chromeos-base/trunks cause build failures in CQ
Resolved issues: - 814514: veyron_rialto-release: No module named telemetry.core
2/12-2/16
sheriffs: snanda, mqg, littlecvr Ongoing issues: - 811210: HWTest failed due to infra issue (code 3)
- 811149: widespready DUT pool shortfalls | shards unable to resolve DUTs | dhclient_conf is not set to "yes" on new shards
- 811217: eve-arcnext-mst-android-pfq:78 failed (PackageBuildFailure: Packages failed in ./build_packages: x11-libs/arc-libdrm)
- b/72697187: quota increase request for ChromeOS Infrastructure
- 811402: Master scheduler crashlooping because of malformed HQE
- 811878: cheets_KeyboardTest fails frequently on multiple boards: retry_count: 2, FAIL: Unhandled AssertionError
- 569819: temporarily remove cheets_KeyboardTest from CQ <- already reverted
- 812425: quipper build seems to be failing unittests occasionally
- 812949: DUT not rebooted in provisioning
- 798618: Intermittent paladin failures with cheets_StartAndroid.stress
- 812581: PackageBuildFailure: Packages failed in ./build_packages: chromeos-base/chromeos-chrome
- 812848: DUTs get stuck in un-abortable state
- 811697: Cyan HQEs queuing up catastrophically in shard (M66: Caroline, Cyan build is RED for 2 days)
2/5-2/9
sheriffs: craigb, adlr, abhishekbh, shenghao
Ongoing issues: 808923: Chrome PFQ failed due to not able to download files from GS 808945: HWTest failed due to infra issue (code 3) 809570: edgar-paladin can't mount chroot 809670: CQ blocker: nyan_kitty-paladin fails due to video test errors 810247: wolf-paladin fails: Start browser timeout 810255: cheets_ContainerMount failed and is blocking CQ 810667: coral-paladin failed on pack_firmware_unittest.py
1/29-2/2
sheriffs: vpalatin, ravisadineni, abhishekbh Ongoing issues:
808434: Canaries are failing at the build_packages stage due to a python error. 808563: bvt-arc suite in Canaries builds failing due as the server side tests do not start after scheduling.
1/22-1/28
Sheriffs: bmgordon, shapiroc, marcochen
Ongoing issues: - 784914: provision failurs: DUT cannot reboot at pre-setup of rootfs update
- 806287: chromeos4-row11-rack10-host13 is failing to provision
- 806013: libbrillo has new ASAN error
- 796275: bvt-arc times out across boards
- 804977: guado_moblab-paladin is failing for HW tests
- 782832: not enough daisy_skate devices to keep bvt pool alive
- b/72397774: intermittent failures to connect to git/gerrit
- 805928: Release builders failing during ManifestVersionSync stage
- This looks like potentially a transient git issue
- 782034: autotest artifacts persist between CQ runs
Resolved issues: - libc++ related changes
- 805691: qhull fails to build with libc++
- 805619: buildpackages failing on numerous packages in bare precq run
- 805722: Clobber incremental builders
- 805657: arc-camera3-hal-intel-ipu3 fails to biuld with libc++
- TKO/mysql issues
- 806019: tko query pileup | tko restart takes over an hour
- 805337: TKO database reached maximum size.
- 804127: shard outage for board:leon, board:nyan_big
- 805724: job_reporter died, causing passing test to appear aborted
- 804425: shard outage for board:orco
- 806011: chromeos-server133 afe serving lots of 5XXs
- 806106: lumpy-incremental-paladin failure
- 806107: Not enough lumpy DUTs
- 806196: Uprev stage fails on cros_mark_as_stable
- 805517: Betty android pfq tests are failing due to VM AU issue
- 805710: gale-paladin fails in ap-daemons unit tests
- 804513: eve-paladin failed to rebuild previously removed ebuild
- 804372: Missing alerts in Sheriff-o-Matic
1/8-1/12 Sheriffs: ahassani, dianders, hiroh
NOTE: Trying idea of just keeping the week's log in a Google doc.
Ongoing Issues: - 782832: not enough daisy_skate devices to keep bvt pool alive
- 795902: fizz-release failing since Dec 8
- 800831: provisioning issue across the board
- 800943: ls-remote-gerrit rate limit exceeded
- 800949: canaries had a problem pushing to gerrit
Resolved Issues: - 800426: race between the Android PFQ and the CQ
- 800132: infrastructure issue caused provisioning failures
- 800886: moblab-paladin autotest failure
1/1-1/5Sheriffs: frankhu, sarthakkukreti, johnylin
Ongoing Issues: - 782832: not enough daisy_skate devices to keep bvt pool alive
- 795902: fizz-release failing since Dec 8
- 788584: kefka/coral-release/paladin: Linksys USB3GIGV1 Ethernet adapter fails to enumerate (r8152, usb X-Y: device not accepting address Z, error -62)
- 797849: paygen tests from previous canary delayed bvt-arc from canary, causing timeout
- 789058: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed
- 799669: banon-release: build fails security_SandboxedServices
- 799604: Lakitu-gpu release: intermittently fails GCETest
- 798540: jetstream_ApiServerDeveloperConfiguration flake: Timed out waiting for AP to appear operational
Resolved Issues:- 798558: factory-strago-7458.B build failure since 7458.357.0
- 798618: veyron_minnie-paladin builds fail with cheets_StartAndroid.stress
- 798649: chromite.scripts.merge_logs_unittest failed due to acrossing year
- 798273: HWTest bvt-arc aborted cheets_CTS_N* and cheets_GTS* tests stuck on "waiting for cache lock"
- 797620: Build failure chrome-pfq: login_CryptohomeIncognito and security_ProfilePermissions.guest are failing across multiple boards
12/25-12/29 Sheriffs: jcliang, semenzato, renyi
Ongoing Issues: - 783832: cheets_StartAndroid.stress fails on release builders and CQ. Trying to reproduce.
- 797620: Build failure chrome-pfq: login_CryptohomeIncognito and security_ProfilePermissions.guest are failing across multiple boards
- 796254: Archive stage fails in build_image
- 796275: bvt-arc times out across boards
- 789058: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed - seen across many canaries.
- 795128: evemu-device did not die resulting in security_SandboxedServices failing
- 794242: chromeos6-row22-jetstream-host5 repeatedly failing tests
- 796684: coral-paladin can't finish very often
- 796737: coral-release hasn't succeeded since build 599
- 782832: not enough daisy_skate devices to keep bvt pool alive
- 795912: kefka-release:1758-1773 failed
- 784914: DUT cannot reboot at pre-setup of rootfs update
- 715011: nvmem ec test crashes
- 795902: fizz-release failing since Dec 8
Resolved Issues: - 797314: cheets_MediaPlayerVideoHWDecodeUsed failing across boards
- 797599: Build failure on coral-release builder: build_packages failed
12/18-12/22 Sheriffs: bmgordon, martinroth, chenghan
Ongoing issues: - 797314: cheets_MediaPlayerVideoHWDecodeUsed failing across boards
- 796254: Archive stage fails in build_image
- 796275: bvt-arc times out across boards
- 789058: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed - seen across many canaries.
- 795128: evemu-device did not die resulting in security_SandboxedServices failing
- 794242: chromeos6-row22-jetstream-host5 repeatedly failing tests
- 796684: coral-paladin can't finish very often
- 796737: coral-release hasn't succeeded since build 599
- 782832: not enough daisy_skate devices to keep bvt pool alive
- 795912: kefka-release:1758-1773 failed
- 784914: DUT cannot reboot at pre-setup of rootfs update
- 715011: nvmem ec test crashes
- 795902: fizz-release failing since Dec 8
Resolved issues: - 796212: graphics_Idle: Unhandled ZeroDivisionError
- 796916: reef unibuild config doesn't validate
12/11-12/15Sheriffs: dtor, ecgh, deanliao
12/04-12/08 Sheriffs: hungte, rspangler, caveh
Ongoing issues: - 767953: cheets_StartAndroid.stress: FAIL: Android did not boot! (first reported 22-Sep; maybe recurring now)
- 789077: release: RootfsUpdateError: Update failed with unexpected update status: UPDATE_STATUS_IDLE'
- 792262: Chrome Pre-Flight Exceptions on M64 Branch
- 792592: ap-demons unit test failing with dbus errors on several canaries
- 792667: CQ failure: Moblab AFE timeout
- 793356: peach-pit-chrome-pfq failed HWTest because of no DUTs
- 793447: M63 builders failing with INVALID_BUILD_DEFINITION on stabilize branch
- 793499: Hwtest provision error on several chrome PFQs and informational PFQs
Resolved Issues: - 791600: Master scheduler is down
- 791786: caroline-tot-chrome-pfq-informational failed HWTest security_OpenFDs
- 791916: Master scheduler down with NoHostIdError
- 792115 -> 791643: TestSimpleChromeWorkflow stage failing due to gsutil creds not updating
- 792565: desktopui_ScreenLocker failing on betty (--> removed from smoke test)
- 792753: chromeos-firmware-coral build issues
- 792985: CQ failure: MySQL Cannot execute statement
- 757625: smbprovider unit tests failing ASAN builds
Misc: - 792536: Need coral testing for branch builds (--> just turned on; may result in new bugs)
11/27-12/01 Sheriffs: drinkcat, athilenius, slavamn
Ongoing issues
Needs attention: - 789058: -release: UnitTest/Archive steps race: WARNING: CreateTarball: tar: source modification time changed
- 787937: elm-paladin: SSP failed with ContainerError('Invalid base container.')
- 791178: No module named pbr.version in container
- 789077: release: RootfsUpdateError: Update failed with unexpected update status: UPDATE_STATUS_IDLE'
Assigned but not fixed (?): - 789062: guado_moblab-paladin failed due to "lxc-clone: command not found"
- 789451: novato-arc64-release: Target image has run out of space
- 740408: sheriffing rotation: No sheriff displayed on Monday morning TPE time
- Patch needs OWNERS review
- 788628: HWTest bvt-arc keeps timing out on a few boards
Deputy stuff: - 784914: provision failurs: DUT cannot reboot at pre-setup of rootfs update
- 788584: kefka/coral-release/paladin: Linksys USB3GIGV1 Ethernet adapter fails to enumerate (r8152, usb X-Y: device not accepting address Z, error -62)
788589: kefka-release: cannot recover from reboot at post check of stateful update // pre-setup of rootfs update (duped to above)
- Missing DUT sadness:
- 782832: not enough daisy_skate devices to keep bvt pool alive
788586: daisy_spring: Not enough DUTs for board: pool: bvt; required: 4, found: 3 - 788596: veyron_rialto: No good devices in pool:bvt
- 780738: M64: FAIL builds of veyron_tiger since 10/28
789352: enguarde: Not enough DUTs for board: enguarde, pool: bvt; required: 4, found: 2 - 789420: pyro-release: bvt-arc suite timeout
Resolved issues - 788455: lxc-start failing in HWTest for electro and basking
- 788595: pyro-release: lxc-start failing in HWTest for pyro
- ultima-release as well
- 788925: File dir-ROOT-A/opt/google/chrome/libwidevinecdm.so contains unsatisfied symbols: set(['\x07\x01'])
- Reverted libwidevinecdm change, hmchen and xhwang are looking
- AI: Could we possibly run ImageTest in Chrome PFQ to avoid this issue next time?
- 789839: chromium-pfq: BuildPackages: chromeos-chrome: Command 'lsb_release -a' returned non-zero exit status 3
- Broke -master and pfq for a few builds...
Misc: - 789461: eve-release: cheets_ContainerMount: Mount points are mismatched with the expected list
- 788017: falco-release times-out at BuildPackage
- 788592: nefario-release: The BuildPackages [afdo_use] stage failed: Packages failed in ./build_packages: sys-boot/depthcharge
Flakes and other issues (not fixed but not consistently failing either): - 789077: -release: RootfsUpdateError: Update failed with unexpected update status: UPDATE_STATUS_IDLE'
- 788591: mccloud: graphics_GLMark2: crash in i915_gem_retire_requests_ring/i915_gem_object_move_to_inactive
11/13-11/17 Sheriffs: benchan, nsanders, hiroh
Ongoing issues - 784462: Provision failure spike in the lab
- (Duplicated) 784222: PaygenTestDev failed on multiple canary builds
- 784225: TestLabException: Not enough DUTs on Chrome-PFQ, Android-PFQ and canary build
- 784686: veyron_rialto-paladin failed at BuildImage staging due to package: chromeos-base/telemetry
- 786159: ImportError: No module named lockfile
- 786159: HWTest failed due to INVALID_OPTIONS
- 786159: AFE is down: google-sso enforced a new config requirement, breaking our apache servers
- 786167: auto-update failed with StatefulUpdateError
- 786395: CQ master failed to push a change with 'git log' errors
- 786487: reef-uni-paladin failed due to no valid hosts for board:reef-uni
- 785552: provision failures: DUT cannot recover from reboot at post check of rootfs update
11/6-11/10 Sheriffs: puthik, ddavenport, cywang
Resolved issues - 782509: video_ChromeHWDecodeUsed mse tests are failed because crosvideo.appspot.com is broken down.
- 781845: desktopui_ScreenLocker failing on amd64-generic and betty
- 781302: slow queries on shards | chromeos-server98 and 104 tick rate is really low
- 783312: video_ChromeHWDecodeUsed failing on tricky, caroline, lumpy, peppy
- 781852: CQ failure when there are no CLs in the CQ run
- 783449: unittest flake in autotest_lib.site_utils.lxc.container_pool.client_unittest.ClientTests.testConnection
Ongoing issues - 776997: cheets_StartAndroid.stress failes and chrome / kernel crashes
- 783832: cheets_StartAndroid.stress timeout
10/30-11/6 Sheriffs: teravest, justincarlson, cywang - 782509: widespread Media.GpuVideoDecoderInitializeStatus not loaded or histogram bucket not found or histogram bucket found at < 100%" - the root cause is "404 in crosvideo.appspot.com". hiroh@ is helping to make a workaround to redirect requests to crosvideo2.appspot.com temporarily.
- 782577: incorrect dependencies of media-libs/arc-camera3-libcamera_jpeg (Fixed)
10/30-11/3 Sheriffs: teravest, justincarlson, fukino - 777920: [kernel 3.18] veyron_speedy provision failure: USB enumeration of ethernet adapter fails with "can't set config #1, error -71"
- 768542: DUT fails to bring up USB ethernet adapter after reboot in provision (chromeos kernel 4.4)
- 779583: General Protection Fault in kernel-list_move_tail called from i915
- Causes graphics_Idle failures
- 780515: daisy_skate-release:1910 failed
- 780045: BuildPackages failing to build chromeos-chrome
- This should be resolved, but keep an eye on the next goma update.
- 780503: cave-release:1635 failed
- 765686: wizpig-paladin Provision failed: Post-provision check for "system-services" being "start/running" can fail
- This needs more attention and debugging.
10/23-10/27 Sheriffs: akahuang, jinsong, mruthven - 777250: HWTest failed to provision on peach_pit and veyron_minnie, let Chrome gardener to triage
- 776919: lakitu-gpu, lakitu, lakitu paladin failed at build_package, should be fixed by CL:735061 and CL:737773
766259: buildstart stage failing with IntegrityError, a flaky failure.- 777829: Most paladins raised exception "process killed by signal 9"
10/16-10/20 Sheriffs: groeck, xiaochu, fukino, tetsui - 775872: M64: Cyan, Eve, Kefka, Samus build is RED for 4 days
10/9-10/13 Sheriffs: jclinton, furquan, posciak - 773185: All Chrome PFQ bots failing starting from 63.0.3237.0 due to a syntax error in DEPS
- 772568: lumpy, peppy, tricky Chrome PFQ failures in vmtest; manual uprev via 773446
10/2-10/8 Sheriffs: ntang, djkurtz, phobbs - 771396: Lab DNS failure caused wide spread master-paladin filaure.
- 771236: Provision failure due to version '9999'
- 772582: Puppet run may interrupt the ssh_config and causes ssh conntection failure.
- 770778: A few cases of shard apache process death, which needs alerting.
- 770865: Shard db inconsistent with master db causes shard_client crashloop
- 770715: Quite a few graphics_drm failure (fixed).
9/25-10/1 Sheriffs: chinyue, vbendeb, mxt - 769099: autotest-server & autotest-web-frontend circular dep
- 769334: betty-arc64-paladin failed VMTest
- 768280: build_image run out of space
9/18-9/24 Sheriffs: puneetster, amstan,
|
