This page describes the Gerrit Commit Queue for Chromium OS (also known by the code name "Paladin") The goal of the Commit Queue is to vet changes (build and test them on multiple platforms) and push them on behalf of developers. It is responsible for not only committing changes, but also doing the Portage voodoo (uprevving ebuilds) necessary for other developers to see each others changes. How do I submit my change to the Commit Queue?If your change has been reviewed and marked as +2 ("Looks good to me, approved"), you can submit your change by selecting the Reply button and marking your change as both "Verified +1" and "Commit-Queue +2". How does the Commit Queue verify changes?We first launch trybots to sanity check your change. This step is called PreCQ (Pre-Commit Queue). The trybots will compile your change, build images, and run unit and VM (virtual machine) tests on major platforms or platforms configured for the corresponding repositories/directories. This check only includes your change and related changes, and takes about 40 minutes. The commit queue runs approximately once every 3-4 hours, and picks up all changes that have been fully verified by PreCQ. It builds a full image (on a set of representative platforms) and runs tests on both virtual machines and real hardware. Each commit queue run takes approximately 3-4 hours, and a new run is kicked off immediately after completing the previous run. How does the commit queue decide when to pick up a change?The Commit Queue only picks up changes that have been approved, verified and marked ready, and only if all their dependencies are similarly approved, verified, and marked ready (see the "How do I specify the dependencies of a change?" section below).
Why wasn't my change picked up by Commit Queue?If your change was picked up by the Commit Queue when the tree was throttled and the run failed, the Commit Queue would attempt to progress by selecting a smaller set of changes in the next run. As the results, your CL may be skipped for the subsequent run. The Commit Queue will continue reducing the number of CLs until a run completes successfully. You do not need to do anything because the Commit Queue will pick up your change for sure once the tree is open or it completes a successful run. If you still suspect there was some wrongdoing of the Commit Queue, please contact the build deputy to triage the issue. When should I mark my changes as dry run ready?The dry run ready flag (CQ+1) lets committers kick off trybots before your change is approved. Select the CQ+1 flag in Gerrit and it'll kick off a trybot. The status will be reported back to Gerrit. If you later mark the same patch as CQ+2, the trybot will already be complete, so your CL may be committed up to an hour faster. You don't have to use the CQ+1 flag -- if you don't, the CQ will kick off trybots anyway when your CL is approved. Will the Commit Queue respect dependent changes in Gerrit?Yes. This means that the Commit Queue will not try a change that depends on another until both it and the change it depends on are marked as ready for commit. How do I specify the dependencies of a change?See the Cq-Depend documentation. How can I launch a non-PreCQ trybot for my change?If you want to test your change with non-PreCQ configs, see the remote trybot documentation. What repositories are currently managed by the Commit Queue?All projects under chromiumos/* and chromeos/* in gerrit and gerrit-int respectively are managed by the commit queue. How do I watch what the Commit Queue is currently doing?In order to see what the Commit Queue is doing, you can view its current work on the internal and external waterfalls.
What happens when the Commit Queue rejects my change?Depending on the scope of the test failures, the CQ may reject one, some, or all of the changes which were attempted during that run. The ChromeOS Commit Bot will set the Commit-Queue mark on any rejected change to 0. The ChromeOS Build Sheriffs and their deputies will attempt to attribute build and test failures to a particular root cause. When they have annotated and finalized their remarks on a failed build, the ChromeOS CL Exonerator Bot will go through and add Commit-Queue +2 back onto the CLs which were previously rejected, such that they may be retried on the next run of the CQ. If the Build Sheriffs suspect that the root cause was due to a particular CL, they will report Verified -1 on the suspicious CL. Note that the threshold of suspicion is low, and that the sheriffs may not be experts on your subsystem! If a sheriff has blamed your CL, then you have been deputized to investigate the build failure in greater detail. It could have been your CL, a problem with a earlier commit, a different CL in the same CQ attempt, etc. Even if your change wasn't the cause of the build failure, you are likely in a better position to investigate the cause than the sheriff was. My change was rejected by the Commit Queue, what can I do?Possibly nothing. If your change was not at fault, then the process of review and exoneration will automatically retry your change until it is committed. If it has been more than 4 hours since your change was rejected and it has not yet been exonerated, then you may mark it Commit-Queue +2 again manually after verifying that your CL did not cause the CQ failure.
When the CQ rejects your change, it adds a comment to Gerrit with additional information about why it was rejected. There will be a link to each of the failed builders' logs with URL's that end in $(board-codename)-paladin/$(build-number). You can also look at the commit-queue builders on the internal and external waterfalls and find the build run which has your CL in the CommitQueueSync stage. These are the same logs other builders would have so delve accordingly. Logfile hints:
There are three places where the commit queue can fail to commit your change.
Will the Commit Queue commit my CL when the Tree is Closed?No. However, it will commit any CLs it is processing if the tree is Throttled. e.g. Tree is open, CQ picks up your CL, tree goes to throttled, CQ finishes testing & merges your CL. How can I bypass the Commit Queue?Please do not bypass the commit queue unless it is necessary. That said, to bypass the commit queue, hit the submit patch button and select "Yes, I'm a chump". Please consult #crosoncall prior to chumping in your change, since it may complicate ongoing failure investigations. Can I bypass the hardware test step?Each repository has a COMMIT-QUEUE.ini file. You can control what steps to skip using this file. Does the Commit Queue ever pick up changes that haven't been verified by (PreCQ) trybots?If the Commit Queue finds that there are no changes that have been verified by trybots, the commit queue will start picking up unverified changes. This should only happen either at off-hours (when few developers are submitting changes) or when the trybot waterfall is having issues. What should I do if the Commit Queue itself is down?If the Commit Queue is down or consistently broken, please close the tree and contact a Build Sheriff or trooper immediately (chrome-troopers [at] google.com). More QuestionsContact: chromium-os-dev@chromium.org |
|
|