This page describes the Gerrit Commit Queue for Chromium OS (also known by the code name "Paladin")

The goal of the Commit Queue is to vet changes (build and test them on multiple platforms) and push them on behalf of developers.  It is responsible for not only committing changes, but also doing the Portage voodoo (uprevving ebuilds) necessary for other developers to see each others changes.

How do I submit my change to the Commit Queue?

If your change has been reviewed and marked as +2 ("Looks good to me, approved"), you can submit your change by selecting the Reply button and marking your change as both "Verified +1" and "Commit-Queue +1".

How does the Commit Queue verify changes?

We first launch trybots to sanity check your change. This step is called PreCQ (Pre-Commit Queue). The trybots will compile your change, build images, and run unit and VM (virtual machine) tests on major platforms or platforms configured for the corresponding repositories/directories. This check only includes your change and related changes, and takes about 40 minutes.

The commit queue runs approximately once every 3-4 hours, and picks up all changes that have been fully verified by PreCQ. It builds a full image (on a set of representative platforms) and runs tests on both virtual machines and real hardware. Each commit queue run takes approximately 3-4 hours, and a new run is kicked off immediately after completing the previous run.

How does the commit queue decide when to pick up a change?

The Commit Queue only picks up changes that have been approved, verified and marked ready, and only if all their dependencies are similarly approved, verified, and marked ready (see the "How do I specify the dependencies of a change?" section below).

• If the tree is open, the Commit Queue will start a normal run and pick up all eligible changes (see How do I submit my change to the Commit Queue above).
• If the tree is throttled, the Commit Queue will pick up Commit-Queue+2 changes if they are available, or pick up a set of changes to test. In the latter case, the changes will not be rejected even if the build fails because the tree is considered not sane (since it is throttled). After a failed run, the subsequent run will attempt with a smaller set of changes.
• If the tree is closed or under maintenance, CQ will not pick up any changes.

Why wasn't my change picked up by Commit Queue?

If your change was picked up by the Commit Queue when the tree was throttled and the run failed, the Commit Queue would attempt to progress by selecting a smaller set of changes in the next run. As the results, your CL may be skipped for the subsequent run. The Commit Queue will continue reducing the number of CLs until a run completes successfully. You do not need to do anything because the Commit Queue will pick up your change for sure once the tree is open or it completes a successful run. If you still suspect there was some wrongdoing of the Commit Queue, please contact the build deputy to triage the issue.

When should I mark my changes as Trybot Ready?

The trybot ready flag lets committers kick off trybots before your change is approved. Select the "Trybot Ready" flag in Gerrit and it'll kick off a trybot. The status will be reported back to Gerrit. If you later mark the same patch as "Commit Ready", the trybot will already be complete, so your CL may be committed up to an hour faster.

You don't have to use the Trybot Ready flag -- if you don't, the CQ will kick off trybots anyway when your CL is approved.

Will the Commit Queue respect dependent changes in Gerrit?

Yes. This means that the Commit Queue will not try a change that depends on another until both it and the change it depends on are marked as ready for commit.

How do I specify the dependencies of a change?

If you commit a stack of CLs to a branch locally, and upload them all together, Gerrit will already know about your dependencies and the commit queue will honor these dependencies.

If you want to specify a dependency on other CLs, you can specify a dependency on any CL in Gerrit by using CQ-DEPEND. Here's how it works:

• Specify dependencies in the last paragraph of your change, using CQ-DEPEND=CL:12345
• You can specify multiple dependencies. Each dependency should be separated with a comma and a space. (E.g. CQ-DEPEND=CL:12345, CL:4321). You can also split dependencies into multiple CQ-DEPEND lines, if needed.
• Internal dependencies should be prefixed with an asterisk. To depend on an internal change, use CL:*12345. To depend on an external change, use CL:12345.
• Atomic transactions within a single repository are supported. If changes across multiple repositories depend on each other, there is a small window where syncs may pull a partial set of the dependent changes.
• If a stack of changes need to go in at once, the first change in the series should have a dependency on the last change in the series. This ensures that the Commit Queue will only push the whole group of CLs at once.

Here's an example:

How can I launch a non-PreCQ trybot for my change?

If you want to test your change with non-PreCQ configs, see the remote trybot documentation.

What repositories are currently managed by the Commit Queue?

All projects under chromiumos/* and chromeos/* in gerrit and gerrit-int respectively are managed by the commit queue.

How do I watch what the Commit Queue is currently doing?

In order to see what the Commit Queue is doing, you can view its current work on the internal and external waterfalls.

• The CommitQueueSync step shows what builds the current Commit Queue build is testing 
• The CommitQueueCompletion step is responsible for submitting changes when they pass these tests

What happens when the Commit Queue rejects my change?

My change was rejected by the Commit Queue, what can I do?

When the CQ rejects your change, it adds a comment to Gerrit with additional information about why it was rejected.  There will be a link to each of the failed builders' logs with URL's that end in $(board-codename)-paladin/$(build-number).  You can also look at the commit-queue builders on the internal and external waterfalls and find the build run which has your CL in the CommitQueueSync stage.  These are the same logs other builders would have so delve accordingly.

Logfile hints:

• master-paladin serves primarily to execute the other builders and aggregate their responses.  The most interesting logs are usually found on the slaved paladins.
• Logs are managed by several systems.  Links into any of the following are links about the logs: Luci, Milo, Stainless, LogDog, and Goldeneye.
• (Googler-specific) Stainless is not multiple-login aware.  If your browser session is logged into both @google.com and @chromium.org credentials, and you are presented with an authorization error (Forbidden 403), they try appending "?authuser=1" to manually select your other credentials.
• In the builder's log, failures in BuildPackages and BuildImage correspond directly to the build_packages and build_image steps in a normal developer workflow.  Failures to compile or package a CL will show up here.
• Failures in tests that are managed by 'autotest' will include a tree of logfiles, including logs from the device under test and logs from the system driving the test.

There are three places where the commit queue can fail to commit your change.

1. When applying a change: If your change conflicts with another change, your change will be rejected and you will need to rebase your change locally and resolve the conflicts.
2. When verifying a change: This is the standard failure where either we fail to build or test your change. Please read the error message and verify that the failure was not caused by your change. If your change was not at fault, you may mark it Commit-Queue+1 again manually.
3. When submitting a change: This can happen if either you already submitted your change while the commit queue was working or someone else bypassed the commit queue and submitted a change that conflicts with your change. If the latter, follow (1).  Additionally, if a revised patchset was published to Gerrit while the CQ was evaluating it, or the change was Verified -1 prior to completing the CQ run, then the change will also be rejected.

Will the Commit Queue commit my CL when the Tree is Closed?

No. However, it will commit any CLs it is processing if the tree is Throttled.

e.g. Tree is open, CQ picks up your CL, tree goes to throttled, CQ finishes testing & merges your CL.

How can I bypass the Commit Queue?

Please do not bypass the commit queue unless it is necessary. That said, to bypass the commit queue, hit the submit patch button and select "Yes, I'm a chump".  Please consult #crosoncall prior to chumping in your change, since it may complicate ongoing failure investigations.

Can I bypass the hardware test step?

Each repository has a COMMIT-QUEUE.ini file. You can control what steps to skip using this file.

Does the Commit Queue ever pick up changes that haven't been verified by (PreCQ) trybots?

If the Commit Queue finds that there are no changes that have been verified by trybots, the commit queue will start picking up unverified changes. This should only happen either at off-hours (when few developers are submitting changes) or when the trybot waterfall is having issues.

What should I do if the Commit Queue itself is down?

If the Commit Queue is down or consistently broken, please close the tree and contact a Build Sheriff or trooper immediately (chrome-troopers [at] google.com).

More Questions

Contact: chromium-os-dev@chromium.org