Chromium‎ > ‎Chromium Security‎ > ‎

Security Release Management


Merging Fixes

TODO(chrome-security@): Identifying fixes to merge.

Security fixes often need to be merged to the Stable or Beta branch. One of the key considerations in picking a fix to merge is assessing the risk of breakage, which the bug’s owner should be able to convey. For risky fixes you should consider bumping it to a later milestone. Generally, you should consider merging the following:
For Critical and High bugs, we aim to fix users fast and work directly with Chrome TPMs to make sure any merge or release is done safely. The following are general recommendations on when / where to merge:
  • Released Critical vulnerability: Emergency, out of band release, target a fix as soon as we have a viable release candidate.
  • Unreleased Critical vulnerability: Target a fix for the next scheduled Stable point release. (An exception here is if we know it will be released prior to the next scheduled release, then we push as fast as needed to address it).
  • Released High vulnerability: A fix should be merged to the next scheduled Stable point release.
  • Unreleased High (and lower severity) vulnerability: Target the merge to the next scheduled Stable milestone release.
Other factors to consider in your merge vs. no-merge calculation are:
  • Bake time. If a fix has been landed on trunk for weeks and made it into various dev channels with no observed screaming, a merge candidate is said to be "well baked". A well baked patch might be considered for merge even if it is of lower severity, or appears scary. Conversely, we should rarely be merging anything that hasn't landed for a few days and been released to a dev channel.
  • Ease of discovery. If a bug has been found independently by multiple efforts (e.g. an external researcher as well as ClusterFuzz internally), this suggests that the bug is readily discoverable and should be merged sooner rather than later.
  • Ease of merge. If the merge is a total nightmare, it may be fair to not merge the fix and just wait for the fix on trunk to "roll into stable", as all work on trunk eventually does. Of course, if the bug is critical or other factors (ease of discovery, highly exploitable, etc.) apply, then we should bite the bullet and undertake the extra work in order to best protect users.
  • Change of behavior. If the fix in fact changes behavior (locks something down, turns something off, etc.) then it is best to have the fix appear as a major revision bump as opposed to a patch on the stable channel. This doesn't mean "don't merge" but it does suggest we might merge to beta but not the existing stable.
Your worklist for merges is defined by the following bug database criteria:
  • All Issues
  • Type-Bug-Security
  • Merge=Approved
(Note: beware of querying on the milestone. At this stage in the bug's life, the milestone label represents the earliest affected release at the time the bug was filed. So if current stable is M27, there might still be bugs in this list which are M25, M26, etc., if we've been a little slow in fixing them. Conversely, anything marked M28 denotes a regression since M27 stable, so it only needs to be merged to the M28 beta.)

You can perform the above query by clicking here.

TODO(bug owner): Merging a security patch.

When you've decided to merge a fix, you should verify with a release manager that the merge window is open for any affected branches.

To avoid accidentally losing a fix for a Chrome release, branch merges should be done in a "newer first" manner. For example, if you're merging a fix to M27 stable and M28 is branched and affected, then merge to M28 first. Such a strategy can also be used to bake a fix on the M28 branch in order to gain confidence about an M27 stable channel merge.

Drover

You should merge fixes with drover, which automates most of the process. If the drover merge does not apply cleanly you may want to reconsider whether the fix will introduce an unacceptable breakage risk. In the event that you can’t use drover to merge a fix, you still have the option of manually merging from a branch checkout. This follows the normal developer normal patch process.

Post-merge bug cleanup.

Once the fix is successfully merged you will need to:

  1. Update the bug with the merge revision numbers (one for each branch you've merged to).
  2. Set the merge status to Merge-Approved to Merge-Merged (if drover didn't do so).
  3. Change the M-# label to reflect the milestone where users of the stable channel will see the fix.
  4. Add Release-? label, where ? indicates whether the fix was merged or will appear in the initial release (Release-0) or a subsequent security patch (Release-1, Release-2, etc.). This is required for us to generate accurate security release notes (see below).
When considering merges, you may decide to NOT merge all of the fixes. In the case of not merging a given fix, you still may need to do label tidyup as noted above. In particular, make sure the bug's milestone will reflect when the fix will hit the stable channel, and use Release-0 to indicate that something will roll into the initial stable release on that branch.





Release Notes

For every new release, we include notes from security bugs that match the following search criteria:
  • Type=Bug-Security
  • M=? (Milestone)
  • Either Release=0 (Denotes that the fix is released in the first Milestone release) OR Release=1 (Denotes the fix is released in the first security patch).
  • -Security_Severity=None
  • -OS=Chrome, -OS=Android (the assumes you're putting together release notes for a desktop release, which is the most common).
For example, this link shows the security fixes that went into the initial Beta -> Stable promotion of Chrome 26, and this link shows the security fixes that went into the first post-stable patch of the Chrome 24 branch.

Every externally reported issue gets assigned a CVE. We coalesce: one CVE for all issues found by the same reporter and hitting the same feature or code area. Google's CVE pool is here (Google internal-only link, sorry). As of Chrome 27, we're focusing the release notes on externally reported issues. This mirrors how Mozilla Firefox arranges their release notes, saves the precious resource of CVEs, saves a lot of time in preparing release notes, and appropriately focuses our security release notes on the excellent contributions and rewards of external researchers. An example of how it looks is here for the Chrome 27 release notes. As can be seen, the internally found issues are bucketed into a single CVE and a single meta bug lists them off in case anyone cares.

security-notify@chromium.org


There is no longer any need to pre-notify security-notify@chromium.org with a copy of release notes. Very few list members were finding it useful or actionable.