ChromeOS security vulnerability management

What is vulnerability management?

The ChromeOS security team maintains a vulnerability management rotation to ensure that incoming security bugs are triaged promptly and fixed according to our security severity guidelines. Each rotation shift lasts a week, from Tuesday to Monday of the following week. The rotation is business hours only, you are not expected to handle bugs during the weekend.

Vulnerability management responsibilities

• The role of the vulnerability manager is to route issues and make sure existing issues get fixed according to the security severity guidelines.
• It is not the manager's responsibility to fix all the bugs coming in.
• Initially, expect to spend 50% of your time dealing with incoming bugs during your shift.
• Triage bugs in the Buganizer dashboard.
• Keep the vulnerability management worklog up to date to help the next person in the rotation.
• Make sure questions to chromeos-security@ don't go unanswered.

How do I handle incoming security bugs?

Open the Buganizer dashboard. The most important thing is to make sure all bugs have appropriate owners.

Buganizer

For Buganizer bugs, make sure that the issues:

• Have an owner.
• Have the correct priority set, per the security severity guidelines.
• Are assigned to the right component, which can be found via go/croscomponents or Buganizer component search.

For critical and severity bugs (all P0 and many P1 bugs), make sure that progress is happening on them. The Buganizer dashboard splits bugs by owner and by priority to help with this.

Third-party packages

If you see a bug reported in a third-party package, check the security-sensitive package list. If the package is in the list, make it a priority to update the package before your shift ends. You can often find newer versions of Portage packages in upstream Gentoo. Do not worry if the version you need to resolve a security issue isn't marked stable. There are instructions for upgrading Portage packages in the portage-stable mirror.

Sometimes a security bug might be in an unused feature of a third party package. If this is the case, you can often disable features during the configuration step.

How do I find an owner?

See go/croscomponents and OWNERS files under the code directory to find the right owner for the security bug.

What to do with syzkaller bug reports

Google maintains a kernel fuzzing project called syzkaller. The syzkaller dashboard is public and therefore folks are able to find syzkaller-found kernel bugs and report them to Chrome/ChromeOS. This is not immediately helpful, for a couple of reasons:

• We receive internal notifications for syzkaller bugs.
• The bugs don't normally reproduce on ChromeOS kernels.

The approach for these bugs is to ask for a repro case that applies to a currently-shipping ChromeOS device (including VM images.)

Process for porting security fixes to Long-Term Support (LTS) branches

The expectation is for all Medium, High, and Critical severity security bugs to be backported to the active LTS branch. This process is managed entirely by the LTS team. A security issue shall be closed after non-LTS branches are fixed per the security severity guidelines. No additional action is required from the vulnerability manager and/or the assignee.

Managing full-chain exploits

Ownership

When a full-chain exploit comes in, the current vulnerability manager owns the issue for the lifetime of the chain. If more than one chain comes in a given rotation week, the next person in the rotation owns the next chain, and so on and so forth. Refer to the vulnerability management rotation to find the next person in the rotation.

Handling a full-chain exploit

When a full-chain exploit comes in, the objective is to break the chain: to fix enough bugs that the exploit as submitted no longer works.

The vulnerability manager should handle a full-chain exploit by breaking the exploit down into its component bugs: each link in the chain should get a separate bug in Buganizer. Individual sub-bugs will be easier to understand and therefore faster to fix. This in turn allows breaking the chain quicker. Smaller sub-bugs also allow for easier merging into release branches, since the CLs should be smaller.

The exit criteria for a full-chain exploit are:

• The chain is broken on the stable branch.
• A short post-mortem is written covering which mitigations were useful and which parts of the system need more hardening. This document doesn't need to be long, it just needs to be comprehensive. Each full-chain exploit is a great prioritization mechanism. The objective of the post-mortem is to write down the information required for prioritization while it's still fresh in our minds.
  • The crash reporter security document has some examples of short analyses of security bugs.
  • A longer analysis for a Chrome browser full chain exploit can be found at A tale of two Pwnies (part 1). Keep in mind that the post-mortem doesn't need to be this long. A one-pager doc should be enough.

Once these two requirements are met, the main issue for the full-chain exploit can be marked as Fixed. Individual sub-bugs don't necessarily need to be closed before marking the main bug as fixed, as long as the exploit chain is successfully broken.