Gatekeeper-NG

gatekeeper-ng is the next generation version of the Chromium gatekeeper. The gatekeeper is responsible for monitoring builders for critical failures and closing the tree when this occurs. It is also responsible for emailing committers and interested parties when this happens.

Rationale

The old version of gatekeeper was embedded in the Buildbot master and was configured on a per-master basis (usually in master_gatekeeper_cfg.py). This had two drawbacks: configuration couldn't be done on the fly (needed a master restart), and the gatekeeper couldn't handle annotated builds gracefully.

gatekeeper-ng is an out-of-process gatekeeper which crawls the external Buildbot JSON for build information. By separating the gatekeeper from Buildbot, it can be reconfigured on the fly and can handle dynamic steps. It consists of two main components: a daemon which crawls Buildbot JSON, and a mailer appengine app which sends notification emails.

Daemon

The gatekeeper-ng daemon lives in scripts/slave/gatekeeper_ng.py. It is invoked with a list of masters to watch. On startup, it loads scripts/slave/gatekeeper.json to determine which steps on which builders to watch, and whom to notify about them. The syntax for gatekeeper.json is documented in scripts/slave/gatekeeper_ng.py (see load_gatekeeper_config).

The daemon is meant to be invoked every minute, and will pick up any new builds since its last run. When it detects a failure, it turns the tree red. It then compiles a list of emails to be notified (committers in that build, sheriffs, additional watchers) and uploads a json representation of the build to the mailer app.

Mailer App

The mailer app lives as part of chromium-build.appspot.com. The daemon POSTs a json representation of the build failure along with a list of emails to be notified. The mailer app generates an HTML representation of the build and sends out the email.

Current Status

gatekeeper-ng is still under testing and is not deployed to the main waterfall yet.
Comments