Blink‎ > ‎

Gardening Blink

Relax. Keeping track of all the Blink commits can be stressful. Sometimes the best approach is to be patient and clean up one mess at a time until the tree is sparkling green again. Blink contributors are only human, and we all make mistakes from time to time.

Goals

There are two overarching goals to gardening Blink:
  1. Increase the revision of Blink we use in Chromium without regressing any tests.
  2. Prevent or fix any Blink regressions in tip of tree Blink.
Ideally, we would increase the Blink revision in Chromium (“roll DEPS”) in small increments, more closely approximating continuous integration. That’s not always possible, but we aim to roll DEPS at least once a day to avoid falling too far behind. When things are rolling smoothly, the gardener should always have a Blink roll in the commit queue. The moment the commit queue lands a roll, queue up the next one.

Prerequisites

Bots

For goal 1, of increasing the Blink revision, fix any failing canary Chromium test buildbots. Once all these bots are green you can roll the Blink revision. You do NOT need to wait for the layout test bots to be green. But use your judgement. If there are hundreds of layout test failures or there are failures you think are likely to cause significant problems in the next Chrome Canary release, then fix those before doing the Blink roll.

For goal 2, fix any failing canary LayoutTest buildbots and failing deps LayoutTests bots using the tools listed below. Getting no failures in garden-o-matic is usually (not always!) sufficient to achieve this.

LayoutTests can have different expected results on different platforms. To avoiding having to store a complete set of results for each platform, most platforms "fall back" to the results used by other platforms if they don't have specific results. Here's a diagram of the expected results fallback graph.

The ASAN bot is slightly unusual; generally speaking we only care about memory failures on that bot (currently we are actually looking at the test results, but that's a bug). You can suppress ASAN-specific failures using the LayoutTests/ASANExpectations file.

The Oilpan bots are currently not tree closers and gardeners can ignore them. They are being used by the Oilpan team, who are responsible for them (though they won't object if you have the time to help keep them running).

Blink gardeners don't have to worry about the GPU bots (the bots whose names start with "GPU").  GPU wranglers will take care of these bots.

Tools

Generally speaking, Blink developers are not supposed to land changes that knowingly break the bots (and the try jobs and the commit queue are supposed to catch failures ahead of time). However, sometimes things slip through ...

Rebaseline-O-Matic

Developers are supposed to know ahead of time which changes will require new baselines, and mark the affected tests as either "NeedsRebaseline" or "NeedsManualRebaseline" in LayoutTests/TestExpectations as part of the change that lands. 

There is a bot (called the Auto-Rebaseline-Bot, or rebaseline-o-matic) that runs the "webkit-patch rebaseline-o-matic" command to periodically update a checkout, look for new entries, run "webkit-patch rebaseline-expectations" to pull down new baselines when they are available, and then automatically land CLs with the new baselines.

So, if you see entries for "NeedsRebaseline", you can ignore them; if you don't see a bot periodically removing these lines from the file and landing baselines, bug someone via the gardening-big-red-button@chromium.org list (or blink-dev if you still don't get a response).

If you see entries with "NeedsManualRebaseline", this is basically equivalent to "Failure" but indicates that someone (the dev who made the change) needs to manually review and update the baselines. You should also ignore these entries, but feel free to nag the people that added them to clean up after themselves if you're bored.

Garden-O-Matic

Garden-O-Matic is a tool that watches the Canary buildbots and clusters test failures with the SVN revisions that might have caused the failures. The tool also lets you examine the failures and download new baselines to your working copy as well as easily monitor all the perf graphs on the canary waterfall.

To use Garden-O-Matic, run the following command from your Blink working copy:
Tools/Scripts/webkit-patch garden-o-matic

This webkit-patch command will spin up a local HTTP server (which is used to download new baselines to your working copy) and will launch the web-based UI.

Rolling back patches

To roll back patches, you can use either git revert or drover.

Flakiness dashboard

The flakiness dashboard is a tool for understanding a test’s behavior over time. Originally designed for managing flaky tests, the dashboard shows a timeline view of the test’s behavior over time. The tool may be overwhelming at first, but the documentation should help.

Contacting patch authors

Use either #blink on irc.freenode.net or comment on the corresponding Rietveld issue to contact the author. It is patch author's responsibility to reply promptly to your query.

Keeping track of ongoing issues

Fixing some problems are out of your control. You did your job filing the bug, and now it is being fixed: a bot is being brought back to life by the infrastructure team or another team is hunting down a flaky bug in browsercontentinteractive_testsorwhatevers. To keep your fellow gardeners informed of these problems, add a gardening-blink label to these bugs. This way, a gardener could always check http://crbug.com/?q=label:gardening-blink whenever an issue pops up and comfort themselves in knowing it is under control.

Workflows

Different gardeners prefer different workflows. One common workflow watches for new failures, attempts to resolve the failures as they occur, and rolls DEPS opportunistically.

Resolving Failures

Garden-O-Matic’s summary tab will help you watch all the canary bots for build failures and LayoutTest failures.

If a canary fails to build, fixing the compile error is the highest priority because build errors prevent us from getting test coverage:
  1. Follow the link from Garden-O-Matic to the compile error.
  2. Determine which patch caused the regression.
  3. Contact the author of the patch and ask him to fix the failure.
  4. If the author fails to respond in reasonable time, roll out the patch.
If the failure appears to be a flaky test (e.g., because it appears only on one cycle of one bot), you can either ignore the failure or mark the test as flaky in TestExpectations. The flakiness dashboard can be helpful in guessing whether a failure is due to flakiness.

If a patch introduces a new failing test, and the author did not create baselines for various Blink configuration:
  1. Determine which patch caused the regression
  2. Contact the author of the patch and ask him to fix the failure.
  3. If the author fails to respond in reasonable time, roll out the patch.
Unfortunately, there is a window of time between when a failure occurs and when you can address the failure. Sometimes patches land during that window that cause more failures. If that happens to you, do not worry. If you calmly address each failure in turn, the process will eventually converge and you’ll have a sparkling green tree again.

Rolling Blink DEPS

Once you’ve got a Blink revision that does not contain any regressions, it’s time to incorporate that revision in Chromium.
src/tools/safely-roll-blink.py new_revision_number

This will do the following:
  1. Change the webkit_revision in Chromium’s DEPS to new_revision_number.
  2. Create a changelist and add it to the commit queue.
The Canary Chromium test buildbots should be an accurate predictor of your try job will fair, but running the job through the commit queue covers more configurations and can ease your peace of mind.  
If you want to check whether you’ve succeeded in creating a blindingly green roll, you can look at the DEPS LayoutTest waterfall, which tests the most recent Chromium revision with the current DEPS revision of Blink.

When safely-roll-blink.py fails with message "Cannot upload with a dirty tree. You must commit locally first.", you may want to run gclient sync again.

Note: Eric runs a script which want to submit autogenerated rolling DEPS changes. Check his changes working fine, and try it manually if not.

Handling Disasters

Sometimes a patch comes through that changes the behavior of a massive number of tests. For example, rendering tree or SVG patches can require updates to a large number of image baselines. Don’t be afraid to ask for help. We’d rather put in the extra effort to handle these cases carefully than to be surprised when a nasty regression sneaks through.

Close the tree by clicking on blink status on a build page, e.g. Canary Console View.

Note: If Garden-O-Matic can’t handle the rebaseline, you can try using the older rebaseline tool.

If all else fails, you are still stuck, and getting further and further behind (say, building up a >150-revision roll), don't hesitate to press the BIG RED BUTTON to summon the few seasoned gardeners who'd seen it all and will gladly help you out of this predicament. No, there's no actual button. That was a metaphor (I think). Just send an email to gardening-big-red-button@chromium.org, and cc blink-dev@chromium.org.

Wrapping up the day

At the end of each day of gardening, write a summary email to blink-dev@chromium.org. As you finish your shift, look over http://crbug.com/?q=label:gardening-blink and remove the gardening-blink label from bugs that don't have useful content for the next gardener.

Final Thoughts

Gardening Blink requires a certain amount of Zen. If you feel yourself getting frustrated, don’t be afraid to stand up from your desk and get some fresh air. If you’re calm and relaxed, you’ll make better decisions and you’ll have more pleasant interactions with other contributors.

Good luck, and thanks for helping to make Blink and Chromium better.

Gardening Schedule
See: Chrome WebKit Sheriff Calendar
Subpages (1): Triaging Gasper Alerts