Layout Test Contest
Why? Fix all test ordering issues so we can enable --order=random-seeded by default. Drive number of lines in test_expectations to 0 so it is easier to maintain. Win fame and prizes!
Who? Everyone! Yes, you can participate even if you've never contributed to Blink before! In fact, this is the perfect way to get some Blink commits under your belt! You don't need to be a Google employee either.
Dates: The contest will begin on 9/30
and end on 10/20 at 11:59pm PST extended to 10/27 at 11:59pm PST!!! All code must be submitted for review during these dates to be eligible.
Scoring: Points will be awarded based on the tasks completed. Point values for each task are explained below. Each participant's score will be the sum of his or her points. See the leaderboard!
There are many chances to win! In addition to standard 1st, 2nd, 3rd Place, there are additional prizes:
- Cheetah: First person to 250 points. Start off the contest with a bang!
- Top Reviewer: Awarded to the participant with the highest number of points from code reviews only. Show off your code review skillz!
- Best Yak Shave: Awarded to the participant with the most ridiculous patch, or series of patches, required to solve a "simple" problem. Show your perseverance!
- Top Slacker: Awarded to the participant with the highest number of points gained, who starts AFTER October 14, 11:59pm PST.
Winners will select a prize from a prize category. Any prize can be exchanged for a "Get out of gardening free pass!" - Priceless!
Your choice of:
Indoor Sky Diving Party for 12
Kitchit Dinner Party for 8
Sennheiser HD 650 Headphones
iRobot Roomba or Scooba
LG Electronics 50LN5400 50-Inch 1080p 120Hz LED-LCD HDTV with Smart Share
1982 Port Ellen 30yr Scotch <img alt="1982 Port Ellen 30 Year Old "K&L Exclusive" Sovereign Single Barrel Cask Strength Single Malt Whisky 750ml" src="http://www.klwines.com/images/skus/1117203l.jpg" height=200 width=83>
Your choice of:
Indoor Sky Diving Party for 5
Kitchit Dinner Party for 4
Nexus 7 32G
Sonos Play 3
Breville Smart Oven Convection Toaster Oven with Element IQ
1996 Opus One Napa Valley Proprietary Red
Top Reviewer: Awarded to the participant with the highest number of points from code reviews only. Show off your code review skillz!
Top Slacker: Awarded to the participant with the highest number of points gained, who starts AFTER October 14, 11:59pm PST.
Your choice of:
Kitchit Dinner Party for 2
SodaStream Pure w/ Soda Starter Kit
Sennheiser HD 558 Headphones
Kindle Paperwhite, 3G
Breville Compact Smart Oven Toaster Oven with Element IQ
Macallan 18 year old Single Malt
Best Yak Shave: Awarded to the participant with the most ridiculous patch, or series of patches, required to solve a "simple" problem.
Your very own Shaving Yak Action Figure!
Indoor Sky Diving for 2
SodaStream Genesis w/ Soda Starter Kit
Sennheiser HD-280 PRO Headphones
Kindle Paperwhite, Wi-Fi
AeroGarden Classic 7-Pod with Gourmet Herb Seed Kit
2003 Dom Perignon <img alt="2003 Moet & Chandon "Dom Pérignon" Brut Champagne" src="http://www.klwines.com/images/skus/1089853l.jpg" height=200 width=83>
- Complete any of the tasks below.
- For each task, submit your participation info using this form. Sorry, any task completed, but not submitted in this manner, will not be eligible.
Points are not additive. Each patch can only apply to one category.
- Points: 50
- Examples: reset some state between test runs or fix a race condition in run-webkit-tests.
- Points: 5/line
- Includes [ Pass Crash ] lines.
Fix a line or flaky test in TestExpectations that doesn't crash
- Points: 3/line
Remove a line from TestExpectations (with no other changes, whitespace and comment lines do *not* count)
- Points: 1/line
- This is for TestExpectations-only changes. If you made a non-TestExpectations-only change that fixed more tests than you realized, you can still submit a TestExpectations-only change for 3 points per line.
- Points: 3
- Again, pure TestExpectations-only changes don't count.
- It's the reviewer's responsibility to fill out the form above.
Here are some ideas for ways to identify test flakiness/ordering issues.
- Run "run-webkit-tests --order=random --no-retry".
- Run "./Tools/Scripts/print-test-ordering" and save the output to a file. This outputs the tests run in the order they were run on each content_shell instance.
- For each test that fails:
- Find which worker it ran on.
- Create a file that contains only the tests run on that worker in the same order as in your saved output file.
- run-webkit-tests --child-processes=1 --order=none --test-list=path/to/file/from/previous/step
- If the test doesn't fail here, then the test itself is probably just flaky. If it does, remove some lines from the file created in step 2.2 and repeat step 3. Continue repeating until you've found the dependency. If the test fails when run by itself, but passes on the bots, that means that it depends on another test to pass. In this case, you need to generate the list of tests run by "run-webkit-tests --order=natural" and repeat this process to find which test causes the test in question to *pass* (e.g. crbug.com/262793).
- File a bug and give it the LayoutTestOrdering label, e.g. crbug.com/262787 or crbug.com/262791
Run "run-webkit-tests --batch-size 1 --no-retry". This starts up a new content_shell instance for each test. Tests that fail when run in isolation but pass when run as part of the full test suite represent some state that we're not properly resetting between test runs or some state that we're not properly setting when starting up content_shell. You might want to run with --timeout-ms=60000 to weed out tests that timeout due to waiting on content_shell startup time.
- Load the flakiness dashboard
- Tweak the flakiness threshold to the desired level of flakiness.
- Click on "layout-tests" to get that list of flaky tests.
- Diagnose the source of flakiness for that test.