SummaryMake Chromium's build process to deterministic. Tracking issue: crbug.com/314403 Handling failures on the deterministic bots GoalImprove cycle time and reduce infrastructure utilization with redundant tasks. Sub goals
So it's actually a 2x multiplier here for each target that becomes deterministic and runs on Swarming. Benefits are both monetary (less hardware is required) and developer time (reduced latency by having less work to do on the TS and CI). We estimate we'd save around >20% of the current testing load. This will result in faster test cycles on the Try Server (TS) and the Continuous Integration (CI) infrastructure. Swarming already dedupe 1~7% of the tasks runtime simply due to incremental builds. BackgroundTest isolation is an on-going effort to generate an exact list of the files needed by a given unit test at runtime. It enables 3 benefits:
Tracking issue: crbug.com/98637 Swarming is the task distributor that leverage Test isolation to run tests simultaneously to reduce latency in getting test results. Normal projects do not have deterministic builds and chromium is one example. A deterministic build is not something that happens naturally. It needs to be done. Swarming knows the relationship between an isolated test and the result when run on a bot with specific features. The specific features are determined by the requests. For example the request may specify bot OS specific version, bitness, metadata like the video card, etc. Google internally uses many tricks to achieve similar performance improves at extremely high cache rates: [link]
Non GoalsMaking the build deterministic is not a goal in these conditions:
Testing planEnforced by bots on the public waterfall on Android, Linux, and Windows; FYI bots on Mac. See also https://chromium.googlesource.com/chromium/src/+/HEAD/docs/deterministic_builds.md. MethodDocumented at http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.htmlOS Specific ChallengesEach toolset is non-deterministic in different ways so the work has to be redone on each platform. WindowsTracking issue: https://crbug.com/330260 Tracking issue: crbug.com/330262 Tracking issue: crbug.com/383340 Builder: Tracking issue: crbug.com/383364
Example workflow on Windows# Do a whitespace change foo.cc to force a compilation. echo "" >> foo_main.cc # It recreates the same foo.obj than before since the code didn't change. compile foo_main.cc -> foo_main.obj
link foo_main.obj -> bar_test.exe
isolate bar_test.exe -> isolateserver
swarming.py run <sha1 of bar_test.isolated> Extension of the projectGetting the build system to be content addressed as described in the Google reference above. This is a secondary task. It is out of scope for this project but is a natural extension. This saves on the build process itself at the cost of calculating hashes for each intermediary files. This will be piling on the fact that the build is itself deterministic. This is not worth doing before the build is itself deterministic and this property is enforced. This will save significantly on the build time itself on the build machines. |