The IDL build is a significant source of generated files, and a complex use of the build system (GYP or GN); see their documentation for possible issues.
Components other than bindings do not need to know about the internal organization of the bindings build. Currently the components
GYP files (.gyp and .gypi) are arranged as follows:
.gyp: There's a generated.gyp file in each directory that generates files (outputting to
.gypi: The .gypi files are named (as usual) as:
There are relatively long lists of .gypi includes in the .gyp files, which are due to factoring for componentization, and are longer than desired due to layering violations, namely
For build performance in compiling a single IDL file, see IDL compiler: Performance
Build performance is one of the IDL compiler goals, though secondary to correctness and performance of the generated code.
The primary concern is minimizing full build time (notably full bindings generation and compilation); the secondary concern is maximizing incrementality of the build (only do full rebuilds when necessary). These priorities are because full builds are common (e.g., on build bots and for any compiler code changes), and changes to individual IDL files are also common, which should only rebuild that file's bindings and any dependents, not trigger a full rebuild.
Current performance (as of Blink r168630 in March 2014) is acceptable: on a fast Linux workstation with 16 cores, full regeneration takes ~3 seconds (~50 seconds of user time, ~80 ms/IDL file), and full rebuilds on IDL changes only occur if a dependency IDL file (partial interface or implemented interface) is touched, which is infrequent.
The coarsest way to profile a full build – which is sufficient for verifying coarse improvements – is to do a build (using ninja), then
In the overall build, bindings compilation and linking are more significant factors, but are difficult to improve: minimizing includes and minimizing and simplifying generated code are the main approaches.
The main improvement would be precise computation of dependencies to minimize full rebuilds, so only dependent files are rebuild if a dependency is touched (Bug 341748), but this would require GYP changes.
IDL build performance contains various components:
Build performance criteria of the IDL build are the time for the following tasks:
The key techniques to optimize these are:
Compilation of multiple IDL files is embarrassingly parallel, as files are almost completely independent (code generation is independent, and reading in is independent other than reading in implemented or referenced interfaces, which is currently minor, and global information, which is handled in a separate step). Thus with enough cores or a distributed build, compilation is very fast: assuming one core per IDL file, full recompile should take 0.1 seconds, plus distribution overhead.
The main areas at the overall build level that suggest further optimization are as follows. These would not help significantly with full rebuilds, but would improve incrementality. These are thus possible, but not high priorities.
This would require new features in GYP, to allow per-file dependency computation in rules.
This would speed up incremental builds a bit, by minimizing the global computation, but may slow down full rebuilds significantly, due to launching O(n) processes.
Currently global computations are done in a single step, which processes all IDL files. Thus touching any IDL file requires reading all of them to recompute the global data. This could be replaced by a 2-step processes: compute the public data one file at a time (n processes), then have a consolidation step that reads these all in and produces the overall global data. Further, if the public data does not change (as is usually the case), the consolidation step need not run. This would thus speed up incremental builds. However, this would require n additional processes, so it would slow down full rebuilds.
Certain mistakes that significantly degrade build performance are easily made; all of these have either occurred or been proposed.
Be careful to use "write only on diff" (correctly) for all global dependency targets.
Touching a global dependency (a file or target that is a dependency of all IDL files) causes a full recompile: if a global dependency changes, all IDL files must be recompiled. Global dependency files are primarily the compiler itself, but also (for now) all dependency IDL files, since we compute dependencies conservatively in the build, not precisely. Global dependency targets (files generated in the preliminary steps) are primarily the global context, but also the generated global constructor IDL files, as these are partial interfaces (hence dependencies, hence treated as global dependencies). Touching any IDL file requires recomputing the global context and these generated global files. However, most changes to IDL files do not change the global dependencies (global context and global constructors), since they only change private information, and thus should not trigger full rebuilds. The "write only on diff" option, supported by some build systems, notably ninja, means that if these files do not change, we do not regenerate the targets, hence avoiding a full rebuild. If this is missed, touching any IDL file cause full regeneration.
Avoid launching processes per-IDL file: preferably only 1 process for the actual compilation, avoiding per-file caching steps.
Launching processes is relatively expensive, particularly on Windows. For global steps this is O(1) in the number of IDL files, so adding an extra global build step is cheap, but for compiling individual IDL files this is O(n), so it's faster to have 1 process per IDL file, i.e., compile each file in a single step. Thus any changes that add a process per IDL file are expensive. These include: splitting global computations into "compute per file one process at a time, then consolidate in one step" (turns O(n) computation in 1 process into O(n) computation in n + 1 processes); splitting the compile into "compute IR, then compile IR" (turns n processes into 2n processes).
Compute global data in a single preliminary step.
Computed data that is needed by more than one IDL file should generally be done a single time, most simply in the global context computation step (or a GYP variable); getting this wrong can turn O(n) work into O(n2) work. For example, public information about an interface should be computed once, rather than computing it every time it is used.
However, in some cases redundant work is preferable to adding extra processes. For example, there is some redundancy in the front end, since some files (notably implemented interfaces) are read during the compile of several IDL files, but this is relatively rare (about 30/600=5% of files are read multiple times) and cheap (actual parsing is cheap relative to parser initialization), and fixing it would require a separate per-file caching step, which would increase the number of processes.
Use precise (not conservative) computations and static lists in GYP, and be careful of GYP rules.
GYP rules generate actions; for the IDL build this is the
See also 350317: Reduce size of individual_generated_bindings.ninja.
Work can be reduces to O(1) if instead of using a Python script to compute dependencies dynamically (at gyp runtime), one determines the dependencies statically (when writing the
Some intuitively appealing optimizations don't actually work well or at all; these are discussed here. These generally have little impact (or negative impact) and involve significant complexity in the build or compiler.
Python and library initialization can be further sped up by processing multiple IDL files in a single compiler run (a single compilation process). However, this significantly complicates the build due to needing to manually distribute the input files to various build steps, ideally one per available core. This interacts poorly with GYP (GYP rules are designed for a single input file and the corresponding output), and is only useful if you cannot fully distribute the build, and thus is not done.
Note that multiprocessing or multithreading Python would not allow a single Python process to compile all files in parallel (thus minimizing initialization time), since the Python interpreter is not concurrency safe (hence not multithreaded; concretely this is due to the Global Interpreter Lock, GIL), and the actual parsing and templating are done in Python. A multiprocessing Python process could launch separate (OS) processes to compile individual files, but this offers no advantage over the build system doing it, and is more complex.
One could start a single Python process, which then forks others once it has initialized to reduce startup time (both of Python and of the libraries): either one per input file or distributing among them. This allows both parallelization and minimal initialization, so it would increase speed. However, this is platform dependent (
The IR is computed multiple times for a few IDL files, namely implemented interfaces that are implemented by several IDL interfaces and targets of
Currently if the global context changes, all bindings are generated. This is conservative and simple, which is why we do it this way, but it causes excess full rebuilds: if the public data of one IDL file changes (e.g., its
This is not done because it would add significant complexity for little benefit (changes to public data are rare, and generally require a significant rebuild anyway), and would require GYP changes (above) for file-by-file dependency computation in rules.