Chrome build with instrumented libraries.
To build chrome or base_unittests with instrumented system libraries use instrumentation flag (asan=1 or msan=1) and this flag in GYP_DEFINES: use_unstrumented_libraries=1
In presence of this flag the build system downloads libraries using apt-get source, builds them and installs to out/Release/instrumented_libraries/asan (in case of Release build and asan=1).
It also adds this path to RPATH of the final binary (chrome or base_unittests).
You can check which libraries are instrumented using this command: ldd base_unittests.
If you build using a custom compiler (not clang from chrome repository), you need to set CC and CXX environment variables when you call ninja. That’s needed because it builds libraries using their ./configure && make and it seems there’s no way to get CC and CXX from gyp and pass it to the build script.
How it works.
The GYP-file with targets is third_party/instrumented_libraries/instrumented_libraries.gyp. It has main target (instrumented_libraries) which depends on many other targets - one target for one instrumented library. They all have format [sanitizer_type]-[library_name], for example, asan-libnspr4.
Most of the targets can be built using download_build_install.py.
The common target looks like this (in instrumented_libraries.gyp):
'library_name': Name of debian package to download and build.
'dependencies=': Should be empty list if this library is built first and should be list of other libraries if they should be built before current library. The “=” suffix is important.
'custom_configure_flags': list or string with flags which should be passed to ./configure which is called before library build.
'run_before_build': command to run after unpacking library and calling ./configure. It’s usually shell script, which patches some sources, makefiles or unpacks archives with code. See examples in third_party/instrumented_libraries/
'includes': should be standard_instrumented_library_target.gypi for most of the targets (which can be built common way).
Libraries are built by special script: third_party/instrumented_libraries/download_build_install.py.
It is called by GYP with parameters of particular library build.
That script takes parameters: which library to build, which instrumentation type to use and where to install the library. It downloads library using apt-get source, builds using ./configure and make and installs using make install.
Parameters of the script are:
'--jobs': number or parallel jobs for make (it just directly passes this parameter to make).
'--library': package name with library to download and build.
'--product-directory': directory where GYP builds chrome or base_unittests (or other executable you want). The script installs libraries in: product_directory/instrumented_libraries/asan (in case of asan=1).
'--intermediate-directory': the directory where the script downloads the package and builds the library.
'--custom-configure-flags': flags which are passed to ./configure to build the library.
'--custom-c-compiler-flags', '--custom-cxx-compiler-flags' and '--custom-linker-flags': CFLAGS, CXXFLAGS and LDFLAGS respectively which are set before ./configure call.
'--sanitizer-type': should be asan or msan. This parameter is determined automatically in instrumented_libraries.gyp (depending on asan=1 or msan=1).
'--verbose': in presence of this flag the script will print to stdout all output from ./configure, make, make install (and you’ll see it after call ninja).
'--check-build-deps': in presence of this flag the script will check that all dependencies of the package are installed (using apt-get build-dep). If not all of them are installed, the build is cancelled. Usage of this flag is not recommended.
'--c-compiler' and '--cxx-compiler': CC and CXX which are set before ./configure call. If environment variables CC and CXX are already set, they will be used and script parameters will be ignored.
'--run-before-build': shell script which is called after downloading the package and running ./configure. Can be used to patch sources and makefiles, extracting archives with sources (necessary for some packages).
How to add a new instrumented library.
To add a new library you need to create an entry in third_party/instrumented_libraries/instrumented_libraries.gyp as described above. Most of the libraries can be built in a standard way (./configure && make && make install). So in that case you need to determine which LDFLAGS, CFLAGS, CXXFLAGS and flags for ./configure are needed to build the library (except for the sanitizer flags - they will be added automatically).
Then you need to add library name to the list: dependencies in target instrumented_libraries in instrumented_libraries.gyp. If this library cannot be built with all sanitizers, you can add it under condition (see examples in instrumented_libraries.gyp).
If the library cannot be built in a usual way (./configure && make && make install), you can add an entry to instrumented_libraries.gyp without including standard_instrumented_library_target.gypi. You need to write a script which builds and installs library and add it as action to the library target in GYP.
When you add new library, ensure that in 'dependencies=' it has all necessary instrumented libs. For example
Instrumented libc++ and libc++abi.
You can turn on usage of instrumented libc++ and libc++abi using flag in GYP_DEFINES: use_custom_libcxx=1. It doesn’t depend on flag use_instrumented_libraries, but instrumentation type is determined the same way (if you have asan=1, that will be asan). You can also use it without any instrumentation - just libc++ and libc++abi instead of libstdc++.
It should be used on bots soon. See https://code.google.com/p/chromium/issues/detail?id=330526
Build order is important. When it builds an instrumented library, all its dependencies should be already instrumented.
During the build some libraries, make produces some binary executables which have dependencies and these dependencies should be instrumented.
But one does not simply recognize all the dependencies. If ninja changes build order (for any reason), new errors related to the dependencies may occur.
Please, add missing dependencies if you catch such errors.
Build with msan.
Some of the libraries produce and run some executable binaries during make or make install. For example, libfontconfig compiles and runs file fc-cache during make install. Other libraries can compile and run some tests which can be shut down using ./configure flag. And they print msan reports very often, so such libraries should be patched even if that binary executable is not needed for final .so.
Running on bots.
One does not simply run that on bots because some of the libraries have build dependencies. You can see the install them using apt-get build-dep. But in fact, they all are not necessary because a lot of them are used to generate documentation and other things which are unrelated to our main purpose (build instrumented .so library). Anyway, some libraries have critical dependencies and couldn’t be built without them. That dependencies should be installed on bots too. Now it’s in progress.
Ubuntu source repositories should be added:
Also, dpkg-dev and library dependencies should be installed.
Here are all commits related to instrumented libraries build:
dpkg-dev is already added to build deps, but seems not installed on bots.
RPATH and $ORIGIN hack.
How can final executable binary (chrome, base_unittests) find our instrumented libraries? Of course, using RPATH. If you turn on use_instrumented_libraries, it adds $ORIGIN/instrumented_libraries/asan to RPATH of final executable (in case of asan=1). But some of the instrumented libraries have other libraries in their dependencies - one instrumented .so file need to link with another instrumented .so file. That means we need to add “current folder” to RPATH of each library because all of them are in one folder. That can be done using adding -Wl,-R,$ORIGIN/. to LDFLAGS. But when it runs ./configure, it passes this string to makefiles (in several libraries) and causes an error due to “$” sign in makefile. To avoid this problem completely we use the next hack: firstly, build library with -Wl,-R,XORIGIN/ in LDFLAGS. After that we use program chrpath (changes RPATH) to replace X with $. See third_party/instrumented_libraries/fix_rpaths.gyp for details.
Program chrpath should be installed if you want to build instrumented libraries.
List of already built libraries and notes about them
chrome: 73 dependencies (.so), 35 of them are instrumented by asan.
base_unittests: 51 dependencies (.so), 29 of them are instrumented by asan.
net_unittests: 54 dependencies (.so), 31 of them are instrumented by asan.
media_unittests: 55 dependencies (.so), 31 of them are instrumented by asan.
unit_tests: 75 dependencies (.so), 35 of them are instrumented by asan.
Continuous testing is to be started in Q1'14.