Sheriff FAQ: Chromium OS ASAN bots

Contents

  1. 1 Overview
  2. 2 Sheriff brief guideline
    1. 2.1 How to suppress a bug
  3. 3 Details
    1. 3.1 BuildTarget step differences
    2. 3.2 VMTest step differences
    3. 3.3 Archive step differences
    4. 3.4 Run time differences
  4. 4 Updating Clang revision periodically
    1. 4.1 SVN <-> Git mapping

Overview

This doc describes Chromium OS ASAN bots. Currently they are:

Triggered on Chromium OS commits:

  • amd64 generic ASAN (config name amd64-generic-asan) incremental bot on chromium os waterfall

Triggered on Chromium browser commits:

Sheriff brief guideline

The main idea of the bots is to use Clang as the compiler and its instrumentation, Address Sanitizer.

AddressSanitizer (ASAN) is a fast memory error detector based on compiler instrumentation (LLVM). It is fully usable for Chrome on Linux and macOS.

If the bot goes red on VMTest step or Archive step, look up artifacts. They should contain top-level files asan_log.1234.txt where 1234 is the PID of the failing process. Logs contain detailed information about the failure. Use it to file a bug, see this one as an example.

Note that Archive step will go red if there is at least one asan_log file found. This is made so because I noticed a rare case when ASAN caught a bug (and created log), but no test failed (and VMTest step remained green).

How to suppress a bug

When you filed a bug, but don't want to revert a change for some reason, you may suppress the offending function (top of your stack trace) like this:

// Mask it out for ASAN before the bug is fixed, see crbug.com/...
#if defined(ADDRESS_SANITIZER)
__attribute__((no_address_safety_analysis))
#endif  // defined(ADDRESS_SANITIZER)
void BuggyFunction() { ... }

Note that, as a result, the bug may become a SEGV and the bot will still be red. If this happens and you want to suppress it, you may add some filtering to the buildbot code that make bot red after asan_log: see GenerateStackTraces() in chromite/cbuildbot/commands.py.

Details

All mentioned Chromium OS ASAN bots are doing the same thing with slight differences:

  • amd64 generic ASAN bot is triggered by Chromium OS commits and tests amd64-generic board
  •  Chromium OS (amd64) ASAN is triggered by Chromium commits and tests amd64-generic board

Note that Chromium OS waterfall bots (1-2) build last known good Chrome, so if the ASAN bug creeps into Chrome code, bots 3-4 will go red immediately, while bots 1-2 may still be green for a day or so.

Following are the differences between Chromium OS ASAN bots and similar incremental / pgq bots (like amd64-generic-incremental / amd64-generic-tot-chrome-pfq-informational).

By default, Chromium OS ASAN report asan errors to var/log/asan/asan.PID. Also, one can modify the environment variable ASAN_OPTIONS="log_path=PATH"

Then every process will write error reports to PATH.PID

BuildTarget step differences

The bots use asan profile (chromeos/src/overlays/overlay-amd64-generic/profiles/asan/) to override some settings.
Mostly it adds USE="asan". So only packages that check that flag are built with Clang/ASAN.

If you ever need to try asan profile locally, do 
setup_board --board=amd64-generic --profile=asan --force 
then run build_packages as usual. Note that most boards do not have an asan profile. For other boards you may do manually what asan profile says.

VMTest step differences

Currently VMTest step runs the same sute_Smoke autotests as other Chromium OS bots do. One small exception: logging_AsanCrash test. It makes main Chrome process simulate memory bug and makes sure ASAN catches it. If not, test goes red.

We are working to add more tests to increase code coverage.

Archive step differences

As mentioned above, Archive step goes red if there is at least one asan_log file in /var/log/chrome directory. Additionally, for every such file, the log is symbolized and put to the top-level artifacts. Note that logging_AsanCrash test remove its asan_logs so they don't break things.

Run time differences

Session manager adds some environment variables to the corresponding processes environment. See "SetUpASAN" and below.

The most important is that it restricts Chrome from using sandbox (adds --no-sandbox flag).

Updating Clang revision periodically

We keep Chromium OS clang and its dependent llvm packages updated. As a reference, I use Gentoo repo of sys-devel. When updating local clang, I try to make it as close as possible to Gentoo.

As a signal to update Chromium OS Clang I watch at Chrome's tools/clang/scripts/update.sh (added myself to the WATCHLIST for 'clang_update'). Chrome Clang guys do the good job to test the new version before they commit.

To try the change, I start remote buildbot for the change: cros tryjob amd64-generic-asan --remote -g <gerrit-change-id>.

Currently Chromium OS ASAN bots are the only users of Clang in Chromium OS. However, llvm package is also used by mesa.

From the bots point of view, we want to use the most recent stable version of Clang to have the most edge ASAN features working. From the mesa point of view, edge llvm may not work. So we use chromiumos-overlay/profiles/targets/chromeos/package.mask to restrict latest llvm to be pulled to target chroots. Remember to update it when llvm version changes.

SVN <-> Git mapping

The git clang/llvm repos are pulled from git.  Since the main llvm repo is maintained in subversion, and the the git repos are mirrors of the svn repo, and the ebuilds are versioned based on the svn repo, you have to map the two.

Every git commit has metadata in it that tells you the svn rev it maps to.  As an example:

Clean up whitespace and indentation a bit

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173960 91177308-0d34-0410-b5e6-96231b3b80d8

This git commit f578c89dc6ca3e79667c2aa9d0ac4fe409da7773 maps to svn rev 173960.

With this information in hand, you can update the git commit sha1 in the ebuilds.

Note that since upstream llvm has a single svn repo for all of its projects, the svn rev in the git tree might not be there exactly.  So you have to find the commit in there that is closet to (but does not exceed) the svn rev you want.

Comments