The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.
For Developers‎ > ‎How-Tos‎ > ‎

Using Dr. Memory

Contents

  1. 1 Dr. Memory is a new open-source tool for finding memory errors - similar to Memcheck (Valgrind) but with Windows support.
  2. 2 OK, I've found a real bug
  3. 3 drmemory_light vs drmemory_full
  4. 4 Suppressing error reports from the bots
  5. 5 Running Chromium under Dr. Memory on a given set of URLs
  6. 6 Running your custom command line under Dr. Memory
  7. 7 Feedback?

Dr. Memory is a new open-source tool for finding memory errors - similar to Memcheck (Valgrind) but with Windows support.

It is not working properly enough to be used on the memory waterfall yet, but we're trying to help it grow up and fulfill our memory error detection needs for Chromium.
We've set up a couple of Dr. Memory bots, you can check them out on the FYI waterfall.

If you want to try Dr. Memory on your machine, please do the following:
  • First of all, you need Windows.  Win7 x64 is preferred.  DrMemory supports Linux, but the support is not as robust because online symbolization is not yet supported.
  • Build Chromium tests on Windows in Debug configuration (you can save some time by building only the small tests like googleurl, ipc, base, net to start with).

    If running full Dr. Memory, you can also try a Release build which will be much faster but you may encounter false positive uninitialized read errors.
    With "light" Dr. Memory Release build is not recommended as callstacks are not as accurate.
    Also note that detection of mismatched free/delete/delete[] is not possible within chrome.dll in current Chromium Release builds.
    For better callstacks in Release build you should disable inlining via this entry in your ~/.gyp/include.gypi file:
    {
      'variables': {
        'build_for_tool': 'tsan',
      },
    }
    or
      set GYP_DEFINES=build_for_tool=drmemory
     gclient runhooks

    Frame pointer optimization has not yet shown to be a problem with callstacks with full Dr. Memory due to a clever callstack walking technique, but keep an eye out for strange callstacks when using Release build and let us know about them.

  • In your .gclient file, add the following lines as a new solution to check out Dr. Memory binaries:
      { "name" : "drmemory.DEPS",
        "url"  : "http://src.chromium.org/chrome/trunk/deps/third_party/drmemory/drmemory.DEPS",
      },
    and then do gclient sync (this will download around 5Mb of Dr. Memory binaries).

  • Now you can run chromium tests like this:
    tools\valgrind\chrome_tests.bat -t <test> --tool drmemory_light # only search for unaddressable accesses and uses-after-free
    or
    tools\valgrind\chrome_tests.bat -t <test> --tool drmemory_full  # search for uninits and leaks etc as well
    Alternatively, if you prefer Cygwin, you can also use tools/valgrind/chrome_tests.sh with the same arguments.  Just be aware that the bots use the .bat script.

  • It will print you some error reports at the end.

OK, I've found a real bug

Great!
Please file a crbug and add the Stability-DrMemory label.

drmemory_light vs drmemory_full

Light: only searches for unaddressable accesses like OOB or use-after-frees, adds moderate execution slowdown
Full: also searches for uninitialized reads and memory leaks but adds a large slowdown.

More on the "full" mode:
If you're familiar with the Valgrind guts you probably know that tools like Valgrind and Dr. Memory need to intercept and handle all system calls to know precisely what data is uninitialized, what is read and so on.
In contrast to the open-source Linux kernel with just a hundred system calls or so, on Windows we have an open-ended problem of handling undocumented system calls.
We're trying hard to do that but we haven't finished yet.

So, before diving into the "full mode" reports and going to file many bugs, you need to know a few things:
  • on XP 32-bit there should be little false positives on tests smaller than unit_tests and just a few dozens of reports on unit_tests
    XP was our primary target recently so it should be "mostly noiseless" - we've added support for a whole bunch of system calls
    and suppressed the rest of false reports until they are fully investigated.

  • on Windows 7 x64 Microsoft added a large number of new system calls we haven't investigated thoroughly yet.
    Be prepared for extra false "uninitialized access" reports

  • If you happen to find a definite false positive, please create a short reproducer and file it into our issue tracker.

  • If the above doesn't help you, use the "light" mode.

Suppressing error reports from the bots

TODO(rnk): When the DrMemory bots have moved to the chromium.memory.fyi waterfall, this information will be moved into the memory sheriff page.
timurrrr: most of this info sounds DrMemory-specific, do we really want to put it onto the "common" page?

The DrMemory suppression file is at tools/valgrind/drmemory/suppressions[_full].txt.
Generally speaking, suppressions in these files are maintained in exactly the same way as those for all of the other memory tools we use for Chromium, except that DrMemory uses a slightly different suppression format.

When the bot generates a report, follow the link to the bot's stdio and search for "hash=#" to find the report quickly.  You should see something similar to:
UNADDRESSABLE ACCESS: reading 0x00000000-0x00000004 4 byte(s)
# 0 TestingProfile::FinishInit           [chrome\test\base\testing_profile.cc:211]
# 1 TestingProfile::TestingProfile       [chrome\test\base\testing_profile.cc:164]
# 2 BrowserAboutHandlerTest_WillHandleBrowserAboutURL_Test::TestBody  [chrome\browser\browser_about_handler_unittest.cc:110]
# 3 testing::Test::Run                   [testing\gtest\src\gtest.cc:2162]
Note: @0:00:08.656 in thread 1900
Note: instruction: mov    (%ebx) -> %esi
Suppression (error hash=#30459DB8320B1FA3#):
{
UNADDRESSABLE ACCESS
name=<insert_a_suppression_name_here>
*!TestingProfile::FinishInit
*!TestingProfile::TestingProfile
*!BrowserAboutHandlerTest_WillHandleBrowserAboutURL_Test::TestBody
*!testing::Test::Run
}

As a starting point, you can copy the suppression (the portion between the curly brackets not including the brackets) into the suppressions.txt file and then widen it so it covers any similar reports on other bots.
The curly brackets are not part of the suppression as in all the other memory tools! DrMemory suppressions are separated by blank lines instead, so make sure your suppression does not run into an existing suppression.

Each frame from the stack trace is of the form "<module>!<function>". For functions from Chrome, we generally link the code statically into many different binaries, so we always use a wildcard for the module.
For third party code, it is generally helpful to identify which DLL the frame came from, especially since we often do not have debug info for windows system libraries.

As in the other tool suppression formats, '*' is a single frame variable-width wildcard, '?' is a single-character wildcard, and '...' will match any number of frames.

DrMemory has support for a handful of other pattern matching mechanisms.
In particular, if you paste the instruction from the report on a line starting with "instruction=", you can match the exact instruction producing the report.
The instruction= qualifier is mostly useful when suppressing bit-level false positive uninitialized reads in third party libraries where we don't have an accurate callstack and want to make a tight suppression.

Once you have your suppression, you can test that it suppresses the existing reports using tools/valgrind/waterfall.sh as you would with the other memory tools.

Running Chromium under Dr. Memory on a given set of URLs

First, put the list of URLs into tools\valgrind\reliability\url_list.txt, one URL per line without the http:// prefix. You can also put in the local URLs like file://c:\mypath\repro.htm
Then, run
tools\valgrind\chrome_tests.bat -t reliability --tool drmemory --tool_flags="-no_check_uninitialized -no_count_leaks"

If you're searching for some stability/security bugs, you may consider running Chromium in the single process mode to increase the probability of the report by applying this patch:
Index: tools/valgrind/chrome_tests.py
===================================================================
--- tools/valgrind/chrome_tests.py      (revision 96002)
+++ tools/valgrind/chrome_tests.py      (working copy)
@@ -271,7 +271,8 @@
   # Valgrind timeouts are in seconds.
   UI_VALGRIND_ARGS = ["--timeout=7200", "--trace_children", "--indirect"]
   # UI test timeouts are in milliseconds.
-  UI_TEST_ARGS = ["--ui-test-action-timeout=120000",
+  UI_TEST_ARGS = ["--single-process=yes",
+                  "--ui-test-action-timeout=120000",
                   "--ui-test-action-max-timeout=280000"]

   def TestAutomatedUI(self):

Running your custom command line under Dr. Memory

Sometimes you may need to run your own program or your own custom command line under Dr. Memory. For example, if you want to prefix your own "chrome.exe --my-flags" command with Dr. Memory, do the following:
tools\valgrind\drmemory.bat myapp.exe --my-flags  # don't forget to increase the timeouts
If the command line runs Chrome, you may consider increasing the timeouts by appending
--ui-test-action-timeout=120000 --ui-test-action-max-timeout=280000 --ui-test-terminate-timeout=120000
to the commandline.

Random notes for the curious ones
You can find the Dr. Memory suppression file for Chrome at tools\valgrind\drmemory\suppressions[_full].txt

The report files are parsed and re-formatted to be more familiar for Valgrind users (see tools\valgrind\drmemory_analyze.py).

Feedback?

Drop drmemory-team@ a message

Comments