32-bit debugging tricks

Imagine that your 32-bit NaCl module is crashing, it can not be run with a command-line sel_ldr, and the issue can not be reproduced with a 64-bit version of the module. Here is what you can do to get at least the stack trace of the crash.

These instructions require a lot of manual work, most of which can (and probably should, unless someone knows a better way) be automated. 


Get and build a 32-bit Chromium:
GYP_DEFINES='target_arch=ia32' gclient runhooks
make -j16 chrome

Enable the NaCl signal handler by editing src/nativeclient/src/trusted/service_runtime/sel_ldr_chrome.c:
  • Remove a NaClSignalHandlerFini call
  • Several lines below that, remove a NaCl*Signal*Assert* call

Enable stack dumping in the signal handler by editing nacl_signal_common.c:
print memory contents up from ctx.stack_ptr in 4-byte words; also print ctx.stack_ptr itself.

Rebuild Chromium. 

Unwind the stack manually by repeating the following steps, starting with the crash address:
  1. Find out which library (or IRT) does the address belong to. See below for details.
  2. Use objdump -d to find out the function and its assembly listing.
  3. Examine the preamble of the function to find out the offset of the next stack frame. Usually, all stack pointer movement is done very close to the beginning of the function.
  4. Examine the next stack frame to find the caller's address.

This is how you could find out the memory layout of the application:
  1. Add a delay to your application somewhere before the crash.
  2. With ps auxw --forest | grep nacl_boot find the most deeply nested nacl_* process. That's where your code runs.
  3. kill -STOP <PID>
  4. cat /proc/<PID>/maps
  5. Find which executable mapping contains the address you are interested in.
  6. Add 0x10000000 to the start address of that mapping. There you will find a non-executable mapping.
  7. Note the file name of the non-executable mapping. It probably says (deleted), but that's ok.
  8. cat /proc/<PID>/fd | grep <mapping file name> gives you the path to the file in question.
  9. readelf -a /proc/<PID>/fd/XX | grep SONAME with tell you what that is.
  10. objdump -d /proc/<PID>/fd/XX will show you its contents.

Comments