Getting the Process ID (PID) of the Native Client Program from Chrome

Chrome -> Developer->Task Manager, look for the link to “Stats for nerds” which will open a more complete task manager, including PIDs. The PID can be used to attach GDB to running Native Client processes.  Or click on a new tab and type "about:memory" into the URL address bar.

Attaching to the Running Native Client Module

Use the technique above to obtain the PID of the NaCl process. Launch GDB and from GDB's command line “attach nnn” where nnn is the PID. Or launch GDB from bash with "gdb --pid=nnn"  Once GDB has finished attaching to the process and loading applicable symbols, hit 'c' followed by enter to continue execution.  When debugging applications that crash at startup, it is recommended that if applicable, debug builds require an extra event to start execution, such as responding to the first mouse click in NPP_HandleEvent(), or at least pause for several seconds. This gives the developer a chance to look up the PID and attach to the NaCl process before the crash occurs and Chrome's task manager kills the process.

Native Client, when targeting x86, has two flavors: x86-32 and x86-64. Each uses a different method to sandbox the untrusted code, and each has unique ramifications when debugging applications.

x86-32:

When nacl-gcc targets 32 bit x86, sandboxing is done via segment registers. This is enforced by validation at load time, which prevents untrusted code from issuing instructions which can manipulate the segment registers. This approach to sandboxing means there are two address ranges inside the NaCl process: first, there is the global address space of the process, and second there is a smaller address space used by the untrusted code. When debugging x86-32 Native Client code in GDB, you must manually juggle between these two address spaces. When GDB breaks on an instruction in x86-32 Native Client application, a thread will either be in trusted code (likely the Native Client Service Runtime) or it will be in untrusted code.

If the thread is in trusted code, GDB will function normally – you should see a callstack, and be able to examine trusted memory and set breakpoints on trusted functions.

If the thread is in untrusted Native Client code, the situation will be different. First, GDB probably won't provide a recognizable call tree. To find out where the break occurred, take the address in the EIP register and look it up in a disassembly of the nexe. This disassembly can be generated with “nacl-objdump -d program.nexe” where program.nexe is the name of the NaCl executable.

Reading the memory contents of the untrusted stack is more challenging. The address reported in the stack register ESP is in the local space of the untrusted module, but GDB when halted will be in the address space of the entire process (both trusted Service Runtime and untrusted nexe.) Before examining the memory contents of untrusted memory – including the stack -- you will need to manually offset the local pointer into the global address space of the process. This is necessary because this global memory space is the one GDB is operating in. Once the stack has been located in the global memory space, the memory can be examined for return addresses. These return addresses obtained by dumping the contents of the stack can be looked up in the disassembly from above. Using this tedious process it is possible to crawl the stack and determine who called who leading up to the break.

How does one obtain the offset to map local untrusted pointers into the global process address space? With the environment variable NACLVERBOSITY=2, at program launch time, look for the string “NaClLdtAllocatePageSelector” The second to last large hex value is the offset to use. Chances are this value will change between each run of your application, so don't reuse values from previous sessions.

[14737,4120116080:17:41:24.138701] NaClLdtAllocatePageSelector(code, 1, 0xb5928000, 0x140
[14737,4120116080:17:41:24.138895] got 0x7
[14737,4120116080:17:41:24.139016] NaClLdtAllocatePageSelector(data, 1, 0xb5928000, 0x3ffff
[14737,4120116080:17:41:24.139132] got 0xf
[14723,4130916176:17:41:24.139234] NaClImcRecvTypedMsg: returning 25

An easier way to get call traces can be done by recompiling your entire code base using nacl-gcc's -finstrument-functions option. This flag is not unique to nacl-gcc, -finstrument-functions is also available to normal gcc. Using –finstrument-functions will instruct nacl-gcc to implicitly add calls at every function entrance and exit to two user supplied functions:

void __cyg_profile_func_enter (void *func, void *caller) __attribute__((no_instrument_function));

void __cyg_profile_func_exit (void *func, void *caller) __attribute__((no_instrument_function));

By providing these functions, it is possible for an application to log and/or record the address of every function recompiled with –finstrument-functions. It is recommended to filter on a thread ID to distinguish multiple call trees interleaved in the output. Use the address of a global TLS variable as a cheap way to mimic thread ID. Use the attribute no_instrument_function to selectively disable functions (required for the __cyg_profile_func_*() themselves to avoid infinite recursion.)

A very simple implementation would be:

__thread int tid;

void __cyg_profile_func_enter(void *this_fn, void *call_site) {
  printf("Thread %p entering %p\n", &tid, this_fn);
}

void __cyg_profile_func_exit(void *this_fn, void *call_site) {
  printf("Thread %p exiting %p\n", &tid, this_fn);
}

Placing breakpoints: It is possible to place breakpoints in untrusted code by manually using the code address offset to translate between the two address spaces. You should explicitly use the manually computed hex address when setting breakpoints in GDB; ie. “break *address”  Due to the sandboxing, normal unmodified GDB will not correctly resolve “break file:line” or “break function” for untrusted code.

x86-64:

When applications are built using nacl64-gcc or nacl-gcc -m64, a different method of sandboxing is used. Instead of segment registers, all read & write operations are sandboxed by masking the memory address to be within a restricted address space by zero extending a 32 bit pointer value to 64 bits, and then adding an offset stored in %r15.  Register %r15 is reserved and should be treated as read-only in all untrusted Native Client x86-64 code.

The untrusted data segment is surrounded on both sides by very large guard pages that will generate exceptions if read or written. By masking all pointers and enforcing this via validation at program load time, the untrusted code will be unable to read or write outside its memory space without triggering an exception. Trusted data is located 'out of reach' from the untrusted code.

By comparison, using GDB to debug x86-64 is somewhat easier than x86-32. Instead of relying on implicit segment registers, the conversion to the address space of the process is explict, and usually is of the form:

   mov %edi, %edi

   add (%r15, %rdi, 1), %eax

Where %r15 is the read-only global offset.  The address in memory is %r15 + %rdi, where %r15 is 64 bits and %rdi is 32 bits, zero extended to 64 bits by the preceeding mov %edi, %edi instruction.

If examining pointer values stored in memory via GDB, you must manually add the offset in %r15 to the pointer before examining the memory it points to. The same needs to be done when setting breakpoints: manually compute the breakpoint address and use the form “break *address” in GDB.

The -finstrument-functions method of capturing call traces explained in the x86-32 section should work the same in x86-64.