This is the documentation for the instrumentation available in GNU
Mach, called db_probes. 

Probes are implemented using the existing ddb infrastructure, most
notably breakpoints and symbol lookup support. A probe is a wrapper
around a breakpoint which has a special flag noting that it is
external, BKPT_EXTERNAL. This is used by ddb so that these breakpoints
can be handled specially, most importantly this is used to dispatch
the probe in db_trap so as not to invoke the commandline interface to
ddb. The other important functionality is to deny the commandline
interface the same abilities to modify that breakpoint; for example
deleting the breakpoint isn't allowed. If this were to happen the
probe could end up floating indefinitely, neither executing nor anyone
paying attention to delete it. Worse, a thread may be waiting for it
and it will never wake.

db_probes have been intentionally designed to rely almost only on ddb
components. This is a desirable feature that should probably be
maintained if possible, it allows porting db_probes to other kernels
that support ddb.

Probes have 4 callbacks of which 2 are used: pre and post. The pre
probe is executed before the instruction being debugged, analogously
for post. Setting a probe requires a valid address, which one can get
from looking up a symbol, handlers (NULL handlers will simply be
ignored and are valid), add_probe, remove_probe along with arm_probe
and disarm_probe constitute the functional part of the in-kernel
db_probes API. add_probe and remove_probe provide a way to allocate
probes without relying on memory management mechanisms like zalloc.

db_probes is implemented in several parts of the kernel:
ddb/db_probes.c, ddb/db_probes.h, i386/i386/db_probes.h,
ddb/db_trap.c; with minor changes to ddb/db_break.c.
kern/startup.c contains the call that initializes db_probes.

Care has been taken to avoid dynamic memory allocation and reliance on
most external mechanisms inside db_probes. As these can be invoked
anywhere ddb can, any kernel mechanism can be in an unsafe state; only
reentrant functions should be called. 

There are 3 user-facing functions in db_probes:

routine mach_arm_probe (
		host		: host_priv_t;
		addr            : vm_address_t;
		name		: kernel_version_t;
		mem_addr	: vm_address_t;
		mem_size	: natural_t;
	out	probe		: vm_address_t;
	out	status		: natural_t);

routine mach_wait_probe (
		host		: host_priv_t;
		probe		: vm_address_t;
	out 	memory		: mach_probe_mem_t, CountInOut;
	out	status		: natural_t);

routine mach_disarm_probe (
		host		: host_priv_t;
		probe		: vm_address_t);

These functions are designed to be called by the user, through the RPC
mechanism, and assume that the kernel is in a safe state. They for
example do memory allocation using kalloc, it is unsafe to call these
from ddb. These are declared in mach_host. This is because mach_debug
does not have RPC stubs generated for it in glibc and seems generally
an entirely unused portion of mach.

- mach_arm_probe:
 Set a probe and arm it. It will begin executing immediately and enqueuing executions.
 - addr 
   Where to set the probe, optional if name is specified
 - name 
   Where to set the probe, will look up the symbol, optional if
   addr is specified. Name will overrule any addr setting. The 
   kernel_version_t type was reused since it's a very large string.
 - mem_addr
   Where to start copying memory that will be returned by the probe
 - mem_size 
   How much to copy. (mem_addr, mem_size) of (0,0) will behave rationally.
 - probe 
   An opaque handle to the probe
 - status
   Currently unused. It existed in previous incarnations and is kept
   since it will be required for more detailed memory information in
   the future.
- mach_wait_probe:
 Wait until a probe executes, pop a probe execution off of the queue
 and return it.
 - memory
   Where to put the memory that this probe will return, and the size
   of the buffer in which you want to put it.
 - status
   Currently unused, same as mach_arm_probe
- mach_disarm_probe:
 Remove a probe, delete it, and delete the breakpoint.

The userland probe capabilities are implemented using the regular
in-kernel probes and have special in-kernel probe handlers. These
handlers manage a queue inside the probe for a particular execution of
the probe. This is a fixed-size queue, each entry records at least why
it exists (pre or post probe) and a memory buffer in which to store
data. The handlers copy the requested memory region into this
buffer. One important note about these queues is that one is never
guaranteed to receive a post probe is a pre probe has been
received. 

While kernel probes support multiple dispatch, multiple probes can
share one breakpoint and so execute on the same memory location;
userland probes intentionally block in order to provide a simple
Hurd-ish interface. You also cannot have two threads waiting on the
same probe, this feature actually existed but I removed it. It makes
no real sense, added a lot of code and buys you no extra power.

What's left to do:

 - Having the time when a probe executed is useful. The problem with
   this is determining if this is really an atomic operation that
   deserves special support. Maybe there's something more general
   here?

 - More robus memory management. How do we want to ask for
   memory. DWARF is an option I've thought of, it allows us to write
   simple code to refer to all types of memory. The problem is writing
   these programs isn't pleasant. 

 - char* name ought to be removed as an argument and userland should
   be forced to look up all the addresses on its own. This makes using
   the API more cumbersome now; probes heavier, and more work on
   getting a probe up and running. The tradeoff is that when we get
   userland breakpoints in ddb this will segway nicely into being able
   to place breakpoints there as well.

 - It's currently possible to mess up the kernel by setting probes and
   then not removing them. You'll eventually end up running out of
   space for them. Perhaps the probes should be attached to tasks
   somehow. I'm unconvinced this actually makes sense.

 - Bindings for your favourite language will make this quite a bit
   more fun.
  
 - More error checking and security