Re: Generic Monitoring Framework Proposal

On Jun 19, 2006, at 4:40 PM, Tom Lane wrote:

> Robert Lor <Robert.Lor@Sun.COM> writes:
>> The main goal for this Generic Monitoring Framework is to provide a
>> common interface for adding instrumentation points or probes to
>> Postgres so its behavior can be easily observed by developers and
>> administrators even in production systems.
>
> What is the overhead of a "probe" when you're not using it?  The  
> answer
> had better not include the phrase "kernel call", or this is  
> unlikely to
> pass muster...
>
>> For DTrace, probes can be enabled using a D script. When the  
>> probes are not enabled, there is absolutely no performance hit  
>> whatsoever.
>
> If you believe that, I have a bridge in Brooklyn you might be  
> interested
> in.

Heh.  Syscall probes and FBT probes in Dtrace have zero overhead.   
User-space probes do have overhead, but it is only a few instructions  
(two I think).  Besically, the probe points are replaced by illegal  
instructions and the kernel infrastructure for Dtrace will fasttrap  
the ops and then act.  So, it is tiny tiny overhead.  Little enough  
that it isn't unreasonable to instrument things like s_lock which are  
tiny.

> What are the criteria going to be for where to put probe calls?  If it
> has to be hard-wired into the source code, I foresee a lot of  
> contention
> about which probes are worth their overhead, because we'll need
> one-size-fits-all answers.
>
>> arg1..arg5  = Any args to pass to the probe such as txn id, lock  
>> id, etc    
> Where is the data type of a probe argument defined?

I assume it would depend on the probe implementation.  In Dtrace they  
are implemented in .d files that will post-instrument the object  
before final linkage.  Dtrace's whole purpose is to be low overhead  
and it really does it in a fantastic way.

As an example, you can take an uninstrumented binary and add dynamic  
instrumentation to the entry, exit and every instruction op-code over  
every single routine in the process.  And clearly, as the binary is  
uninstrumented, the overhead is indeed zero when the probes are not  
enabled.

The reason that Robert proposes user-space probes (I assume) is that  
tracing C functions can be too granular and not conveniently expose  
the "right" information to make tracing useful.

// Theo Schlossnagle
// CTO -- http://www.omniti.com/~jesus/
// OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
// Ecelerity: Run with it.