Re: PostgreSQL insights: does it use DMA?

Thank you a lot, Craig, that's really insightful and exhaustively
complete explanation I've expected!

2011/9/10 Craig Ringer <ringerc@ringerc.id.au>:
> On 10/09/2011 1:55 AM, Antonio Rodriges wrote:
>>
>> Hello,
>>
>> Does anyone know whether PostgreSQL uses DMA (Direct Memory Access) in
>> certain cases to improve networking IO performance?
>
> From what you described in your message it sounds like what you really want
> is not DMA, but use of something like the sendfile() system call
> (http://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2
> <http://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2>), where the
> kernel is told to send data over the network with no further interaction
> with the application.
>
> PostgreSQL does not, and can not, do this for regular query results. The
> sample query you posted most certainly DOES require CPU processing during
> its execution: Any index being used must be traversed, which requires logic.
> Date comparisons must be performed, possibly including timezone conversions,
> and non-matching rows must be discarded. If a sequential scan is being done,
> a bitmap showing holes in the file may be consulted to decide where to scan
> and where to skip, and checks of row versions to determine visibility must
> be done. Once the data has been selected, it must be formatted into proper
> PostgreSQL v3 network protocol messages, which involves function calls to
> data type output functions among many other things. Only then does the data
> get written to a network socket.
>
> Needless to say, it's not like Pg is just picking a file to open and doing a
> loop where it reads from the file and writes to a socket.
>
> That said, PostgreSQL benefits from the DMA the operating system does when
> handling system calls. For example, if your network interface supports DMA
> buffer access then PostgreSQL will benefit from that. Similarly, Pg benefits
> from the kernel-to-hardware DMA for disk I/O etc. Beyond that I doubt
> there's much. PostgreSQL's use of shared_buffers for read data means data
> will get copied on read, and writes go through the OS's buffer cache, so
> there's unlikely to be direct DMA between PostgreSQL buffers and the disk
> hardware for example.
>
>
> Theoretically PostgreSQL could use something like sendfile() for sending
> large object (blob) data and bytea data, but to do so it'd have to change
> how that data is stored. Currently blob data is stored in (often compressed)
> 8kb chunks in the pg_largeobject table. This data has to be assembled and
> possibly decompressed to be transmitted. Similar things apply for bytea
> fields of tables. In addition, the data is usually sent over the text
> protocol, which means it has to be encoded to hex or (for older versions)
> octal escapes. That encoding is incompatible with the use of an API like
> sendfile() .
>
> So, in practice, PostgreSQL can _not_ use the kinds of direct kernel-level
> disk-to-network sending you seem to be referring to. That sort of thing is
> mostly designed for file servers, web servers, etc where at some point in
> the process they end up dumping a disk file down a network socket without
> transforming the data. Even they don't benefit from it as much these days
> because of the wider use of encryption and compression.
>
> --
> Craig Ringer
>



--
Kind regards,
Antonio Rodriges