Re: Is there a cache consistent interface to tables ?

Thanks for the replies.  

@Craig

Realtime ? well not really. I guess it's your definition of realtime. I usually think of micro to nano seconds as real-time. If I were still designing chips, I'd be calling picosecs real-time these days.  

"How sure are you that it's not viable for SQL queries?"

I'm topping off at 20K/s inserts due to locking and scheduling issues. There are peaks reaching 50K inserts, but not sustained. I also placed the login on a separate disk to prevent collisions. I have backing storage with 50us latency and 400mb/s bandwidth, so backing storage is not the issue.

"And, if not, what makes you think that a lower level interface will help you? "

Well, I was thinking sql parsing and planning is an unnecessary step that can be removed from the pipeline. Instead of parsing the data stream to another format and creating a query, I'll just have the stream parser(s) generate the table structure and write directly to the tables.

 "Has profiling and tracing/timing shown that significant time/delays are arising from layers you can bypass in a sensible way?"

"Sensible" is why I'm posting the question here.  I'm not familiar enough with the code and processing pipelines to understand subtleties effecting high volume insert performance.  There is a lot of "stuff" going on, much of it event driven.  My first impulse is a pipeline scheduler taking advantage of processor affinity.  Its an ugly, brute force approach, but it does work.

The final design will likely require some type of shared block storage.  I've always liked the Solaris ZFS LVM layer, although is not distributed.  Fire and forget is not an option here. I was hoping to leverage existing postgres facilities for that heavy lifting. That's why I had originally looked at the wal interface. 

I'm also looking at approaches from a project called "Bottled-Water" https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/  

One more fact I forgot to add.. The insert load into the database is about 2kb/record or about 200MB/s.