Обсуждение: [JDBC] Reading and writing off-heap data
Hi all
After original discussion back in March[1] we've finally gotten around to scheduling this work. I've submitted a pull-request to support writing data from off-heap locations using the discussed interface here [2].
We'd like to do the flip of this too, though: read incoming data into a caller-controlled buffer in some way. The interface would look something like this:
// provided by driver
interface ByteStreamReader<T implements Closeable> {
T readByteStream(int length, InputStream stream) throws IOException;
}
// user code
class MyCustomByteStreamReader implements ByteStreamReader<MyBufferHandle> {
...
}
preparedStatement.registerByteStreamReader(new MyCustomByteStreamReader());
...
MyBufferHandle b = (MyBufferHandle) resultSet.getObject(2);
There are a couple of issues:
1. Internally, the driver passes incoming tuples around as byte[][] instances, which doesn't leave much ability to do something else with the incoming data. I've submitted a PR [3] that introduces a Tuple class as a wrapper to pass around, which then allows us to do more interesting things with the data.
2. How should we register the reader? We have to do it ahead of execution of the query, as the driver has already read at least some data rows by the time we return the ResultSet.
Some potential options are:
a) Register against the statement and use it for all columns of binary type. This would look like the above.
b) Register against the statement but for individual columns:
statement.registerByteStreamReader(2, new MyCustomByteStreamReader());
statement.registerByteStreamReader("foo", new MyCustomByteStreamReader());
c) Mark incoming columns in some other way that the driver can recognise. This requires getting creative. An example would be to create a domain over the bytea type and then register the reader for that type. Then queries would have to have results cast to that type.
d) Register a higher-level object like the connection or driver and use it for all columns of binary type.
Option a) is the simplest in that it neither requires us to keep track of readers for individual columns nor requires users having to mess with their database schema. It's definitely enough for my use-case, but I'm interested in hearing other opinions on whether that's flexible enough.
Is there general support for the feature? I'm again happy to code up a PR and have time allocated to do that fairly soon if there's likelihood of it being merged.
Thanks
Tom
On 20 September 2017 at 22:43, Tom Dunstan <pgsql@tomd.cc> wrote:
Hi allAfter original discussion back in March[1] we've finally gotten around to scheduling this work. I've submitted a pull-request to support writing data from off-heap locations using the discussed interface here [2].We'd like to do the flip of this too, though: read incoming data into a caller-controlled buffer in some way. The interface would look something like this:// provided by driverinterface ByteStreamReader<T implements Closeable> {T readByteStream(int length, InputStream stream) throws IOException;}// user codeclass MyCustomByteStreamReader implements ByteStreamReader<MyBufferHandle> { ...}preparedStatement.registerByteStreamReader(new MyCustomByteStreamReader()); ...MyBufferHandle b = (MyBufferHandle) resultSet.getObject(2); There are a couple of issues:1. Internally, the driver passes incoming tuples around as byte[][] instances, which doesn't leave much ability to do something else with the incoming data. I've submitted a PR [3] that introduces a Tuple class as a wrapper to pass around, which then allows us to do more interesting things with the data.2. How should we register the reader? We have to do it ahead of execution of the query, as the driver has already read at least some data rows by the time we return the ResultSet.Some potential options are:a) Register against the statement and use it for all columns of binary type. This would look like the above.b) Register against the statement but for individual columns:statement.registerByteStreamReader(2, new MyCustomByteStreamReader()); statement.registerByteStreamReader("foo" , new MyCustomByteStreamReader()); c) Mark incoming columns in some other way that the driver can recognise. This requires getting creative. An example would be to create a domain over the bytea type and then register the reader for that type. Then queries would have to have results cast to that type.
def don't like C
d) Register a higher-level object like the connection or driver and use it for all columns of binary type.Option a) is the simplest in that it neither requires us to keep track of readers for individual columns nor requires users having to mess with their database schema. It's definitely enough for my use-case, but I'm interested in hearing other opinions on whether that's flexible enough.
I think I prefer this as it is simplest
Is there general support for the feature? I'm again happy to code up a PR and have time allocated to do that fairly soon if there's likelihood of it being merged.
I would think so , however as noted on the PR I'd like to see some timing of the Tuple code. I don't expect it would be heinous but one never knows.
>I've submitted a PR [3] that introduces a Tuple class as a wrapper to pass around, which then allows us to do more interesting things with the data.
I wonder how would you wire things since "new Tuple" is created before "new ResultSet", so it would be not that trivial to call user code (enable user to read bytes off the wire) from there.
On the other hand, if you postpone processing the tuple till resultSet is fully initialized, then Tuple would not differ from byte[][] much.
Vladimir
On 22 September 2017 at 01:35, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>I've submitted a PR [3] that introduces a Tuple class as a wrapper to pass around, which then allows us to do more interesting things with the data.I wonder how would you wire things since "new Tuple" is created before "new ResultSet", so it would be not that trivial to call user code (enable user to read bytes off the wire) from there.
Code affecting where incoming data goes would have to be registered with the statement object or similar before execution.
Cheers
Tom
(Sorry, re-sending from address that is subscribed to the list)
On 22 September 2017 at 01:01, Dave Cramer <pg@fastcrypt.com> wrote:
Is there general support for the feature? I'm again happy to code up a PR and have time allocated to do that fairly soon if there's likelihood of it being merged.I would think so , however as noted on the PR I'd like to see some timing of the Tuple code. I don't expect it would be heinous but one never knows.
Currently it's a very thin wrapper around the array and construction in most places is still just filling that array and the constructing the wrapper once that's done, so I do expect almost zero change. I'll try to quantify it though. Do we have any existing test for these sorts of things? If not I'll just generate a bunch of data and pull it through - seems easy enough.
Thanks
Tom
(Sorry, re-sending from address that is subscribed to the list)
On 22 September 2017 at 01:35, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>I've submitted a PR [3] that introduces a Tuple class as a wrapper to pass around, which then allows us to do more interesting things with the data.I wonder how would you wire things since "new Tuple" is created before "new ResultSet", so it would be not that trivial to call user code (enable user to read bytes off the wire) from there.
Code affecting where incoming data goes would have to be registered with the statement object or similar before execution.
Cheers
Tom
Tom>I'll try to quantify it though. Do we have any existing test for these sorts of things? If not I'll just generate a bunch of data and pull it through - seems easy enough.
>Code affecting where incoming data goes would have to be registered with the statement object or similar before execution.
That is of no question. The question is "do you intend to call user-provided code right from org.postgresql.core.v3.QueryExecutorImpl#processResults, case 'D': // Data Transfer (ongoing Execute response)" ?
That is of no question. The question is "do you intend to call user-provided code right from org.postgresql.core.v3.QueryExecutorImpl#processResults, case 'D': // Data Transfer (ongoing Execute response)" ?
Vladimir
Hi Vladimir!
On 25 September 2017 at 17:50, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
>
> That is of no question. The question is "do you intend to call user-provided code right from org.postgresql.core.v3.QueryExecutorImpl#processResults, case 'D': // Data Transfer (ongoing Execute response)" ?
We'd have to call it inside PGStream#receiveTupleV3 I think, as that's where the byte arrays for each field are being allocated. Or potentially it could be moved into the tuple class itself.
>
>
> That is of no question. The question is "do you intend to call user-provided code right from org.postgresql.core.v3.QueryExecutorImpl#processResults, case 'D': // Data Transfer (ongoing Execute response)" ?
We'd have to call it inside PGStream#receiveTupleV3 I think, as that's where the byte arrays for each field are being allocated. Or potentially it could be moved into the tuple class itself.
As with PR 953 https://github.com/pgjdbc/pgjdbc/pull/953 we'd call the user-provided code on a guarded InputStream so that we know they can't mess the stream up.
Cheers
Tom
Sorry, gmail keeps defaulting to the wrong email and I catch it afterwards. Posted again to the list properly.
On 25 September 2017 at 20:17, Tom Dunstan <tom@tomd.cc> wrote:
Hi Vladimir!On 25 September 2017 at 17:50, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
>
> That is of no question. The question is "do you intend to call user-provided code right from org.postgresql.core.v3.QueryExecutorImpl# processResults, case 'D': // Data Transfer (ongoing Execute response)" ?
We'd have to call it inside PGStream#receiveTupleV3 I think, as that's where the byte arrays for each field are being allocated. Or potentially it could be moved into the tuple class itself.As with PR 953 https://github.com/pgjdbc/pgjdbc/pull/953 we'd call the user-provided code on a guarded InputStream so that we know they can't mess the stream up. CheersTom