Обсуждение: Problem with the default registration of the JSON adapter
Hello list,
ticket #172 [1] suggests I have been too eager with JSON and Postgres
9.2. The ticket is from an user who just wants to read the json from the
database and forward it to the web clients, so a json -> python
conversion would be just a waste of time. Psycopg 2.5 instead registers
the json adapter on the PG 9.2 oids automatically.
If anybody else finds the same problem and wants to receive json as text
in 9.2 too, the interim solution with psycopg 2.5 and 2.5.1 is the
following snippet:
import psycopg2.extensions as ext
if hasattr(ext, 'JSON'):
ext.string_types.pop(ext.JSON.values[0], None)
ext.string_types.pop(ext.JSONARRAY.values[0], None)
Another solution would be of course to cast the result to text instead.
What I would do is to introduce in the next release an unregister_json()
function doing roughly (but less roughly) what this snippet does.
I've taken in consideration the idea of dropping the automatic
registration of the json adapter after giving a few months of warning,
but thinking about that more coolly I think it would be a bad reaction.
The feature was legitimately added on a major release (2.5) and if
somebody finds it broken he can still install the last 2.4 available
(pip install "psycopg2<2.5"). The right moment to think about a
different behaviour (e.g. require register_json() to be called) is the
next major release.
Comments? Better ideas?
[1] http://psycopg.lighthouseapp.com/projects/62710-psycopg/tickets/172
--
Daniele
On Jul 19, 2013, at 11:41 AM, Daniele Varrazzo wrote: > Another solution would be of course to cast the result to text instead. The good news is that a SELECT json_field::TEXT ... doesn't do any computation, so you get a text value containing the JSONback without conversion, efficiently. -- -- Christophe Pettus xof@thebuild.com
On 19/07/2013 20:41, Daniele Varrazzo wrote: > I've taken in consideration the idea of dropping the automatic > registration of the json adapter after giving a few months of warning, > but thinking about that more coolly I think it would be a bad reaction. > The feature was legitimately added on a major release (2.5) and if > somebody finds it broken he can still install the last 2.4 available > (pip install "psycopg2<2.5"). The right moment to think about a > different behaviour (e.g. require register_json() to be called) is the > next major release. IMO, dropping authomatic registration in the next minor release and having users explicitly calling register_json() is better. Having to explicitly register a type caster always was the psycopg's way while having to deregister it is counter-intuitive. I don't think there is a whole legion of programmers out there already depending on the json type-caster so it is much better to introduce the change now. federico
On Jul 22, 2013, at 2:04 AM, Federico Di Gregorio wrote: > IMO, dropping authomatic registration in the next minor release and > having users explicitly calling register_json() is better. You don't have to register TEXT; why would you have to register JSON? -- -- Christophe Pettus xof@thebuild.com
On 22/07/2013 17:02, Christophe Pettus wrote: > > On Jul 22, 2013, at 2:04 AM, Federico Di Gregorio wrote: >> IMO, dropping authomatic registration in the next minor release and >> having users explicitly calling register_json() is better. > > You don't have to register TEXT; why would you have to register JSON? Because with TEXT and DECIMAL and FLOAT and all the base types that have d direct 1-on-1 mapping to Python you don't have a choice. In more than 10 years of psycopg nobody ever asked «can I please get the floats as text instead of Python float objects?». JSON is different because the representation you get is exactly the representation you may want to send to the the client (probably over HTTP) and the conversion text->json objects->text uses quite a bit of cpu/memory. So it does make sense to make it either opt-in or opt-out. Given that psycopg never had opt-out adapters/typecasters my vote is to make it opt-in. federico
On Jul 22, 2013, at 8:20 AM, Federico Di Gregorio wrote: > JSON is different because the representation you get is exactly the > representation you may want to send to the the client (probably over > HTTP) and the conversion text->json objects->text uses quite a bit of > cpu/memory. If you want the text version, you can ask for the text version; I don't think it's worth breaking an existing API over. We already have a precedent here in hstore. -- -- Christophe Pettus xof@thebuild.com
On 22/07/2013 17:55, Christophe Pettus wrote: > On Jul 22, 2013, at 8:20 AM, Federico Di Gregorio wrote: > >>> JSON is different because the representation you get is exactly >>> the representation you may want to send to the the client >>> (probably over HTTP) and the conversion text->json objects->text >>> uses quite a bit of cpu/memory. > If you want the text version, you can ask for the text version; I > don't think it's worth breaking an existing API over. We already > have a precedent here in hstore. Apart from the fact that hstore in text format is pretty much unusable: there was no other type that has multiple useful representations until PostgreSQL introduced json. Looking forward, I'd say that if a type has only one useful representation, that one should be used; but if it has multiple useful representations the *simplest* one should be the default. federico
On Jul 22, 2013, at 9:06 AM, Federico Di Gregorio wrote: > Looking forward, I'd say that if a type has only one useful > representation, that one should be used; but if it has multiple useful > representations the *simplest* one should be the default. I have to say that I find the current situation pleasing: psycopg2 does the most natural thing with the type (converts itto a data structure, which is after all what json is encoding) if you ask for the base type. If you want the text version,just cast it to the text version in the query, and that's what you get, parallel with every other type. -- -- Christophe Pettus xof@thebuild.com
On Mon, Jul 22, 2013 at 5:13 PM, Christophe Pettus <xof@thebuild.com> wrote:
>
> On Jul 22, 2013, at 9:06 AM, Federico Di Gregorio wrote:
>
>> Looking forward, I'd say that if a type has only one useful
>> representation, that one should be used; but if it has multiple useful
>> representations the *simplest* one should be the default.
>
> I have to say that I find the current situation pleasing: psycopg2 does the most natural thing with the type
(convertsit to a data structure, which is after all what json is encoding) if you ask for the base type. If you want
thetext version, just cast it to the text version in the query, and that's what you get, parallel with every other
type.
I've been thinking about this issue for a while, and I think too I
like what we do better than the alternatives. Without assumption of
what people does or doesn't do with their data, JSON is structured
data and getting it in structured way in Python is a sensible default.
>>> cur.execute("""select '{"data": "here", "more": [1,2,3]}'::json""")
>>> cur.fetchone()[0]
{u'data': u'here', u'more': [1, 2, 3]}
It's just as natural as this: json is structure and we get a
reasonable structure by default, with plenty of customization
possibilities if the defaults are not right for the task. If somebody
doesn't want the structured data but just a string to pass through,
querying json as text is a perfect workaround, requiring no knowledge
of psycopg internals. The best workaround in psycopg is probably not
to unregister the adapter as suggested in the first thread message,
but to register the text (or unicode) adapter on the json oids, which
can be done at smaller scope than globally and doesn't leak the
string_types implementation:
ext.register_type(ext.new_type(ext.JSON.values, 'JSON2STR',
psycopg2.STRING))
I would mention this in the docs but leave the json handling as it is.
For future data types we will discuss whether handling them by default
would be wise or not.
-- Daniele