Обсуждение: Range Type Support
Hi, I am using range types in an application I am writing and am planning on writing some mapping to have support with built-inPython ranges, and some simple extensions for dealing with date ranges. I am going to write this on my own regardless,but is anyone working on this for psycopg2? If not, I would be willing to contribute the code I am working on. I was planning on writing the adapter in Python as I am unfamiliar with writing C code for Python. Are there any thoughtswhere such an adapter would be placed, e.g. in lib.extras Thanks, Jonathan
On Tue, Sep 18, 2012 at 5:57 PM, Jonathan S. Katz <jonathan.katz@excoventures.com> wrote: > Hi, > > I am using range types in an application I am writing and am planning on writing some mapping to have support with built-inPython ranges, and some simple extensions for dealing with date ranges. I am going to write this on my own regardless,but is anyone working on this for psycopg2? If not, I would be willing to contribute the code I am working on. What is a "built-in Python range"? You mean some Python object to be written with the same semantics of a postgres range? (subtype, handling of boundaries, empty singleton). > I was planning on writing the adapter in Python as I am unfamiliar with writing C code for Python. Are there any thoughtswhere such an adapter would be placed, e.g. in lib.extras lib.extras is the current kitchen sink and contains all the extra data types. I'm toying with the idea of reorganizing the package submodules but this will come later. -- Daniele
On Sep 18, 2012, at 12:07 PM, Daniele Varrazzo wrote: > On Tue, Sep 18, 2012 at 5:57 PM, Jonathan S. Katz > <jonathan.katz@excoventures.com> wrote: >> Hi, >> >> I am using range types in an application I am writing and am planning on writing some mapping to have support with built-inPython ranges, and some simple extensions for dealing with date ranges. I am going to write this on my own regardless,but is anyone working on this for psycopg2? If not, I would be willing to contribute the code I am working on. > > What is a "built-in Python range"? You mean some Python object to be > written with the same semantics of a postgres range? (subtype, > handling of boundaries, empty singleton). Good point, my mistake in terminology. I was thinking of the "range" function for integers, but that outputs an array ofnumbers. This works well for small data sets but if your range type in the DB is something like [1, 10000000] we couldhave a bad day. I would like to propose having a special class to handle the ranges with the same semantics of the Postgres ranges. I wouldlike to have a base class that would handle the range mechanics that could then be inherited, thus mimicking how rangetypes can be extended. Additionally, this would allow us to support continuous ranges (e.g. time, numeric/float/decimal). My concern is I don't want to deviate too far from the standard way of doing things in Python (lists as ranges) for the sakeof other libs (i.e. ORMs) interfacing with psycopg2, so I can make strong arguments on both sides. >> I was planning on writing the adapter in Python as I am unfamiliar with writing C code for Python. Are there any thoughtswhere such an adapter would be placed, e.g. in lib.extras > > lib.extras is the current kitchen sink and contains all the extra data > types. I'm toying with the idea of reorganizing the package submodules > but this will come later. More or less what I figured which is why I suggested putting initial code in there, particularly if we are defining our ownrange class. Jonathan
On 18/09/2012 19:07, Daniele Varrazzo wrote: > On Tue, Sep 18, 2012 at 5:57 PM, Jonathan S. Katz > <jonathan.katz@excoventures.com> wrote: >>> Hi, >>> >>> I am using range types in an application I am writing and am >>> planning on writing some mapping to have support with built-in >>> Python ranges, and some simple extensions for dealing with date >>> ranges. I am going to write this on my own regardless, but is >>> anyone working on this for psycopg2? If not, I would be willing >>> to contribute the code I am working on. > What is a "built-in Python range"? You mean some Python object to be > written with the same semantics of a postgres range? (subtype, > handling of boundaries, empty singleton). > > We'll probably need a range Python type too here. federico -- Federico Di Gregorio federico.digregorio@dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it 99.99999999999999999999% still isn't 100% but sometimes suffice. -- Me
On 18/09/2012 18:57, Jonathan S. Katz wrote: > I am using range types in an application I am writing and am planning > on writing some mapping to have support with built-in Python ranges, > and some simple extensions for dealing with date ranges. I am going > to write this on my own regardless, but is anyone working on this for > psycopg2? If not, I would be willing to contribute the code I am > working on. > > I was planning on writing the adapter in Python as I am unfamiliar > with writing C code for Python. Are there any thoughts where such an > adapter would be placed, e.g. in lib.extras It would be nice to place it in lib.extras or even in lib.extensions. I don't have the time to write that myself but I can surely help you by checking your code and applying your patches. federico -- Federico Di Gregorio federico.digregorio@dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it But not all bugs are an interesting challenge. Some are just a total waste of my time, which usually is much more valuable than the time of the submitter. -- Md
On Tue, Sep 18, 2012 at 6:20 PM, Jonathan S. Katz <jonathan.katz@excoventures.com> wrote: > On Sep 18, 2012, at 12:07 PM, Daniele Varrazzo wrote: >> On Tue, Sep 18, 2012 at 5:57 PM, Jonathan S. Katz >> <jonathan.katz@excoventures.com> wrote: >>> Hi, >>> >>> I am using range types in an application I am writing and am planning on writing some mapping to have support with built-inPython ranges, and some simple extensions for dealing with date ranges. I am going to write this on my own regardless,but is anyone working on this for psycopg2? If not, I would be willing to contribute the code I am working on. >> >> What is a "built-in Python range"? You mean some Python object to be >> written with the same semantics of a postgres range? (subtype, >> handling of boundaries, empty singleton). > > Good point, my mistake in terminology. I was thinking of the "range" function for integers, but that outputs an arrayof numbers. This works well for small data sets but if your range type in the DB is something like [1, 10000000] wecould have a bad day. You are being misled by the common "range" term. In Python, a range is an object primarily made to be iterated, in Postgres is primarily for membership/intersection test. A python range (in python 2 xrange() would do, in python 3 there's really a range type) is only equivalent to a postgres int range with left bound included. OTOH there is no step in postgres range. They are definitely two different beasts. > I would like to propose having a special class to handle the ranges with the same semantics of the Postgres ranges. Iwould like to have a base class that would handle the range mechanics that could then be inherited, thus mimicking how rangetypes can be extended. Additionally, this would allow us to support continuous ranges (e.g. time, numeric/float/decimal). Yes, that would be the thing to adapt to. I don't know how much it would be useful in Python but I see no more useful thing to adapt to. > My concern is I don't want to deviate too far from the standard way of doing things in Python (lists as ranges) for thesake of other libs (i.e. ORMs) interfacing with psycopg2, so I can make strong arguments on both sides. I don't see any standard python object to adapt to without leaving half of the model away. Ah, postgres ranges can also be open left/right/both. So I'd have: - Basic class with all attributes and implementation, and subclasses specifying a type as you mentioned. - The basic class should have the following attributes, which are a direct mapping to the postgres ones and are required to inspect the object: - lower/upper attributes - lower_inc/upper_inc attributes - lower_inf/upper_inf attributes - isempty attibute - Possibly it should support an "in" operator and an & operator. But then, should we mimic all the operators? <http://www.postgresql.org/docs/9.2/static/functions-range.html> Probably not. - Parsing interval should resort to the base type parser, so when constructing a concrete range type a typecaster should be probably specified. - If a new range type is created in postgres, it would be nice to have something like register_range() to introspect pg_range and use the right typecaster (rngsubtype) for parsing. - test and docs Having the range iterable seems tempting but the Postgres model doesn't have it (even for discrete ranges, it mandates a canonicalization function, not a step delta) so the two types wouldn't map automatically (not possible to introspect the catalog to get such a step) What do you think? -- Daniele
On Sep 18, 2012, at 1:08 PM, Daniele Varrazzo wrote: >> Good point, my mistake in terminology. I was thinking of the "range" function for integers, but that outputs an arrayof numbers. This works well for small data sets but if your range type in the DB is something like [1, 10000000] wecould have a bad day. > > You are being misled by the common "range" term. In Python, a range is > an object primarily made to be iterated, in Postgres is primarily for > membership/intersection test. A python range (in python 2 xrange() > would do, in python 3 there's really a range type) is only equivalent > to a postgres int range with left bound included. OTOH there is no > step in postgres range. They are definitely two different beasts. Thanks for the clarification! >> I would like to propose having a special class to handle the ranges with the same semantics of the Postgres ranges. Iwould like to have a base class that would handle the range mechanics that could then be inherited, thus mimicking how rangetypes can be extended. Additionally, this would allow us to support continuous ranges (e.g. time, numeric/float/decimal). > > Yes, that would be the thing to adapt to. I don't know how much it > would be useful in Python but I see no more useful thing to adapt to. I can tell you I have real world uses for it at the app level :-) >> My concern is I don't want to deviate too far from the standard way of doing things in Python (lists as ranges) for thesake of other libs (i.e. ORMs) interfacing with psycopg2, so I can make strong arguments on both sides. > > I don't see any standard python object to adapt to without leaving > half of the model away. > > > Ah, postgres ranges can also be open left/right/both. So I'd have: > > - Basic class with all attributes and implementation, and subclasses > specifying a type as you mentioned. > > - The basic class should have the following attributes, which are a > direct mapping to the postgres ones and are required to inspect the > object: > - lower/upper attributes > - lower_inc/upper_inc attributes > - lower_inf/upper_inf attributes > - isempty attibute All of the above is good. > - Possibly it should support an "in" operator and an & operator. But > then, should we mimic all the operators? > <http://www.postgresql.org/docs/9.2/static/functions-range.html> > Probably not. By "&" do you mean "&&"? Part of me thinks we should support overlaps, but part of the power of the range types featureis that Postgres can handle overlap operations (which are nontrivial to build). I see "in" being used a lot. > - Parsing interval should resort to the base type parser, so when > constructing a concrete range type a typecaster should be probably > specified. > > - If a new range type is created in postgres, it would be nice to have > something like register_range() to introspect pg_range and use the > right typecaster (rngsubtype) for parsing. Well, it is really easy to create new ranges in Postgres as is. For example with one line of SQL you can have an INET range. I'm not quite sure I understand what you are suggesting to help with this issue (perhaps I need to dive more intopsycopg2 internals?). Perhaps by having the base class available, people using the adapter can write their own extensions. > - test and docs Absolutely! > Having the range iterable seems tempting but the Postgres model > doesn't have it (even for discrete ranges, it mandates a > canonicalization function, not a step delta) so the two types wouldn't > map automatically (not possible to introspect the catalog to get such > a step) Perhaps we can put iteration on the onus of the user. I know that with dates, for instance, quite often I will need to iterateover them. Thoughts? Jonathan
On Wed, Sep 19, 2012 at 5:59 PM, Jonathan S. Katz <jonathan.katz@excoventures.com> wrote: > > On Sep 18, 2012, at 1:08 PM, Daniele Varrazzo wrote: >> Yes, that would be the thing to adapt to. I don't know how much it >> would be useful in Python but I see no more useful thing to adapt to. > > I can tell you I have real world uses for it at the app level :-) Can you post an example of this use? It could be interesting to know to design the object to adapt to. >> - The basic class should have the following attributes, which are a >> direct mapping to the postgres ones and are required to inspect the >> object: >> - lower/upper attributes >> - lower_inc/upper_inc attributes >> - lower_inf/upper_inf attributes >> - isempty attibute > > All of the above is good. > >> - Possibly it should support an "in" operator and an & operator. But >> then, should we mimic all the operators? >> <http://www.postgresql.org/docs/9.2/static/functions-range.html> >> Probably not. > > By "&" do you mean "&&"? Part of me thinks we should support overlaps, but part of the power of the range types featureis that Postgres can handle overlap operations (which are nontrivial to build). I see "in" being used a lot. There's no && operator in python. But you are sorta right: I was thinking about an operator saying if two intervals match, not to return their intersection. But would it be any useful? It's definitely little thing compared to the richness of the postgres range type and implementing all of them is a project on its own. "in" otoh would be easy to implement. >> - Parsing interval should resort to the base type parser, so when >> constructing a concrete range type a typecaster should be probably >> specified. >> >> - If a new range type is created in postgres, it would be nice to have >> something like register_range() to introspect pg_range and use the >> right typecaster (rngsubtype) for parsing. > > Well, it is really easy to create new ranges in Postgres as is. For example with one line of SQL you can have an INETrange. I'm not quite sure I understand what you are suggesting to help with this issue (perhaps I need to dive moreinto psycopg2 internals?). Perhaps by having the base class available, people using the adapter can write their ownextensions. Just making my mind about what to do. There's not really the need of a separate python class for each different range type supported by postgres: the typecaster for dates range is a generic range typecaster (one knowing the syntax "[from,to)" and its nuances) plus a reference to the dates typecaster to ask parsing the "from" and "to" parts. What changes in the int typecaster is only the sub-typecaster it refers to. The composite typecaster solves the same type of problem, probably even a more complex one. >> Having the range iterable seems tempting but the Postgres model >> doesn't have it (even for discrete ranges, it mandates a >> canonicalization function, not a step delta) so the two types wouldn't >> map automatically (not possible to introspect the catalog to get such >> a step) > > Perhaps we can put iteration on the onus of the user. I know that with dates, for instance, quite often I will need toiterate over them. I don't want to add features to the typecaster that then I would have to maintain before knowing there is a use for them, so for the moment the object returned would be just a record to retrieve what's in the database. Of course you can come with a very rich model of a python range, maybe deserving a pypi project on its own, and have a typecaster to get these objects from the db, instead of a more basic one that psycopg would provide. -- Daniele
On 20/09/2012 02:18, Daniele Varrazzo wrote: >>> - Possibly it should support an "in" operator and an & operator. But >>> >> then, should we mimic all the operators? >>> >> <http://www.postgresql.org/docs/9.2/static/functions-range.html> >>> >> Probably not. >> > >> > By "&" do you mean "&&"? Part of me thinks we should support overlaps, but part of the power of the range types featureis that Postgres can handle overlap operations (which are nontrivial to build). I see "in" being used a lot. > There's no && operator in python. But you are sorta right: I was > thinking about an operator saying if two intervals match, not to > return their intersection. But would it be any useful? It's definitely > little thing compared to the richness of the postgres range type and > implementing all of them is a project on its own. "in" otoh would be > easy to implement. > After we have a basic range type we can add operators as methods (IMO, in this case operator "overloading" in Python is not what we want). So you can have: range1.overlaps(range2) range1.inside(range2) and so on. "in" can be an alias for one of such methods. In fact a range type is so useful that I am amazed it doesn't already exists in Python. federico -- Federico Di Gregorio federico.digregorio@dndg.it Studio Associato Di Nunzio e Di Gregorio http://dndg.it Qu'est ce que la folie? Juste un sentiment de liberté si fort qu'on en oublie ce qui nous rattache au monde... -- J. de Loctra