Обсуждение: Python 3.1 support
Here's the patch to support Python >=3.1 with PL/Python. The compatibility code is mostly in line with the usual 2->3 C porting practice and is documented inline. I needed to create an arguably weird hack to manage the regression tests. Instead of creating a new expected file for pretty much every test file and also for some input files (where Python syntax had changed), a sed script creates a complete Python 3 compatible set of input and output files. Doesn't look pretty but works quite well. If anyone has a better idea, please let me know.
Вложения
Peter Eisentraut <peter_e@gmx.net> writes: > Here's the patch to support Python >=3.1 with PL/Python. The > compatibility code is mostly in line with the usual 2->3 C porting > practice and is documented inline. There was considerable debate earlier about whether we wanted to treat Python 3 as a separate PL so it could be available in parallel with plpython 2, because of the user-level coding incompatibilities. It looks like this patch simply ignores that problem. What is going to happen to plpython functions that depend on 2.x behavior? regards, tom lane
On Nov 12, 2009, at 12:54 PM, Peter Eisentraut wrote: > Here's the patch to support Python >=3.1 with PL/Python. :\ I was hoping to be able to use Python 3 to draw a clear distinction between plpython and the would-be "plpython3" that I'vebeen working on. I understand that you're not in favor of a brand new implementation for Python 3. Despite my dislikefor that position(well, it would seem to be in opposition to my initiative, so naturally =), I don't entirely disagreewith your rationale[wrt doing things more incrementally]. For me, plpython has never been what I would call a pleasureto use, and many of the gripes that I have with it are, IMO, entrenched far enough into the implementation that anyefforts to change it would(should? =) cause unacceptable breakage in user applications(?). Well, as far as additionalPython interfaces are concerned, a lot of redundant functionality, but that's not even the half of it. [I was hoping to get to a status message this weekend,but it seems like I should follow-up here. =] So here's where I'm at: -- Mostly documentation improvements since I last pinged -hackers. Still, *sigh*, filling in documentation and fighting bugs as I go. Currently resolving a bug instantiating MD arrays from nested lists. Once I'm "finished" with the docs, I'm going to start looking for refcount leaks. No major additions or changes are planned, but I have been making some minor additions as I write more docs. Overview/Features: http://wiki.postgresql.org/wiki/WIP:plpython3 Documentation: http://python.projects.postgresql.org/pldocs/plpython3.html git repo[see the plpython3 branch]: http://git.postgresql.org/gitweb?p=plpython3.git;a=summary Most of the documented interfaces have tests. I only have two platforms at my disposal, so I do fear that this will not "justwork" on all of PG's supported platforms. Specifically, I've ran the tests on freebsd/amd64 and Mac10.6/intel(of course10.5 as well for some earlier revisions). [err, actually, it's been a while since I ran the tests on freebsd.] -- plpython3 is turning out to be kinda beefy(~974K diff[eh, there is some fluff in there]), and I can't say that I've seenmuch interest in it, so I can't really blame anyone if -hackers ends up taking a pass on it. (python3 is too far awayfor most folk to care? folk are content with plpython?) eh, cheers, either way. =)
On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > Peter Eisentraut <peter_e@gmx.net> writes: > > Here's the patch to support Python >=3.1 with PL/Python. The > > compatibility code is mostly in line with the usual 2->3 C porting > > practice and is documented inline. > > There was considerable debate earlier about whether we wanted to treat > Python 3 as a separate PL so it could be available in parallel with > plpython 2, because of the user-level coding incompatibilities. It > looks like this patch simply ignores that problem. Exactly how to package that is something to be determined by the packagers, and we can give them the support they need. But first you need code that works with Python 3, which is what this patch does. > What is going to happen to plpython functions that depend on 2.x behavior? The porting path from 2 to 3 is pretty well established. You first port to 2.6, then remove all the old features, then move to 3.x. This is not something we have to reinvent. The only significant differences that you can't use in 2.6 without future imports are unicode literals and the print function, both of which are not in common use in PL/Python.
On tor, 2009-11-12 at 18:42 -0700, James Pye wrote: > For me, plpython has never been what I would call a pleasure to use, > and many of the gripes that I have with it are, IMO, entrenched far > enough into the implementation that any efforts to change it > would(should? =) cause unacceptable breakage in user applications(?). Has this list of gripes ever been brought up and discussed in this forum?
On Nov 13, 2009, at 4:47 AM, Peter Eisentraut wrote: > Has this list of gripes ever been brought up and discussed in this > forum? Some are TODOs, so in part by other people. Some were briefly touched on in the recent past discussions(around the time thatI announced the WIP). Native typing vs conversion, function fragments vs function modules. I don't think native typing has seen any actual discussion, but I do recall mentioning it as something that I wanted to do(implicitlygriped?). ... There is a difference in the situation from the discussion before. Prior, it was, "I would like to implement a new PL forPython 3 with these features", and now, it is, "I have implemented a new PL for Python 3 with these features". Simply, -hackers can choose among moving forward with Python 3 support in plpython or going with "plpython3" or even both,I suppose(?). Naturally, I'm biased toward something that involves plpython3, so I don't think I can(should?) be ofmuch help to -hackers as a Python & PG user in any such decision. Of course, excepting the provision of justificationsfor my implementation/design choices... I would really love to see some input from Python users. I certainly don't want to waste time trying to get something into pgsql that Python users don't want. [here's a gripe that I haven't brought up as I think it is a matter of taste] I find (plpython) trigger functions to be a bit odd. I think it's the way in which manipulation/suppression decisions aremade in BEFORE ROW triggers(return "OK", "SKIP", etc).. [label this as opinion at this point as I have yet to be ableto nail down what, specifically, is "wrong" or un-pythonic about them.] Also, having distinct entry points to handle trigger events helps reduce potential errors by forcing the user to explicitlystate the events that the trigger function can handle. Currently, in plpython, users *should* do sanity checks. Function modules opened the door for implementing this in a natural way, multiple functions(entry points) in the functionmodule. http://python.projects.postgresql.org/pldocs/plpython3-programming.html#PLPYTHON3-FUNCTIONS-TRIGGERS
On fre, 2009-11-13 at 11:27 -0700, James Pye wrote: > Some are TODOs, so in part by other people. Some were briefly touched > on in the recent past discussions(around the time that I announced the > WIP). Native typing vs conversion, function fragments vs function > modules. I'm of course only one user, but these two features don't excite me at all, and certainly not enough to go through the pain of dealing with a second implementation.
On Nov 15, 2009, at 6:37 AM, Peter Eisentraut wrote: > but these two features don't excite me at all, hrm.. at all? I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits ofa *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's sidewas also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemedlike a win all around... AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 providesproperties on PG's datetime types to access the date_part()'s of the object. OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no directutility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interestin native typing is quite understandable.) [looking at the PL/Python todo list..] Excepting DB-API and trusted, I believe all the current PL/Python TODOs are fulfilled or N/A in plpython3... ugh, the docsare not yet complete, but I like to think of them as "better" anyways. :P > the pain of dealing with a second implementation. What pain are you anticipating? Maintenance?
On sön, 2009-11-15 at 18:39 -0700, James Pye wrote: > I can see how function modules might look like a half-step backwards from function fragments at first, but the benefitsof a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer'sside was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source.It seemed like a win all around... The question is whether it helps the user, not the implementer. As far as I can tell, it just creates more typing for no benefit whatsoever. Also, it's inconsistent with normal Python script files and with other PLs. > AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 providesproperties on PG's datetime types to access the date_part()'s of the object. > > OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of nodirect utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interestin native typing is quite understandable.) Right, if I use PL/Python, I do it because I want to use Python. I don't need another PostgreSQL implementation on top of Python. The maintenance effort required to keep those two consistent aside. Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing. > > the pain of dealing with a second implementation. > > What pain are you anticipating? Maintenance? Right.
> Again, I'm only one user. But so far I haven't seen anyone else speak > up here, and clearly accepting this for inclusion will need nontrivial > convincing. Well, FWIW, I am excited about better type integration. Also, I am a little skeptical about this patch. I am sorry if this has already been discussed, but would this mean that I need to choose whether pl/python is built against Python 2.* or Python 3.*? -Nathan
Nathan Boley <npboley@gmail.com> writes: > Also, I am a little skeptical about this patch. I am sorry if this has > already been discussed, but would this mean that I need to choose > whether pl/python is built against Python 2.* or Python 3.*? Yes. That's exactly what I was complaining about upthread. I'm not a Python user, but from what I can gather of the 2-to-3 changes, having to choose one at package build time is going to be a disaster. regards, tom lane
On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: > Nathan Boley <npboley@gmail.com> writes: > > Also, I am a little skeptical about this patch. I am sorry if this has > > already been discussed, but would this mean that I need to choose > > whether pl/python is built against Python 2.* or Python 3.*? > > Yes. That's exactly what I was complaining about upthread. I'm not > a Python user, but from what I can gather of the 2-to-3 changes, > having to choose one at package build time is going to be a disaster. > Agreed. We really need to have a plpython and plpython3. Heck this would play nicely too because we support backward compatibility but also upward version differences. Joshua D. Drake > regards, tom lane > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
"Joshua D. Drake" <jd@commandprompt.com> writes: > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: >> Yes. That's exactly what I was complaining about upthread. I'm not >> a Python user, but from what I can gather of the 2-to-3 changes, >> having to choose one at package build time is going to be a disaster. > Agreed. We really need to have a plpython and plpython3. Peter was concerned about duplicative maintenance effort, but what I think this patch shows is that (at least for the near future) both could be built from a single source file. What we need is configure and makefile support to do that. regards, tom lane
On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote: > "Joshua D. Drake" <jd@commandprompt.com> writes: > > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: > >> Yes. That's exactly what I was complaining about upthread. I'm not > >> a Python user, but from what I can gather of the 2-to-3 changes, > >> having to choose one at package build time is going to be a disaster. > > > Agreed. We really need to have a plpython and plpython3. > > Peter was concerned about duplicative maintenance effort, but what I > think this patch shows is that (at least for the near future) both > could be built from a single source file. What we need is configure > and makefile support to do that. Ahh, so we would have: --enable-plpython2=/usr/bin/python2 --enable-plpython3=/usr/bin/python3 ? That seems reasonable if we can run both. Although I wonder if longer term (2.x is going to be support a long time) we will end up with frustration within the single source file trying to keep things straight. Joshua D. Drake > > regards, tom lane > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
"Joshua D. Drake" <jd@commandprompt.com> writes: > On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote: >> Peter was concerned about duplicative maintenance effort, but what I >> think this patch shows is that (at least for the near future) both >> could be built from a single source file. > That seems reasonable if we can run both. Although I wonder if longer > term (2.x is going to be support a long time) we will end up with > frustration within the single source file trying to keep things > straight. Once it gets to the point where it's more trouble to keep them together than not, we can split the source. But judging from this patch, a single source file is the ticket for the moment. regards, tom lane
> Here's the patch to support Python >=3.1 with PL/Python. The > compatibility code is mostly in line with the usual 2->3 C porting > practice and is documented inline. I took a cursory look at this patch and, while the logic seems sound and roughly in line with the suggested python porting procedure, I'm not quite certain what this implies for potential future patches. For instance, if I wanted to write a type converter for bytea -> the python 3 byte type would the expectation be that I ensure that it works in Python 2? Or is an ifdef that ignores it in the case of Python 2 OK, and we can just put a note in the docs. Also, how far back do we want to maintain 2.x compatibility? 2.0? If I wanted to submit a patch that makes use of the list sort method, do I need to ensure that it can either use the cmp arguments or a key argument? What if I wanted to implement a set returning function that made use of an iterators next() method. Would I just put ifdefs around the code or a preprocessor definition that defines NEXT as next() for Python 2.x and __next__() for 3.x? I guess that my first impression is that Python broke compatibility for a reason, and that either plpython can't evolve, or it will quickly become impossible to maintain. That being said, I mostly buy the maintenance arguments from the previous discussion, but if we want to have plpython and plpython3, a bunch of defines and ifdefs does not seem like the best way to do this. Would a better approach be to maintain compatibility layer? ie plython_compat.h/c plython2.c plython3.c Then patches that apply to a python3 can be applied to plython3.c and any changed function can be ripped out of plython_compat and moved into plpython2. I'm sorry to snipe from the sidelines like this. If we didn't expect plpython to evolve then this patch seems like the correct approach, but there is clearly some desire to expand plpython and following this path seems like it will end in a much more painful split in the future or a necessary rewrite. If there is some consensus that this is the best approach, then I will do a more comprehensive review. -Nathan
On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote: > The question is whether it helps the user, not the implementer. Sure, but do you have a patch waiting to implement tracebacks? I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without somesignificant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack. What helped the implementer here would help the user. > As far > as I can tell, it just creates more typing for no benefit whatsoever. "def main(*args):" is annoying, but not entirely lamentable... It's explicit, as well(no need to document munging that occurs behind the scenes). Also, compare the cases where you need to cache some initialized data: if 'key' not in SD:...SD['key'] = my_newly_initialized_data ... With function modules, SD is not needed as you have your module globals to keep your locally cached data in: ... data = my_newly_initialized_data def main(*args):... > Also, it's inconsistent with normal Python script files Hinges on whether "normal" is actually normal. I often use the __name__ convention in script files myself: if __name__ == '__main__':main(...) That is, using that convention, the script can be import'd and used without executing the "script functionality". (It hasproven to be very handy a few times now) > and with other PLs. I don't understand why that's a significant enough interest to note. > I don't need another PostgreSQL implementation on top of Python. Indeed, and I do understand that. That is, I have removed some features with that very thought in mind. (OTOH, I considerthe date_part properties on datetime types to be special: too likely useful.) [tho, "PostgreSQL implementation"? I think I understand what you were getting at, but..] > The maintenance effort required to keep those two consistent aside. I don't think there are many consistency issues here. What did you have in mind? > Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusionwill need nontrivial convincing. Agreed. It would seem quite doomed. At this point, I'm not going to try getting it into PG. (apparent futility and such)
On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: > Nathan Boley <npboley@gmail.com> writes: > > Also, I am a little skeptical about this patch. I am sorry if this has > > already been discussed, but would this mean that I need to choose > > whether pl/python is built against Python 2.* or Python 3.*? > > Yes. That's exactly what I was complaining about upthread. I'm not > a Python user, but from what I can gather of the 2-to-3 changes, > having to choose one at package build time is going to be a disaster. > Agreed. We really need to have a plpython and plpython3. Heck this would play nicely too because we support backward compatibility but also upward version differences. Joshua D. Drake > regards, tom lane > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote: > "Joshua D. Drake" <jd@commandprompt.com> writes: > > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: > >> Yes. That's exactly what I was complaining about upthread. I'm not > >> a Python user, but from what I can gather of the 2-to-3 changes, > >> having to choose one at package build time is going to be a disaster. > > > Agreed. We really need to have a plpython and plpython3. > > Peter was concerned about duplicative maintenance effort, but what I > think this patch shows is that (at least for the near future) both > could be built from a single source file. What we need is configure > and makefile support to do that. Ahh, so we would have: --enable-plpython2=/usr/bin/python2 --enable-plpython3=/usr/bin/python3 ? That seems reasonable if we can run both. Although I wonder if longer term (2.x is going to be support a long time) we will end up with frustration within the single source file trying to keep things straight. Joshua D. Drake > > regards, tom lane > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
On Nov 18, 2009, at 1:36 PM, James Pye wrote: > At this point, I'm not going to try getting it into PG. (apparent futility and such) ugh, on second thought, I think I've written a bit too much code to stop now. I'm going to get plpython3 as far as I canand submit it to the next commitfest.
On ons, 2009-11-18 at 09:48 -0800, Joshua D. Drake wrote: > Although I wonder if longer > term (2.x is going to be support a long time) we will end up with > frustration within the single source file trying to keep things > straight. There are five million Python modules with C code out there with the same problem. Considerable effort has been put in by Python upstream to make the effort manageable. No one in their right mind is going to create two separate source files just because in the future the mythical differences will be too big, when clearly the effort is going into a direction to reduce the differences. If you look into the source file, there is already special code for Python 2.2, 2.3, 2.4, 2.5, 2.6, and now 3.1. The chunk for 3.1 is a bit bigger, but only a bit, and well, that's why it's 3.x and not 2.x. No one has ever suggested, we might need to support Python 2.2 for a long time, let's create a separate source file. I agree, there will probably need to be some configuration/build support on top of this, but that's something we should work out independently of how to manage the source file.
On ons, 2009-11-18 at 08:43 -0800, Nathan Boley wrote: > > Again, I'm only one user. But so far I haven't seen anyone else speak > > up here, and clearly accepting this for inclusion will need nontrivial > > convincing. > > Well, FWIW, I am excited about better type integration. Let's clarify, as there are two different models being proposed here. The first approach, which is currently implemented (and some patches pending), is to convert a PostgreSQL type to the "nearest" Python type. For example, text to string, int to int, array to list, timestamp to datetime.datetime, etc. The other approach, which is what James Pye's new implementation proposes (as I understand it), is to convert PostgreSQL types into specially made Python objects, such as Postgres.types.record or Postgres.types.timestamp. > Also, I am a little skeptical about this patch. I am sorry if this has > already been discussed, but would this mean that I need to choose > whether pl/python is built against Python 2.* or Python 3.*? Yeah, see later discussion about how to resolve this. But I think in practice, unless you use lots of print statements in your stored procedures (?!?), this problem is exaggerated.
On ons, 2009-11-18 at 11:32 -0800, Nathan Boley wrote: > I took a cursory look at this patch and, while the logic seems sound > and roughly in line with the suggested python porting procedure, I'm > not quite certain what this implies for potential future patches. > > For instance, if I wanted to write a type converter for bytea -> the > python 3 byte type would the expectation be that I ensure that it > works in Python 2? Or is an ifdef that ignores it in the case of > Python 2 OK, and we can just put a note in the docs. Note that this is already implemented. The main point of the patch is to provide a small compatibility layer so that these kinds of issues are practically nonexistent. The fact that you didn't notice might prove that the patch does its job. ;-) > Also, how far back do we want to maintain 2.x compatibility? 2.0? We handle this on an ad hoc basis. We currently support Python 2.2 and later, and this cutoff exists -- this is my interpretation of history -- because 2.2 introduced iterators and no one bothered(?) to put ifdefs around the code in PL/Python that provides iterator support. Over the years, we will probably drop support for other older Python versions, but there is no process or plan for that. Right now, the support for Python 2.2 is about three lines, so it's not a bother, but when someone comes and implements a major feature that, say, requires Python 2.3, we can probably drop 2.2. But when the major feature requires 2.6, we probably don't want to drop 2.5 quite yet at this time. It's a judgment call. > If I > wanted to submit a patch that makes use of the list sort method, do I > need to ensure that it can either use the cmp arguments or a key > argument? Any patch one is likely to submit will be a C patch, not a Python patch. But anyway, the "key" argument was introduced in Python 2.4, and so we'd have to come to a decision in the community about whether Python 2.3 support is worth keeping versus the value of that new feature. See above. But anyway, this problem has nothing to do with my patch; it has already existed in the same form forever. > What if I wanted to implement a set returning function that made use > of an iterators next() method. Would I just put ifdefs around the code > or a preprocessor definition that defines NEXT as next() for Python > 2.x and __next__() for 3.x? Again, you would likely submit a C patch, and the iterator API is the same between 2.x and 3.x. > I guess that my first impression is that Python broke compatibility > for a reason, and that either plpython can't evolve, or it will > quickly become impossible to maintain. I think this is an exaggeration of reality. Python 3 removed deprecated features. There is a perfectly good migration path that covers most code: Switch to Python 2.6, switch to the new features, remove the old features, switch to Python 3.x. This applies both on the Python and the C level. They did not break compatibility with the intention of making every module author out there reimplement their thing from scratch. Otherwise Python 2.6 would make very little sense at all. Take a look at an example closer to home: PostgreSQL breaks C API compatibility in almost every major release. We do this to remove cruft and support new features. The intent is not to make Slony and PostGIS and all the other modules reimplement their product from scratch every time. They put in a few ifdefs, sometimes they complain about it ;-), and then the problem is solved. > That being said, I mostly buy > the maintenance arguments from the previous discussion, but if we want > to have plpython and plpython3, a bunch of defines and ifdefs does not > seem like the best way to do this. These ifdefs were not my idea. They are in some cases directly and in some cases in spirit from the Python 2.6 header files, so they are the official way to do this. > Would a better approach be to maintain compatibility layer? ie > plython_compat.h/c > plython2.c > plython3.c > > Then patches that apply to a python3 can be applied to plython3.c and > any changed function can be ripped out of plython_compat and moved > into plpython2. As I tried to explain above, we have always had a rolling feature model of sorts, even across various Python 2.x versions. If you want to try it out, you could take the current source and split it up into plpython22.c, plpython23.c, etc. and see if that becomes useful.
On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote: > The other approach, which is what James Pye's > new implementation proposes (as I understand it), is to convert > PostgreSQL types into specially made Python objects, such as > Postgres.types.record or Postgres.types.timestamp. Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum).So more like "copied into Python memory", and associated with its respective type. Wrapped in a Python object? One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no typeI/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it couldbe quite beneficial.
On ons, 2009-11-18 at 12:28 -0500, Tom Lane wrote: > "Joshua D. Drake" <jd@commandprompt.com> writes: > > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote: > >> Yes. That's exactly what I was complaining about upthread. I'm not > >> a Python user, but from what I can gather of the 2-to-3 changes, > >> having to choose one at package build time is going to be a disaster. > > > Agreed. We really need to have a plpython and plpython3. > > Peter was concerned about duplicative maintenance effort, but what I > think this patch shows is that (at least for the near future) both > could be built from a single source file. What we need is configure > and makefile support to do that. By the way, it occurred to me that having two different versions of libpython loaded into the same process is probably not going to work sanely. So whatever solution we come up with for the Python 3 transition, the possibilities for a jolly back-and-forth are probably going to be quite limited.
On ons, 2009-11-18 at 13:36 -0700, James Pye wrote: > On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote: > > The question is whether it helps the user, not the implementer. > > Sure, but do you have a patch waiting to implement tracebacks? > > I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *withoutsome significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack. > > What helped the implementer here would help the user. But you wouldn't, for example, get away with breaking SQL (or even improving it incompatibly) to facilitate a better elog. > > As far > > as I can tell, it just creates more typing for no benefit whatsoever. > > "def main(*args):" is annoying, but not entirely lamentable... > It's explicit, as well(no need to document munging that occurs behind the scenes). > > Also, compare the cases where you need to cache some initialized data: > > if 'key' not in SD: > ... > SD['key'] = my_newly_initialized_data > ... > > > With function modules, SD is not needed as you have your module globals to keep your locally cached data in: > > ... > data = my_newly_initialized_data > > def main(*args): > ... I can see that this creates other options for structuring code, but it doesn't actually match my way of thinking. (Obviously, I'm biased, but anyway.) I think of a PL/Python function as a Python script file stored in the database. When you call it, arguments are passed just like a Python script receives arguments from the shell. When Python scripts want to share data, they might use a file (or perhaps a database server in advanced cases) and do if not file exists: create the file fill it with data This is in my mind quite analogous to how the SD business works. The analogy to your approach, as I understand it, would be that multiple instances of the same script file will automatically share their global variables. That could be quite interesting, actually, but it's not how it works, and in most cases it's better that way.
Peter Eisentraut <peter_e@gmx.net> writes: > By the way, it occurred to me that having two different versions of > libpython loaded into the same process is probably not going to work > sanely. Why not? There's no way they'd even know about each other. We tell the loader not to make the symbols globally visible. But in any case, my main concern here is that I don't want to have to predetermine which python version a user of Red Hat/Fedora will have to use. If they can only use one at a time, that's still a good bit better than not having a choice at all. regards, tom lane
On Nov 19, 2009, at 11:32 AM, Peter Eisentraut wrote: > But you wouldn't, for example, get away with breaking SQL (or even > improving it incompatibly) to facilitate a better elog. This doesn't fit the situation. I'm not breaking PL/Python. I'm trying to add PL/Python3. =) > I think of a PL/Python function as a Python script file stored > in the database. For Python, I think that's a mistake. Python scripts are independent applications. [tho, I think this does illuminate our perspectives...]
On tor, 2009-11-19 at 13:43 -0500, Tom Lane wrote: > But in any case, my main concern here is that I don't want to have > to predetermine which python version a user of Red Hat/Fedora will > have to use. If they can only use one at a time, that's still a > good bit better than not having a choice at all. By the way, mod_wsgi supports Python 3 already (same patch as here, in principle). From the Fedora wiki page, I gather that no one has really looked into packaging that yet for Python 3, but if someone does, maybe we can cross-inspire ourselves.
On tor, 2009-11-19 at 13:12 -0700, James Pye wrote: > > I think of a PL/Python function as a Python script file stored > > in the database. > > For Python, I think that's a mistake. Python scripts are independent applications. Is there any precedent for the sort of behavior that you are implementing, that is, automatic sharing of variables between independent executions of the same source container?
On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote: > Is there any precedent for the sort of behavior that you are > implementing, that is, automatic sharing of variables between > independent executions of the same source container? import foo # bar is a regular, def'd function. foo.bar() ... # even in another thread, doesn't matter.. foo.bar() In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary). A plpython3 function *is* a Python module.
On fre, 2009-11-20 at 01:20 -0700, James Pye wrote: > On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote: > > Is there any precedent for the sort of behavior that you are > > implementing, that is, automatic sharing of variables between > > independent executions of the same source container? > > import foo > > # bar is a regular, def'd function. > foo.bar() > > ... > > # even in another thread, doesn't matter.. > foo.bar() > > > In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary). That's not what I meant, because this is the same execution of the same source container, with threads explicitly started somewhere. You could do the same in a plpython function (in theory, at least). What I mean is more like, you execute the same source file twice in a row, and the global variables are saved for the second run.
On Nov 20, 2009, at 1:26 AM, Peter Eisentraut wrote: > because this is the same execution Hrm, not necessarily. foo could be imported by another, completely independent part of the program. foo is cached in sys.modules.bar() is executed and it's still the same globals(). shared.
Am 19.11.2009 18:01, schrieb James Pye: > On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote: >> The other approach, which is what James Pye's >> new implementation proposes (as I understand it), is to convert >> PostgreSQL types into specially made Python objects, such as >> Postgres.types.record or Postgres.types.timestamp. > > Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum).So more like "copied into Python memory", and associated with its respective type. Wrapped in a Python object? Yes "wrapped" is the term commonly used for that. And I must say I like it and I used plpy where I could. > One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's notype I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it couldbe quite beneficial. Exactly and you have all the meta information about the original postgres type. IIRC there were some thoughts of having something like that in a DBAPI interface as well (similar for example to cx_Oracle). Tino
On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > There was considerable debate earlier about whether we wanted to treat > Python 3 as a separate PL so it could be available in parallel with > plpython 2, because of the user-level coding incompatibilities. It > looks like this patch simply ignores that problem. What is going to > happen to plpython functions that depend on 2.x behavior? I have a proposal for how to handle this, and a prototype patch attached. This follows essentially what the CPython distribution itself does, which will make this tolerably easy to follow for users. We install plpython as plpython2.so or plpython3.so, depending on the version used to build it. Then, plpython.so is a symlink to plpython2.so. We then create three language definition templates: plpythonu -> plpython.so plpython2u -> plpython2.so plpython3u -> plpython3.so In the far future we flip the default symlink to plpython3.so, maybe in about 5 years when Python 2.x expires. This gives the users the following options and scenarios: - Existing users don't have to do anything, until maybe in five years they will notice that their OS doesn't ship Python 2 anymore and they will have to act anyway. In practice, by then they might have adjusted their coding style to Python 2.6/2.7 norms and their functions will migrate to 3.x without change. - Users who know that they have heavily Python 2.x dependent code and don't want to ever change it can make use of the plpython2u language name, just like they should probably change their scripts to use something like #!/usr/bin/python2. - Users who want to dive into Python 3 can use the plpython3u language name, which will basically keep working forever by today's standards. Those users would probably also use #!/usr/bin/python3 or the like in their scripts. In the far future they might like to remove the "3". - Daredevils can change symlink manually and make plpython3.so the default plpythonu implementation. Those people would probably also make /usr/bin/python be version 3. Comments?
Вложения
On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote: > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > > There was considerable debate earlier about whether we wanted to treat > > Python 3 as a separate PL so it could be available in parallel with > > plpython 2, because of the user-level coding incompatibilities. It > > looks like this patch simply ignores that problem. What is going to > > happen to plpython functions that depend on 2.x behavior? > > I have a proposal for how to handle this, and a prototype patch > attached. This follows essentially what the CPython distribution itself > does, which will make this tolerably easy to follow for users. > > We install plpython as plpython2.so or plpython3.so, depending on the > version used to build it. Then, plpython.so is a symlink to > plpython2.so. > > We then create three language definition templates: > > plpythonu -> plpython.so > plpython2u -> plpython2.so > plpython3u -> plpython3.so > > > Comments? Well as a Python guy... makes sense to me :) Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote: > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > > There was considerable debate earlier about whether we wanted to treat > > Python 3 as a separate PL so it could be available in parallel with > > plpython 2, because of the user-level coding incompatibilities. It > > looks like this patch simply ignores that problem. What is going to > > happen to plpython functions that depend on 2.x behavior? > > I have a proposal for how to handle this, and a prototype patch > attached. This follows essentially what the CPython distribution itself > does, which will make this tolerably easy to follow for users. > > We install plpython as plpython2.so or plpython3.so, depending on the > version used to build it. Then, plpython.so is a symlink to > plpython2.so. > > We then create three language definition templates: > > plpythonu -> plpython.so > plpython2u -> plpython2.so > plpython3u -> plpython3.so > > > Comments? Well as a Python guy... makes sense to me :) Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
I wrote: > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > > There was considerable debate earlier about whether we wanted to treat > > Python 3 as a separate PL so it could be available in parallel with > > plpython 2, because of the user-level coding incompatibilities. It > > looks like this patch simply ignores that problem. What is going to > > happen to plpython functions that depend on 2.x behavior? > > I have a proposal for how to handle this, and a prototype patch > attached. This follows essentially what the CPython distribution itself > does, which will make this tolerably easy to follow for users. > > We install plpython as plpython2.so or plpython3.so, depending on the > version used to build it. Then, plpython.so is a symlink to > plpython2.so. So here is the potentially final patch for this, including the original port of plpython.c itself, build system adjustments, and documentation.
On Mon, Dec 14, 2009 at 1:42 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > I wrote: >> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: >> > There was considerable debate earlier about whether we wanted to treat >> > Python 3 as a separate PL so it could be available in parallel with >> > plpython 2, because of the user-level coding incompatibilities. It >> > looks like this patch simply ignores that problem. What is going to >> > happen to plpython functions that depend on 2.x behavior? >> >> I have a proposal for how to handle this, and a prototype patch >> attached. This follows essentially what the CPython distribution itself >> does, which will make this tolerably easy to follow for users. >> >> We install plpython as plpython2.so or plpython3.so, depending on the >> version used to build it. Then, plpython.so is a symlink to >> plpython2.so. > > So here is the potentially final patch for this, including the original > port of plpython.c itself, build system adjustments, and documentation. I think you forgot to actually attach it... ...Robert
I wrote: > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > > There was considerable debate earlier about whether we wanted to treat > > Python 3 as a separate PL so it could be available in parallel with > > plpython 2, because of the user-level coding incompatibilities. It > > looks like this patch simply ignores that problem. What is going to > > happen to plpython functions that depend on 2.x behavior? > > I have a proposal for how to handle this, and a prototype patch > attached. This follows essentially what the CPython distribution itself > does, which will make this tolerably easy to follow for users. > > We install plpython as plpython2.so or plpython3.so, depending on the > version used to build it. Then, plpython.so is a symlink to > plpython2.so. So here is the potentially final patch for this, including the original port of plpython.c itself, build system adjustments, and documentation. Really attached this time.