Обсуждение: problem with copy_expert on cursor
What would cause the following error when I call copy_expert on a
DictCursor?
SystemError: 'null argument to internal routine'
The sql I pass in is the copy, in a string. The file is a
cStringIO.StringO object. I am using a custom row factory on the
DictCursor. So there are a number of things that could be the source of
the error. However, I am not familiar enough with psycopg to dig this
out.
Incidently, the problem is actually when I am using skytools for
replication, and not actually in any code I have written. I am using
psycopg2 2.3.2. Thanks.
-eric
This email message is intended for the use of the person to whom it has been sent, and may contain information that is
confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are
notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender
immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that
thisemail is error or virus free. Thank you.
If my StringIO object gets a MemoryError, will the cursor's copy_expert
handle that, or is that the SystemError I am seeing? Thanks.
-eric
-----Original Message-----
From: Eric Snow
Sent: Wednesday, March 02, 2011 2:21 PM
To: 'psycopg@postgresql.org'
Subject: problem with copy_expert on cursor
What would cause the following error when I call copy_expert on a
DictCursor?
SystemError: 'null argument to internal routine'
The sql I pass in is the copy, in a string. The file is a
cStringIO.StringO object. I am using a custom row factory on the
DictCursor. So there are a number of things that could be the source of
the error. However, I am not familiar enough with psycopg to dig this
out.
Incidently, the problem is actually when I am using skytools for
replication, and not actually in any code I have written. I am using
psycopg2 2.3.2. Thanks.
-eric
This email message is intended for the use of the person to whom it has been sent, and may contain information that is
confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are
notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender
immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that
thisemail is error or virus free. Thank you.
On Wed, Mar 2, 2011 at 9:20 PM, Eric Snow <esnow@verio.net> wrote:
> What would cause the following error when I call copy_expert on a
> DictCursor?
>
> SystemError: 'null argument to internal routine'
>
> The sql I pass in is the copy, in a string. The file is a
> cStringIO.StringO object. I am using a custom row factory on the
> DictCursor. So there are a number of things that could be the source of
> the error. However, I am not familiar enough with psycopg to dig this
> out.
>
> Incidently, the problem is actually when I am using skytools for
> replication, and not actually in any code I have written. I am using
> psycopg2 2.3.2. Thanks.
Hi Eric,
the description of the problem is not very clear, e.g. I've not
understood if the problem happened in a COPY FROM STDIN or in COPY TO
STDOUT. I've tried to reproduce the problem but without success, both
in 2.3.2 and in 2.4.0.
To try and reproduce the problem I've added the test below to
tests/test_copy.py. Can you modify the test until you are able to make
it crash?
In order to run the test suite, create a database called
"psycopg2_test". Then, from the psycopg directory, execute "make",
then "make check". If you need to specify host/username/password to
connect to the database, set the env variables PSYCOPG2_TESTDB_HOST
and so on.
Thank you.
-- Daniele
def test_copy_strange_cursor(self):
from psycopg2.extras import DictCursor
f = StringIO()
for i, c in enumerate(string.ascii_letters):
f.write("%s\t%s\n" % (i, c))
f.seek(0)
curs = self.conn.cursor(cursor_factory=DictCursor)
curs.copy_expert('COPY tcopy FROM STDIN', f)
curs.close()
curs = self.conn.cursor()
curs.execute("select * from tcopy order by id;")
self.assertEqual(curs.fetchall(),
list(enumerate(string.ascii_letters)))
f1 = StringIO()
curs = self.conn.cursor(cursor_factory=DictCursor)
curs.copy_expert('COPY tcopy TO STDOUT', f1)
curs.close()
f.seek(0)
f1.seek(0)
self.assertEqual(f1.read(), f.read())
On Wed, Mar 2, 2011 at 11:19 PM, Eric Snow <esnow@verio.net> wrote: > If my StringIO object gets a MemoryError, will the cursor's copy_expert > handle that, or is that the SystemError I am seeing? Thanks. I assume you are writing into the StringIO. Yes, it should be handled (https://github.com/dvarrazzo/psycopg/blob/2_3_2/psycopg/pqpath.c#L1162): if write() returns a MemoryError the function bails out with the exception set by write() itself. -- Daniele
Awesome! Thanks, Daniele. -eric -----Original Message----- From: Daniele Varrazzo [mailto:daniele.varrazzo@gmail.com] Sent: Wednesday, March 02, 2011 5:29 PM To: Eric Snow Cc: psycopg@postgresql.org Subject: Re: [psycopg] problem with copy_expert on cursor On Wed, Mar 2, 2011 at 11:19 PM, Eric Snow <esnow@verio.net> wrote: > If my StringIO object gets a MemoryError, will the cursor's copy_expert > handle that, or is that the SystemError I am seeing? Thanks. I assume you are writing into the StringIO. Yes, it should be handled (https://github.com/dvarrazzo/psycopg/blob/2_3_2/psycopg/pqpath.c#L1162): if write() returns a MemoryError the function bails out with the exception set by write() itself. -- Daniele This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that thisemail is error or virus free. Thank you.
I am trying to figure out the conditions under which I get that
SystemError. Here is a simple way that I found to reproduce the
problem:
>>> import psycopg2
>>> psycopg2.__version__
'2.3.2 (dt dec pq3 ext)'
>>> conn = psycopg2.connect("...")
>>> cur = conn.cursor()
>>> cur.execute(";")
Traceback (most recent call last):
File "<stdin>", line 1, in ?
SystemError: null argument to internal routine
I am getting the same exception during a call to copy_expert. How would
the data I pass in there trigger the same exception as when I pass a
single semicolon? I am guessing that it does not even make it over the
wire to the postgres server when the exception is raised. Thanks.
-eric
This email message is intended for the use of the person to whom it has been sent, and may contain information that is
confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are
notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender
immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that
thisemail is error or virus free. Thank you.
On 03/03/11 20:52, Eric Snow wrote:
> I am trying to figure out the conditions under which I get that
> SystemError. Here is a simple way that I found to reproduce the
> problem:
>
>>>> import psycopg2
>>>> psycopg2.__version__
> '2.3.2 (dt dec pq3 ext)'
>>>> conn = psycopg2.connect("...")
>>>> cur = conn.cursor()
>>>> cur.execute(";")
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> SystemError: null argument to internal routine
>
> I am getting the same exception during a call to copy_expert. How would
> the data I pass in there trigger the same exception as when I pass a
> single semicolon? I am guessing that it does not even make it over the
> wire to the postgres server when the exception is raised. Thanks.
Aaah, that rings a bell. I was getting the same error with txpostgres
and even once found out why, but forgot, and now I remembered.
I believe that the problem is in pqpath.c, in pq_fetch. There's a switch
statement there that checks the result of PQresultStatus and does not
take PGRES_EMPTY_QUERY into consideration.
The code that jumps to the default: label, which leads it to pq_raise.
Towards the end there's a
if (code != NULL)
exc = exception_from_sqlstate(code);
which AFAIR never gets executed, because code *is* NULL. And then it
hits psyco_set_error with a NULL exception, this leads to the "NULL
argument to internal routine" error.
I might have messed up some details, but I'm pretty sure the problem is
not handling PGRES_EMPTY_QUERY correctly.
Cheers,
Jan
On Thu, Mar 3, 2011 at 8:02 PM, Jan Urbański <wulczer@wulczer.org> wrote:
> On 03/03/11 20:52, Eric Snow wrote:
>> I am trying to figure out the conditions under which I get that
>> SystemError. Here is a simple way that I found to reproduce the
>> problem:
>>
>>>>> import psycopg2
>>>>> psycopg2.__version__
>> '2.3.2 (dt dec pq3 ext)'
>>>>> conn = psycopg2.connect("...")
>>>>> cur = conn.cursor()
>>>>> cur.execute(";")
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in ?
>> SystemError: null argument to internal routine
>>
>> I am getting the same exception during a call to copy_expert. How would
>> the data I pass in there trigger the same exception as when I pass a
>> single semicolon? I am guessing that it does not even make it over the
>> wire to the postgres server when the exception is raised. Thanks.
>
> Aaah, that rings a bell. I was getting the same error with txpostgres
> and even once found out why, but forgot, and now I remembered.
>
> I believe that the problem is in pqpath.c, in pq_fetch. There's a switch
> statement there that checks the result of PQresultStatus and does not
> take PGRES_EMPTY_QUERY into consideration.
>
> The code that jumps to the default: label, which leads it to pq_raise.
> Towards the end there's a
>
> if (code != NULL)
> exc = exception_from_sqlstate(code);
>
> which AFAIR never gets executed, because code *is* NULL. And then it
> hits psyco_set_error with a NULL exception, this leads to the "NULL
> argument to internal routine" error.
>
> I might have messed up some details, but I'm pretty sure the problem is
> not handling PGRES_EMPTY_QUERY correctly.
Thank you for the report: open ticket #46
<http://psycopg.lighthouseapp.com/projects/62710-psycopg/tickets/46>
If anyone wants to try to solve it, please add a comment to the
tracker. Otherwise I will take a look at it as soon as I have a
moment.
-- Daniele
Thanks Jan, that helps explain that. So you are saying that the problem is in pq_raise at the end of the function. If
"code"is NULL then "exc" is still NULL, which results in the SystemError. While the handling of the empty query is one
thing,the handling of errors where "code" is NULL is another. Seems like two separate issue that should both be
addressed. Should there be separate tickets for this, or is the ticket Daniele made sufficient? Thanks.
-eric
-----Original Message-----
From: Jan Urbański [mailto:wulczer@wulczer.org]
Sent: Thursday, March 03, 2011 1:02 PM
To: Eric Snow
Cc: psycopg@postgresql.org
Subject: Re: [psycopg] SystemError: null argument to internal routine
On 03/03/11 20:52, Eric Snow wrote:
> I am trying to figure out the conditions under which I get that
> SystemError. Here is a simple way that I found to reproduce the
> problem:
>
>>>> import psycopg2
>>>> psycopg2.__version__
> '2.3.2 (dt dec pq3 ext)'
>>>> conn = psycopg2.connect("...")
>>>> cur = conn.cursor()
>>>> cur.execute(";")
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> SystemError: null argument to internal routine
>
> I am getting the same exception during a call to copy_expert. How would
> the data I pass in there trigger the same exception as when I pass a
> single semicolon? I am guessing that it does not even make it over the
> wire to the postgres server when the exception is raised. Thanks.
Aaah, that rings a bell. I was getting the same error with txpostgres
and even once found out why, but forgot, and now I remembered.
I believe that the problem is in pqpath.c, in pq_fetch. There's a switch
statement there that checks the result of PQresultStatus and does not
take PGRES_EMPTY_QUERY into consideration.
The code that jumps to the default: label, which leads it to pq_raise.
Towards the end there's a
if (code != NULL)
exc = exception_from_sqlstate(code);
which AFAIR never gets executed, because code *is* NULL. And then it
hits psyco_set_error with a NULL exception, this leads to the "NULL
argument to internal routine" error.
I might have messed up some details, but I'm pretty sure the problem is
not handling PGRES_EMPTY_QUERY correctly.
Cheers,
Jan
This email message is intended for the use of the person to whom it has been sent, and may contain information that is
confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are
notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender
immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that
thisemail is error or virus free. Thank you.
On Fri, Mar 4, 2011 at 12:03 AM, Eric Snow <esnow@verio.net> wrote: > Thanks Jan, that helps explain that. So you are saying that the problem is in pq_raise ... I've fixed the unhandled exception in pq_raise and the handling of empty queries. Patches in my devel branch. Eric, is your problem related to the "empty query" result not handled or is it something else related to the copy command, that psycopg may still not handle correctly? Thanks, -- Daniele
That is a great question. To be honest, I ran into the problem using londiste (skytools) where I was getting the SystemError. It happens right around 500MB every time (out of around 1.5GB) during the copy_expert (with a copy_to). Istumbled on the empty query thing accidently when I was trying to reproduce my problem with generic data. Since it is thesame error I suppose it could be related. However, I have not looked at the psycopg code enough to see what sort of problemsmight trigger the SystemError. So I am not sure if an empty query is slipping in there somehow, or if it's somethingelse that triggers the same failure. I have been working with Marko on this a little, but also was trying to understand the error better. Thanks for the effortso far. I would certainly be grateful for any further help! -eric -----Original Message----- From: Daniele Varrazzo [mailto:daniele.varrazzo@gmail.com] Sent: Friday, March 04, 2011 3:16 PM To: Eric Snow Cc: Jan Urbański; psycopg@postgresql.org Subject: Re: [psycopg] SystemError: null argument to internal routine On Fri, Mar 4, 2011 at 12:03 AM, Eric Snow <esnow@verio.net> wrote: > Thanks Jan, that helps explain that. So you are saying that the > problem is in pq_raise ... I've fixed the unhandled exception in pq_raise and the handling of empty queries. Patches in my devel branch. Eric, is your problem related to the "empty query" result not handled or is it something else related to the copy command,that psycopg may still not handle correctly? Thanks, -- Daniele This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that thisemail is error or virus free. Thank you.
Daniele, Where could I get a hold of that patch? Thanks. -eric -----Original Message----- From: Daniele Varrazzo [mailto:daniele.varrazzo@gmail.com] Sent: Friday, March 04, 2011 3:16 PM To: Eric Snow Cc: Jan Urbański; psycopg@postgresql.org Subject: Re: [psycopg] SystemError: null argument to internal routine On Fri, Mar 4, 2011 at 12:03 AM, Eric Snow <esnow@verio.net> wrote: > Thanks Jan, that helps explain that. So you are saying that the problem is in pq_raise ... I've fixed the unhandled exception in pq_raise and the handling of empty queries. Patches in my devel branch. Eric, is your problem related to the "empty query" result not handled or is it something else related to the copy command, that psycopg may still not handle correctly? Thanks, -- Daniele This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that thisemail is error or virus free. Thank you.
On Mon, Mar 7, 2011 at 5:59 PM, Eric Snow <esnow@verio.net> wrote: > Daniele, > > Where could I get a hold of that patch? Thanks. They are in the devel branch in my git repository. https://github.com/dvarrazzo/psycopg -- Daniele
Daniele, Looks good. However, apparently that wasn't my problem. I am still getting the SystemError for the "null argument to internalroutine". Where would be a good place to look for another source of that error in psycopg? I am guessing that thatSystemError is a pretty generic one when it comes to python C extensions. Thanks. -eric -----Original Message----- From: Daniele Varrazzo [mailto:daniele.varrazzo@gmail.com] Sent: Friday, March 04, 2011 3:16 PM To: Eric Snow Cc: Jan Urbański; psycopg@postgresql.org Subject: Re: [psycopg] SystemError: null argument to internal routine On Fri, Mar 4, 2011 at 12:03 AM, Eric Snow <esnow@verio.net> wrote: > Thanks Jan, that helps explain that. So you are saying that the problem is in pq_raise ... I've fixed the unhandled exception in pq_raise and the handling of empty queries. Patches in my devel branch. Eric, is your problem related to the "empty query" result not handled or is it something else related to the copy command, that psycopg may still not handle correctly? Thanks, -- Daniele This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidentialor legally protected. If you are not the intended recipient or have received this message in error, you are notauthorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediatelyby return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that thisemail is error or virus free. Thank you.
On Mon, Mar 7, 2011 at 11:25 PM, Eric Snow <esnow@verio.net> wrote: > Daniele, > > Looks good. However, apparently that wasn't my problem. I am still getting the SystemError for the "null argument tointernal routine". Where would be a good place to look for another source of that error in psycopg? I am guessing thatthat SystemError is a pretty generic one when it comes to python C extensions. Thanks. If you have applied my patches, probably the problem is not in one of the pgstatus values not checked, as it would raise a different exception. You may build the library using PSYCOPG_DEBUG in setup.cfg, then run your test with the PSYCOPG_DEBUG environment variable set: this will spit out plenty of informations about what's currently happening into psycopg. Anyway, if you have managed to package a test to reproduce the bug, that would be more than enough for us to investigate. Thank you! -- Danele