Обсуждение: support for hyper-long transactions ...
at the moment i am working on an application which is supposed to run extremely large transactions (a lot of server side stored procedure stuff which can hardly be split into small transactions for visibility reasons).
so, from time to time it happens that i exceed my CommandCounter (> 2.000.000.000 statements inside the same transaction).
my idea is: how about adding a configure option to compile postgres with a 64 bit command counter. this would allow larger transactions for special purpose applications while it would not have an impact on normal applications.
comments are welcome ...
best regards,
hans
--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, 2700 Wiener Neustadt
Tel: +43/1/205 10 35 / 340
www.postgresql.at, www.cybertec.at
Hans-Juergen Schoenig wrote: > at the moment i am working on an application which is supposed to run > extremely large transactions (a lot of server side stored procedure > stuff which can hardly be split into small transactions for visibility > reasons). > so, from time to time it happens that i exceed my CommandCounter (> > 2.000.000.000 statements inside the same transaction). Interesting. What kind of a stored procedure is that? One optimization that might be possible is to not increment it for statements that do on-disk changes. > my idea is: how about adding a configure option to compile postgres with > a 64 bit command counter. this would allow larger transactions for > special purpose applications while it would not have an impact on normal > applications. One objection to expanding TransactionId to 64-bits has been that we depend on assignment of TransactionId to be atomic. That might not be an issue with CommandIds; I don't think we store commandids in shared memory structures. It still doesn't feel like a good idea to me, though. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Hans-Juergen Schoenig wrote: > at the moment i am working on an application which is supposed to run > extremely large transactions (a lot of server side stored procedure > stuff which can hardly be split into small transactions for visibility > reasons). > so, from time to time it happens that i exceed my CommandCounter (> > 2.000.000.000 statements inside the same transaction). AFAICS, maximum number of command ids is actually 2^32-1, or over 4 billion. Are you sure you bumped into that limit and not something else? What's the error message you're getting? What version of Postgres is this? PG 8.3 will have another related limit on the number of combocids you can have. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas írta: > Hans-Juergen Schoenig wrote: > >> at the moment i am working on an application which is supposed to run >> extremely large transactions (a lot of server side stored procedure >> stuff which can hardly be split into small transactions for visibility >> reasons). >> so, from time to time it happens that i exceed my CommandCounter (> >> 2.000.000.000 statements inside the same transaction). >> > > Interesting. What kind of a stored procedure is that? > > One optimization that might be possible is to not increment it for > statements that do on-disk changes. > > >> my idea is: how about adding a configure option to compile postgres with >> a 64 bit command counter. this would allow larger transactions for >> special purpose applications while it would not have an impact on normal >> applications. >> > > One objection to expanding TransactionId to 64-bits has been that we > depend on assignment of TransactionId to be atomic. That might not be an > issue with CommandIds; I don't think we store commandids in shared > memory structures. It still doesn't feel like a good idea to me, though. > Isn't 64-bit assignment atomic on 64-bit machines? With a little autoconf magic, the conditional can be disabled for 32-bit platforms. So that's not a real obstacle for going to 64-bit TransactionIds. -- ---------------------------------- Zoltán Böszörményi Cybertec Geschwinde & Schönig GmbH http://www.postgresql.at/
AFAICS, maximum number of command ids is actually 2^32-1, or over 4billion. Are you sure you bumped into that limit and not something else?What's the error message you're getting?What version of Postgres is this? PG 8.3 will have another related limiton the number of combocids you can have.
it is clearly caused by the CommandCounter - it is indicated by the error message. i don't have the message on my notebook here but this is exactly what is going on if we run out of statements.
i would not see atomic as a problem here as we can support it for 64 bit boxes only.
i would vote for some autoconf flag which is off by default to make sure that other applications don't waste space here.
to answer the question you had before:
it is an application going through some enormous amount of raw data and trying to do some filtering, analysis and preaggregation (which is not an issue here). the thing is that filtering and analysis are quite complex and have to be done on a per entry level (a lot of conditional lookups, if statements, custom aggregated, status changed and so on). if you are forced to do this fancy logic for 1xx mio records you can easily run out of commands.
many thanks,
hans
--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, 2700 Wiener Neustadt
Tel: +43/1/205 10 35 / 340
www.postgresql.at, www.cybertec.at