Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: ECPG FETCH readahead, was: Re: ECPG fixes

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: ECPG FETCH readahead, was: Re: ECPG fixes

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: ECPG FETCH readahead

ECPG FETCH readahead

Hi,

we improved ECPG quite a lot in 9.0 because we worked and
still working with an Informix to PostgreSQL migration project.

We came across a pretty big performance problem that can be seen in
every "naive" application that uses only FETCH 1, FETCH RELATIVE
or FETCH ABSOLUTE. These are almost the only FETCH variations
usable in Informix, i.e. it doesn't have the grammar for fetching N rows
at once. Instead, the Client SDK libraries do caching themselves
behind the scenes to reduce network turnaround time.

This is what we implemented for ECPG, so by default it fetches 256 rows
at once if possible and serves the application from memory. The number
of cached rows can be changed using the ECPGFETCHSZ environment
variable. The cursor readahead is activated by "ecpg -r fetch_readahead".

The implementation splits ECPGdo() and ecpg_execute() in ecpglib/execute.c
so the different parts are callable from the newly introduced cursor.c code.
Three new API calls are introduced: ECPGopen(), ECPGfetch() and
ECPGclose(). Obviously, OPEN and CLOSE use ECPGopen() and
ECPGclose(), respectively. They build and tear down the cache structures
besides calling the main ECPGdo() behind the scenes.

ECPGopen() also discovers the total number of records in the recordset,
so the previous ECPG "deficiency" (backend limitation) that sqlca.sqlerrd[2]
didn't report the (possibly estimated) number of rows in the resultset
is now
overcome. This slows down OPEN for cursors serving larger datasets
but it makes possible to position the readahead window using MOVE
ABSOLUTE no matter what FORWARD/BACKWARD/ABSOLUTE/RELATIVE
variants are used by the application. And the caching is more than
overweighs
the slowdown in OPEN it seems.

ECPGfetch() is the more interesting one, this handles FETCH and MOVE
statements and follows the absolute position of the cursor in the
client, too.

In Informix, the DECLARE statement is used for creating a cursor descriptor,
it can be OPENed/CLOSEd several times and the "FREE curname" statement
tears down the cursor descriptor. In our implementation, OPEN and CLOSE
sets up and tears down the caching structure, The DECLARE statement
didn't lose its declarative nature and the FREE statement is still only
usable
only for prepared statements. I chose this path because this way this
feature
can be used in native mode as well. It is usable even if the application
itself
uses FETCH N. The readahead window can be set externally to the
application to squeeze out more performance in batch programs.

The patch size is over 2MB because I introduced a new regression test
called fetch2.pgc that does a lot of work on a recordset having 400 rows.
It browses the recordset back and forth with:
- FETCH FORWARD 1/FETCH BACKWARD 1
- FETCH FORWARD 5/FETCH BACKWARD 5
- FETCH ABSOLUTE +N/FETCH ABSOLUTE -N
- FETCH FORWARD 3+MOVE FORWARD 7, also backwards
- FETCH RELATIVE +2/FETCH RELATIVE -2
This test is compiled both with and without "-r fetch_readahead", so
I was able to verify that the two runs produce the same output.
Also, fetch.pgc, dyntest.pgc and sqlda.pgc are also compiled with and
without "-r fetch_readahead", for verifying that both SQL and SQLDA
descriptors are working the same way as before. E.g. PGresult for
SQL descriptors are not simply assigned anymore, they are copied
using PQcopyResult() without tuples and a bunch of PQsetvalue() calls
to copy only the proper rows from the cache or all rows if no cache.

The split parts of ecpg_execute() are intentionally kept the original
wording (especially the "ecpg_execute" function name) in ecpg_log()
messages to eliminate any impact on other regression tests. If this is
not desired, a patch for this can come later.

Because of the patch size, the compressed version is attached.

Comments?

Best regards,
Zoltán Böszörményi

-- 
Bible has answers for everything. Proof:
"But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil." (Matthew 5:37) - basics of digital technology.
"May your kingdom come" - superficial description of plate tectonics

----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

2010-10-14 11:56 keltezéssel, Boszormenyi Zoltan írta:
> Hi,
>
> Robert Haas írta:
>> On Thu, Jun 24, 2010 at 8:19 AM, Michael Meskes  wrote:
>>   
>>> On Thu, Jun 24, 2010 at 12:04:30PM +0300, Heikki Linnakangas wrote:
>>>     
>>>> Is there a reason not to enable it by default? I'm a bit worried
>>>> that it will receive no testing if it's not always on.
>>>>       
>>> Yes, there is a reason, namely that you don't need it in native mode at all.
>>> ECPG can read as many records as you want in one go, something ESQL/C
>>> apparently cannot do. This patch is a workaround for that restriction. I still
>>> do not really see that this feature gives us an advantage in native mode
>>> though.
>>>     
>> So, who has the next action on this patch?  Does Zoltan need to revise
>> it, or does Michael need to review it, or where are we?
>>   
> Michael reviewed it shortly in private and I need to send
> a new revision anyway, regardless of his comments.
> I will refresh it ASAP.

The ASAP took a little long. The attached patch is in git diff format,
because (1) the regression test intentionally doesn't do ECPGdebug()
so the patch isn't dominated by a 2MB stderr file, so this file is empty
and (2) regular diff cannot cope with empty new files.

Anyway, here is the new version with a new feature.
Implementation details:

- New ecpglib/cursor.c handles the client-side accounting of cursors.
- Functions in ecpglib/execute.c are split into smaller functions
  so useful operations can be used by the new cursor.c
- ecpg -r fetch_readahead enables readahead by default for all cursors.
- Default readahead size is 256 rows, it can be modified by an environment
  variable, ECPGFETCHSZ.
- *NEW FEATURE* Readahead can be individually enabled or disabled
  by ECPG-side grammar:
        DECLARE curname [ [ NO ] READAHEAD ] CURSOR FOR ...
  Without [ NO ] READAHEAD, the default behaviour is used for cursors.
- Since the server and the client may disagree on the cursor position
  if readahead is used, ECPG preprocessor throws an error if
  WHERE CURRENT OF is used on such cursors.
- The default assumed behaviour of cursors with readahead is NO SCROLL.
  If you want readahead and backward scrolling, SCROLL keyword is mandatory.

The regression test creates a table with 513 rows, so it spans 2 full and
1 incomplete readahead window. It reads the table with two cursors, one
with readahead and one without by 1 records forward and backward.
This is repeated with reading 5 records at a time. Then the whole table is
read into a chain of sqlda structures forward and backward. All other
regression tests pass as well.

The original regression tests also pass with these changes, the split of
execute.c was risky in this regard. Now the split is done more cleanly
than in the previous version, the file is not as rearranged as before.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
     http://www.postgresql.at/

Re: ECPG FETCH readahead

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de    http://www.postgresql.at/

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

2012-03-15 21:59 keltezéssel, Robert Haas írta:
> I think we need either an updated version of this patch that's ready for commit real 
> soon now, or we need to postpone it to 9.3. 

Sorry for the delay, I had been busy with other tasks and I rewrote this code
to better cope with unknown result size, scrollable cursors and negative
cursor positions.

I think all points raised by Noah is addressed: per-cursor readahead window size,
extensive comments, documentation and not enabling result set size discovery.

Also, I noticed this in passing:

static void
free_variable(struct variable * var)
{
         struct variable *var_next;

         if (var == NULL)
                 return;
         var_next = var->next;
         ecpg_free(var);

         while (var_next)
         {
                 var = var_next;
                 var_next = var->next;
                 ecpg_free(var);
         }
}

I rewrote this as below to eliminate manual unrolling of the loop,
they are equivalent:

static void
free_variable(struct variable * var)
{
         struct variable *var_next;

         while (var)
         {
                 var_next = var->next;
                 ecpg_free(var);
                 var = var_next;
         }
}


The problem with WHERE CURRENT OF is solved by a little more grammar
and ecpglib code, which effectively does a MOVE ABSOLUTE N before
executing the DML with WHERE CURRENT OF clause. No patching of the
backend. This way, the new ECPG caching code is compatible with older
servers but obviously reduces the efficiency of caching.

Attached are two patches, the first one is the feature patch.

The second patch makes all cursor statements go through the new
caching functions with 1 tuple readahead window size. It only changes
the preprocessed code and stderr logs of the regression tests affected,
not their results.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig&  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead

2012-03-29 02:43 keltezéssel, Noah Misch írta:
> On Sat, Mar 24, 2012 at 10:49:07AM +0100, Boszormenyi Zoltan wrote:
>> Sorry for the delay, I had been busy with other tasks and I rewrote this code
>> to better cope with unknown result size, scrollable cursors and negative
>> cursor positions.
>>
>> I think all points raised by Noah is addressed: per-cursor readahead window size,
>> extensive comments, documentation and not enabling result set size discovery.
> The new comments are a nice improvement; thanks.
>
>> The problem with WHERE CURRENT OF is solved by a little more grammar
>> and ecpglib code, which effectively does a MOVE ABSOLUTE N before
>> executing the DML with WHERE CURRENT OF clause. No patching of the
>> backend. This way, the new ECPG caching code is compatible with older
>> servers but obviously reduces the efficiency of caching.
> Good plan.
>
>> diff -dcrpN postgresql.orig/doc/src/sgml/ecpg.sgml postgresql/doc/src/sgml/ecpg.sgml
>> *** postgresql.orig/doc/src/sgml/ecpg.sgml	2012-03-12 09:24:31.699560098 +0100
>> --- postgresql/doc/src/sgml/ecpg.sgml	2012-03-24 10:15:00.538924601 +0100
>> *************** EXEC SQL COMMIT;
>> *** 454,459 ****
>> --- 454,479 ----
>>       details.
>>      
>>
>> +
>> +    ECPG may use cursor readahead to improve performance of programs
>> +    that use single-row FETCH statements. Option-R number
>> +    option for ECPG modifies the default for all cursors fromNO READAHEAD
> This sentence duplicates the word "option".

Fixed.

>
>> +    toREADAHEAD number. ExplicitREADAHEAD number  or
>> +NO READAHEAD  turns cursor readahead on (withnumber
>> +    as the size of the readahead window) or off for the specified cursor,
>> +    respectively. For cursors using a non-0 readahead window size is 256 rows,
> The number 256 is no longer present in your implementation.

Indeed. Oversight, that part of the sentence is deleted.

>
>> +    the window size may be modified by setting theECPGFETCHSZ
>> +    environment variable to a different value.
> I had in mind that DECLARE statements adorned with READAHEAD syntax would
> always behave precisely as written, independent of ECPGFETCHSZ or "ecpg -R".
> Unadorned DECLARE statements would use ECPGFETCHSZ if set, then the value
> passed to "ecpg -R" if provided, and finally a value of 1 (no readahead) in
> the absence of both ECPGFETCHSZ and "ecpg -R".  Did you do it differently for
> a particular reason?  I don't particularly object to what you've implemented,
> but I'd be curious to hear your thinking.

What I had in mind was:

NO READAHEAD == READAHEAD 0. This will translate into 1 tuple sized
readahead window that cannot be modified by ECPGFETCHSZ.

READAHEAD 1 also means uncached by default but ECPGFETCHSZ may
modify the readahead window size.

Values greater than 1 means cached by default with the specified window size
but it can also be overridden (even disabled) just in case. This part can be debated,
I agree. This is in add_cursor():

-----8<-----8<-----8<-----8<-----8<-----8<-----
         desc->cachable = (ra_size > 0);
         /*
          * If this cursor is allowed to be cached, the readahead
          * windows is also allowed to be overridden (possibly
          * even disabled) by default_fetch_size if it's set.
          */
         if (desc->cachable)
                 desc->ra_size = (default_fetch_size > 0 ?
                                                 default_fetch_size :
                                                 ra_size);
         else
                 desc->ra_size = 1;
-----8<-----8<-----8<-----8<-----8<-----8<-----

The check (default_fetch_size > 0) can be modified so it reads
(default_fetch_size > ra_size) so the override can only happen upwards.

This part is policy that can be debated and modified accordingly,
it doesn't affect the internals of the caching code.

>
>> +
>> +
>> +
>> +UPDATE  orDELETE  with the
>> +WHERE CURRENT OF  clause, cursor readahead may or may not
>> +    actually improve performance, asMOVE  statement has to be
>> +    sent to the backend before the DML statement to ensure correct cursor position
>> +    in the backend.
>> +
> This sentence seems to be missing a word near its beginning.

Sounds like a corner case to me, but I am not a native english speaker.
Which one sounds better:

    UPDATE or DELETE with the WHERE CURRENT OF clause...

or

    UPDATE or DELETE statements with the WHERE CURRENT OF clause...
?

I went with the second. The committer can change the wording in any way
he wants.

>
>> *************** DECLAREc
>> *** 6639,6649 ****
>>          
>>         
>>        
>>
>>        
>> !      For the meaning of the cursor options,
>> !      see.
>>        
>>       
>>
>>       
>> --- 6669,6728 ----
>>          
>>         
>>        
>> +
>> +
>> +
>> +Cursor options
>>
>>        
>> !      For the meaning of other cursor options, see.
>>        
>> +
>> +
>> +
>> +READAHEAD number
>> +NO READAHEAD
>> +
>> +
>> +READAHEAD number  makes the ECPG preprocessor and
>> +          runtime library use a client-side cursor accounting and data readahead
>> +          duringFETCH. This improves performance for programs
>> +          that use single-rowFETCH  statements.
>> +
>> +
>> +
>> +NO READAHEAD  disables data readahead in case
>> +-R number  is used for compiling the file.
>> +
>> +
>> +
> One of the new sections about readahead should somehow reference the hazard
> around volatile functions.

Done.

>> +
>> +
>> +OPEN RETURNS LAST ROW POSITION
>> +OPEN RETURNS 0 FOR LAST ROW POSITION
>> +
>> +
>> +          When the cursor is opened, it's possible to discover the size of the result set
>> +          usingMOVE ALL  which traverses the result set and move
>> +          the cursor back to the beginning usingMOVE ABSOLUTE 0.
>> +          The size of the result set is returned in sqlca.sqlerrd[2].
>> +
>> +
>> +
>> +          This slows down opening the cursor from the application point of view
>> +          but may also have side effects. If the cursor query contains volatile function
>> +          calls with side effects, they will be evaluated twice because of traversing
>> +          the result set this way duringOPEN.
>> +
>> +
>> +
>> +          The default is not to discover.
>> +
> I mildly oppose having syntax to enable this per-cursor.  Readahead is a
> generally handy feature that I might use in new programs, but this feature is
> more of a hack for compatibility with some old Informix version.  For new
> code, authors should do their own MOVEs instead of using this option.
>
> The patch also adds an undocumented ECPGOPENRETURNSRESULTSETSIZE environment
> variable to control this.  I see no use for such a thing, because a program's
> dependency on sqlerrd[2] is fixed at build time.  If a program does not
> reference sqlerrd[2] for a cursor, there's no benefit from choosing at runtime
> to populate that value anyway.  If a program does reference it, any option
> needed to have it set correctly had better be set at build time and apply to
> every run of the final program.
>
> IOW, I suggest having only the "ecpg"-time option to enable this behavior.
> Thoughts?

I wanted to make it available for non-Informix mode and a way to
disable it if the command line option enables it.

But the extra environment variable seems to be silly indeed.
I deleted that part of the code.

>> +
>> +
>> +
>> +
>> +
>>       
>>
>>       
>> diff -dcrpN postgresql.orig/doc/src/sgml/ref/ecpg-ref.sgml postgresql/doc/src/sgml/ref/ecpg-ref.sgml
>> *** postgresql.orig/doc/src/sgml/ref/ecpg-ref.sgml	2011-08-07 11:29:16.003256959 +0200
>> --- postgresql/doc/src/sgml/ref/ecpg-ref.sgml	2012-03-24 10:13:08.679284063 +0100
>> *************** PostgreSQL documentation
>> *** 171,176 ****
>> --- 171,196 ----
>>        
>>
>>        
>> +
>> +
>> +
>> +        Turn on cursor readahead by default usingnumber
>> +        as readahead window size.
>> +
>> +
>> +
>> +
>> +
>> +
>> +
>> +
>> +        Detect the cursor result set size duringOPEN  and
>> +        return it in sqlca.sqlerrd[2].
> Make reference to this option in the ecpg-sqlca section, where sqlerrd[2] has
> its primary documentation.

Done.

>> +
>> +
>> +
>> +
>> +
>>         
>>         
>>          
>
> I did not re-review the code changes in the same detail as before, but nothing
> caught my attention when scanning through them.  If some particular section
> has changed in a tricky way and deserves a close look, let me know.

Well, the extra comments should explain everything. The rewrite did not change
the execute.c function split, only comments were added. The changes in cursor.c
were mainly about being able to properly deal with negative cursor positions,
i.e. traversing the cursor backwards, e.g. with FETCH LAST or FETCH ABSOLUTE -N.

> The documentation-cosmetic and policy questions I raise above shouldn't entail
> large-scale patch churn, so I've marked the patch Ready for Committer.

Thanks. The patch with the above changes is attached.
Also attached the second patch from the previous mail,
in case the committer wants to consider it.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig&  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

2012-03-29 19:03 keltezéssel, Noah Misch írta:
> To me, it feels too magical to make READAHEAD 1 and READAHEAD 0 differ only in
> whether ECPGFETCHSZ can override.  I'm willing to go with whatever consensus
> arises on this topic, though.

I now removed this distinction because the grammar does it so
the value 0 doesn't even reach ECPGopen().

>> This part is policy that can be debated and modified accordingly,
>> it doesn't affect the internals of the caching code.
> Agreed.
>
>>>> +
>>>> +
>>>> +
>>>> +UPDATE   orDELETE   with the
>>>> +WHERE CURRENT OF   clause, cursor readahead may or may not
>>>> +    actually improve performance, asMOVE   statement has to be
>>>> +    sent to the backend before the DML statement to ensure correct cursor position
>>>> +    in the backend.
>>>> +
>>> This sentence seems to be missing a word near its beginning.
>> Sounds like a corner case to me, but I am not a native english speaker.
>> Which one sounds better:
>>
>>     UPDATE or DELETE with the WHERE CURRENT OF clause...
>>
>> or
>>
>>     UPDATE or DELETE statements with the WHERE CURRENT OF clause...
>> ?
> Here is the simplest change to make the original grammatical:
>
>    Given an UPDATE or DELETE with the WHERE CURRENT OF clause, ...
>
> I might write the whole thing this way:
>
>    To execute an UPDATE or DELETE statement bearing the WHERE CURRENT OF
>    clause, ECPG may first execute an implicit MOVE to synchronize the server
>    cursor position with the local cursor position.  This can reduce or
>    eliminate the performance benefit of readahead for affected cursors.

OK, I used your text.

>>> one of the new sections about readahead should somehow reference the hazard
>>> around volatile functions.
>> Done.
> I don't see the mention in your latest patch.  You do mention it for the
> sqlerrd[2] compatibility stuff.

sqlerrd[2] compatibility stuff? I mentioned it in section "ecpg-sqlca", this is the main
documentation section, not the compatibility one AFAIK. Anyway, I now reference the volatile
function hazard in the first paragraphs added to section "ecpg-cursors".


> I can understand the desire to have a way to disable it.  Perhaps you have a
> bunch of old Informix code, so you enable the global option and later discover
> some expensive cursors that don't use sqlerrd[2].  It would be nice to locally
> suppress the behavior for those cursors.
>
> Still, we're looking at dedicated ECPG syntax, quite visible even to folks
> with no interest in Informix.  We have eschewed littering our syntax with
> compatibility aids, and I like it that way.  IMO, an option to the "ecpg"
> preprocessor is an acceptable level of muddle to provide this aid.  A DECLARE
> CURSOR syntax extension goes too far.

I have now deleted this grammar extension.

Attached is the new core feature patch. Summary of changes:
- documentation changes according to your query
- cursor.c doesn't distinguish between 0 and 1, this is the job of the grammar.
- I saved some MOVE statements in case the cursor position in the backend
   is what we want at the moment. This means READAHEAD 1 is at the same level
   of performance like without caching, there are no extra implicit MOVEs.
   WHERE CURRENT OF also avoids MOVEs in this case. The caching code is now
   clearly faster in case when there are a lot of MOVEs without actual FETCHes.
   This is only an if () condition around the previously coded MOVE block in
   ecpg_cursor_execute(). It doesn't change the semantics, only omits the
   MOVE in case it's not needed.

I also refreshed the second patch that drives all cursors with the new
cursor functions. Because the runtime doesn't handle 0, 1 is the default
readahead window size. Now, NO READAHEAD would mean READAHEAD 1
but ECPGFETCHSZ can also override it. This means that NO READAHEAD
makes no sense, so it's removed from the grammar and the documentation
is changed accordingly.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig&  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Hi,

2012-04-10 16:55 keltezéssel, Michael Meskes írta:
> On Tue, Apr 10, 2012 at 10:37:22AM -0400, Noah Misch wrote:
>>> Only a non-decorated cursor can be overridden, even if
>>> a different default readahead window size is specified with
>>> e.g. "ecpg -R 8". If ECPGFETCHSZ is not present, 8 will be used,
>>> if ECPGFETCHSZ is present, its value will be used. ECPGopen()
>>> will need an extra bool argument to distinguish this.
>>>
>>> Is this acceptable? Noah, Michael?
>> Sounds perfect.
> Fine by me.
>
> Michael

you commented on "two new options were added and they should
be suboptions to -r". I looked at "man getopt_long" to see what I can do
about the "-R" option and there seems to be a getsubopt() call which is
an extension to getopt_long. My manpage under Fedora 16 says this:

NAME
        getsubopt - parse suboption arguments from a string

SYNOPSIS
        #include 

        int getsubopt(char **optionp, char * const *tokens, char **valuep);

    Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

        getsubopt():
            _XOPEN_SOURCE >= 500 || _XOPEN_SOURCE && _XOPEN_SOURCE_EXTENDED
            || /* Since glibc 2.12: */ _POSIX_C_SOURCE >= 200809L

I wonder whether the manual parsing of "-r" suboptions may be rewritten
using this function or PostgreSQL supports systems without the above
X/Open or POSIX support levels.

Anyway, to make it possible to rewrite using the above call, I modified "-R"
and it's now "-r readahead=number". Documentation is adjusted.

With the above, it would be possible to use a comma separated list of "-r"
suboptions, e.g. "-r prepare,questionmarks,readahead=16" in one option.

Summary of other changes:
- The result set size detection is a suboption of "-r", documentation is adjusted.
- Only undecorated cursors use ECPGFETCHSZ, documentation is adjusted
- "ecpg --help says ...default 0 (disabled)..." fixed.
- Comment in cursor-readahead.pgc is fixed.
- New regression test that exercises ECPGFETCHSZ=8 and a "non-readahead"
   cursor. The stderr file shows the "fetch forward 8" executed by the runtime.
- Also added a note to the documentation about a possible performance trap
   if a previously written ECPG application uses its own custom readahead via
   multi-row FETCH statements.

This patch should be applied over the two patches I last sent.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig&  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Hi,

I am restarting this old thread... :-)

2012-04-24 10:17 keltezéssel, Michael Meskes írta:
>> OK, I will implement #2. Another question popped up: what to do
>> with FETCH ALL? The current readahead window size or temporarily
>> bumping it to say some tens of thousands can be used. We may not
>> know how much is the "all records". This, although lowers performance,
>> saves memory.
> I would say doing a large fetch in two or three batches won't cost too much in
> terms of performance.
>
>> Please, don't apply this patch yet. I discovered a rather big hole
>> that can confuse the cursor position tracking if you do this:
>> ...
>> That will also need a new round of review. Sorry for that.
> No problem, better to find it now instead of after release.
>
> Anyway, I moved the patch to 2012-next (I hope I did it correctly) so 2012-1
> can be closed. Let's try to get this patch done in the next commit fest.
>
> Michael

I had time to look into this patch of mine again after about 1.5 years.
Frankly, this time was too long to remember every detail of the patch
and looking at parts of the patch as a big entity was confusing.

So I started fresh and to make review easier, I broke the patch up
into small pieces that all build on each other. I have also fixed quite
a few bugs, mostly in my code, but some in the ECPG parser and
the regression tests as well.

I have put the broken up patchset into a GIT tree of mine at GitHub:
https://github.com/zboszor/ecpg-readahead/
but the huge compressed patch is also attached for reference.
It was generated with

$ git diff 221e92f64c6e136e550ec2592aac3ae0d4623209..870922676e6ae0faa4ebbf94b92e0b97ec418e16

ECPG regression tests are now Valgrind-clean except two of them
but both are pre-existing bugs.

1. ecpg/test/compat_informix/rfmtlong.pgc points out a problem in
    ecpg/compatlib/informix.c

==5036== 1 errors in context 1 of 4:
==5036== Invalid read of size 4
==5036==    at 0x4E3453C: rfmtlong (informix.c:941)
==5036==    by 0x4007DA: fmtlong.constprop.0 (rfmtlong.pgc:22)
==5036==    by 0x4006BE: main (rfmtlong.pgc:45)
==5036==  Address 0x60677d8 is 24 bytes inside a block of size 25 alloc'd
==5036==    at 0x4C28409: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==5036==    by 0x4E34268: rfmtlong (informix.c:783)
==5036==    by 0x4007DA: fmtlong.constprop.0 (rfmtlong.pgc:22)
==5036==    by 0x4006BE: main (rfmtlong.pgc:45)

The same error is reported 4 times.

2. ecpg_add_mem() seems to leak memory:

==5463== 256 bytes in 16 blocks are definitely lost in loss record 1 of 1
==5463==    at 0x4C2A121: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==5463==    by 0x4E3E153: ecpg_alloc (memory.c:21)
==5463==    by 0x4E3E212: ecpg_add_mem (memory.c:110)
==5463==    by 0x4E3542B: ecpg_store_result (execute.c:409)
==5463==    by 0x4E37E5A: ecpg_process_output (execute.c:1777)
==5463==    by 0x4E38CCA: ecpg_do (execute.c:2137)
==5463==    by 0x4E38D8A: ECPGdo (execute.c:2159)
==5463==    by 0x400A82: fn (alloc.pgc:51)
==5463==    by 0x5152C52: start_thread (pthread_create.c:308)
==5463==    by 0x545C13C: clone (clone.S:113)

The last two issue we talked about in this thread are also implemented:
- permanently raise the readahead window if the application sends a
   bigger FETCH command, and
- temporarily raise the readahead window for FETCH ALL commands

The cursor position tracking was completely rewritten, so the client side
properly follows the cursor position known by the backend and doesn't
skip MOVE statements where it shouldn't. The previously known
bug is completely eliminated this way.

Please, review that patch.

Thanks in advance and best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

New ECPG idea, was: Re: ECPG FETCH readahead

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de    http://www.postgresql.at/

Re: ECPG FETCH readahead

2013-08-17 12:08 keltezéssel, Boszormenyi Zoltan írta:
>
> I have put the broken up patchset into a GIT tree of mine at GitHub:
> https://github.com/zboszor/ecpg-readahead/
> but the huge compressed patch is also attached for reference.

I merged current PG GIT HEAD in the above tree and fixed a merge conflict
caused by commit 673b527534893a4a8adb3cdef52fc645c13598ce

The huge patch is attached for reference.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead

2013-09-07 03:25 keltezéssel, Peter Eisentraut írta:
> On Wed, 2013-09-04 at 10:06 +0200, Boszormenyi Zoltan wrote:
>> 2013-08-17 12:08 keltezéssel, Boszormenyi Zoltan írta:
>>> I have put the broken up patchset into a GIT tree of mine at GitHub:
>>> https://github.com/zboszor/ecpg-readahead/
>>> but the huge compressed patch is also attached for reference.
>> I merged current PG GIT HEAD in the above tree and fixed a merge conflict
>> caused by commit 673b527534893a4a8adb3cdef52fc645c13598ce
>>
>> The huge patch is attached for reference.
> The documentation doesn't build:
>
> openjade:ecpg.sgml:478:8:E: end tag for "LITERAL" omitted, but OMITTAG NO was specified
> openjade:ecpg.sgml:477:40: start tag was here
> openjade:ecpg.sgml:478:8:E: end tag for "LITERAL" omitted, but OMITTAG NO was specified
> openjade:ecpg.sgml:477:20: start tag was here
> openjade:ecpg.sgml:478:8:E: end tag for "LITERAL" omitted, but OMITTAG NO was specified
> openjade:ecpg.sgml:473:81: start tag was here
> openjade:ecpg.sgml:478:8:E: end tag for "LITERAL" omitted, but OMITTAG NO was specified
> openjade:ecpg.sgml:473:56: start tag was here

Thanks, fixed.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead

2013-09-10 03:04 keltezéssel, Peter Eisentraut írta:
> You need to update the dblink regression tests.

Done.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Proposal: UPDATE/DELETE ... WHERE OFFSET n OF cursor_name, was: Re: New ECPG idea, was: Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead

2013-11-12 07:01 keltezéssel, Noah Misch írta:
> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>> The old contents of my GIT repository was removed so you need to
>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>> I won't post the humongous patch again, since sending a 90KB
>> compressed file to everyone on the list is rude.
> Patches of that weight show up on a regular basis.  I don't think it's rude.

OK, here it is.

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG fixes, was: Re: ECPG FETCH readahead

2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>> The old contents of my GIT repository was removed so you need to
>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>> I won't post the humongous patch again, since sending a 90KB
>>> compressed file to everyone on the list is rude.
>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>
> OK, here it is.

I have rebased the patchset after "ecpg: Split off mmfatal() from mmerror()"
since it caused merge conflicts.

It's at the usual place again, you need to clone it from scratch if you are
interested in looking at git diff/log

I have removed some previous ecpg_log() debug output and
the total patch size is not so huge any more but I am going to post
the split-up set in parts.

Attached is the first few patches that are strictly generic ECPG fixes.
They can be applied independently and obvious enough.

Subsequent patches will come as reply to this email.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Modify the DECLARE CURSOR command tag depending on the scrollable flag

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

Attached is the patch that modified the command tag returned by
the DECLARE CURSOR command. It returns "DECLARE SCROLL CURSOR"
or "DECLARE NO SCROLL CURSOR" depending on the cursor's
scrollable flag that can be determined internally even if neither is
asked explicitly.

It is expected by the ECPG cursor readahead code.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG infrastructure changes part 1, was: Re: ECPG fixes

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

Infrastructure changes in ecpglib/execute.c to split up
ECPGdo and ecpg_execute() and expose the parts as
functions internal to ecpglib.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG infrastructure changes, part 2, was: Re: ECPG fixes

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

ecpg_log() fixes after part 1 that produces a lot of regression test changes.
This patch is over 200K in itself so I send it separately and compressed.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG infrastructure changes, part 3, was: Re: ECPG fixes

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

Further ecpglib/execute.c changes.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG infrastructure changes, part 4, was: Re: ECPG fixes

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

This is another, semi independent subfeature of ECPG readahead.
It's about 150K by itself, so I send it compressed.

The purpose of this patch is to track (sub-)transactions and cursors
in ecpglib to reduce network turnaround and speed up the application
in certain cases. E.g. cursors are discarded upon ROLLBACK TO
SAVEPOINT and ecpglib needs to know about it. When an unknown
savepoint or cursor name is sent, ecpglib would not send the command
to the server in an open transaction after this patch. Instead, it flips
a "client-side error" flag and returns the same error the backend
would in this case.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG infrastructure changes, part 5, was: Re: ECPG fixes

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

Patch 24 slightly speeds up ECPGsetcommit() at runtime, shifting
the strcmp() cost to the ecpg preprocessor. It also avoids the
confusion coming from a manually crafted C source that sends
a string different from "on" and "off".

Patch 25 is another subfeature of the ECPG readahead code.
ECPGopen() gets the user-specified (or unspecified) scrollable flag.
ECPGfetch() gets the amount of tuples to fetch separately as
a fixed value, which may or may not be also present in the variable
argument list. It makes it unnecessary to parse the FETCH/MOVE
query and extract the amount of tuples that way.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

ECPG FETCH readahead, was: Re: ECPG fixes

2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>> The old contents of my GIT repository was removed so you need to
>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>> I won't post the humongous patch again, since sending a 90KB
>>>> compressed file to everyone on the list is rude.
>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>
>> OK, here it is.
>
> ...
> Subsequent patches will come as reply to this email.

$SUBJECT, the real thing. Needs all previous patches.
Compared the the previous incarnation, a lot of debugging
ecpg_log() lines are removed. The newly introduced regression
tests have much smaller .stderr outputs because of this,
the patch is about 800K smaller in total.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Followup patches for ECPG readahead, was: Re: ECPG FETCH readahead

2013-11-20 15:25 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>>> The old contents of my GIT repository was removed so you need to
>>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>>> I won't post the humongous patch again, since sending a 90KB
>>>>> compressed file to everyone on the list is rude.
>>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>>
>>> OK, here it is.
>>
>> ...
>> Subsequent patches will come as reply to this email.
>

Followup patches/subfeatures for the ECPG readahead code.

The last (31st) patch drives all cursors through the readahead code
and changes a lot of regression tests. It's over 100K so I send it compressed.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de    http://www.postgresql.at/

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: ECPG infrastructure changes part 1, was: Re: ECPG fixes

2013-11-20 14:53 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>>> The old contents of my GIT repository was removed so you need to
>>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>>> I won't post the humongous patch again, since sending a 90KB
>>>>> compressed file to everyone on the list is rude.
>>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>>
>>> OK, here it is.
>>
>> ...
>> Subsequent patches will come as reply to this email.
>
> Infrastructure changes in ecpglib/execute.c to split up
> ECPGdo and ecpg_execute() and expose the parts as
> functions internal to ecpglib.

Rebased after killing the patch that changed the DECLARE CURSOR command tag.
All infrastructure patches are attached, some of them compressed.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead, was: Re: ECPG fixes

2013-11-20 15:25 keltezéssel, Boszormenyi Zoltan írta:
> 2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>>> The old contents of my GIT repository was removed so you need to
>>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>>> I won't post the humongous patch again, since sending a 90KB
>>>>> compressed file to everyone on the list is rude.
>>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>>
>>> OK, here it is.
>>
>> ...
>> Subsequent patches will come as reply to this email.
>
> $SUBJECT, the real thing. Needs all previous patches.
> Compared the the previous incarnation, a lot of debugging
> ecpg_log() lines are removed. The newly introduced regression
> tests have much smaller .stderr outputs because of this,
> the patch is about 800K smaller in total.

Rebased after killing the patch that changed the DECLARE CURSOR command tag.
The ECPG readahead patch and all the small following patches are attached.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

2013-12-03 16:48 keltezéssel, Antonin Houska írta:
> 22.patch
> --------
>
> tuples_left > 0
>
> instead of just
>
> tuples_left
>
> seems to me safer in for-loops. Not sure if there's a convention for
> this though.

I decided not to change this. The "tuples_left" variable starts at
a non-negative value and decreased by one in every iteration.
The previous code also used the same "test for non-zero" test with
the variable "ntuples".

> 23.patch
> --------
>
> git apply --verbose ~/cybertec/patches/ecpq/23.patch
> /home/anton/cybertec/patches/ecpq/23.patch:494: space before tab in indent.
>                          /*------
> /home/anton/cybertec/patches/ecpq/23.patch:495: space before tab in indent.
>                             translator: this string will be truncated at
> 149 characters expanded.  */
> /home/anton/cybertec/patches/ecpq/23.patch:4019: trailing whitespace.
>          exec sql close :curname;

I fixed this in this patch and all subsequent ones.

> Tests - 23.patch
> ----------------
>
> src/interfaces/ecpg/test/sql/cursorsubxact.pgc
>
>
>          /*
>           * Test the implicit RELEASE SAVEPOINT if a SAVEPOINT
>           * is used with an already existing name.
>           */
>
> Shouldn't it be "... if a CURSOR is used with an already existing
> name?". Or just "... implicit RELEASE SAVEPOINT after an error"?
> I'd also appreciate a comment where exactly the savepoint is
> (implicitly) released.

I have already answered this in my previous answer.

All patches are attached again for completeness.

> 23.patch and 24.patch
> ---------------------
>
> SO_MAJOR_VERSION and also interfaces/ecpg/include/ecpglib.h is changed
>
> Thus all client applications probably need to be preprocessed & compiled
> against the PG 9.4. I don't know how this is usually enforced. I'm
> mentioning it for the sake of completeness.
>
> // Antonin Houska (Tony)
>
>
> On 11/28/2013 03:21 PM, Boszormenyi Zoltan wrote:
>> 2013-11-20 14:53 keltezéssel, Boszormenyi Zoltan írta:
>>> 2013-11-20 14:41 keltezéssel, Boszormenyi Zoltan írta:
>>>> 2013-11-12 07:15 keltezéssel, Boszormenyi Zoltan írta:
>>>>> 2013-11-12 07:01 keltezéssel, Noah Misch írta:
>>>>>> On Mon, Nov 11, 2013 at 10:17:54AM +0100, Boszormenyi Zoltan wrote:
>>>>>>> The old contents of my GIT repository was removed so you need to
>>>>>>> clone it fresh. https://github.com/zboszor/ecpg-readahead.git
>>>>>>> I won't post the humongous patch again, since sending a 90KB
>>>>>>> compressed file to everyone on the list is rude.
>>>>>> Patches of that weight show up on a regular basis.  I don't think it's rude.
>>>>> OK, here it is.
>>>> ...
>>>> Subsequent patches will come as reply to this email.
>>> Infrastructure changes in ecpglib/execute.c to split up
>>> ECPGdo and ecpg_execute() and expose the parts as
>>> functions internal to ecpglib.
>> Rebased after killing the patch that changed the DECLARE CURSOR command tag.
>> All infrastructure patches are attached, some of them compressed.
>>
>> Best regards,
>> Zoltán Böszörményi
>>
>>
>>
>>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead, was: Re: ECPG fixes

2013-11-28 15:23 keltezéssel, Boszormenyi Zoltan írta:
> Rebased after killing the patch that changed the DECLARE CURSOR command tag.
> The ECPG readahead patch and all the small following patches are attached.

Fixed the extra spaces that "git apply" complains about.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

2013-12-04 14:51 keltezéssel, Boszormenyi Zoltan írta:
> 2013-12-03 16:48 keltezéssel, Antonin Houska írta:
>
>> Tests - 23.patch
>> ----------------
>>
>> src/interfaces/ecpg/test/sql/cursorsubxact.pgc
>>
>>
>>          /*
>>           * Test the implicit RELEASE SAVEPOINT if a SAVEPOINT
>>           * is used with an already existing name.
>>           */
>>
>> Shouldn't it be "... if a CURSOR is used with an already existing
>> name?". Or just "... implicit RELEASE SAVEPOINT after an error"?
>> I'd also appreciate a comment where exactly the savepoint is
>> (implicitly) released.
>
> I have already answered this in my previous answer.

And I was wrong in that. The comments in the test were rearranged
a little and the fact in the above comment is now actually tested.

Some harmless unused variables were also removed and an
uninitialized variable usage was fixed. Because of these and the above
changes a lot of patches need to be rebased.

All patches are attached again for completeness.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ECPG FETCH readahead, was: Re: ECPG fixes

Rebased patches after the regression test and other details were fixed
in the infrastructure part.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: ECPG FETCH readahead, was: Re: ECPG fixes

Re: Review: ECPG infrastructure changes part 1, was: Re: ECPG fixes

Re: ECPG FETCH readahead, was: Re: ECPG fixes

ECPG FETCH readahead

Hi,


Alvaro Herrera wrote:
> Boszormenyi Zoltan escribió:
>> Rebased patches after the regression test and other details were fixed
>> in the infrastructure part.
>
> This thread started in 2010, and various pieces have been applied
> already and some others have changed in nature.  Would you please post a
> new patchset, containing rebased patches that still need application, in
> a new email thread to be linked in the commitfest entry?

I hope Thunderbird did the right thing and didn't include the reference
message ID when I told it to "edit as new". So supposedly this is a new
thread but with all the cc: addresses kept.

I have rebased all remaining patches and kept the numbering.
All the patches from 18 to 25 that were previously in the
"ECPG infrastructure" CF link are included here.

There is no functional change.

Best regards,
Zoltán Böszörményi

Re: ECPG FETCH readahead

2014-01-18 18:08 keltezéssel, Boszormenyi Zoltan írta:
> Hi,
>
>
> Alvaro Herrera wrote:
>> Boszormenyi Zoltan escribió:
>>> Rebased patches after the regression test and other details were fixed
>>> in the infrastructure part.
>>
>> This thread started in 2010, and various pieces have been applied
>> already and some others have changed in nature.  Would you please post a
>> new patchset, containing rebased patches that still need application, in
>> a new email thread to be linked in the commitfest entry?
>
> I hope Thunderbird did the right thing and didn't include the reference
> message ID when I told it to "edit as new". So supposedly this is a new
> thread but with all the cc: addresses kept.
>
> I have rebased all remaining patches and kept the numbering.
> All the patches from 18 to 25 that were previously in the
> "ECPG infrastructure" CF link are included here.
>
> There is no functional change.

Because of the recent bugfixes in the ECPG area, the patchset
didn't apply cleanly anymore. Rebased with no functional change.

Best regards,
Zoltán Böszörményi

Re: ECPG FETCH readahead

Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Review: ECPG FETCH readahead

Re: Modify the DECLARE CURSOR command tag depending on the scrollable flag

Re: ECPG FETCH readahead

Re: ECPG FETCH readahead