Обсуждение: [QUESTIONS] lo_write cannot > 640Kb? memory leaks?

Поиск
Список
Период
Сортировка

[QUESTIONS] lo_write cannot > 640Kb? memory leaks?

От
"Park, Chul-Su"
Дата:
Hi all,

    Can anybody pinpoint where is wrong on lo_write(and possibly
lo_read)?  I tried to lo_import(and lo_write)  few MegaBytes of data
with large_object but it was painfully SLOW!  So I simplified the
problem, just test "testlo.c" under test/examples directory

    with few MegaBytes of file X.dat

    and printout debug messages like this with lo_import ()->
importFile() in main()

    IT CANNOT LO_WRITE > 640KB !!!  WHY?

testlo.c-------------------
...
in importFile()
{
...
//    with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e.  1K, 8K, 64K, ...
 /*
  * read in from the Unix file and write to the inversion file
  */
 while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
 {
   printf("COUNT [%d]   %d Kb\n", ++count, count*BUFSIZE/1024);
  tmp = lo_write(conn, lobj_fd, buf, nbytes);
...
if (tmp < nbytes)
  {
   fprintf(stderr, "error while reading \"%s\"", filename);
  }
  printf("READ %d , WRITE %d\n", nbytes, tmp);
_________________________

./testlo template1 X.dat out

....
importing file "in" ...
COUNT [1]   0 Kb
READ 65536 , WRITE 65536
COUNT [2]   64 Kb
...
READ 65536 , WRITE 65536
COUNT [9]   512 Kb
error while reading "in"READ 65536 , WRITE -1
COUNT [10]   576 Kb
error while reading "in"READ 65536 , WRITE -1
COUNT [11]   640 Kb
________________________________________________

and "strace ./testlo template1 in out" shows

...
write(3, "F \0", 3)                     = 3
write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512
write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20
read(4, "EERROR:  no empty local buffer.\n"..., 1024) = 33
write(2, "error while reading \"in\"", 24error while reading "in") = 24
read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536
write(1, "COUNT [10]   576 Kb\n", 20COUNT [10]   576 Kb
)   = 20
write(3, "F \0", 3)                     = 3
write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512
write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20
read(4, "EERROR:  cannot read block 62 of"..., 1024) = 45
write(2, "error while reading \"in\"", 24error while reading "in") = 24
read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536
write(1, "COUNT [11]   640 Kb\n", 20COUNT [11]   640 Kb
...
___________________________________________________________

So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop,
it solved the problem, which means MEM LEAKS or buffer size problem?
And just
   lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ);
...
 /*
  * read in from the Unix file and write to the inversion file
  */
 while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
 {
   printf("COUNT [%d]   %d Kb\n", ++count, count*BUFSIZE/1024);
   o_lseek(conn, lobj_fd, 0, SEEK_END);
   tmp = lo_write(conn, lobj_fd, buf, nbytes);
...

Also solved the problem...  But I want reliable(lo_read/lo_write) large
object with BIG data.

Please Help me out.

C.S.Park

Re: [HACKERS] [QUESTIONS] lo_write cannot > 640Kb? memory leaks?

От
dg@illustra.com (David Gould)
Дата:
> Hi all,
>
>     Can anybody pinpoint where is wrong on lo_write(and possibly
> lo_read)?  I tried to lo_import(and lo_write)  few MegaBytes of data
> with large_object but it was painfully SLOW!  So I simplified the
> problem, just test "testlo.c" under test/examples directory
>
>     with few MegaBytes of file X.dat
>
>     and printout debug messages like this with lo_import ()->
> importFile() in main()
>
>     IT CANNOT LO_WRITE > 640KB !!!  WHY?
>
> testlo.c-------------------
> ...
> in importFile()
> {
> ...
> //    with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e.  1K, 8K, 64K, ...
>  /*
>   * read in from the Unix file and write to the inversion file
>   */
>  while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
>  {
>    printf("COUNT [%d]   %d Kb\n", ++count, count*BUFSIZE/1024);
>   tmp = lo_write(conn, lobj_fd, buf, nbytes);
> ...
> if (tmp < nbytes)
>   {
>    fprintf(stderr, "error while reading \"%s\"", filename);
>   }
>   printf("READ %d , WRITE %d\n", nbytes, tmp);
> _________________________
>
> ./testlo template1 X.dat out
>
> ....
> importing file "in" ...
> COUNT [1]   0 Kb
> READ 65536 , WRITE 65536
> COUNT [2]   64 Kb
> ...
> READ 65536 , WRITE 65536
> COUNT [9]   512 Kb
> error while reading "in"READ 65536 , WRITE -1
> COUNT [10]   576 Kb
> error while reading "in"READ 65536 , WRITE -1
> COUNT [11]   640 Kb
> ________________________________________________
>
> and "strace ./testlo template1 in out" shows
>
> ...
> write(3, "F \0", 3)                     = 3
> write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
> write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512
> write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20
> read(4, "EERROR:  no empty local buffer.\n"..., 1024) = 33
> write(2, "error while reading \"in\"", 24error while reading "in") = 24
> read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536
> write(1, "COUNT [10]   576 Kb\n", 20COUNT [10]   576 Kb
> )   = 20
> write(3, "F \0", 3)                     = 3
> write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024
> write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512
> write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20
> read(4, "EERROR:  cannot read block 62 of"..., 1024) = 45
> write(2, "error while reading \"in\"", 24error while reading "in") = 24
> read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536
> write(1, "COUNT [11]   640 Kb\n", 20COUNT [11]   640 Kb
> ...
> ___________________________________________________________
>
> So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop,
> it solved the problem, which means MEM LEAKS or buffer size problem?
> And just
>    lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ);
> ...
>  /*
>   * read in from the Unix file and write to the inversion file
>   */
>  while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
>  {
>    printf("COUNT [%d]   %d Kb\n", ++count, count*BUFSIZE/1024);
>    o_lseek(conn, lobj_fd, 0, SEEK_END);
>    tmp = lo_write(conn, lobj_fd, buf, nbytes);
> ...
>
> Also solved the problem...  But I want reliable(lo_read/lo_write) large
> object with BIG data.
>
> Please Help me out.
>
> C.S.Park

Have you tried increasing the number of postgres buffer cache buffers? I am
speculating that the lo_write() is using these buffers but perhaps not
flushing them. As the default configuration has way too few buffers anyway
this might be the problem. Could you try increasing the buffers to say 1024 or
so and rerun your test and post the results?

Thanks,
-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
 and someone who knows less will correct me if I'm right."
               --David Palmer (palmer@tybalt.caltech.edu)

Re: [HACKERS] [QUESTIONS] lo_write cannot > 640Kb? memory leaks?

От
Chul Su Park
Дата:
Hi,

> Have you tried increasing the number of postgres buffer cache buffers? I am
> speculating that the lo_write() is using these buffers but perhaps not
> flushing them. As the default configuration has way too few buffers anyway
> this might be the problem. Could you try increasing the buffers to say 1024 or
> so and rerun your test and post the results?
>
> -dg

    Thanks for your reply, but as I posted I tested many different size of
buffers in lotest.c, e.g. 1Kb, 4Kb, 8Kb, 9Kb, 10Kb, 16Kb, 32Kb, 64Kb, 640Kb,
1Mb, 10Mb, ...  or your "cache" or "buffer" size have some diffferent meaning?

but lo_write cannot exceed 640kb always, only open/write(less than
640kb)/close looping OR open LOOP lo_write(buffer<640Kb)) , lo_lseek to
SEEK_END, write, ... seems to solve this problem(as in my last email),  and I
found that 16Kb buffer seems to be the optimal buffer size for the SPEED(not
8Kb) for network & socket connections why?...

and I think that above lo_lseek write loop is not a real solution, because
lo_read seems does not have such defects(but 16kb buffer seems to be the
optimal size also), there must be some problems in lo_write or buffer
managing...  does anybody have some guess or fixes?

c.s.park

Re: [HACKERS] [QUESTIONS] lo_write cannot > 640Kb? memory leaks?

От
dg@illustra.com (David Gould)
Дата:
> Hi,
>
> > Have you tried increasing the number of postgres buffer cache buffers? I am
> > speculating that the lo_write() is using these buffers but perhaps not
> > flushing them. As the default configuration has way too few buffers anyway
> > this might be the problem. Could you try increasing the buffers to say 1024 or
> > so and rerun your test and post the results?
> >
> > -dg
>
>     Thanks for your reply, but as I posted I tested many different size of
> buffers in lotest.c, e.g. 1Kb, 4Kb, 8Kb, 9Kb, 10Kb, 16Kb, 32Kb, 64Kb, 640Kb,
> 1Mb, 10Mb, ...  or your "cache" or "buffer" size have some diffferent meaning?

Yes, my "cache" or "buffer" size has a different meaning. I am referring to
the size of the postgres buffer pool which can be changed by a command line
option when you start the postmaster. I forget what the exact option is but
it is something like "-o -B <number_of_buffers>". It is documented I think.

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
"Of course, someone who knows more about this will correct me if I'm wrong,
 and someone who knows less will correct me if I'm right."
               --David Palmer (palmer@tybalt.caltech.edu)