Re: psql and bytea

** Reply to message from Féliciano Matias <feliciano.matias@free.fr> on 16 May
2003 06:56:02 +0200
Hi,
 I'm interested :)

Regards,
Wayne

> Le jeu 15/05/2003 à 16:47, Tom Lane a écrit :
> > =?ISO-8859-1?Q?F=E9liciano?= Matias <feliciano.matias@free.fr> writes:
> > > one=3D> -- this is not the original size (20M : 20971520).
> > > I don't have any problems with smaller files (10Mo is always fine).
> > > All of this is used with postgresql 7.3.2 shipped with Red Hat Linux 9 .
> >
> > Hmm.  There used to be some off-by-one type bugs in psql's
> > variable-substitution code, but those were fixed long before 7.3.2.
> > In any case, it's hard to see why such a problem would only arise
> > when you got past 10Mb string lengths.
> >
>
> More then 10Mb string lengths :
> > > $ tobytea < data | wc
> > > 0   81574 74379444     (71 Mo)
>
> > I couldn't duplicate the problem here,
>
> It seems it's a bug in glibc-2.3.2-27.9.i686.rpm coming with RH9.
>
> > so I'm going to suggest that
> > maybe you have a hardware problem?  Perhaps there's a flaky RAM chip in
> > an area of memory that doesn't get used until you push up the size
> > of psql quite a bit.  It'd be worth running memtest86 for awhile to
> > check.
>
> The hardware is ok.
>
> I found the problem. It's bug in sprintf()
> ===============================================
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #define SIZE (1024*70000)
> int main(void) {
>     char * s = malloc(SIZE) ;
>     char * d = malloc(SIZE) ;
>     memset(s, 'a', SIZE-1) ;
>     d[SIZE-1] = '\0' ;
>     sprintf(d,"%s",s) ;
>     printf("%zi\n", strlen(d)) ;
>     return 0 ;
> }
> ===============================================
> $ ./a.out
> 67108863    (2^26-1)
>
> My libc have a problem. I will fill a bug report to
> http://bugzilla.redhat.com/ .
>
> I apply this patch to the postgresql source :
> ===============================================
> diff -urN postgresql-7.3.2.orig/src/bin/psql/mainloop.c postgresql-7.3.2/src/bin/psql/mainloop.c
> --- postgresql-7.3.2.orig/src/bin/psql/mainloop.c       2002-10-13 01:09:34.000000000 +0200
> +++ postgresql-7.3.2/src/bin/psql/mainloop.c    2003-05-16 04:46:03.000000000 +0200
> @@ -389,8 +389,9 @@
>                                                 exit(EXIT_FAILURE);
>                                         }
>
> -                                       sprintf(new, "%.*s%s%s", i, line, value,
> -                                                       &line[i + thislen + in_length]);
> +                                       sprintf(new, "%.*s", i, line) ;
> +                                       strcat(&new[i], value) ;
> +                                       strcat(&new[i+out_length], &line[i + thislen + in_length]) ;
>
>                                         free(line);
>                                         line = new;
> ===============================================
>
> Also, i want to know if someone is interesting by tools such as
> (to|from)bytea. I can put this little toys (with some enhancement.
> "--help" :-) )  in contrib and update the man page of psql to show how
> to use this tools to copy the content of a file (or stream) into a
> field.
> Since Postgresql have binary string, it is interesting and better than
> \lo_* for little files (this use more memory than \lo_*).
>
> By the way, i can add an option to psql that toggle the output of
> carrier return at the end of each records. Perhaps this can be add to
> \t.
>
> >
> >             regards, tom lane
> >
>
> --
> Féliciano Matias <feliciano.matias@free.fr>