Обсуждение: bytea size limit?

On Wed, Jan 21, 2009 at 05:21:03AM -0800, paulo matadr wrote:
> I everyone, need help!!!
> My aplication return erro:
>
> 2009-01-20 10:22:03,264 INFO  [STDOUT] Caused by: org.postgresql.util.
> PSQLException: ERROR: invalid memory alloc request size 1705447581
>
>  TABLE batch.relatorio_gerado
>
>   rege_id integer NOT NULL,
>   fuin_id integer NOT NULL,
>   rela_id integer NOT NULL,
>   rege_tmultimaalteracao timestamp without time zone NOT NULL DEFAULT now(),
>   rege_nnpaginas integer NOT NULL,
>   rege_binario bytea,
>  rege_pdf bytea
>
> I get this erro above  refers the column "rege_pdf bytea"   , when try generate report in pdf.
>
> thanks for help
>

I believe that the default size limit for a bytea or text field is
currently 1GB.

Ken

there's no real limit (its size is described with 32bit number, and
that's the only limitation here).
But you need to be aware, that content is sent over at once, so memory
is the limit in your case.

http://www.postgresql.org/docs/8.3/static/datatype-binary.html

For such large objects, it might be actually better to store them
separate as files, and just store file name.

On Wed, Jan 21, 2009 at 01:55:28PM +0000, Grzegorz Ja??kiewicz wrote:
> there's no real limit (its size is described with 32bit number, and
> that's the only limitation here).
> But you need to be aware, that content is sent over at once, so memory
> is the limit in your case.
>
> http://www.postgresql.org/docs/8.3/static/datatype-binary.html
>
> For such large objects, it might be actually better to store them
> separate as files, and just store file name.
>

The TOAST implementation however only allows 30-bits for the
size of the TOAST entry which caps the size at 2^30 or 1GB. I
agree that he could very well be limited also by the memory on
his system.

Cheers,
Ken

On Wed, Jan 21, 2009 at 2:06 PM, Kenneth Marshall <ktm@rice.edu> wrote:

> The TOAST implementation however only allows 30-bits for the
> size of the TOAST entry which caps the size at 2^30 or 1GB. I
> agree that he could very well be limited also by the memory on
> his system.

i wasn't aware of that, and also - it doesn't say anything about it in docs.
As for limitations, that also depends on db drivers he is using, etc,
etc. I use bytea to store 100-200MB objects in many dbs, but I
wouldn't go as far as 1.5GB ...



--
GJ

On Wed, Jan 21, 2009 at 02:09:01PM +0000, Grzegorz Ja??kiewicz wrote:
> On Wed, Jan 21, 2009 at 2:06 PM, Kenneth Marshall <ktm@rice.edu> wrote:
>
> > The TOAST implementation however only allows 30-bits for the
> > size of the TOAST entry which caps the size at 2^30 or 1GB. I
> > agree that he could very well be limited also by the memory on
> > his system.
>
> i wasn't aware of that, and also - it doesn't say anything about it in docs.
> As for limitations, that also depends on db drivers he is using, etc,
> etc. I use bytea to store 100-200MB objects in many dbs, but I
> wouldn't go as far as 1.5GB ...
>
The reference is in:
    http://www.postgresql.org/docs/8.3/static/storage-toast.html

Here is the pertinent excerpt:

Only certain data types support TOAST -- there is no need to impose the
overhead on data types that cannot produce large field values. To support
TOAST, a data type must have a variable-length (varlena) representation,
in which the first 32-bit word of any stored value contains the total
length of the value in bytes (including itself). TOAST does not constrain
the rest of the representation. All the C-level functions supporting a
TOAST-able data type must be careful to handle TOASTed input values.
(This is normally done by invoking PG_DETOAST_DATUM before doing anything
with an input value, but in some cases more efficient approaches are possible.)

TOAST usurps two bits of the varlena length word (the high-order bits on
big-endian machines, the low-order bits on little-endian machines),
thereby limiting the logical size of any value of a TOAST-able data type
to 1 GB (230 - 1 bytes). When both bits are zero, the value is an ordinary
un-TOASTed value of the data type, and the remaining bits of the length
word give the total datum size (including length word) in bytes. When the
highest-order or lowest-order bit is set, the value has only a single-byte
header instead of the normal four-byte header, and the remaining bits give
the total datum size (including length byte) in bytes. As a special case,
if the remaining bits are all zero (which would be impossible for a
self-inclusive length), the value is a pointer to out-of-line data stored
in a separate TOAST table. (The size of a TOAST pointer is given in the
second byte of the datum.) Values with single-byte headers aren't aligned
on any particular boundary, either. Lastly, when the highest-order or
lowest-order bit is clear but the adjacent bit is set, the content of the
datum has been compressed and must be decompressed before use. In this
case the remaining bits of the length word give the total size of the
compressed datum, not the original data. Note that compression is also
possible for out-of-line data but the varlena header does not tell whether
it has occurred -- the content of the TOAST pointer tells that, instead.

Cheers,
Ken

you don't have to quote everything :)
I ment, there's nothing on bytea on its doc page, where one would
expect to read it.

On Wed, Jan 21, 2009 at 3:02 PM, paulo matadr <saddoness@yahoo.com.br> wrote:
> My system have very large ram size, so its possible review postgresql.conf ?
>
all depends on how you access DB, what type of drivers (odbc, libpq, etc, etc).
See, every time you pass a row , they usually have to allocate that
much memory, not only its quite inefficient, but also slow.

As for the configuration option, I am interested in knowing myself too :)

--
GJ

On Wed, Jan 21, 2009 at 03:07:13PM +0000, Grzegorz Ja??kiewicz wrote:
> On Wed, Jan 21, 2009 at 3:02 PM, paulo matadr <saddoness@yahoo.com.br> wrote:
> > My system have very large ram size, so its possible review postgresql.conf ?
> >
> all depends on how you access DB, what type of drivers (odbc, libpq, etc, etc).
> See, every time you pass a row , they usually have to allocate that
> much memory, not only its quite inefficient, but also slow.
>
> As for the configuration option, I am interested in knowing myself too :)
>
I do not think that the size limit is a runtime option. It is currently
compiled into the server.

Ken

On 1/21/09, paulo matadr <saddoness@yahoo.com.br> wrote:
>
> I everyone, need help!!!
> My aplication return erro:
>
> 2009-01-20 10:22:03,264 INFO  [STDOUT] Caused by: org.postgresql.util.
> PSQLException: ERROR: invalid memory alloc request size 1705447581

What exactly were you doing when you got the error?
How big is the item?
How are you trying to pull it?? libpq, php, etc?

merlin

paulo matadr wrote:
> I everyone, need help!!!
> My aplication return erro:
> 
> 2009-01-20 10:22:03,264 INFO  [STDOUT] Caused by: org.postgresql.util.
> PSQLException: ERROR: invalid memory alloc request size 1705447581
> 
> 
>  TABLE batch.relatorio_gerado 
> 
>   rege_id integer NOT NULL,
>   fuin_id integer NOT NULL,
>   rela_id integer NOT NULL,
>   rege_tmultimaalteracao timestamp without time zone NOT NULL DEFAULT now(),
>   rege_nnpaginas integer NOT NULL,
>   rege_binario bytea,
>   rege_pdf bytea
> 
> I get this erro above  refers the column "rege_pdf bytea"   , 
> when try generate report in pdf.

What are you doing in terms of SQL?
INSERT, UPDATE, DELETE?

How big are the binary objects involved?

What are the values of the database parameters shared_buffers and work_mem?

Yours,
Laurenz Albe

Please don't top post.

paulo matadr wrote:
>>> My aplication return erro:
>>> 
>>> 2009-01-20 10:22:03,264 INFO  [STDOUT] Caused by: org.postgresql.util.
>>> PSQLException: ERROR: invalid memory alloc request size 1705447581
>>
>> What are you doing in terms of SQL?
>> INSERT, UPDATE, DELETE?
>> 
>> How big are the binary objects involved?
>> 
>> What are the values of the database parameters shared_buffers 
>> and work_mem?
>
> The size of object depend on report  for a user request,
> shared_buffers = 2048MB
> work_mem = 12MB 
> 
> -------
> Server conf
> 16 GB RAM
> Red Hat Enterprise Linux Server release 5
> 
> Using apliccation web based , with Jboss apliccation server on jdbc driver.
> Lets see  scenario :
> Apliccation request a report,if this bigger , hangs .
> No able to select or others report in this table.

You need to find out the SQL statement that triggers the error.

The error you quoted does not come from the JDBC driver, but from the database
server.

I suggest that you set
log_min_messages to ERROR or lower,
log_min_error_statement to ERROR or lower, and
log_statement to all.

Then reproduce the error and look into the log file to find out the
statement that got the error. We can proceed from there.

Yours,
Laurenz Albe

checkpoints don't have anything to do with it.

Please don't top post.

paulo matadr wrote:
> I think identified the problem
> lts's check log below:
> 
> Query:
> SELECTSTATEMENT:  select relatorios0_.fuin_id as fuin5_1_, relatorios0_.rege_id as rege1_1_, relatorios0_.rege_id as
r
> ege1_624_0_, relatorios0_.rege_tmultimaalteracao as rege2_624_0_, relatorios0_.rege_nnpaginas as rege3_624_0_,
relatorios0_.rege_pdfas rege4_624_0_, relatorios0_.fuin_id as fui
 
> n5_624_0_, relatorios0_.rela_id as rela6_624_0_ from batch.relatorio_gerado relatorios0_ where relatorios0_.fuin_id
in($1, $2)
 
>  
> Error:
> 2009-01-18 00:05:28 BRT LOG:  checkpoints are occurring too frequently (25 seconds apart)
> 2009-01-18 00:05:28 BRT HINT:  Consider increasing the configuration parameter "checkpoint_segments".
> gsan_comercial gcom_batch 10.1.1.5 2009-01-18 00:05:39 BRT SELECTERROR:  invalid memory alloc request size
1705447581

> we  need make report bigger than 200 M.
> suggestions?

I tried selecting large bytea values on my 8.3.5 system, and
I ran into similar problems; I could select a value of length 200000000
but got the same error with a value of 268435456 bytes.

I tried with psql.

Anyone knows what causes this?

I guess it might be the best approach for you to either use large objects,
which can contain up to 2GB and provide functions to read them in smaller parts,
or to change your table structure so that large values are split into
parts and stored separately...

Yours,
Laurenz Albe