Re: Large Databases

elein wrote:
> What is the linux and/or postgres limitation for very
> large databases, if any?  We are looking at 6T-20T.
> My understanding is that if the hardware supports it,
> then it can be done in postgres.  But can hardware
> support that?

I've recently been going through a project to support what will become a
5 to 6 TB Postgres database (initially it will be about 300GB after
conversion from the source system). A few significant things I've
learned along the way:

1) The linux 2.4 kernel has a block device size limit of 2 TB.

2) The linux 2.6 kernel supports *huge* block device size -- I don't
    have it in front of me, but IIRC it was in the peta-bytes range.

3) xfs, jfs, and ext3 all can handle more than the 6TB we needed them to
    handle.

4) One of the leading SAN vendors initially claimed to be able to
    support our desire to have a single 6TB volume. We found that when
    pushed hard, we would get disk corruption (archives are down, but see
    HACKERS on 8/21/04 for a message I posted on the topic). Now we are
    being told that they don't support the linux 2.6 kernel, and
    therefore don't support > 2TB volumes.

So the choices seem to be:
a) Use symlinks or Postgres 8.0.0beta tablespaces to split your data
    across multiple 2 TB volumes.

b) Use NFS mounted NAS.

We are already a big NetApp shop, so NFS mounted NAS is the direction
we'll likely take. It appears (from their online docs) that NetApp can
have individual volumes up to 16 TB. We should be confirming that with
them in the next day or two.

HTH,

Joe