Re: 10 TB database

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Arthur,

We have a database that has a table growing ~1,5M rows each month.
The overall growth for the db is ~1GB/month.
PostgreSQL 8.2.9 on x86_64 - a very modest Dell R200 with 4GB of ram.

Although poor planning was made in the beginning (i.e. no clustering,
no partitioning...) - we weren't expecting the boom :] - that
particular server runs like clockwork with hundreds of queries per
minute and still doing so without any noticeable speed loss.

We're, of course, planning for load balancing in the beginning of next
year ... :)

BR,

Pedro Doria Meunier
GSM: +351 96 17 20 188
Skype: pdoriam




Michelle Konzack wrote:
> Hi Artur,
>
> I am owner of a database about War, Worcrime and Terroism with more
> then 1,6 TByte and I am already fscked...
>
> Am 2009-06-15 14:00:05, schrieb Artur:
>> Hi!
>>
>> We are thinking to create some stocks related search engine. It
>> is experimental project just for fun.
>>
>> The problem is that we expect to have more than 250 GB of data
>> every month.
>
> I have only 500 MByte per month...
>
>> This data would be in two tables. About 50.000.000 new rows every
>> month.
>
> arround 123.000 new rows per month
>
>> We want to have access to all the date mostly for generating user
>>  requesting reports (aggregating). We would have about 10TB of
>> data in three years.
>>
>> Do you think is it possible to build this with postgresql and
>> have any idea how to start? :)
>
> You have to use a physical cluster like me. Searches in  a
> Database  of more then 1 TByte even under using "tablespace" and
> "tablepartitioning" let you run into performance issues...
>
> I have now splited my Database in chunks of 250 GByte using a
> Cluster of 1U Servers from Sun Microsystems.  Currently I run  8
> servers  with  one proxy.  Each server cost me 2.300 Euro.
>
> Note:   On Friday I have a meeting with a  Sun  Partner  in Germany
> because a bigger project...  where  I  have to increase the
> performance of my database servers. I have to calculate with
> 150.000 customers.
>
> Thanks, Greetings and nice Day/Evening Michelle Konzack
> Systemadministrator Tamay Dogan Network Debian GNU/Linux Consultant
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFKN13P2FH5GXCfxAsRAkIiAJ95GvbQhBrOglzK2d57F5Qv7E5NdgCfcKga
bFpRiWf2vSY0oMOD40PgSsg=
=4OB3
-----END PGP SIGNATURE-----