Re: pgBackRest for a 50 TB database

Поиск
Список
Период
Сортировка
От Stephen Frost
Тема Re: pgBackRest for a 50 TB database
Дата
Msg-id CAOuzzgrxuvvwv7UfaEzGc5QhQ4d8V9ptykQTKhoqHxktAod2Pw@mail.gmail.com
обсуждение исходный текст
Ответ на Re: pgBackRest for a 50 TB database  (Abhishek Bhola <abhishek.bhola@japannext.co.jp>)
Ответы Re: pgBackRest for a 50 TB database
Список pgsql-general
Greetings,

On Thu, Oct 5, 2023 at 03:10 Abhishek Bhola <abhishek.bhola@japannext.co.jp> wrote:
Here is the update with compress-type=zst in the config file
Process-max is still 30. But it longer than before, around 27 hours 50 mins

full backup: 20231004-130621F
            timestamp start/stop: 2023-10-04 13:06:21+09 / 2023-10-05 15:56:03+09
            wal start/stop: 000000010001AC0E00000054 / 000000010001AC0E00000054
            database size: 38249.0GB, database backup size: 38249.0GB
            repo1: backup size: 5799.8GB

Do you think I could be missing something?

Sounds like there’s something else which is the bottleneck once you have process-max at 30. I suspect you could reduce that process-max value and have around the same time still with zstd.  Ultimately if you want it to be faster then you’ll need to figure out what the bottleneck is (seemingly not CPU, unlikely to be memory, so that leaves network or storage) and address that. 

We’ve seen numbers approaching 10TB/hr with lots of processes and zstd and fast storage on high end physical hardware. 

Thanks,

Stephen

В списке pgsql-general по дате отправления:

Предыдущее
От: Nick Cleaton
Дата:
Сообщение: Re: Gradual migration from integer to bigint?
Следующее
От: Lauri Kajan
Дата:
Сообщение: Re: Index scan is not pushed down to union all subquery