Re: parallelizing the archiver

Поиск
Список
Период
Сортировка
От Andrey Borodin
Тема Re: parallelizing the archiver
Дата
Msg-id FA9D5725-17CA-4476-8B3D-919BD32AFD96@yandex-team.ru
обсуждение исходный текст
Ответ на parallelizing the archiver  ("Bossart, Nathan" <bossartn@amazon.com>)
Ответы Re: parallelizing the archiver  (Julien Rouhaud <rjuju123@gmail.com>)
Список pgsql-hackers

> 8 сент. 2021 г., в 03:36, Bossart, Nathan <bossartn@amazon.com> написал(а):
>
> Anyway, I'm curious what folks think about this.  I think it'd help
> simplify server administration for many users.

BTW this thread is also related [0].

My 2 cents.
It's OK if external tool is responsible for concurrency. Do we want this complexity in core? Many users do not enable
archivingat all. 
Maybe just add parallelism API for external tool?
It's much easier to control concurrency in external tool that in PostgreSQL core. Maintaining parallel worker is a
tremendouslyharder than spawning goroutine, thread, task or whatever. 
External tool needs to know when xlog segment is ready and needs to report when it's done. Postgres should just ensure
thatexternal archiever\restorer is running. 
For example external tool could read xlog names from stdin and report finished files from stdout. I can prototype such
toolswiftly :) 
E.g. postgres runs ```wal-g wal-archiver``` and pushes ready segment filenames on stdin. And no more listing of
archive_statusand hacky algorithms to predict next WAL name and completition time! 

Thoughts?

Best regards, Andrey Borodin.

[0]
https://www.postgresql.org/message-id/flat/CA%2BTgmobhAbs2yabTuTRkJTq_kkC80-%2Bjw%3DpfpypdOJ7%2BgAbQbw%40mail.gmail.com


В списке pgsql-hackers по дате отправления:

Предыдущее
От: Dilip Kumar
Дата:
Сообщение: Re: Toast compression method options
Следующее
От: "houzj.fnst@fujitsu.com"
Дата:
Сообщение: RE: Added schema level support for publication.