Re: parallelizing the archiver
От | Bossart, Nathan |
---|---|
Тема | Re: parallelizing the archiver |
Дата | |
Msg-id | E9035E94-EC76-436E-B6C9-1C03FBD8EF54@amazon.com обсуждение исходный текст |
Ответ на | Re: parallelizing the archiver (Andrey Borodin <x4mmm@yandex-team.ru>) |
Ответы |
Re: parallelizing the archiver
(Andrey Borodin <x4mmm@yandex-team.ru>)
|
Список | pgsql-hackers |
On 9/10/21, 10:42 AM, "Robert Haas" <robertmhaas@gmail.com> wrote: > I had kind of been thinking that the way to attack this problem is to > go straight to allowing for a background worker, because the other > problem with archive_command is that running a shell command like cp, > scp, or rsync is not really safe. It won't fsync your data, it might > not fail if the file is in the archive already, and it definitely > won't succeed without doing anything if there's a byte for byte > identical file in the archive and fail if there's a file with > different contents already in the archive. Fixing that stuff by > running different shell commands is hard, but it wouldn't be that hard > to do it in C code, and you could then also extend whatever code you > wrote to do batching and parallelism; starting more workers isn't > hard. > > However, I can't see the idea of running a shell command going away > any time soon, in spite of its numerous and severe drawbacks. Such an > interface provides a huge degree of flexibility and allows system > admins to whack around behavior easily, which you don't get if you > have to code every change in C. So I think command-based enhancements > are fine to pursue also, even though I don't think it's the ideal > place for most users to end up. I've given this quite a bit of thought. I hacked together a batching approach for benchmarking, and it seemed to be a decent improvement, but you're still shelling out every N files, and all the stuff about shell commands not being ideal that you mentioned still applies. Perhaps it's still a good improvement, and maybe we should still do it, but I get the idea that many believe we can still do better. So, I looked into adding support for setting up archiving via an extension. The attached patch is a first try at adding alternatives for archive_command, restore_command, archive_cleanup_command, and recovery_end_command. It adds the GUCs archive_library, restore_library, archive_cleanup_library, and recovery_end_library. Each of these accepts a library name that is loaded at startup, similar to shared_preload_libraries. _PG_init() is still used for initialization, and you can use the same library for multiple purposes by checking the new exported variables (e.g., process_archive_library_in_progress). The library is then responsible for implementing the relevant function, such as _PG_archive() or _PG_restore(). The attached patch also demonstrates a simple implementation for an archive_library that is similar to the sample archive_command in the documentation. I tested the sample archive_command in the docs against the sample archive_library implementation in the patch, and I saw about a 50% speedup. (The archive_library actually syncs the files to disk, too.) This is similar to the improvement from batching. Of course, there are drawbacks to using an extension. Besides the obvious added complexity of building an extension in C versus writing a shell command, the patch disallows changing the libraries without restarting the server. Also, the patch makes no effort to simplify error handling, memory management, etc. This is left as an exercise for the extension author. I'm sure there are other ways to approach this, but I thought I'd give it a try to see what was possible and to get the conversation started. Nathan
Вложения
В списке pgsql-hackers по дате отправления:
Предыдущее
От: Amit KapilaДата:
Сообщение: Re: Failed transaction statistics to measure the logical replication progress
Следующее
От: Amit KapilaДата:
Сообщение: Re: Some thoughts about the TAP tests' wait_for_catchup()