Re: [PERFORM] Backup taking long time !!!

От: Stephen Frost
Тема: Re: [PERFORM] Backup taking long time !!!
Дата: ,
Msg-id: 20170120134054.GH18360@tamriel.snowman.net
(см: обсуждение, исходный текст)
Ответ на: Re: [PERFORM] Backup taking long time !!!  (Vladimir Borodin)
Список: pgsql-performance

Скрыть дерево обсуждения

[PERFORM] Backup taking long time !!!  (Dinesh Chandra 12108, )
 Re: [PERFORM] Backup taking long time !!!  ("Madusudanan.B.N", )
  Re: [PERFORM] Backup taking long time !!!  (Dinesh Chandra 12108, )
   Re: [PERFORM] Backup taking long time !!!  (Pavel Stehule, )
    Re: [PERFORM] Backup taking long time !!!  ("Madusudanan.B.N", )
    Re: [PERFORM] Backup taking long time !!!  (Dinesh Chandra 12108, )
     Re: [PERFORM] Backup taking long time !!!  (Pavel Stehule, )
      Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
  Re: [PERFORM] Backup taking long time !!!  (Vladimir Borodin, )
   Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (Vladimir Borodin, )
  Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (Vladimir Borodin, )
  Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
   Re: [PERFORM] Backup taking long time !!!  (Jim Nasby, )
    Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
     Re: [PERFORM] Backup taking long time !!!  (Jim Nasby, )
   Re: [PERFORM] Backup taking long time !!!  (David Steele, )
  Re: [PERFORM] Backup taking long time !!!  (Torsten Zuehlsdorff, )
   Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (Vladimir Borodin, )
  Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (julyanto SUTANDANG, )
  Re: [PERFORM] Backup taking long time !!!  (julyanto SUTANDANG, )
   Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
    Re: [PERFORM] Backup taking long time !!!  (Jeff Janes, )
     Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
     Re: [PERFORM] Backup taking long time !!!  (Simon Riggs, )
      Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
      Re: [PERFORM] Backup taking long time !!!  (Jeff Janes, )
       Re: [PERFORM] Backup taking long time !!!  (Rick Otten, )
        Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
     Re: [PERFORM] Backup taking long time !!!  (julyanto SUTANDANG, )
  Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (julyanto SUTANDANG, )
  Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (julyanto SUTANDANG, )
  Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
   Re: [PERFORM] Backup taking long time !!!  (Jim Nasby, )
    Re: [PERFORM] Backup taking long time !!!  (Jeff Janes, )
     Re: [PERFORM] Backup taking long time !!!  (Stephen Frost, )
 Re: [PERFORM] Backup taking long time !!!  (julyanto SUTANDANG, )

Vladimir,

* Vladimir Borodin () wrote:
> > 20 янв. 2017 г., в 15:22, Stephen Frost <> написал(а):
> >> This process can be automatized by some applications like barman
> >> http://www.pgbarman.org/
> >
> > Last I checked, barman is still single-threaded.
> >
> > If the database is large enough that you need multi-process backup, I'd
> > suggest looking at pgbackrest- http://www.pgbackrest.org.
> >
> > pgbackrest has parallel backup, incremental/differential/full backup
> > support, supports compression, CRC checking, and a whole ton of other
> > good stuff.
>
> Increments in pgbackrest are done on file level which is not really efficient. We have done parallelism, compression
andpage-level increments (9.3+) in barman fork [1], but unfortunately guys from 2ndquadrant-it don’t hurry to work on
it.

We're looking at page-level incremental backup in pgbackrest also.  For
larger systems, we've not heard too much complaining about it being
file-based though, which is why it hasn't been a priority.  Of course,
the OP is on 9.1 too, so.

As for your fork, well, I can't say I really blame the barman folks for
being cautious- that's usually a good thing in your backup software. :)

I'm curious how you're handling compressed page-level incremental
backups though.  I looked through barman-incr and it wasn't obvious to
me what was going wrt how the incrementals are stored, are they ending
up as sparse files, or are you actually copying/overwriting the prior
file in the backup repository?  Apologies, python isn't my first
language, but the lack of any comment anywhere in that file doesn't
really help.

> And actually it would be much better to do a good backup and recovery manager part of the core postgres.

Sure, but that's not going to happen for 9.1, or even 9.6, and I doubt
PG10 is going to suddenly get parallel base-backup with compression.

I've been discussing ways to improve the situation with Magnus and we do
have some ideas about it, but that's really an independent effort as
we're still going to need a tool for released versions of PG.

Thanks!

Stephen

Вложения

В списке pgsql-performance по дате сообщения:

От: Stephen Frost
Дата:
Сообщение: Re: [PERFORM] Backup taking long time !!!
От: julyanto SUTANDANG
Дата:
Сообщение: Re: [PERFORM] Backup taking long time !!!