Re: Storing files: 2.3TBytes, 17M file count

Поиск
Список
Период
Сортировка
От Thomas Güttler
Тема Re: Storing files: 2.3TBytes, 17M file count
Дата
Msg-id 8d88644c-27b8-3f60-4263-168fe1a3ab1d@thomas-guettler.de
обсуждение исходный текст
Ответ на Re: Storing files: 2.3TBytes, 17M file count  (Adrian Klaver <adrian.klaver@aklaver.com>)
Ответы Re: Storing files: 2.3TBytes, 17M file count
Список pgsql-general

Am 28.11.2016 um 16:01 schrieb Adrian Klaver:
> On 11/28/2016 06:28 AM, Thomas Güttler wrote:
>> Hi,
>>
>> PostgreSQL is rock solid and one of the most reliable parts of our
>> toolchain.
>>
>>    Thank you
>>
>> Up to now, we don't store files in PostgreSQL.
>>
>> I was told, that you must not do this .... But this was 20 years ago.
>>
>>
>> I have 2.3TBytes of files. File count is 17M
>>
>> Up to now we use rsync (via rsnapshot) to backup our data.
>>
>> But it takes longer and longer for rsync to detect
>> the changes. Rsync checks many files. But daily only
>> very few files really change. More than 99.9% don't.
>
> Are you rsyncing over all the files at one time?

Yes, we rsyncing every night.

> Or do break it down into segments over the day?

No, up to now it is one rsync run.

> The closest I remember is Bacula:
>
> http://blog.bacula.org/documentation/documentation/
>
> It uses a hybrid solution where the files are stored on a file server and data about the files is stored in a
database.
> Postgres is one of the database backends it can work with.

I heard of Bacula, but I was not aware of the fact, that they can use postfres for the meta data.

>>
>> I have the hope, that it would be easier to backup only the files which
>> changed.
>
> Backup to where and how?
> Are you thinking of using replication?

No, replication is not the current issue. Plain old backup is my current issue.

Backup where and how? ... That's what this question is about :-)

--
Thomas Guettler http://www.thomas-guettler.de/


В списке pgsql-general по дате отправления:

Предыдущее
От: rob
Дата:
Сообщение: Re: Invoice Table Design
Следующее
От: Thomas Güttler
Дата:
Сообщение: We reached the limit of inotify. Was: Storing files: 2.3TBytes, 17M file count