Обсуждение: pgbackrest creating new directories (messing up cron jobs).
Hello, My pgbackrest creates the files in /var/lib/pgbackrest/archive/main/10/0000000100000000 Each file looks something like: 000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz. However, when the last two digits of the section before the hyphen become FF, pgbackrest creates a new directory: /var/lib/pgbackrest/archive/main/10/0000000100000001. This is annoying as I have a cron job set up to focus on files in the 0000000100000000 directory. Is there a way to force bgbackrest to only use one directory (e.g. 0000000100000000)? Thanks.
Greetings, * mimble9@danwin1210.me (mimble9@danwin1210.me) wrote: > My pgbackrest creates the files in > /var/lib/pgbackrest/archive/main/10/0000000100000000 Yes, those are files which are created in the repo. > Each file looks something like: > 000000010000000100000022-033bbe600913c068e89c3d063afb1f0527bf513d.gz. That's currently the naming, but we anticipate changing it in the future and so I strongly recommend that you do not make assumptions or depend on that exact naming. > However, when the last two digits of the section before the hyphen become > FF, pgbackrest creates a new directory: > /var/lib/pgbackrest/archive/main/10/0000000100000001. Yes. > This is annoying as I have a cron job set up to focus on files in the > 0000000100000000 directory. What is the cronjob doing..? You really shouldn't be hacking around with things in the repo- there's commands available like archive-get to extract out WAL files from the repo. > Is there a way to force bgbackrest to only use one directory (e.g. > 0000000100000000)? No. Thanks, Stephen
Вложения
Thank you for your answer! > >> This is annoying as I have a cron job set up to focus on files in the >> 0000000100000000 directory. >> > > What is the cronjob doing..? You really shouldn't be hacking around > with things in the repo- there's commands available like archive-get to > extract out WAL files from the repo. > The cronjob is using 'mutt' to mail me the .gz files at a set period. It says: cd /var/lib/pgbackrest/archive/main/10/0000000100000000/ echo "" | mutt -s "Ttile" test@example.com -a *.gz Of course, once 0000000100000000 becomes 0000000100000001, this no longer works. Is there, perhaps, a better way to achieve my goal? Thanks again - I do appreciate all your help.
Greetings, * mimble9@danwin1210.me (mimble9@danwin1210.me) wrote: > >> This is annoying as I have a cron job set up to focus on files in the > >> 0000000100000000 directory. > > > > What is the cronjob doing..? You really shouldn't be hacking around > > with things in the repo- there's commands available like archive-get to > > extract out WAL files from the repo. > > The cronjob is using 'mutt' to mail me the .gz files at a set period. > > It says: > > cd /var/lib/pgbackrest/archive/main/10/0000000100000000/ > echo "" | mutt -s "Ttile" test@example.com -a *.gz Isn't that going to mail you the same WAL over and over again if you set it up as a cronjob..? Is that really what you want? > Of course, once 0000000100000000 becomes 0000000100000001, this no longer > works. That doesn't seem like the worst of the problems here. > Is there, perhaps, a better way to achieve my goal? Well, you could use archive-get with pgbackrest to pull out the files by requesting each segment number, but I'm not sure what the idea here is exactly- *why* are you email'ing them? Thanks, Stephen
Вложения
I think I need to take a step backwards and ask something else before moving forward. > > Isn't that going to mail you the same WAL over and over again if you set > it up as a cronjob..? Is that really what you want? > I took a full backup when I started. Then I used type=diff. This creates two archives every x hours (depending on the time period set in cron). For example: -rw-r----- 1 postgres postgres 27145 Mar 11 00:00 0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz -rw-r----- 1 postgres postgres 27126 Mar 11 00:00 0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz Only one of these relates to the database I setup in PostgreSQL. I don't know what the other one is but it might be related to the "default" database 'postgres'. (This is just a guess). Irrespective, I would only want a new archive file if something has changed in the database. Yet I seem to always receive two new files every x hours even if nothing has changed. So I tried type-incr but that seems to produce the same results as type=diff. I wonder what I am doing wrong?
Greetings, * mimble9@danwin1210.me (mimble9@danwin1210.me) wrote: > I think I need to take a step backwards and ask something else before > moving forward. Good idea. > > Isn't that going to mail you the same WAL over and over again if you set > > it up as a cronjob..? Is that really what you want? > > I took a full backup when I started. Then I used type=diff. This creates > two archives every x hours (depending on the time period set in cron). WAL files are not "archives" in the sense that they are a complete database backup- they're just the write-ahead-logs from the ongoing running of PG. You need the actual data files as well as the WAL to perform a restore. > For example: > > -rw-r----- 1 postgres postgres 27145 Mar 11 00:00 > 0000000100000001000000B9-6f3902fe5c3bdebc3c1c124ec6821c7206e350da.gz > > -rw-r----- 1 postgres postgres 27126 Mar 11 00:00 > 0000000100000001000000BA-214e7142c6eda0a350577f6bd624c3db203e184f.gz > > Only one of these relates to the database I setup in PostgreSQL. I don't > know what the other one is but it might be related to the "default" > database 'postgres'. (This is just a guess). No, that's not how WAL works. Those are two WAL files and they're both generated as part of running the PG system- they are not specific to one database or another in PG. > Irrespective, I would only want a new archive file if something has > changed in the database. Yet I seem to always receive two new files every > x hours even if nothing has changed. These individuals files are not archives of the entire PG system or anything you can use to reconstruct a running PG system with just one file. > So I tried type-incr but that seems to produce the same results as type=diff. The difference between incremental and differential backups is documented at https://pgbackrest.org > I wonder what I am doing wrong? I'm afraid there are some pretty fundamental misunderstandings that you have about the PG WAL, PG backups, and what you can do with a PG cluster andf file-level backup/restore. Documentation about the WAL can be found here: https://www.postgresql.org/docs/current/wal-intro.html If what you're actually looking for is a single file that has a backup of your entire database, you can get that by running: pgbackrest backup --archive-copy and then do: pgbackrest restore --pg-path=/path/to/somewhere tar -czf pgdata.tar.gz /path/to/somewhere or so. Check the pgbackrest docs for the specific command syntax and such. Thanks, Stephen