Обсуждение: how long should Archive logs be retained

Поиск
Список
Период
Сортировка

how long should Archive logs be retained

От
"Subramanian,Ramachandran"
Дата:

Hello,

 

 

    Kindly excuse my lack of knowledge.  I am new to Postgres and I  want to make sure that I am doing the right thing.

 

 

How long should archived logs be retained?

 

I make pg_basebackups once  a day  with wal-method ‚stream‘  and postgresql.conf has wal_level = replica and  archive_mode=on

 

Case 1:  Recovery to the point in time when the basebackup ended--  Am I correct in assuming that I do not need anything else other than the base backup if I want to just recover to the point in time when the basebackup ended?   Please correct me if I am wrong.

 

Case 2: Recovery to the latest point – For example if a server crashes, I can simply restart the server and postgres and it will rollforward all the active logs since the last flush oft he data buffers to disk happened . Is my understanding correct ?  While I understand that all the log buffers are written as soon as a commit is issued, how does Postgres keep track of when the data buffers were flushed to disk? In other words, how does Postgres know from which log sequence number it must begin rolling forward?  Can we as users also see it?

 

Case 3:  No archive logging and transaction that does not commit:  Let us say that we have not enabled archive logging and we have a long running update that fills up all the active logs (WALs) and it has wrapped around. What happens now?  Will the system Hang?  Will Postgres cancel this thread and rollback?  Should we manually cancel the thread?  How can we find out which thread is writing without commits?

 

Case 4: Recovery to the latest point with Archive logging enabled:  If  I have archive logging enabled and I have transactions that wrap around the logs without commit and the server crashes,  is it correct to assume that  I need

  1. my base backup ( with all the ‚then copied WALs‘ )
  2.  the current WALs in the active WAL directory
  3. All the archived WALs from the time the base backup began to the time the crash happened

 

to bring the system to a consistent state?

 

 

My question in short is, Is it enough if I hold on to my archive logs since the last backup for any recovery scenareo?

 

 

Thank you so much for your time

 

 

LG

 

Ram


Freundliche Grüße

i. A. Ramachandran Subramanian

Zentralbereich Informationstechnologie

Alte Leipziger Lebensversicherung a. G.

Hallesche Krankenversicherung a. G.

______________________

ALH Gruppe
Alte Leipziger-Platz 1, 61440 Oberursel
Tel: +49 (6171) 66-4882
Fax: +49 (6171) 66-800-4882
E-Mail: ramachandran.subramanian@alte-leipziger.de
www.alte-leipziger.de
www.hallesche.de

Alte Leipziger Lebensversicherung a. G., Alte Leipziger-Platz 1, 61440 Oberursel

Vors. des Aufsichtsrats: Dr. Walter Botermann · Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.), Frank Kettnaker, Dr. Jochen Kriegmeier, Alexander Mayer, Christian Pape, Wiltrud Pekarek, Udo Wilcsek

Sitz Oberursel (Taunus) · Rechtsform VVaG · Amtsgericht Bad Homburg v. d. H. HRB 1583 · USt.-IdNr. DE 114106814

Hallesche Krankenversicherung a. G., Löffelstraße 34-38, 70597 Stuttgart

Vors. des Aufsichtsrats: Dr. Walter Botermann · Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.), Frank Kettnaker, Dr. Jochen Kriegmeier, Alexander Mayer, Christian Pape, Wiltrud Pekarek, Udo Wilcsek

Sitz Stuttgart · Rechtsform VVaG · Amtsgericht Stuttgart HRB 2686 · USt.-IdNr. DE 147802285

Beiträge zu privaten Kranken- und Pflegekrankenversicherungen unterliegen nicht der Versicherungsteuer (§ 4 (1) Nr. 5 b VersStG) · Versicherungsleistungen sowie Umsätze aus Versicherungsvertreter-/Maklertätigkeiten sind umsatzsteuerfrei

Pflichtangaben der ALH Gruppe gemäß § 35a GmbHG bzw. § 80 AktG

Re: how long should Archive logs be retained

От
Laurenz Albe
Дата:
On Thu, 2025-11-27 at 10:34 +0000, Subramanian,Ramachandran wrote:
> How long should archived logs be retained?
>  
> I make pg_basebackups once  a day  with wal-method ‚stream‘  and postgresql.conf has wal_level = replica and
 archive_mode=on
>  
> Case 1:  Recovery to the point in time when the basebackup ended--  Am I correct in
>          assuming that I do not need anything else other than the base backup if I want
>          to just recover to the point in time when the basebackup ended?
>          Please correct me if I am wrong.

Correct.

> Case 2: Recovery to the latest point – For example if a server crashes, I can simply
>         restart the server and postgres and it will rollforward all the active logs since
>         the last flush oft he data buffers to disk happened . Is my understanding correct ?
>         While I understand that all the log buffers are written as soon as a commit is
>         issued, how does Postgres keep track of when the data buffers were flushed to disk?
>         In other words, how does Postgres know from which log sequence number it must begin
>         rolling forward?  Can we as users also see it?

The redo LSN of the latest checkpoint is stored in the control file, so that PostgreSQL
can start recovering from there.

You can use the pg_controldata command or the pg_control_checkpoint() table function
to get that information.

PostgreSQL doesn't need the archive for crash recovery, it uses the WAL in pg_wal.

> Case 3:  No archive logging and transaction that does not commit:  Let us say that we have
>          not enabled archive logging and we have a long running update that fills up all
>          the active logs (WALs) and it has wrapped around. What happens now?  Will the
>          system Hang?  Will Postgres cancel this thread and rollback?  Should we manually
>          cancel the thread?  How can we find out which thread is writing without commits?

Transactions are independent of checkpoints.  The WAL from the uncommitted transaction will
just be archived and removed from pg_wal after a checkpoint.  PostgreSQL never needs to
undo the modifications from an uncommitted transaction.

> Case 4: Recovery to the latest point with Archive logging enabled:  If  I have archive
>         logging enabled and I have transactions that wrap around the logs without commit
>         and the server crashes,  is it correct to assume that  I need
>    a. my base backup ( with all the ‚then copied WALs‘ )
>    b.  the current WALs in the active WAL directory
>    c. All the archived WALs from the time the base backup began to the time the crash happened
>  
> to bring the system to a consistent state?

No, the WAL in pg_wal is enough.  You don't need an archive for that.
The uncommitted data just get recovered while remaining invisible, and the next VACUUM
run will remove them.

> My question in short is, Is it enough if I hold on to my archive logs since the last backup
> for any recovery scenareo?

You need to retain WAL from the *start* of the earliest backup you need to recover on.

I'd recommend that you look into a ready-made backup solution like pgBackRest.

Yours,
Laurenz Albe



AW: how long should Archive logs be retained

От
"Subramanian,Ramachandran"
Дата:
Hello

  Thank you so much for your Answers.  

I am unable to understand the answer to Case 3.  May I restate it again with your permission.

Archive logging is NOT enabled  and a transaction keeps filling up the logs one by one  till all the logs are full
WITHOUTAny COMMIT. Then what happens?  Does Postgres hang or does Postgres terminate this thread ? 
 

After reading your answers, I understand that Archive logs are needed ONLY if we want to go back in time to a past
pointof consistency ( example  Current date minus 5 days ) .  Ofcourse a begining backup is needed on this day as well
(a backup taken on current day - 5 days ) . Then using the backup and the Logs that were archived on this day , we can
rollforwardto a Point in time on this day.   1. Did I understand correctly ?  2. Is there any other scenareo where
Archivelogs will be used?
 


I will look into a pre made solution like pgbackrest. But I want to understand the concepts before that. 


Thank you again for taking the time for my questions.

LG

Ram



Freundliche Grüße

i. A. Ramachandran Subramanian
Zentralbereich Informationstechnologie

Alte Leipziger Lebensversicherung a.G.


Hallesche Krankenversicherung a.G.







Alte Leipziger Lebensversicherung a.G., Alte Leipziger-Platz 1, 61440 Oberursel
Vors. des Aufsichtsrats: Dr. Walter Botermann · Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.),
FrankKettnaker, Dr. Jochen Kriegmeier, Alexander Mayer, Christian Pape, Wiltrud Pekarek, Udo Wilcsek
 
Sitz Oberursel (Taunus) · Rechtsform VVaG · Amtsgericht Bad Homburg v. d. H. HRB 1583 · USt.-IdNr. DE 114106814





 
Hallesche Krankenversicherung a.G.,  Löffelstraße 34-38, 70597 Stuttgart
Vors. des Aufsichtsrats: Dr. Walter Botermann · Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.),
FrankKettnaker, Dr. Jochen Kriegmeier, Alexander Mayer, Christian Pape,
 
Wiltrud Pekarek, Udo Wilcsek
Sitz Stuttgart · Rechtsform VVaG · Amtsgericht Stuttgart HRB 2686 · USt.-IdNr. DE 147802285
Beiträge zu privaten Kranken- und Pflegekrankenversicherungen unterliegen nicht der Versicherungsteuer (§ 4 Nr. 5
VersStG)· Versicherungsleistungen sowie Umsätze aus Versicherungsvertreter-/Maklertätigkeiten sind umsatzsteuerfrei
 
 



 
Die Pflichtangaben der ALH Gruppe gemäß § 35a GmbHG bzw. § 80 AktG finden Sie hier:
https://www.alte-leipziger.de/impressum
 





______________________

ALH Gruppe
Alte Leipziger-Platz 1, 61440 Oberursel
Tel.: +49 (6171) 66-4882
Fax: +49 (6171) 66-800-4882
E-Mail: ramachandran.subramanian@alte-leipziger.de
www.alte-leipziger.de
www.hallesche.de



-----Ursprüngliche Nachricht-----
Von: Laurenz Albe <laurenz.albe@cybertec.at> 
Gesendet: Donnerstag, 27. November 2025 15:00
An: Subramanian,Ramachandran IT-md-db <ramachandran.subramanian@alte-leipziger.de>; pgsql-novice@lists.postgresql.org
Betreff: Re: how long should Archive logs be retained

On Thu, 2025-11-27 at 10:34 +0000, Subramanian,Ramachandran wrote:
> How long should archived logs be retained?
>  
> I make pg_basebackups once  a day  with wal-method ‚stream‘  and 
> postgresql.conf has wal_level = replica and  archive_mode=on
>  
> Case 1:  Recovery to the point in time when the basebackup ended--  Am I correct in
>          assuming that I do not need anything else other than the base backup if I want
>          to just recover to the point in time when the basebackup ended?
>          Please correct me if I am wrong.

Correct.

> Case 2: Recovery to the latest point – For example if a server crashes, I can simply
>         restart the server and postgres and it will rollforward all the active logs since
>         the last flush oft he data buffers to disk happened . Is my understanding correct ?
>         While I understand that all the log buffers are written as soon as a commit is
>         issued, how does Postgres keep track of when the data buffers were flushed to disk?
>         In other words, how does Postgres know from which log sequence number it must begin
>         rolling forward?  Can we as users also see it?

The redo LSN of the latest checkpoint is stored in the control file, so that PostgreSQL can start recovering from
there.

You can use the pg_controldata command or the pg_control_checkpoint() table function to get that information.

PostgreSQL doesn't need the archive for crash recovery, it uses the WAL in pg_wal.

> Case 3:  No archive logging and transaction that does not commit:  Let us say that we have
>          not enabled archive logging and we have a long running update that fills up all
>          the active logs (WALs) and it has wrapped around. What happens now?  Will the
>          system Hang?  Will Postgres cancel this thread and rollback?  Should we manually
>          cancel the thread?  How can we find out which thread is writing without commits?

Transactions are independent of checkpoints.  The WAL from the uncommitted transaction will just be archived and
removedfrom pg_wal after a checkpoint.  PostgreSQL never needs to undo the modifications from an uncommitted
transaction.

> Case 4: Recovery to the latest point with Archive logging enabled:  If  I have archive
>         logging enabled and I have transactions that wrap around the logs without commit
>         and the server crashes,  is it correct to assume that  I need
>    a. my base backup ( with all the ‚then copied WALs‘ )
>    b.  the current WALs in the active WAL directory
>    c. All the archived WALs from the time the base backup began to the 
> time the crash happened
>  
> to bring the system to a consistent state?

No, the WAL in pg_wal is enough.  You don't need an archive for that.
The uncommitted data just get recovered while remaining invisible, and the next VACUUM run will remove them.

> My question in short is, Is it enough if I hold on to my archive logs 
> since the last backup for any recovery scenareo?

You need to retain WAL from the *start* of the earliest backup you need to recover on.

I'd recommend that you look into a ready-made backup solution like pgBackRest.

Yours,
Laurenz Albe

Re: how long should Archive logs be retained

От
Laurenz Albe
Дата:
On Thu, 2025-11-27 at 15:49 +0000, Subramanian,Ramachandran wrote:
> Archive logging is NOT enabled  and a transaction keeps filling up the logs one by one
> till all the logs are full WITHOUT Any COMMIT. Then what happens?  Does Postgres hang
> or does Postgres terminate this thread ?

Neither.
As I said, PostgreSQL will happily remove WAL segments that contain uncommitted
transactions.  PostgreSQL will never undo any work.  If a transaction is rolled back,
all the uncommitted data are still there, they are only invisible.
Later, the autovacuum daemon (a kind of garbage collection) will remove these data,
but that's a separate activity (that is again logged to WAL).

> After reading your answers, I understand that Archive logs are needed ONLY if we want
> to go back in time to a past point of consistency ( example  Current date minus 5 days ) .

No, archive logs are needed whenever you you need to go back in time at all.
With the WAL in pg_wal, all you can do is recover after a crash, which will recover
all committed transactions.

> Ofcourse a begining backup is needed on this day as well ( a backup taken on current
> day - 5 days ) . Then using the backup and the Logs that were archived on this day ,
> we can rollforward to a Point in time on this day.   1. Did I understand correctly ?  2.
> Is there any other scenareo where Archive logs will be used?

I am not sure I can follow.

You need a WAL archive whenever you want to undo *anything*.  For that, you have to
restore a base backup that *finished* before the point in time that you want to
recover to.  Also, you need all WAL from the start of the base backup until the point
in time to which you want to recover.

Yours,
Laurenz Albe



AW: how long should Archive logs be retained

От
"Subramanian,Ramachandran"
Дата:
Thank you.. You answered my question  with the below comment.

--------------------------------------------------------

You need a WAL archive whenever you want to undo *anything*.  For that, you have to restore a base backup that
*finished*before the point in time that you want to recover to.  Also, you need all WAL from the start of the base
backupuntil the point in time to which you want to recover.
 


Regarding your below comment
---------------------------------------------
As I said, PostgreSQL will happily remove WAL segments that contain uncommitted transactions.  PostgreSQL will never
undoany work.  If a transaction is rolled back, all the uncommitted data are still there, they are only invisible.
 
Later, the autovacuum daemon (a kind of garbage collection) will remove these data, but that's a separate activity
(thatis again logged to WAL).
 

Question >>>  Wow !!  That is so new to me.  So
1. if log wrap around happens  and some Wal-Files containing uncomitted log records are being overwritten,   
2. AND  iin the mean time some data updates pertaining to these uncomitted transactions have been flushed from the data
buffersto disk
 
3. AND after the data buffers have been flushed the transaction fails ( and I assume releases all the locks ) 

 how does postgres know that  a particular unlocked row it finds in the disk ( or even the bufferpool for that matter )
isnot in a valid state  even though it physically exists in the table?  How can it decode this information?
 


LG

Ram 






Freundliche Grüße

i. A. Ramachandran Subramanian
Zentralbereich Informationstechnologie

Alte Leipziger Lebensversicherung a.G.


Hallesche Krankenversicherung a.G.







Alte Leipziger Lebensversicherung a.G., Alte Leipziger-Platz 1, 61440 Oberursel
Vors. des Aufsichtsrats: Dr. Walter Botermann · Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.),
FrankKettnaker, Dr. Jochen Kriegmeier, Alexander Mayer, Christian Pape, Wiltrud Pekarek, Udo Wilcsek
 
Sitz Oberursel (Taunus) · Rechtsform VVaG · Amtsgericht Bad Homburg v. d. H. HRB 1583 · USt.-IdNr. DE 114106814





 
Hallesche Krankenversicherung a.G.,  Löffelstraße 34-38, 70597 Stuttgart
Vors. des Aufsichtsrats: Dr. Walter Botermann · Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.),
FrankKettnaker, Dr. Jochen Kriegmeier, Alexander Mayer, Christian Pape,
 
Wiltrud Pekarek, Udo Wilcsek
Sitz Stuttgart · Rechtsform VVaG · Amtsgericht Stuttgart HRB 2686 · USt.-IdNr. DE 147802285
Beiträge zu privaten Kranken- und Pflegekrankenversicherungen unterliegen nicht der Versicherungsteuer (§ 4 Nr. 5
VersStG)· Versicherungsleistungen sowie Umsätze aus Versicherungsvertreter-/Maklertätigkeiten sind umsatzsteuerfrei
 
 



 
Die Pflichtangaben der ALH Gruppe gemäß § 35a GmbHG bzw. § 80 AktG finden Sie hier:
https://www.alte-leipziger.de/impressum
 





______________________

ALH Gruppe
Alte Leipziger-Platz 1, 61440 Oberursel
Tel.: +49 (6171) 66-4882
Fax: +49 (6171) 66-800-4882
E-Mail: ramachandran.subramanian@alte-leipziger.de
www.alte-leipziger.de
www.hallesche.de



-----Ursprüngliche Nachricht-----
Von: Laurenz Albe <laurenz.albe@cybertec.at> 
Gesendet: Donnerstag, 27. November 2025 20:34
An: Subramanian,Ramachandran IT-md-db <ramachandran.subramanian@alte-leipziger.de>; pgsql-novice@lists.postgresql.org
Betreff: Re: how long should Archive logs be retained

On Thu, 2025-11-27 at 15:49 +0000, Subramanian,Ramachandran wrote:
> Archive logging is NOT enabled  and a transaction keeps filling up the 
> logs one by one till all the logs are full WITHOUT Any COMMIT. Then 
> what happens?  Does Postgres hang or does Postgres terminate this thread ?

Neither.
As I said, PostgreSQL will happily remove WAL segments that contain uncommitted transactions.  PostgreSQL will never
undoany work.  If a transaction is rolled back, all the uncommitted data are still there, they are only invisible.
 
Later, the autovacuum daemon (a kind of garbage collection) will remove these data, but that's a separate activity
(thatis again logged to WAL).
 

> After reading your answers, I understand that Archive logs are needed 
> ONLY if we want to go back in time to a past point of consistency ( example  Current date minus 5 days ) .

No, archive logs are needed whenever you you need to go back in time at all.
With the WAL in pg_wal, all you can do is recover after a crash, which will recover all committed transactions.

> Ofcourse a begining backup is needed on this day as well ( a backup 
> taken on current day - 5 days ) . Then using the backup and the Logs 
> that were archived on this day , we can rollforward to a Point in time on this day.   1. Did I understand correctly
? 2.
 
> Is there any other scenareo where Archive logs will be used?

I am not sure I can follow.

You need a WAL archive whenever you want to undo *anything*.  For that, you have to restore a base backup that
*finished*before the point in time that you want to recover to.  Also, you need all WAL from the start of the base
backupuntil the point in time to which you want to recover.
 

Yours,
Laurenz Albe

AW: how long should Archive logs be retained

От
"Subramanian,Ramachandran"
Дата:
I googled and came upon your article about levels of isolation in cybertec.

 Which lead me to a concept called MVCC . 


Wow!!!   Now I understand how it works!! 

  Each transaction has it's own private copy of the data it is interested in, and each private copy of the data has
it'sunique transaction number associated with it. Depending on weather the transaction (TranID) committed or not, and
whocommitted what in which order, a final picture emerges. 
 

Coming from the mainframe world, this is new to me.


Thank you for taking the time to write.


Regards

Ram 





Freundliche Grüße

i. A. Ramachandran Subramanian
Zentralbereich Informationstechnologie

Alte Leipziger Lebensversicherung a.G.


Hallesche Krankenversicherung a.G.






______________________

ALH Gruppe
Alte Leipziger-Platz 1, 61440 Oberursel
Tel.: +49 (6171) 66-4882
Fax: +49 (6171) 66-800-4882
E-Mail: ramachandran.subramanian@alte-leipziger.de
www.alte-leipziger.de
www.hallesche.de



-----Ursprüngliche Nachricht-----
Von: Subramanian,Ramachandran <ramachandran.subramanian@alte-leipziger.de> 
Gesendet: Freitag, 28. November 2025 05:46
An: Laurenz Albe <laurenz.albe@cybertec.at>; pgsql-novice@lists.postgresql.org
Betreff: AW: how long should Archive logs be retained

Thank you.. You answered my question  with the below comment.

--------------------------------------------------------

You need a WAL archive whenever you want to undo *anything*.  For that, you have to restore a base backup that
*finished*before the point in time that you want to recover to.  Also, you need all WAL from the start of the base
backupuntil the point in time to which you want to recover.
 


Regarding your below comment
---------------------------------------------
As I said, PostgreSQL will happily remove WAL segments that contain uncommitted transactions.  PostgreSQL will never
undoany work.  If a transaction is rolled back, all the uncommitted data are still there, they are only invisible.
 
Later, the autovacuum daemon (a kind of garbage collection) will remove these data, but that's a separate activity
(thatis again logged to WAL).
 

Question >>>  Wow !!  That is so new to me.  So
1. if log wrap around happens  and some Wal-Files containing uncomitted log records are being overwritten,   
2. AND  iin the mean time some data updates pertaining to these uncomitted transactions have been flushed from the data
buffersto disk 3. AND after the data buffers have been flushed the transaction fails ( and I assume releases all the
locks) 
 

 how does postgres know that  a particular unlocked row it finds in the disk ( or even the bufferpool for that matter )
isnot in a valid state  even though it physically exists in the table?  How can it decode this information?
 


LG

Ram 






Freundliche Grüße

i. A. Ramachandran Subramanian
Zentralbereich Informationstechnologie

Alte Leipziger Lebensversicherung a.G.


Hallesche Krankenversicherung a.G.







Alte Leipziger Lebensversicherung a.G., Alte Leipziger-Platz 1, 61440 Oberursel Vors. des Aufsichtsrats: Dr. Walter
Botermann· Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.), Frank Kettnaker, Dr. Jochen Kriegmeier,
AlexanderMayer, Christian Pape, Wiltrud Pekarek, Udo Wilcsek Sitz Oberursel (Taunus) · Rechtsform VVaG · Amtsgericht
BadHomburg v. d. H. HRB 1583 · USt.-IdNr. DE 114106814
 





 
Hallesche Krankenversicherung a.G.,  Löffelstraße 34-38, 70597 Stuttgart Vors. des Aufsichtsrats: Dr. Walter Botermann
·Vorstand: Christoph Bohn (Vors.), Dr. Jürgen Bierbaum (stv. Vors.), Frank Kettnaker, Dr. Jochen Kriegmeier, Alexander
Mayer,Christian Pape, Wiltrud Pekarek, Udo Wilcsek Sitz Stuttgart · Rechtsform VVaG · Amtsgericht Stuttgart HRB 2686 ·
USt.-IdNr.DE 147802285 Beiträge zu privaten Kranken- und Pflegekrankenversicherungen unterliegen nicht der
Versicherungsteuer(§ 4 Nr. 5 VersStG) · Versicherungsleistungen sowie Umsätze aus
Versicherungsvertreter-/Maklertätigkeitensind umsatzsteuerfrei
 
 



 
Die Pflichtangaben der ALH Gruppe gemäß § 35a GmbHG bzw. § 80 AktG finden Sie hier:
https://www.alte-leipziger.de/impressum
 





______________________

ALH Gruppe
Alte Leipziger-Platz 1, 61440 Oberursel
Tel.: +49 (6171) 66-4882
Fax: +49 (6171) 66-800-4882
E-Mail: ramachandran.subramanian@alte-leipziger.de
www.alte-leipziger.de
www.hallesche.de



-----Ursprüngliche Nachricht-----
Von: Laurenz Albe <laurenz.albe@cybertec.at>
Gesendet: Donnerstag, 27. November 2025 20:34
An: Subramanian,Ramachandran IT-md-db <ramachandran.subramanian@alte-leipziger.de>; pgsql-novice@lists.postgresql.org
Betreff: Re: how long should Archive logs be retained

On Thu, 2025-11-27 at 15:49 +0000, Subramanian,Ramachandran wrote:
> Archive logging is NOT enabled  and a transaction keeps filling up the 
> logs one by one till all the logs are full WITHOUT Any COMMIT. Then 
> what happens?  Does Postgres hang or does Postgres terminate this thread ?

Neither.
As I said, PostgreSQL will happily remove WAL segments that contain uncommitted transactions.  PostgreSQL will never
undoany work.  If a transaction is rolled back, all the uncommitted data are still there, they are only invisible.
 
Later, the autovacuum daemon (a kind of garbage collection) will remove these data, but that's a separate activity
(thatis again logged to WAL).
 

> After reading your answers, I understand that Archive logs are needed 
> ONLY if we want to go back in time to a past point of consistency ( example  Current date minus 5 days ) .

No, archive logs are needed whenever you you need to go back in time at all.
With the WAL in pg_wal, all you can do is recover after a crash, which will recover all committed transactions.

> Ofcourse a begining backup is needed on this day as well ( a backup 
> taken on current day - 5 days ) . Then using the backup and the Logs 
> that were archived on this day , we can rollforward to a Point in time on this day.   1. Did I understand correctly
? 2.
 
> Is there any other scenareo where Archive logs will be used?

I am not sure I can follow.

You need a WAL archive whenever you want to undo *anything*.  For that, you have to restore a base backup that
*finished*before the point in time that you want to recover to.  Also, you need all WAL from the start of the base
backupuntil the point in time to which you want to recover.
 

Yours,
Laurenz Albe

Re: AW: how long should Archive logs be retained

От
Laurenz Albe
Дата:
On Fri, 2025-11-28 at 04:46 +0000, Subramanian,Ramachandran wrote:
> Question >>>  Wow !!  That is so new to me.  So
> 1. if log wrap around happens  and some Wal-Files containing uncomitted log records are being overwritten,

No, they are deleted (well, perhaps recycled, but that amounts to the same).

> 2. AND  iin the mean time some data updates pertaining to these uncomitted transactions have been flushed from the
databuffers to disk 
> 3. AND after the data buffers have been flushed the transaction fails ( and I assume releases all the locks )
>
> how does postgres know that  a particular unlocked row it finds in the disk ( or even the bufferpool
> for that matter ) is not in a valid state  even though it physically exists in the table?
> How can it decode this information?

From the commit log that stores the state of every transaction.

Each table row knows which transaction created it and which transaction removed it.
So if the creating transaction is not committed, or if the deleting transaction is committed,
you cannot see the data.

Yours,
Laurenz Albe



Re: AW: how long should Archive logs be retained

От
Laurenz Albe
Дата:
On Fri, 2025-11-28 at 05:28 +0000, Subramanian,Ramachandran wrote:
> Each transaction has it's own private copy of the data it is interested in,
> and each private copy of the data has it's unique transaction number associated
> with it.

A snapshot is not a copy of the data.  It is the list of transaction IDs that
are visible and the ones that are not.  That snapshot determines which of the
data in the database you can see.

Yours,
Laurenz Albe