RE: bdr replication breaks down

Hi again,

So I checked the x_logs and found the following lines.

It seems like it faild at commiting a long transaction…

I am still not sure what the pg_temp30 schema is used for.

But I was able to “fix” it with

select bdr.skip_changes_upto('6208877715678412212', 1, 2942745, '7D/2E314530');

I guess the listed transaction is now missing in some of the nodes.

rmgr: Standby     len (rec/tot):     28/    60, tx:          0, lsn: 7D/2E313238, prev 7D/2E313208, bkp: 0000, desc: running xacts: nextXid 106791716 latestCompletedXid 106791714 oldestRunningXid 106791715; 1 xacts: 106791715

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313278, prev 7D/2E313238, bkp: 0000, desc: new_cid: rel 1663/2942745/12834; tid 14/1; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3132C0, prev 7D/2E313278, bkp: 0000, desc: delete: rel 1663/2942745/12834; tid 14/1 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313300, prev 7D/2E3132C0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/82; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313348, prev 7D/2E313300, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/82 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313388, prev 7D/2E313348, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/81; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3133D0, prev 7D/2E313388, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/81 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313410, prev 7D/2E3133D0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/80; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313458, prev 7D/2E313410, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/80 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313498, prev 7D/2E313458, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/79; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3134E0, prev 7D/2E313498, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/79 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313520, prev 7D/2E3134E0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/78; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313568, prev 7D/2E313520, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/78 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3135A8, prev 7D/2E313568, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/77; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3135F0, prev 7D/2E3135A8, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/77 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313630, prev 7D/2E3135F0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/76; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313678, prev 7D/2E313630, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/76 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3136B8, prev 7D/2E313678, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/75; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313700, prev 7D/2E3136B8, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/75 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313740, prev 7D/2E313700, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/74; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313788, prev 7D/2E313740, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/74 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3137C8, prev 7D/2E313788, bkp: 0000, desc: new_cid: rel 1663/2942745/12747; tid 12/10; cmin: 4294967295, cmax: 1, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313810, prev 7D/2E3137C8, bkp: 0000, desc: delete: rel 1663/2942745/12747; tid 12/10 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313850, prev 7D/2E313810, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/72; cmin: 4294967295, cmax: 1, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313898, prev 7D/2E313850, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/72 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3138D8, prev 7D/2E313898, bkp: 0000, desc: new_cid: rel 1663/2942745/12747; tid 12/9; cmin: 4294967295, cmax: 2, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313920, prev 7D/2E3138D8, bkp: 0000, desc: delete: rel 1663/2942745/12747; tid 12/9 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313960, prev 7D/2E313920, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/71; cmin: 4294967295, cmax: 2, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3139A8, prev 7D/2E313960, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/71 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3139E8, prev 7D/2E3139A8, bkp: 0000, desc: new_cid: rel 1663/2942745/12759; tid 74/33; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313A30, prev 7D/2E3139E8, bkp: 0000, desc: delete: rel 1663/2942745/12759; tid 74/33 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313A70, prev 7D/2E313A30, bkp: 0000, desc: new_cid: rel 1663/2942745/12759; tid 74/34; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313AB8, prev 7D/2E313A70, bkp: 0000, desc: delete: rel 1663/2942745/12759; tid 74/34 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313AF8, prev 7D/2E313AB8, bkp: 0000, desc: new_cid: rel 1663/2942745/12759; tid 74/35; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313B40, prev 7D/2E313AF8, bkp: 0000, desc: delete: rel 1663/2942745/12759; tid 74/35 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313B80, prev 7D/2E313B40, bkp: 0000, desc: new_cid: rel 1663/2942745/12770; tid 14/33; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313BC8, prev 7D/2E313B80, bkp: 0000, desc: delete: rel 1663/2942745/12770; tid 14/33 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313C08, prev 7D/2E313BC8, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/73; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313C50, prev 7D/2E313C08, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/73 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313C90, prev 7D/2E313C50, bkp: 0000, desc: new_cid: rel 1664/0/12906; tid 1/49; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313CD8, prev 7D/2E313C90, bkp: 0000, desc: delete: rel 1664/0/12906; tid 1/49 KEYS_UPDATED

rmgr: Heap        len (rec/tot):    160/  1660, tx:  106791715, lsn: 7D/2E313D18, prev 7D/2E313CD8, bkp: 1000, desc: insert: rel 1663/2942745/2943410; tid 0/7

rmgr: Transaction len (rec/tot):    352/   384, tx:  106791715, lsn: 7D/2E3143B0, prev 7D/2E313D18, bkp: 0000, desc: commit: 2019-01-28 01:12:34.348577 PST; inval msgs: catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 46 catcache 45 catcache 64 catcache 63 catcache 64 catcache 63 catcache 49 relcache 23783862 snapshot 2608 snapshot 1214 snapshot 2608 snapshot 2608 relcache 23783862 snapshot 2608

rmgr: Standby     len (rec/tot):     44/    76, tx:          0, lsn: 7D/2E314530, prev 7D/2E3143B0, bkp: 0000, desc: nontransactional message with 35 bytes

Hi all,

I have a bdr replication setup that worked fine until recently.

But it seems like one command cannot be replicated, and it breaks down the whole process on all hosts where it is sent to.

I appended part of the log file, it gets repeated over and over.

Can I somehow omit this command and try if the next one works?

Best Regards,

Daniel Fink

< 2019-02-08 00:00:01.658 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:01.658 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:01.661 CET >LOG:  worker process: bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27640) exited with exit code 1

< 2019-02-08 00:00:06.666 CET >LOG:  starting background worker process "bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:07.641 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:07.641 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:07.644 CET >LOG:  worker process: bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27645) exited with exit code 1

< 2019-02-08 00:00:12.649 CET >LOG:  starting background worker process "bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:13.633 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:13.633 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

DANIEL FINK
Senior Software Engineer

tel (+49) 89.767062.20
fax(+49) 89.767062.11
email daniel.fink@pdf.com

PDF Solutions GmbH 
(a PDF Solutions Company)
Managing Director: Kimon Michaels
Schwanthalerstr. 10
D-80336 München, Germany  

München HRB 87307
DE 128214899

www.pdf.com