Обсуждение: BUG #14180: Segmentation fault on replication slave

VGhlIGZvbGxvd2luZyBidWcgaGFzIGJlZW4gbG9nZ2VkIG9uIHRoZSB3ZWJz
aXRlOgoKQnVnIHJlZmVyZW5jZTogICAgICAxNDE4MApMb2dnZWQgYnk6ICAg
ICAgICAgIEJvIMOYcnN0ZWQgQW5kcmVzZW4KRW1haWwgYWRkcmVzczogICAg
ICBib2FAbmVvZ3JpZC5kawpQb3N0Z3JlU1FMIHZlcnNpb246IDkuNS4zCk9w
ZXJhdGluZyBzeXN0ZW06ICAgVWJ1bnR1IDE2LjA0IExUUwpEZXNjcmlwdGlv
bjogICAgICAgIAoKSGVsbG8sDQoNCldlIGhhdmUgYSByZXBsaWNhdGlvbiBz
bG90IHNldHVwIHdoZXJlIHRoZSByZXBsaWNhdGlvbiBjYXVzZXMgYSBzZWdt
ZW50YXRpb24KZmF1bHQgd2l0aGluIGVpZ2h0IGhvdXJzIGFmdGVyIGEgcmVi
dWlsZCBvZiB0aGUgc2xhdmUuDQoNCkluIHRoZSBmb2xsb3dpbmcgdGhlIG1h
c3RlciBpcyBJUCAxMC4wLjAuMiBhbmQgdGhlIHNsYXZlIGlzIElQIDEwLjAu
MC4zLg0KDQpPbiB0aGUgbWFzdGVyIHdlIGhhdmUgdGhlIGRlZmF1bHRzIGFu
ZCB0aGUgZm9sbG93aW5nOg0KDQovZXRjL3Bvc3RncmVzcWwvOS41L21haW4v
cG9zdGdyZXNxbC5jb25mDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tDQpsaXN0ZW5fYWRkcmVzc2VzID0gJyonDQpwb3J0ID0g
NTQzMw0Kd2FsX2xldmVsID0gYXJjaGl2ZQ0KYXJjaGl2ZV9tb2RlID0gb24N
CmFyY2hpdmVfY29tbWFuZCA9ICd0ZXN0ICEgLWYgL21udC9wb3N0Z3Jlc19h
cmNoaXZlLyVmICYmIGNwICVwCi9tbnQvcG9zdGdyZXNfYXJjaGl2ZS8lZicN
Cm1heF93YWxfc2VuZGVycyA9IDUNCndhbF9rZWVwX3NlZ21lbnRzID0gNDAw
MA0KbWF4X3JlcGxpY2F0aW9uX3Nsb3RzID0gMQ0KdGltZXpvbmUgPSAnVVRD
Jw0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K
DQovZXRjL3Bvc3RncmVzcWwvOS41L21haW4vcGdfaGJhLmNvbmYNCi0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCmhvc3QgICAg
cmVwbGljYXRpb24gICAgIHBvc3RncmVzICAgICAgICAgMTAuMC4wLjMvMzIg
ICAgICAgIHRydXN0DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tDQoNCk9uIHRoZSBzbGF2ZSB3ZSBoYXZlIHRoZSBkZWZhdWx0
cyBhbmQgdGhlIHRpbWV6b25lIGNoYW5nZWQ6DQoNCi9ldGMvcG9zdGdyZXNx
bC85LjUvbWFpbi9wb3N0Z3Jlc3FsLmNvbmYuDQotLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQp0aW1lem9uZSA9ICdVVEMnDQot
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCk9u
IHRoZSBtYXN0ZXIgd2UgcnVuIHRoZSBTUUwgcXVlcnk6DQoNClNFTEVDVCAq
IEZST00gcGdfY3JlYXRlX3BoeXNpY2FsX3JlcGxpY2F0aW9uX3Nsb3QoJ3Ns
YXZlJyk7DQoNCk9uIHRoZSBzbGF2ZSB3ZSBydW4gdGhlIGNvbW1hbmQ6DQoN
CnBnX2Jhc2ViYWNrdXAgLVAgLVIgLVggc3RyZWFtIC1jIGZhc3QgLWggMTAu
MC4wLjIgLXAgNTQzMyAtVSBwb3N0Z3JlcyAtRAovdmFyL2xpYi9wb3N0Z3Jl
c3FsLzkuNS9tYWluDQoNCkFmdGVyIHRoaXMgcmVjb3ZlcnkuY29uZiBsb29r
cyBsaWtlIHRoaXMgKHdoZXJlIHdlIGFkZGVkIHRoZSBzbG90IGxpbmUpOg0K
DQpzdGFuZGJ5X21vZGUgPSAnb24nDQpwcmltYXJ5X2Nvbm5pbmZvID0gJ3Vz
ZXI9cG9zdGdyZXMgaG9zdD0xMC4wLjAuMiBwb3J0PTU0MzMgc3NsbW9kZT1w
cmVmZXIKc3NsY29tcHJlc3Npb249MSBrcmJzcnZuYW1lPXBvc3RncmVzJw0K
cHJpbWFyeV9zbG90X25hbWUgPSAnc2xhdmUnDQoNClRoZW4gd2UgZml4IHRo
ZSBvd25lcnNoaXAgYW5kIHN0YXJ0IHRoZSBzbGF2ZSBkYXRhYmFzZS4gQWZ0
ZXIgYSB3aGlsZSAtCmFueXRoaW5nIGJldHdlZW4gdGVuIG1pbnV0ZXMgYW5k
IGVpZ2h0IGhvdXJzIHdlIGdldCB0aGlzIGVycm9yIGluIHRoZSBsb2cKZmls
ZToNCg0KMjAxNi0wNi0wMyAwNTo1NToyNyBVVEMgWzI3MzAzLTRdIExPRzog
IHN0YXJ0dXAgcHJvY2VzcyAoUElEIDI3MzA1KSB3YXMKdGVybWluYXRlZCBi
eSBzaWduYWwgMTE6IFNlZ21lbnRhdGlvbiBmYXVsdA0KMjAxNi0wNi0wMyAw
NTo1NToyNyBVVEMgWzI3MzAzLTVdIExPRzogIHRlcm1pbmF0aW5nIGFueSBv
dGhlciBhY3RpdmUgc2VydmVyCnByb2Nlc3Nlcw0KDQpJZiB3ZSBhdHRhY2gg
d2l0aCBnZGIgYmVmb3JlIHRoZSBzZWdtZW50YXRpb24gZmF1bHQgd2UgZ2V0
Og0KDQojIGdkYiAtcCAzMDUyNA0KR05VIGdkYiAoVWJ1bnR1IDcuMTEtMHVi
dW50dTEpIDcuMTENCkNvcHlyaWdodCAoQykgMjAxNiBGcmVlIFNvZnR3YXJl
IEZvdW5kYXRpb24sIEluYy4NCkxpY2Vuc2UgR1BMdjMrOiBHTlUgR1BMIHZl
cnNpb24gMyBvciBsYXRlcgo8aHR0cDovL2dudS5vcmcvbGljZW5zZXMvZ3Bs
Lmh0bWw+DQpUaGlzIGlzIGZyZWUgc29mdHdhcmU6IHlvdSBhcmUgZnJlZSB0
byBjaGFuZ2UgYW5kIHJlZGlzdHJpYnV0ZSBpdC4NClRoZXJlIGlzIE5PIFdB
UlJBTlRZLCB0byB0aGUgZXh0ZW50IHBlcm1pdHRlZCBieSBsYXcuICBUeXBl
ICJzaG93IGNvcHlpbmciDQphbmQgInNob3cgd2FycmFudHkiIGZvciBkZXRh
aWxzLg0KVGhpcyBHREIgd2FzIGNvbmZpZ3VyZWQgYXMgIng4Nl82NC1saW51
eC1nbnUiLg0KVHlwZSAic2hvdyBjb25maWd1cmF0aW9uIiBmb3IgY29uZmln
dXJhdGlvbiBkZXRhaWxzLg0KRm9yIGJ1ZyByZXBvcnRpbmcgaW5zdHJ1Y3Rp
b25zLCBwbGVhc2Ugc2VlOg0KPGh0dHA6Ly93d3cuZ251Lm9yZy9zb2Z0d2Fy
ZS9nZGIvYnVncy8+Lg0KRmluZCB0aGUgR0RCIG1hbnVhbCBhbmQgb3RoZXIg
ZG9jdW1lbnRhdGlvbiByZXNvdXJjZXMgb25saW5lIGF0Og0KPGh0dHA6Ly93
d3cuZ251Lm9yZy9zb2Z0d2FyZS9nZGIvZG9jdW1lbnRhdGlvbi8+Lg0KRm9y
IGhlbHAsIHR5cGUgImhlbHAiLg0KVHlwZSAiYXByb3BvcyB3b3JkIiB0byBz
ZWFyY2ggZm9yIGNvbW1hbmRzIHJlbGF0ZWQgdG8gIndvcmQiLg0KQXR0YWNo
aW5nIHRvIHByb2Nlc3MgMzA1MjQNClJlYWRpbmcgc3ltYm9scyBmcm9tIC91
c3IvbGliL3Bvc3RncmVzcWwvOS41L2Jpbi9wb3N0Z3Jlcy4uLlJlYWRpbmcg
c3ltYm9scwpmcm9tCi91c3IvbGliL2RlYnVnLy5idWlsZC1pZC9jNi83NDQ0
Y2FlMmRiYzZiY2FjNDZlODA1MjkyMWMwMWMwNjc4MGQ3Mi5kZWJ1Zy4uLmRv
bmUuDQpkb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9saWIveDg2
XzY0LWxpbnV4LWdudS9saWJ4bWwyLnNvLjIuLi5SZWFkaW5nCnN5bWJvbHMg
ZnJvbQovdXNyL2xpYi9kZWJ1Zy8uYnVpbGQtaWQvYTEvNTVjN2JjMzQ1ZDBl
MGI3MTFiZTA5MTIwMjA0YmQ4OGY0NzVmOWUuZGVidWcuLi5kb25lLg0KZG9u
ZS4NClJlYWRpbmcgc3ltYm9scyBmcm9tIC9saWIveDg2XzY0LWxpbnV4LWdu
dS9saWJwYW0uc28uMC4uLihubyBkZWJ1Z2dpbmcKc3ltYm9scyBmb3VuZCku
Li5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL2xpYi94ODZfNjQtbGlu
dXgtZ251L2xpYnNzbC5zby4xLjAuMC4uLlJlYWRpbmcgc3ltYm9scwpmcm9t
Ci91c3IvbGliL2RlYnVnLy5idWlsZC1pZC84Mi8yNzU0Njk1ZTRiMzFhZTgy
OTM3MjU4YmRmZjNkNTJlZmEwYmEzNi5kZWJ1Zy4uLmRvbmUuDQpkb25lLg0K
UmVhZGluZyBzeW1ib2xzIGZyb20gL2xpYi94ODZfNjQtbGludXgtZ251L2xp
YmNyeXB0by5zby4xLjAuMC4uLlJlYWRpbmcKc3ltYm9scyBmcm9tCi91c3Iv
bGliL2RlYnVnLy5idWlsZC1pZC9iNy81YTk2YzU5YmUxYjViNTRmYmYxYTkx
ZWQ3MjJiZWM5NDA2Mjg4ZS5kZWJ1Zy4uLmRvbmUuDQpkb25lLg0KUmVhZGlu
ZyBzeW1ib2xzIGZyb20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJn
c3NhcGlfa3JiNS5zby4yLi4uKG5vCmRlYnVnZ2luZyBzeW1ib2xzIGZvdW5k
KS4uLmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvbGliL3g4Nl82NC1s
aW51eC1nbnUvbGlicnQuc28uMS4uLlJlYWRpbmcgc3ltYm9scyBmcm9tCi91
c3IvbGliL2RlYnVnLy9saWIveDg2XzY0LWxpbnV4LWdudS9saWJydC0yLjIz
LnNvLi4uZG9uZS4NCmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvbGli
L3g4Nl82NC1saW51eC1nbnUvbGliZGwuc28uMi4uLlJlYWRpbmcgc3ltYm9s
cyBmcm9tCi91c3IvbGliL2RlYnVnLy9saWIveDg2XzY0LWxpbnV4LWdudS9s
aWJkbC0yLjIzLnNvLi4uZG9uZS4NCmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMg
ZnJvbSAvbGliL3g4Nl82NC1saW51eC1nbnUvbGlibS5zby42Li4uUmVhZGlu
ZyBzeW1ib2xzIGZyb20KL3Vzci9saWIvZGVidWcvL2xpYi94ODZfNjQtbGlu
dXgtZ251L2xpYm0tMi4yMy5zby4uLmRvbmUuDQpkb25lLg0KUmVhZGluZyBz
eW1ib2xzIGZyb20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJsZGFw
X3ItMi40LnNvLjIuLi5SZWFkaW5nCnN5bWJvbHMgZnJvbQovdXNyL2xpYi9k
ZWJ1Zy8uYnVpbGQtaWQvYWQvZjZmNDFmMjIzZDQyMTkzMTY1ZmEwYzU1ODcx
ZjAyZDkxNWZiMTkuZGVidWcuLi5kb25lLg0KZG9uZS4NClJlYWRpbmcgc3lt
Ym9scyBmcm9tIC9saWIveDg2XzY0LWxpbnV4LWdudS9saWJjLnNvLjYuLi5S
ZWFkaW5nIHN5bWJvbHMgZnJvbQovdXNyL2xpYi9kZWJ1Zy8vbGliL3g4Nl82
NC1saW51eC1nbnUvbGliYy0yLjIzLnNvLi4uZG9uZS4NCmRvbmUuDQpSZWFk
aW5nIHN5bWJvbHMgZnJvbSAvdXNyL2xpYi94ODZfNjQtbGludXgtZ251L2xp
YmljdXVjLnNvLjU1Li4uUmVhZGluZwpzeW1ib2xzIGZyb20KL3Vzci9saWIv
ZGVidWcvLmJ1aWxkLWlkLzMyLzNlNDg3ODA3M2JiNGUwZDdiMTc0YWUyNGUz
ODNlYzVlMDVkNjhhLmRlYnVnLi4uZG9uZS4NCmRvbmUuDQpSZWFkaW5nIHN5
bWJvbHMgZnJvbSAvbGliL3g4Nl82NC1saW51eC1nbnUvbGliei5zby4xLi4u
UmVhZGluZyBzeW1ib2xzIGZyb20KL3Vzci9saWIvZGVidWcvLmJ1aWxkLWlk
LzM0LzBiN2I0NjNmOTgxYjhhMGZiMzQ1MTc1MWY4ODFkZjFiMGMyZjc0LmRl
YnVnLi4uZG9uZS4NCmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvbGli
L3g4Nl82NC1saW51eC1nbnUvbGlibHptYS5zby41Li4uKG5vIGRlYnVnZ2lu
ZwpzeW1ib2xzIGZvdW5kKS4uLmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJv
bSAvbGliL3g4Nl82NC1saW51eC1nbnUvbGliYXVkaXQuc28uMS4uLihubyBk
ZWJ1Z2dpbmcKc3ltYm9scyBmb3VuZCkuLi5kb25lLg0KUmVhZGluZyBzeW1i
b2xzIGZyb20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJrcmI1LnNv
LjMuLi4obm8gZGVidWdnaW5nCnN5bWJvbHMgZm91bmQpLi4uZG9uZS4NClJl
YWRpbmcgc3ltYm9scyBmcm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUv
bGliazVjcnlwdG8uc28uMy4uLihubwpkZWJ1Z2dpbmcgc3ltYm9scyBmb3Vu
ZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL2xpYi94ODZfNjQt
bGludXgtZ251L2xpYmNvbV9lcnIuc28uMi4uLlJlYWRpbmcgc3ltYm9scwpm
cm9tIC91c3IvbGliL2RlYnVnLy9saWIveDg2XzY0LWxpbnV4LWdudS9saWJj
b21fZXJyLnNvLjIuMS4uLmRvbmUuDQpkb25lLg0KUmVhZGluZyBzeW1ib2xz
IGZyb20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJrcmI1c3VwcG9y
dC5zby4wLi4uKG5vCmRlYnVnZ2luZyBzeW1ib2xzIGZvdW5kKS4uLmRvbmUu
DQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvbGliL3g4Nl82NC1saW51eC1nbnUv
bGlicHRocmVhZC5zby4wLi4uUmVhZGluZyBzeW1ib2xzCmZyb20KL3Vzci9s
aWIvZGVidWcvLmJ1aWxkLWlkL2I3Lzc4NDdjYzljYWNiY2EzYjU3NTNkMGQy
NWEzMmU1Nzk1YWZlNzViLmRlYnVnLi4uZG9uZS4NCmRvbmUuDQpbVGhyZWFk
IGRlYnVnZ2luZyB1c2luZyBsaWJ0aHJlYWRfZGIgZW5hYmxlZF0NClVzaW5n
IGhvc3QgbGlidGhyZWFkX2RiIGxpYnJhcnkgIi9saWIveDg2XzY0LWxpbnV4
LWdudS9saWJ0aHJlYWRfZGIuc28uMSIuDQpSZWFkaW5nIHN5bWJvbHMgZnJv
bSAvbGliNjQvbGQtbGludXgteDg2LTY0LnNvLjIuLi5SZWFkaW5nIHN5bWJv
bHMgZnJvbQovdXNyL2xpYi9kZWJ1Zy8vbGliL3g4Nl82NC1saW51eC1nbnUv
bGQtMi4yMy5zby4uLmRvbmUuDQpkb25lLg0KUmVhZGluZyBzeW1ib2xzIGZy
b20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJsYmVyLTIuNC5zby4y
Li4uUmVhZGluZwpzeW1ib2xzIGZyb20KL3Vzci9saWIvZGVidWcvLmJ1aWxk
LWlkLzZiLzlmNDA2MWExZDQ0ODEzYTU0ZGE0ZGJiMDA4OGY1MjlkOGQ3OGVh
LmRlYnVnLi4uZG9uZS4NCmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAv
bGliL3g4Nl82NC1saW51eC1nbnUvbGlicmVzb2x2LnNvLjIuLi5SZWFkaW5n
IHN5bWJvbHMKZnJvbSAvdXNyL2xpYi9kZWJ1Zy8vbGliL3g4Nl82NC1saW51
eC1nbnUvbGlicmVzb2x2LTIuMjMuc28uLi5kb25lLg0KZG9uZS4NClJlYWRp
bmcgc3ltYm9scyBmcm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvbGli
c2FzbDIuc28uMi4uLihubyBkZWJ1Z2dpbmcKc3ltYm9scyBmb3VuZCkuLi5k
b25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9saWIveDg2XzY0LWxp
bnV4LWdudS9saWJnc3NhcGkuc28uMy4uLihubwpkZWJ1Z2dpbmcgc3ltYm9s
cyBmb3VuZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9s
aWIveDg2XzY0LWxpbnV4LWdudS9saWJnbnV0bHMuc28uMzAuLi4obm8KZGVi
dWdnaW5nIHN5bWJvbHMgZm91bmQpLi4uZG9uZS4NClJlYWRpbmcgc3ltYm9s
cyBmcm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvbGliaWN1ZGF0YS5z
by41NS4uLihubwpkZWJ1Z2dpbmcgc3ltYm9scyBmb3VuZCkuLi5kb25lLg0K
UmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdu
dS9saWJzdGRjKysuc28uNi4uLihubwpkZWJ1Z2dpbmcgc3ltYm9scyBmb3Vu
ZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL2xpYi94ODZfNjQt
bGludXgtZ251L2xpYmdjY19zLnNvLjEuLi5SZWFkaW5nIHN5bWJvbHMKZnJv
bSAvdXNyL2xpYi9kZWJ1Zy8vbGliL3g4Nl82NC1saW51eC1nbnUvbGliZ2Nj
X3Muc28uMS4uLmRvbmUuDQpkb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20g
L2xpYi94ODZfNjQtbGludXgtZ251L2xpYmtleXV0aWxzLnNvLjEuLi4obm8g
ZGVidWdnaW5nCnN5bWJvbHMgZm91bmQpLi4uZG9uZS4NClJlYWRpbmcgc3lt
Ym9scyBmcm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvbGliaGVpbW50
bG0uc28uMC4uLihubwpkZWJ1Z2dpbmcgc3ltYm9scyBmb3VuZCkuLi5kb25l
Lg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9saWIveDg2XzY0LWxpbnV4
LWdudS9saWJrcmI1LnNvLjI2Li4uKG5vIGRlYnVnZ2luZwpzeW1ib2xzIGZv
dW5kKS4uLmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvdXNyL2xpYi94
ODZfNjQtbGludXgtZ251L2xpYmFzbjEuc28uOC4uLihubyBkZWJ1Z2dpbmcK
c3ltYm9scyBmb3VuZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20g
L3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJoY3J5cHRvLnNvLjQuLi4o
bm8KZGVidWdnaW5nIHN5bWJvbHMgZm91bmQpLi4uZG9uZS4NClJlYWRpbmcg
c3ltYm9scyBmcm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvbGlicm9r
ZW4uc28uMTguLi4obm8KZGVidWdnaW5nIHN5bWJvbHMgZm91bmQpLi4uZG9u
ZS4NClJlYWRpbmcgc3ltYm9scyBmcm9tIC91c3IvbGliL3g4Nl82NC1saW51
eC1nbnUvbGlicDExLWtpdC5zby4wLi4uKG5vCmRlYnVnZ2luZyBzeW1ib2xz
IGZvdW5kKS4uLmRvbmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvdXNyL2xp
Yi94ODZfNjQtbGludXgtZ251L2xpYmlkbi5zby4xMS4uLihubyBkZWJ1Z2dp
bmcKc3ltYm9scyBmb3VuZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZy
b20gL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9saWJ0YXNuMS5zby42Li4u
KG5vIGRlYnVnZ2luZwpzeW1ib2xzIGZvdW5kKS4uLmRvbmUuDQpSZWFkaW5n
IHN5bWJvbHMgZnJvbSAvdXNyL2xpYi94ODZfNjQtbGludXgtZ251L2xpYm5l
dHRsZS5zby42Li4uKG5vCmRlYnVnZ2luZyBzeW1ib2xzIGZvdW5kKS4uLmRv
bmUuDQpSZWFkaW5nIHN5bWJvbHMgZnJvbSAvdXNyL2xpYi94ODZfNjQtbGlu
dXgtZ251L2xpYmhvZ3dlZWQuc28uNC4uLihubwpkZWJ1Z2dpbmcgc3ltYm9s
cyBmb3VuZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9s
aWIveDg2XzY0LWxpbnV4LWdudS9saWJnbXAuc28uMTAuLi4obm8gZGVidWdn
aW5nCnN5bWJvbHMgZm91bmQpLi4uZG9uZS4NClJlYWRpbmcgc3ltYm9scyBm
cm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvbGlid2luZC5zby4wLi4u
KG5vIGRlYnVnZ2luZwpzeW1ib2xzIGZvdW5kKS4uLmRvbmUuDQpSZWFkaW5n
IHN5bWJvbHMgZnJvbSAvdXNyL2xpYi94ODZfNjQtbGludXgtZ251L2xpYmhl
aW1iYXNlLnNvLjEuLi4obm8KZGVidWdnaW5nIHN5bWJvbHMgZm91bmQpLi4u
ZG9uZS4NClJlYWRpbmcgc3ltYm9scyBmcm9tIC91c3IvbGliL3g4Nl82NC1s
aW51eC1nbnUvbGliaHg1MDkuc28uNS4uLihubyBkZWJ1Z2dpbmcKc3ltYm9s
cyBmb3VuZCkuLi5kb25lLg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL3Vzci9s
aWIveDg2XzY0LWxpbnV4LWdudS9saWJzcWxpdGUzLnNvLjAuLi5SZWFkaW5n
CnN5bWJvbHMgZnJvbQovdXNyL2xpYi9kZWJ1Zy8uYnVpbGQtaWQvZDkvNzgy
YmEwMjNjYWVjMjZiMTVkODY3NmUzYTVkMDdiNTVlMTIxZWYuZGVidWcuLi5k
b25lLg0KZG9uZS4NClJlYWRpbmcgc3ltYm9scyBmcm9tIC9saWIveDg2XzY0
LWxpbnV4LWdudS9saWJjcnlwdC5zby4xLi4uUmVhZGluZyBzeW1ib2xzCmZy
b20gL3Vzci9saWIvZGVidWcvL2xpYi94ODZfNjQtbGludXgtZ251L2xpYmNy
eXB0LTIuMjMuc28uLi5kb25lLg0KZG9uZS4NClJlYWRpbmcgc3ltYm9scyBm
cm9tIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvbGliZmZpLnNvLjYuLi5S
ZWFkaW5nIHN5bWJvbHMKZnJvbQovdXNyL2xpYi9kZWJ1Zy8uYnVpbGQtaWQv
OWQvOWM5NThmMWY0ODk0YWZlZjZhZWNkOTBkMWM0MzBlYTI5YWMzNGYuZGVi
dWcuLi5kb25lLg0KZG9uZS4NClJlYWRpbmcgc3ltYm9scyBmcm9tIC9saWIv
eDg2XzY0LWxpbnV4LWdudS9saWJuc3NfZmlsZXMuc28uMi4uLlJlYWRpbmcK
c3ltYm9scyBmcm9tCi91c3IvbGliL2RlYnVnLy9saWIveDg2XzY0LWxpbnV4
LWdudS9saWJuc3NfZmlsZXMtMi4yMy5zby4uLmRvbmUuDQpkb25lLg0KMHgw
MDAwN2Y4MTk5MjVkZTcwIGluIF9fcG9sbF9ub2NhbmNlbCAoKSBhdAouLi9z
eXNkZXBzL3VuaXgvc3lzY2FsbC10ZW1wbGF0ZS5TOjg0DQo4NCAgICAgIC4u
L3N5c2RlcHMvdW5peC9zeXNjYWxsLXRlbXBsYXRlLlM6IE5vIHN1Y2ggZmls
ZSBvciBkaXJlY3RvcnkuDQooZ2RiKSBzZXQgcGFnaW5hdGlvbiBvZmYNCihn
ZGIpIHNldCBsb2dnaW5nIGZpbGUgL3RtcC9nZGIubG9nDQooZ2RiKSBzZXQg
bG9nZ2luZyBvbg0KQ29weWluZyBvdXRwdXQgdG8gL3RtcC9nZGIubG9nDQoo
Z2RiKSBoYW5kbGUgU0lHVVNSMSBub3N0b3ANClNpZ25hbCAgICAgICAgU3Rv
cCAgICAgIFByaW50ICAgUGFzcyB0byBwcm9ncmFtIERlc2NyaXB0aW9uDQpT
SUdVU1IxICAgICAgIE5vICAgICAgICBZZXMgICAgIFllcyAgICAgICAgICAg
ICBVc2VyIGRlZmluZWQgc2lnbmFsIDENCihnZGIpIGhhbmRsZSBTSUdVU1Ix
IG5vcHJpbnQNClNpZ25hbCAgICAgICAgU3RvcCAgICAgIFByaW50ICAgUGFz
cyB0byBwcm9ncmFtIERlc2NyaXB0aW9uDQpTSUdVU1IxICAgICAgIE5vICAg
ICAgICBObyAgICAgIFllcyAgICAgICAgICAgICBVc2VyIGRlZmluZWQgc2ln
bmFsIDENCihnZGIpIGNvbnQNCkNvbnRpbnVpbmcuDQoNClByb2dyYW0gcmVj
ZWl2ZWQgc2lnbmFsIFNJR1NFR1YsIFNlZ21lbnRhdGlvbiBmYXVsdC4NCl9i
dF9yZXN0b3JlX3BhZ2UgKHBhZ2U9MHg3ZjgxNmZjZTJiNDAgIiIsIGZyb209
MHg1NWEwOTQ1YWJiNzAgIlwwMzYiLApsZW49PG9wdGltaXplZCBvdXQ+KSBh
dAovYnVpbGQvcG9zdGdyZXNxbC05LjUteHA5dXRIL3Bvc3RncmVzcWwtOS41
LTkuNS4zL2J1aWxkLy4uL3NyYy9iYWNrZW5kL2FjY2Vzcy9uYnRyZWUvbmJ0
eGxvZy5jOjU3DQo1NyAgICAgCi9idWlsZC9wb3N0Z3Jlc3FsLTkuNS14cDl1
dEgvcG9zdGdyZXNxbC05LjUtOS41LjMvYnVpbGQvLi4vc3JjL2JhY2tlbmQv
YWNjZXNzL25idHJlZS9uYnR4bG9nLmM6Ck5vIHN1Y2ggZmlsZSBvciBkaXJl
Y3RvcnkuDQooZ2RiKSBidA0KIzAgIF9idF9yZXN0b3JlX3BhZ2UgKHBhZ2U9
MHg3ZjgxNmZjZTJiNDAgIiIsIGZyb209MHg1NWEwOTQ1YWJiNzAgIlwwMzYi
LApsZW49PG9wdGltaXplZCBvdXQ+KSBhdAovYnVpbGQvcG9zdGdyZXNxbC05
LjUteHA5dXRIL3Bvc3RncmVzcWwtOS41LTkuNS4zL2J1aWxkLy4uL3NyYy9i
YWNrZW5kL2FjY2Vzcy9uYnRyZWUvbmJ0eGxvZy5jOjU3DQojMSAgMHgwMDAw
MDAwMDAwMDAwMDAwIGluID8/ICgpDQooZ2RiKSBwIGZyb20NCiQxID0gMHg1
NWEwOTQ1YWJiNzAgIlwwMzYiDQooZ2RiKSBwIGVuZA0KJDIgPSAweDU1YTA5
NDVhYzkyOCAiXDMwNWNPIg0KKGdkYikgcCBpDQokMyA9IDMzMjQNCihnZGIp
IHAgbGVuDQokNCA9IDxvcHRpbWl6ZWQgb3V0Pg0KKGdkYikgcCAmaXR1cGRh
dGENCiQ1ID0gKEluZGV4VHVwbGVEYXRhICopIDB4N2ZmZTgzZWE4NGUwDQoo
Z2RiKSBwIGl0ZW1zDQokNiA9IHsweDAgPHJlcGVhdHMgNDA4IHRpbWVzPn0N
CihnZGIpIHAgJml0ZW1zDQokNyA9IChJdGVtICgqKVs0MDhdKSAweDdmZmU4
M2VhODgyMA0KKGdkYikgcCBpdGVtc3oNCiQ4ID0gPG9wdGltaXplZCBvdXQ+
DQooZ2RiKSBydW4NClRoZSBwcm9ncmFtIGJlaW5nIGRlYnVnZ2VkIGhhcyBi
ZWVuIHN0YXJ0ZWQgYWxyZWFkeS4NClN0YXJ0IGl0IGZyb20gdGhlIGJlZ2lu
bmluZz8gKHkgb3IgbikgeQ0KU3RhcnRpbmcgcHJvZ3JhbTogL3Vzci9saWIv
cG9zdGdyZXNxbC85LjUvYmluL3Bvc3RncmVzDQpbVGhyZWFkIGRlYnVnZ2lu
ZyB1c2luZyBsaWJ0aHJlYWRfZGIgZW5hYmxlZF0NClVzaW5nIGhvc3QgbGli
dGhyZWFkX2RiIGxpYnJhcnkgIi9saWIveDg2XzY0LWxpbnV4LWdudS9saWJ0
aHJlYWRfZGIuc28uMSIuDQoicm9vdCIgZXhlY3V0aW9uIG9mIHRoZSBQb3N0
Z3JlU1FMIHNlcnZlciBpcyBub3QgcGVybWl0dGVkLg0KVGhlIHNlcnZlciBt
dXN0IGJlIHN0YXJ0ZWQgdW5kZXIgYW4gdW5wcml2aWxlZ2VkIHVzZXIgSUQg
dG8gcHJldmVudA0KcG9zc2libGUgc3lzdGVtIHNlY3VyaXR5IGNvbXByb21p
c2UuICBTZWUgdGhlIGRvY3VtZW50YXRpb24gZm9yDQptb3JlIGluZm9ybWF0
aW9uIG9uIGhvdyB0byBwcm9wZXJseSBzdGFydCB0aGUgc2VydmVyLg0KW0lu
ZmVyaW9yIDEgKHByb2Nlc3MgMTg4NykgZXhpdGVkIHdpdGggY29kZSAwMV0N
CihnZGIpIHF1aXQKCg==

Hi,

thanks for reporting this issue.

On 2016-06-07 09:16:18 +0000, boa@neogrid.dk wrote:
> Program received signal SIGSEGV, Segmentation fault.
> _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70 "\036",
> len=<optimized out>) at
> /build/postgresql-9.5-xp9utH/postgresql-9.5-9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:57
> 57
> /build/postgresql-9.5-xp9utH/postgresql-9.5-9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:
> No such file or directory.
> (gdb) bt
> #0  _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70 "\036",
> len=<optimized out>) at
> /build/postgresql-9.5-xp9utH/postgresql-9.5-9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:57
> #1  0x0000000000000000 in ?? ()
> (gdb) p from
> $1 = 0x55a0945abb70 "\036"
> (gdb) p end
> $2 = 0x55a0945ac928 "\305cO"
> (gdb) p i
> $3 = 3324
> (gdb) p len
> $4 = <optimized out>
> (gdb) p &itupdata
> $5 = (IndexTupleData *) 0x7ffe83ea84e0
> (gdb) p items
> $6 = {0x0 <repeats 408 times>}
> (gdb) p &items
> $7 = (Item (*)[408]) 0x7ffe83ea8820
> (gdb) p itemsz
> $8 = <optimized out>

Uhm, this is a bit odd. There's no backtrace, but types and such are
known?  I guess you do have the debug symbols installed?

Greetings,

Andres Freund

> On 2016-06-07 19:21, Andres Freund wrote:
> > Program received signal SIGSEGV, Segmentation fault.
> > _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70 "\036",
> > len=<optimized out>) at
> > /build/postgresql-9.5-xp9utH/postgresql-9.5-
> 9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:57
> > 57
> > /build/postgresql-9.5-xp9utH/postgresql-9.5-
> 9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:
> > No such file or directory.
> > (gdb) bt
> > #0  _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70
> > "\036", len=<optimized out>) at
> > /build/postgresql-9.5-xp9utH/postgresql-9.5-9.5.3/build/../src/backend
> > /access/nbtree/nbtxlog.c:57
> > #1  0x0000000000000000 in ?? ()
> > (gdb) p from
> > $1 = 0x55a0945abb70 "\036"
> > (gdb) p end
> > $2 = 0x55a0945ac928 "\305cO"
> > (gdb) p i
> > $3 = 3324
> > (gdb) p len
> > $4 = <optimized out>
> > (gdb) p &itupdata
> > $5 = (IndexTupleData *) 0x7ffe83ea84e0
> > (gdb) p items
> > $6 = {0x0 <repeats 408 times>}
> > (gdb) p &items
> > $7 = (Item (*)[408]) 0x7ffe83ea8820
> > (gdb) p itemsz
> > $8 = <optimized out>
>
> Uhm, this is a bit odd. There's no backtrace, but types and such are known?  I
> guess you do have the debug symbols installed?

Yeah, it's confusing. I installed the result of ./list-dbgsym-packages-v2.1.sh -p $pid which gave:

libcomerr2-dbg libffi6-dbg libgcc1-dbg libicu55-dbg libldap-2.4-2-dbg libsqlite3-0-dbg libssl1.0.0-dbg libxml2-dbg
postgresql-9.5-dbgzlib1g-dbg 

Not sure what else I can do short of recompiling postgresql mysql.

Greetings,

Bo Ørsted Andresen

On 2016-06-07 17:36:38 +0000, Bo Ørsted Andresen wrote:
> > On 2016-06-07 19:21, Andres Freund wrote:
> > > Program received signal SIGSEGV, Segmentation fault.
> > > _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70 "\036",
> > > len=<optimized out>) at
> > > /build/postgresql-9.5-xp9utH/postgresql-9.5-
> > 9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:57
> > > 57
> > > /build/postgresql-9.5-xp9utH/postgresql-9.5-
> > 9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:
> > > No such file or directory.
> > > (gdb) bt
> > > #0  _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70
> > > "\036", len=<optimized out>) at
> > > /build/postgresql-9.5-xp9utH/postgresql-9.5-9.5.3/build/../src/backend
> > > /access/nbtree/nbtxlog.c:57
> > > #1  0x0000000000000000 in ?? ()
> > > (gdb) p from
> > > $1 = 0x55a0945abb70 "\036"
> > > (gdb) p end
> > > $2 = 0x55a0945ac928 "\305cO"
> > > (gdb) p i
> > > $3 = 3324
> > > (gdb) p len
> > > $4 = <optimized out>
> > > (gdb) p &itupdata
> > > $5 = (IndexTupleData *) 0x7ffe83ea84e0
> > > (gdb) p items
> > > $6 = {0x0 <repeats 408 times>}
> > > (gdb) p &items
> > > $7 = (Item (*)[408]) 0x7ffe83ea8820
> > > (gdb) p itemsz
> > > $8 = <optimized out>
> >
> > Uhm, this is a bit odd. There's no backtrace, but types and such are known?  I
> > guess you do have the debug symbols installed?
>
> Yeah, it's confusing. I installed the result of ./list-dbgsym-packages-v2.1.sh -p $pid which gave:
>
> libcomerr2-dbg libffi6-dbg libgcc1-dbg libicu55-dbg libldap-2.4-2-dbg libsqlite3-0-dbg libssl1.0.0-dbg libxml2-dbg
postgresql-9.5-dbgzlib1g-dbg 
>
> Not sure what else I can do short of recompiling postgresql mysql.

Any chance the running version of postgres is out of date with the
installed binaries / debug symbols?

> On 2016-06-07 19:41, Andres Freund wrote:
> > > > Program received signal SIGSEGV, Segmentation fault.
> > > > _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70
> > > > "\036", len=<optimized out>) at
> > > > /build/postgresql-9.5-xp9utH/postgresql-9.5-
> > > 9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:57
> > > > 57
> > > > /build/postgresql-9.5-xp9utH/postgresql-9.5-
> > > 9.5.3/build/../src/backend/access/nbtree/nbtxlog.c:
> > > > No such file or directory.
> > > > (gdb) bt
> > > > #0  _bt_restore_page (page=0x7f816fce2b40 "", from=0x55a0945abb70
> > > > "\036", len=<optimized out>) at
> > > > /build/postgresql-9.5-xp9utH/postgresql-9.5-9.5.3/build/../src/bac
> > > > kend
> > > > /access/nbtree/nbtxlog.c:57
> > > > #1  0x0000000000000000 in ?? ()
> > > > (gdb) p from
> > > > $1 = 0x55a0945abb70 "\036"
> > > > (gdb) p end
> > > > $2 = 0x55a0945ac928 "\305cO"
> > > > (gdb) p i
> > > > $3 = 3324
> > > > (gdb) p len
> > > > $4 = <optimized out>
> > > > (gdb) p &itupdata
> > > > $5 = (IndexTupleData *) 0x7ffe83ea84e0
> > > > (gdb) p items
> > > > $6 = {0x0 <repeats 408 times>}
> > > > (gdb) p &items
> > > > $7 = (Item (*)[408]) 0x7ffe83ea8820
> > > > (gdb) p itemsz
> > > > $8 = <optimized out>
> > >
> > > Uhm, this is a bit odd. There's no backtrace, but types and such are
> > > known?  I guess you do have the debug symbols installed?
> >
> > Yeah, it's confusing. I installed the result of ./list-dbgsym-packages-v2.1.sh -
> p $pid which gave:
> >
> > libcomerr2-dbg libffi6-dbg libgcc1-dbg libicu55-dbg libldap-2.4-2-dbg
> > libsqlite3-0-dbg libssl1.0.0-dbg libxml2-dbg postgresql-9.5-dbg
> > zlib1g-dbg
> >
> > Not sure what else I can do short of recompiling postgresql mysql.
>
> Any chance the running version of postgres is out of date with the installed
> binaries / debug symbols?

You mean that I upgraded without restarting postgres before the segfault?

Before this I have had the same problem with Ubuntu 14.04 with postgresql 9.5.3 from a ppa. I then tried building a new
VMbased on Ubuntu 16.04 on the 3rd of June to see if there was something wrong with the slave VM. So it's a clean
installwhere I installed latest postgresql once and ran it. I've reproduced the problem three times since that happened
withone reboot in between. Of course what hasn't changed during all of this is the master. 

The version of postgresql-client-9.5, libpq5, postgreql-9.5, postgresql-contrib-9.5 and postgresql-9.5-dbg are all at
9.5.3-0ubuntu0.16.04.

Regards,
Bo Ørsted Andresen

On 2016-06-07 17:53:46 +0000, Bo Ørsted Andresen wrote:
> > On 2016-06-07 19:41, Andres Freund wrote:
> > > Not sure what else I can do short of recompiling postgresql mysql.
> >
> > Any chance the running version of postgres is out of date with the installed
> > binaries / debug symbols?
>
> You mean that I upgraded without restarting postgres before the segfault?

Yes, that's what I was wondering. But alas, that's aparently not the
reason.

This is going to be a bit more complicated, sorry :(

Could you try to reproduce the problem, and do 'p/x ReadRecPtr'? That
should give you something like 0x5434343496. If you rewrite this as
first-four-bytes/last-four-bytes e.g. 54/34343496 you get the LSN. With
that, could you try
pg_xlogdump -p /path/to/data/directory -s 54/34343496 -n 100
and send the output?

Regards,

Andres

> On 2016-06-07 20:00, Andres Freund wrote:
> > > On 2016-06-07 19:41, Andres Freund wrote:
> > > > Not sure what else I can do short of recompiling postgresql mysql.
> > >
> > > Any chance the running version of postgres is out of date with the
> > > installed binaries / debug symbols?
> >
> > You mean that I upgraded without restarting postgres before the segfault?
>
> Yes, that's what I was wondering. But alas, that's aparently not the reason.
>
> This is going to be a bit more complicated, sorry :(
>
> Could you try to reproduce the problem, and do 'p/x ReadRecPtr'? That
> should give you something like 0x5434343496. If you rewrite this as first-four-
> bytes/last-four-bytes e.g. 54/34343496 you get the LSN. With that, could you
> try pg_xlogdump -p /path/to/data/directory -s 54/34343496 -n 100 and send
> the output?

Will do. May take a while.
Regards,
Bo Ørsted Andresen

On 2016-06-07 18:04:50 +0000, Bo Ørsted Andresen wrote:
> > On 2016-06-07 20:00, Andres Freund wrote:
> > > > On 2016-06-07 19:41, Andres Freund wrote:
> > > > > Not sure what else I can do short of recompiling postgresql mysql.
> > > >
> > > > Any chance the running version of postgres is out of date with the
> > > > installed binaries / debug symbols?
> > >
> > > You mean that I upgraded without restarting postgres before the segfault?
> >
> > Yes, that's what I was wondering. But alas, that's aparently not the reason.
> >
> > This is going to be a bit more complicated, sorry :(
> >
> > Could you try to reproduce the problem, and do 'p/x ReadRecPtr'? That
> > should give you something like 0x5434343496. If you rewrite this as first-four-
> > bytes/last-four-bytes e.g. 54/34343496 you get the LSN. With that, could you
> > try pg_xlogdump -p /path/to/data/directory -s 54/34343496 -n 100 and send
> > the output?
>
> Will do. May take a while.

To clarify: After the crash recovery continues successfully? Or do you
have to scrap te standby?

Regards,

Andres

Bo Ørsted Andresen <boa@neogrid.dk> writes:
>> On 2016-06-07 19:41, Andres Freund wrote:
>> Any chance the running version of postgres is out of date with the installed
>> binaries / debug symbols?

> You mean that I upgraded without restarting postgres before the segfault?

I think the reason for the lack of useful backtrace info is that we've
smashed the stack.  Note that the original report shows i == 3324 which is
much larger than the available length of the local items[] array (408).
So presumably, the passed-in "len" was bogus (much too large).

If you're prepared to build a custom version of Postgres, you could
try adding this to _bt_restore_page():
    /* Need to copy tuple header due to alignment considerations */    memcpy(&itupdata, from, sizeof(IndexTupleData));
  itemsz = IndexTupleDSize(itupdata);    itemsz = MAXALIGN(itemsz); 

+        if (i >= lengthof(items))
+            elog(PANIC, "too many items on btree page");
+    items[i] = (Item) from;    itemsizes[i] = itemsz;    i++;
    from += itemsz;

and then you should get a core dump before the stack is clobbered.

I wonder whether we shouldn't add such a check to the regular sources...
        regards, tom lane

> On 2016-06-07 20:07, Andres Freund wrote:
> > > > > > Not sure what else I can do short of recompiling postgresql mysql.
> > > > >
> > > > > Any chance the running version of postgres is out of date with
> > > > > the installed binaries / debug symbols?
> > > >
> > > > You mean that I upgraded without restarting postgres before the
> segfault?
> > >
> > > Yes, that's what I was wondering. But alas, that's aparently not the
> reason.
> > >
> > > This is going to be a bit more complicated, sorry :(
> > >
> > > Could you try to reproduce the problem, and do 'p/x ReadRecPtr'?
> > > That should give you something like 0x5434343496. If you rewrite
> > > this as first-four- bytes/last-four-bytes e.g. 54/34343496 you get
> > > the LSN. With that, could you try pg_xlogdump -p
> > > /path/to/data/directory -s 54/34343496 -n 100 and send the output?
> >
> > Will do. May take a while.
>
> To clarify: After the crash recovery continues successfully? Or do you have to
> scrap te standby?

The crash recovery does not continue successfully. I don't know of a way to attach in gdb to the process that crashes
beforeit already crashed, which does not involve scrapping the standby. 

Regards,
Bo Ørsted Andresen

On 2016-06-07 18:15:14 +0000, Bo Ørsted Andresen wrote:
> The crash recovery does not continue successfully. I don't know of a way to attach in gdb to the process that crashes
beforeit already crashed, which does not involve scrapping the standby. 

gdb --args /path/to/postgres --single postgres -D /path/to/datadir
(gdb) run

should probably do the trick.

Regards,

Andres

> Fra: [mailto:andres@anarazel.de]
> Sendt: 07 June 2016 20:17
> Til: Bo Ørsted Andresen <boa@neogrid.dk>
> Cc: pgsql-bugs@postgresql.org
> Emne: Re: SV: [BUGS] BUG #14180: Segmentation fault on replication slave
>
> On 2016-06-07 20:17, Andres Freund wrote:
> > The crash recovery does not continue successfully. I don't know of a way to
> attach in gdb to the process that crashes before it already crashed, which
> does not involve scrapping the standby.
>
> gdb --args /path/to/postgres --single postgres -D /path/to/datadir
> (gdb) run
>
> should probably do the trick.

Will try that next time, thanks. Unfornately it's gone. :(

Regards,
Bo Ørsted Andresen

> On 2016-06-07 20:08, Tom Lane wrote:
> I think the reason for the lack of useful backtrace info is that we've smashed
> the stack.  Note that the original report shows i == 3324 which is much larger
> than the available length of the local items[] array (408).
> So presumably, the passed-in "len" was bogus (much too large).
>
> If you're prepared to build a custom version of Postgres, you could try adding
> this to _bt_restore_page():
>
>         /* Need to copy tuple header due to alignment
> considerations */
>         memcpy(&itupdata, from, sizeof(IndexTupleData));
>         itemsz = IndexTupleDSize(itupdata);
>         itemsz = MAXALIGN(itemsz);
>
> +        if (i >= lengthof(items))
> +            elog(PANIC, "too many items on btree page");
> +
>         items[i] = (Item) from;
>         itemsizes[i] = itemsz;
>         i++;
>
>         from += itemsz;
>
> and then you should get a core dump before the stack is clobbered.
>
> I wonder whether we shouldn't add such a check to the regular sources...

Will give it a shot tomorrow.

Regards,
Bo Ørsted Andresen

> On 2016-06-07 20:00, Andres Freund wrote:
> > > > Not sure what else I can do short of recompiling postgresql mysql.
> > >
> > > Any chance the running version of postgres is out of date with the
> > > installed binaries / debug symbols?
> >
> > You mean that I upgraded without restarting postgres before the segfault?
>
> Yes, that's what I was wondering. But alas, that's aparently not the reason.
>
> This is going to be a bit more complicated, sorry :(
>
> Could you try to reproduce the problem, and do 'p/x ReadRecPtr'? That
> should give you something like 0x5434343496. If you rewrite this as first-four-
> bytes/last-four-bytes e.g. 54/34343496 you get the LSN. With that, could you
> try pg_xlogdump -p /path/to/data/directory -s 54/34343496 -n 100 and send
> the output?

Output attached.

Thanks,
Bo Ørsted Andresen

> > On 2016-06-07 20:08, Tom Lane wrote:
> > I think the reason for the lack of useful backtrace info is that we've
> > smashed the stack.  Note that the original report shows i == 3324
> > which is much larger than the available length of the local items[] array
> (408).
> > So presumably, the passed-in "len" was bogus (much too large).
> >
> > If you're prepared to build a custom version of Postgres, you could
> > try adding this to _bt_restore_page():
> >
> >         /* Need to copy tuple header due to alignment
> considerations */
> >         memcpy(&itupdata, from, sizeof(IndexTupleData));
> >         itemsz = IndexTupleDSize(itupdata);
> >         itemsz = MAXALIGN(itemsz);
> >
> > +        if (i >= lengthof(items))
> > +            elog(PANIC, "too many items on btree page");
> > +
> >         items[i] = (Item) from;
> >         itemsizes[i] = itemsz;
> >         i++;
> >
> >         from += itemsz;
> >
> > and then you should get a core dump before the stack is clobbered.
> >
> > I wonder whether we shouldn't add such a check to the regular sources...

Logged:

LOG:  started streaming WAL from primary at 631/7000000 on timeline 1
PANIC:  too many items on btree page
CONTEXT:  xlog redo Btree/SPLIT_R: level 0, firstright 139

Bacttrace:

# gdb -p 10069
GNU gdb (Ubuntu 7.11-0ubuntu1) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 10069
Reading symbols from /usr/local/pgsql/bin/postgres...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/librt-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libc-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from
/usr/lib/debug/.build-id/b7/7847cc9cacbca3b5753d0d25a32e5795afe75b.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/ld-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done.
done.
0x00007ffff73f3e70 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:84
84      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) set pagination off
(gdb) set logging file /tmp/debuglog-20160608-2.txt
(gdb) set logging on
Copying output to /tmp/debuglog-20160608-2.txt.
(gdb) handle SIGUSR1 nostop
Signal        Stop      Print   Pass to program Description
SIGUSR1       No        Yes     Yes             User defined signal 1
(gdb) handle SIGUSR1 noprint
Signal        Stop      Print   Pass to program Description
SIGUSR1       No        No      Yes             User defined signal 1
(gdb) cont
Continuing.
(gdb)
Program received signal SIGABRT, Aborted.
0x00007ffff732e418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff732e418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff733001a in __GI_abort () at abort.c:89
#2  0x000000000078ccaa in errfinish (dummy=dummy@entry=0) at elog.c:551
#3  0x000000000079074a in elog_finish (elevel=elevel@entry=22, fmt=fmt@entry=0x7cb187 "too many items on btree page")
atelog.c:1368 
#4  0x00000000004ae437 in _bt_restore_page (page=page@entry=0x7fffefa2cb40 "", from=<optimized out>,
from@entry=0xc52e70"\036", len=<optimized out>) at nbtxlog.c:58 
#5  0x00000000004ae8a4 in btree_xlog_split (onleft=onleft@entry=0 '\000', isroot=isroot@entry=0 '\000',
record=record@entry=0xc3b840)at nbtxlog.c:241 
#6  0x00000000004aee1c in btree_redo (record=0xc3b840) at nbtxlog.c:984
#7  0x00000000004d5c2b in StartupXLOG () at xlog.c:6825
#8  0x000000000064e212 in StartupProcessMain () at startup.c:215
#9  0x00000000004e3168 in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7fffffffe3e0) at bootstrap.c:418
#10 0x000000000064b698 in StartChildProcess (type=StartupProcess) at postmaster.c:5199
#11 0x000000000064dc84 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xc1b9f0) at postmaster.c:1284
#12 0x0000000000467950 in main (argc=3, argv=0xc1b9f0) at main.c:228

Regards,
Bo Ørsted Andresen