Обсуждение: Tainted kernel possible caused by postgresql
Hi,
I've been experiencing tainted kernel which is my guess caused by postgresql and I don't know where to start. I also guess that this might be a kernel issue. I tried 3.13.0-49 and 3.13.0-53 but no good results. This is happening on 2 of our Ubuntu 14.04.2 servers with postgresql-9.3 version 9.3.5-0ubuntu0.14.04.1. I am also using the same postgresql version and kernel to other 10 14.04.2 servers and doesn't seem to encounter this problem. Below is the kernel log:
kernel: [651454.831139] ------------[ cut here ]------------
kernel: [651454.831155] WARNING: CPU: 1 PID: 20500 at /build/buildd/linux-3.13.0/net/core/dst.c:285 dst_release+0x45/0x60()
kernel: [651454.831158] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs i2c_piix4 serio_raw mac_hid drbd lru_cache libcrc32c lp parport pata_acpi
kernel: [651454.831179] CPU: 1 PID: 20500 Comm: postgres Not tainted 3.13.0-53-generic #89-Ubuntu
kernel: [651454.831181] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
kernel: [651454.831183] 0000000000000009 ffff88002e785b18 ffffffff81722e1e 0000000000000000
kernel: [651454.831186] ffff88002e785b50 ffffffff810677fd ffff88003aa78000 00000000ffffffff
kernel: [651454.831189] ffff88003abb8440 0000000000000000 0000000000000000 ffff88002e785b60
kernel: [651454.831192] Call Trace:
kernel: [651454.831200] [<ffffffff81722e1e>] dump_stack+0x45/0x56
kernel: [651454.831205] [<ffffffff810677fd>] warn_slowpath_common+0x7d/0xa0
kernel: [651454.831208] [<ffffffff810678da>] warn_slowpath_null+0x1a/0x20
kernel: [651454.831211] [<ffffffff8162f4e5>] dst_release+0x45/0x60
kernel: [651454.831258] [<ffffffff816102e9>] sk_dst_check+0xb9/0xe0
kernel: [651454.831264] [<ffffffff816c2f1f>] ip6_sk_dst_lookup_flow+0x2f/0x1b0
kernel: [651454.831268] [<ffffffff81612af8>] ? release_sock+0x118/0x170
kernel: [651454.831272] [<ffffffff816dde8e>] udpv6_sendmsg+0x61e/0xb10
kernel: [651454.831277] [<ffffffff8166d322>] ? tcp_recvmsg+0x6c2/0xb80
kernel: [651454.831283] [<ffffffff81695364>] inet_sendmsg+0x64/0xb0
kernel: [651454.831303] [<ffffffff81314527>] ? apparmor_socket_sendmsg+0x17/0x20
kernel: [651454.831307] [<ffffffff8160de3b>] sock_sendmsg+0x8b/0xc0
kernel: [651454.831314] [<ffffffff8172eef4>] ? __do_page_fault+0x204/0x570
kernel: [651454.831317] [<ffffffff8160e381>] SYSC_sendto+0x121/0x1c0
kernel: [651454.831322] [<ffffffff8104f2bf>] ? kvm_clock_get_cycles+0x1f/0x30
kernel: [651454.831327] [<ffffffff810cd9da>] ? __getnstimeofday+0x3a/0xc0
kernel: [651454.831330] [<ffffffff810cda6e>] ? getnstimeofday+0xe/0x30
kernel: [651454.831333] [<ffffffff810cdaaa>] ? do_gettimeofday+0x1a/0x50
kernel: [651454.831336] [<ffffffff8160ed6e>] SyS_sendto+0xe/0x10
kernel: [651454.831339] [<ffffffff8173391d>] system_call_fastpath+0x1a/0x1f
kernel: [651454.831341] ---[ end trace 5e06bdd4ad2125b1 ]---
kernel: [653314.515991] ------------[ cut here ]------------
kernel: [653314.516010] WARNING: CPU: 1 PID: 32055 at /build/buildd/linux-3.13.0/net/core/dst.c:285 dst_release+0x45/0x60()
kernel: [653314.516056] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs i2c_piix4 serio_raw mac_hid drbd lru_cache libcrc32c lp parport pata_acpi
kernel: [653314.516073] CPU: 1 PID: 32055 Comm: postgres Tainted: G W 3.13.0-53-generic #89-Ubuntu
kernel: [653314.516075] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
kernel: [653314.516076] 0000000000000009 ffff88003a445b58 ffffffff81722e1e 0000000000000000
kernel: [653314.516079] ffff88003a445b90 ffffffff810677fd ffff88003aa78000 00000000ffffffff
kernel: [653314.516081] ffff88003abb87b8 00000000000003d0 ffff88003aa78000 ffff88003a445ba0
kernel: [653314.516083] Call Trace:
kernel: [653314.516090] [<ffffffff81722e1e>] dump_stack+0x45/0x56
kernel: [653314.516095] [<ffffffff810677fd>] warn_slowpath_common+0x7d/0xa0
kernel: [653314.516097] [<ffffffff810678da>] warn_slowpath_null+0x1a/0x20
kernel: [653314.516099] [<ffffffff8162f4e5>] dst_release+0x45/0x60
kernel: [653314.516103] [<ffffffff8161033c>] sk_setup_caps+0x2c/0xd0
kernel: [653314.516107] [<ffffffff816ddb8c>] udpv6_sendmsg+0x31c/0xb10
kernel: [653314.516111] [<ffffffff8166d322>] ? tcp_recvmsg+0x6c2/0xb80
kernel: [653314.516116] [<ffffffff81695364>] inet_sendmsg+0x64/0xb0
kernel: [653314.516123] [<ffffffff81314527>] ? apparmor_socket_sendmsg+0x17/0x20
kernel: [653314.516125] [<ffffffff8160de3b>] sock_sendmsg+0x8b/0xc0
kernel: [653314.516130] [<ffffffff81157b50>] ? free_pages.part.64+0x40/0x50
kernel: [653314.516133] [<ffffffff81157b73>] ? free_pages+0x13/0x20
kernel: [653314.516137] [<ffffffff81176635>] ? tlb_finish_mmu+0x35/0x60
kernel: [653314.516139] [<ffffffff8160e381>] SYSC_sendto+0x121/0x1c0
kernel: [653314.516144] [<ffffffff8104f2bf>] ? kvm_clock_get_cycles+0x1f/0x30
kernel: [653314.516148] [<ffffffff810cd9da>] ? __getnstimeofday+0x3a/0xc0
kernel: [653314.516150] [<ffffffff810cda6e>] ? getnstimeofday+0xe/0x30
kernel: [653314.516152] [<ffffffff810cdaaa>] ? do_gettimeofday+0x1a/0x50
kernel: [653314.516155] [<ffffffff8160ed6e>] SyS_sendto+0xe/0x10
kernel: [653314.516158] [<ffffffff8173391d>] system_call_fastpath+0x1a/0x1f
kernel: [653314.516160] ---[ end trace 5e06bdd4ad2125b2 ]---
Thanks,
Norbert
Norbert Kiam Maclang wrote: > I've been experiencing tainted kernel which is my guess caused by postgresql and I don't know where to > start. I also guess that this might be a kernel issue. I tried 3.13.0-49 and 3.13.0-53 but no good > results. This is happening on 2 of our Ubuntu 14.04.2 servers with postgresql-9.3 version 9.3.5- > 0ubuntu0.14.04.1. I am also using the same postgresql version and kernel to other 10 14.04.2 servers > and doesn't seem to encounter this problem. Below is the kernel log: [...] > kernel: [653314.516073] CPU: 1 PID: 32055 Comm: postgres Tainted: G W 3.13.0-53-generic #89-Ubuntu [...] I don't see how PostgreSQL, which runs entirely in user space, could taint a kernel. The line above indicates that there has been a kernel warning. Try to locate it in the kernel log, it might help you understand what has happened. Yours, Laurenz Albe
On Tue, Jun 9, 2015 at 3:32 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote: > Norbert Kiam Maclang wrote: >> I've been experiencing tainted kernel which is my guess caused by postgresql and I don't know where to >> start. I also guess that this might be a kernel issue. I tried 3.13.0-49 and 3.13.0-53 but no good >> results. This is happening on 2 of our Ubuntu 14.04.2 servers with postgresql-9.3 version 9.3.5- >> 0ubuntu0.14.04.1. I am also using the same postgresql version and kernel to other 10 14.04.2 servers >> and doesn't seem to encounter this problem. Below is the kernel log: > > [...] >> kernel: [653314.516073] CPU: 1 PID: 32055 Comm: postgres Tainted: G W 3.13.0-53-generic #89-Ubuntu > [...] > > I don't see how PostgreSQL, which runs entirely in user space, could taint a kernel. Agreed. Even if postgres caused the kernel issues, the problem fundamentally lies with the kernel since userspace programs are not supposed to be able to harm the underlying operating system. Kernel bugs do happen and so do hardware issues which can sometimes masquerade as software issues. Is this system virtualized? merlin
Hi Merlin,
Yes it is running on kvm.
Regards,
Norbert
On Jun 11, 2015 2:37 AM, "Merlin Moncure" <mmoncure@gmail.com> wrote:
On Tue, Jun 9, 2015 at 3:32 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> Norbert Kiam Maclang wrote:
>> I've been experiencing tainted kernel which is my guess caused by postgresql and I don't know where to
>> start. I also guess that this might be a kernel issue. I tried 3.13.0-49 and 3.13.0-53 but no good
>> results. This is happening on 2 of our Ubuntu 14.04.2 servers with postgresql-9.3 version 9.3.5-
>> 0ubuntu0.14.04.1. I am also using the same postgresql version and kernel to other 10 14.04.2 servers
>> and doesn't seem to encounter this problem. Below is the kernel log:
>
> [...]
>> kernel: [653314.516073] CPU: 1 PID: 32055 Comm: postgres Tainted: G W 3.13.0-53-generic #89-Ubuntu
> [...]
>
> I don't see how PostgreSQL, which runs entirely in user space, could taint a kernel.
Agreed. Even if postgres caused the kernel issues, the problem
fundamentally lies with the kernel since userspace programs are not
supposed to be able to harm the underlying operating system. Kernel
bugs do happen and so do hardware issues which can sometimes
masquerade as software issues. Is this system virtualized?
merlin