Re: pg13.2: invalid memory alloc request size NNNN

On Fri, Feb 12, 2021 at 06:44:54PM +0100, Tomas Vondra wrote:
> > (gdb) p len
> > $1 = -4
> > 
> > This VM had some issue early today and I killed the VM, causing PG to execute
> > recovery.  I'm tentatively blaming that on zfs, so this could conceivably be a
> > data error (although recovery supposedly would have resolved it).  I just
> > checked and data_checksums=off.
> 
> This seems very much like a corrupted varlena header - length (-4) is
> clearly bogus, and it's what triggers the problem, because that's what wraps
> around to 18446744073709551613 (which is 0xFFFFFFFFFFFFFFFD).
> 
> This has to be a value stored in a table, not some intermediate value
> created during execution. So I don't think the exact query matters. Can you
> try doing something like pg_dump, which has to detoast everything?

Right, COPY fails and VACUUM FULL crashes.

message                | invalid memory alloc request size 18446744073709551613
query                  | COPY child.tt TO '/dev/null';

> The question is whether this is due to the VM getting killed in some strange
> way (what VM system is this, how is the storage mounted?) or whether the
> recovery is borked and failed to do the right thing.

This is qemu/kvm, with block storage:
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/data/postgres'/>

And then more block devices for ZFS vdevs:
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/data/zfs2'/>
      ...

Those are LVM volumes (I know that ZFS/LVM is discouraged).

$ zpool list -v
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs          762G   577G   185G        -         -    71%    75%  1.00x    ONLINE  -
  vdj        127G  92.7G  34.3G        -         -    64%  73.0%      -  ONLINE  
  vdd        127G  95.6G  31.4G        -         -    74%  75.2%      -  ONLINE  
  vdf        127G  96.0G  31.0G        -         -    75%  75.6%      -  ONLINE  
  vdg        127G  95.8G  31.2G        -         -    74%  75.5%      -  ONLINE  
  vdh        127G  95.5G  31.5G        -         -    74%  75.2%      -  ONLINE  
  vdi        128G   102G  25.7G        -         -    71%  79.9%      -  ONLINE  

This is recently upgraded to ZFS 2.0.0, and then to 2.0.1:

Jan 21 09:33:26 Installed: zfs-dkms-2.0.1-1.el7.noarch
Dec 23 08:41:21 Installed: zfs-dkms-2.0.0-1.el7.noarch

The VM has gotten "wedged" and I've had to kill it a few times in the last 24h
(needless to say this is not normal).  That part seems like a kernel issue and
not postgres problem.  It's unclear if that's due to me trying to tickle the
postgres ERROR.  It's the latest centos7 kernel: 3.10.0-1160.15.2.el7.x86_64

-- 
Justin