First, shmget is asking for a lot less than shmmax - why is it failing?
Check to see if things work as expected when you have shmmax and shmall set to a shade less than 2GB and fail when they are a shade more. If so, it would seem there's a signed-integer-overflow bug somewhere in the kernel's handling of shmem requests ... which would be a reportable kernel bug.