Re: Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

On 02/12/2014 05:42 PM, Florian Pflug wrote:
> On Feb12, 2014, at 12:55 , MauMau <maumau307@gmail.com> wrote:
>> From: "Andres Freund" <andres@2ndquadrant.com>
>>> It's x86, right? Then it's unlikely to be actual unordered memory
>>> accesses, but if the compiler reordered:
>>>    LOG_LWDEBUG("LWLockRelease", T_NAME(l), T_ID(l), "release waiter");
>>>    proc = head;
>>>    head = proc->lwWaitLink;
>>>    proc->lwWaitLink = NULL;
>>>    proc->lwWaiting = false;
>>>    PGSemaphoreUnlock(&proc->sem);
>>> to
>>>    LOG_LWDEBUG("LWLockRelease", T_NAME(l), T_ID(l), "release waiter");
>>>    proc = head;
>>>    proc->lwWaiting = false;
>>>    head = proc->lwWaitLink;
>>>    proc->lwWaitLink = NULL;
>>>    PGSemaphoreUnlock(&proc->sem);
>>> which it is permitted to do, yes, that could cause symptoms like you
>>> describe.
>>
>> Yes, the hang occurred with 64-bit PostgreSQL 9.2.4 running on RHEL6 for x86_64.
>> The PostgreSQL was built with GCC.
>
> The relevant part of the disassembled binary you attached seems to be
>
> Dump of assembler code for function LWLockRelease:
> ...
> 0x0000000000647f47 <LWLockRelease+519>:    lea    0x10(%rcx),%rdi
> 0x0000000000647f4b <LWLockRelease+523>:    movq   $0x0,0x48(%rcx)
> 0x0000000000647f53 <LWLockRelease+531>:    movb   $0x0,0x41(%rcx)
> 0x0000000000647f57 <LWLockRelease+535>:    callq  0x606210 <PGSemaphoreUnlock>
>
> I haven't checked the offsets, but since lwWaitLink is an 8-byte quantity
> and lwWaiting a single-byte quantity, it's pretty much certain that the
> first store updates lwWaitLink and the second lwWaiting. Thus, no reordering
> seems to have taken place here...
>
> best regards,
> Florian Pflug
>
>
>

Even if reordering was not done by compiler, it still can happen while execution.
There is no warranty that two subsequent assignments will be observed by all CPU cores in the same order.
So if one thread is performing

p->x = 1;
p->y = 2;
p->x = 3;
p->y = 4;

then other thread can see the following combinations of (x,y):

(1,2)
(1,4)
(3,2)
(3,4)

It is necessary to explicitly insert write barrier to prevent such non-deterministic behaviour.