> transaction. No passing connections by hand anywhere, everything should be
> nicely thread-bound. Still, if not here, where could it go wrong?
>
I have seen two cases in my career where there was an "evil box" on the
network that corrupted the traffic.
The first was a very long time ago (in the late '80s) but the second was
only a couple of years ago and presented
with very similar symptoms to your report. This happened at a consulting
client's site (actually between two sites).
Weird "franken-packets" showed up once in a while, leading to a protocol
decode failure. Luckily we had been
involved in writing both the client and the server, and therefore had a
high degree of confidence that they were
correct. The network administrators denied strongly that they had any
equipment deployed that touched the
payload of any packet. They denied this several times. Eventually we
were able to take packet traces
on both client and server machines, correlate the traffic (not
necessarily an easy task), and prove
conclusively that what had been sent from one end did not show up intact
at the other end.
A few days later the network people revealed that they had some sort of
firewall/traffic management box
that was mangling the traffic.
Having said that, bugs in buffer management code are also not uncommon,
and can manifest intermittently
since they may be triggered by specific boundary conditions, specific
received data buffer size, and so on.
I have also seen once case of data leaking between threads in an
unpleasant and intermittent way
in a Java application, in buffer management code that attempted to avoid
GC overhead by re-using
buffers across sessions. So that's definitely a non-zero possibility too.