The above numbers are taken from the minimum of two runs of each scenario.
I can see, when we have TAR_SEND_SIZE as 32kb or 128kb, it is giving us a good performance whereas, for 1Mb it is taking 2.5x more time.
Please let me know your thoughts/suggestions on the same.
So the patch came out slightly faster at 8kB and slightly slower in the other tests. That's kinda strange. I wonder if it's just noise. How much do the results vary run to run?
I would've expected (and I think Andres thought the same) that a smaller block size would be bad for the patch and a larger block size would be good, but that's not what these numbers show.
I wouldn't worry too much about the regression at 1MB. Probably what's happening there is that we're losing some concurrency - perhaps with smaller block sizes the OS can buffer the entire chunk in the pipe connecting pg_basebackup to the server and start on the next one, but when you go up to 1MB it doesn't fit and ends up spending a lot of time waiting for data to be read from the pipe. Wait event profiling might tell you more. Probably what this suggests is that you want the largest buffer size that doesn't cause you to overrun the network/pipe buffer and no larger. Unfortunately, I have no idea how we'd figure that out dynamically, and I don't see a reason to believe that everyone will have the same size buffers.