At Tue, 24 Dec 2019 11:57:39 -0800, Noah Misch <noah@leadboat.com> wrote in
> On Mon, Dec 23, 2019 at 07:41:49PM +0900, Kyotaro Horiguchi wrote:
> > If I understand it correctly, it is mentioning the number of the all
> > segment files in a fork, not the length of md_seg_fds arrays at a
> > certain moment. But actually _fdvec_resize is called for every segment
> > opening during mdnblocks (just-after mdopen), and every segment
> > closing during mdclose and mdtruncate as mentioned here. We are going
> > to omit pallocs only in the decreasing case.
>
> That is a good point. How frequently one adds 1 GiB of data is not the main
> issue. mdclose() and subsequent re-opening of all segments will be more
> relevant to overall performance.
Yes, that's exactly what I meant.
> > If we regard repalloc as far faster than FileOpen/FileClose or we care
> > about only increase of segment number of mdopen'ed files and don't
> > care the frequent resize that happens during the functions above, then
> > the comment is right and we may resize the array in the
> > segment-by-segment manner.
>
> In most cases, the array will fit into a power-of-two chunk, so repalloc()
> already does the right thing. Once the table has more than ~1000 segments (~1
> TiB table size), the allocation will get a single-chunk block, and every
> subsequent repalloc() will call realloc(). Even then, repalloc() probably is
> far faster than File operations. Likely, I should just accept the extra
> repalloc() calls and drop the "else if" change in _fdvec_resize().
I'm not sure which is better. If we say we know that
repalloc(AllocSetRealloc) doesn't free memory at all, there's no point
in calling repalloc for shrinking and we could omit that under the
name of optimization. If we say we want to free memory as much as
possible, we should call repalloc pretending to believe that that
happens.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center