On Wed, Aug 9, 2023 at 8:56 PM David Rowley <dgrowleyml@gmail.com> wrote:
>
> On Thu, 10 Aug 2023 at 03:12, Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> > Thinking more about it, I think memory used is the only right metrics.
> > It's an optimization in MemoryContext implementation that malloc'ed
> > memory is not freed when it is returned using free().
>
> I guess it depends on the problem you're trying to solve. I had
> thought you were trying to do some work to reduce the memory used by
> the planner, so I imagined you wanted this so you could track your
> progress and also to help ensure we don't make too many mistakes in
> the future which makes memory consumption worse again.
Right.
> For that, I
> imagined you'd want to know how much memory is held to ransom by the
> context with malloc(), not palloc(). Is it really useful to reduce the
> palloc'd memory by the end of planning if it does not reduce the
> malloc'd memory?
AFAIU, the memory freed by the end of planning can be used by later
stages of query processing. The memory malloc'ed during planning can
also be used at the time of execution if it was not palloc'ed or
palloc'ed but pfree'd. So I think it's useful to reduce palloc'ed
memory by the end of planning.
>
> Another way we might go about reducing planner memory is to make
> changes to the allocators themselves. For example, aset rounds up to
> the next power of 2. If we decided to do something like add more
> freelists to double the number so we could add a mid-way point
> freelist between each power of 2, then we might find it reduces the
> planner memory by, say 12.5%. If we just tracked what was consumed by
> palloc() that would appear to save us nothing, but it might actually
> save us several malloced blocks.
>
This won't just affect planner but every subsystem of PostgreSQL, not
just planner, unless we device a new context type for planner.
My point is what's relevant here is how much net memory planner asked
for. Internally how much memory PostgreSQL allocated seems relevant
for the memory context infra as such but not so much for planner
alone.
If you think that memory allocated is important, I will add both used
and (net) allocated memory to EXPLAIN output.
--
Best Wishes,
Ashutosh Bapat