Обсуждение: pgsql: llvmjit: Use explicit LLVMContextRef for inlining
llvmjit: Use explicit LLVMContextRef for inlining When performing inlining LLVM unfortunately "leaks" types (the types survive and are usable, but a new round of inlining will recreate new structurally equivalent types). This accumulation will over time amount to a memory leak which for some queries can be large enough to trigger the OOM process killer. To avoid accumulation of types, all IR related data is stored in an LLVMContextRef which is dropped and recreated in order to release all types. Dropping and recreating incurs overhead, so it will be done only after 100 queries. This is a heuristic which might be revisited, but until we can get the size of the context from LLVM we are flying a bit blind. This issue has been reported several times, there may be more references to it in the archives on top of the threads linked below. This is a backpatch of 9dce22033d5 to all supported branches. Reported-By: Justin Pryzby <pryzby@telsasoft.com> Reported-By: Kurt Roeckx <kurt@roeckx.be> Reported-By: Jaime Casanova <jcasanov@systemguards.com.ec> Reported-By: Lauri Laanmets <pcspets@gmail.com> Author: Andres Freund and Daniel Gustafsson Discussion: https://postgr.es/m/7acc8678-df5f-4923-9cf6-e843131ae89d@www.fastmail.com Discussion: https://postgr.es/m/20201218235607.GC30237@telsasoft.com Discussion: https://postgr.es/m/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA@mail.gmail.com Backpatch-through: v12 Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/3b991f81c45720515144b928bb8d1f03577628f1 Modified Files -------------- src/backend/jit/llvm/llvmjit.c | 141 ++++++++++++++++++++++++++++---- src/backend/jit/llvm/llvmjit_deform.c | 77 ++++++++--------- src/backend/jit/llvm/llvmjit_expr.c | 72 ++++++++-------- src/backend/jit/llvm/llvmjit_inline.cpp | 33 +++++--- src/include/jit/llvmjit.h | 8 ++ src/include/jit/llvmjit_emit.h | 30 ++++--- 6 files changed, 252 insertions(+), 109 deletions(-)
Daniel Gustafsson <dgustafsson@postgresql.org> writes:
> llvmjit: Use explicit LLVMContextRef for inlining
It looks like the v12 back-port of this wasn't quite right:
canebrake and urutu have both been showing assertion failures
down inside LLVMContextDispose since it went in. They aren't
complaining about v13 and later, though.
No idea what the cause may be, but perhaps the right answer
is just to revert this and not try to fix the bug in v12?
regards, tom lane
> On 19 Nov 2023, at 17:05, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Daniel Gustafsson <dgustafsson@postgresql.org> writes: >> llvmjit: Use explicit LLVMContextRef for inlining > > It looks like the v12 back-port of this wasn't quite right: > canebrake and urutu have both been showing assertion failures > down inside LLVMContextDispose since it went in. They aren't > complaining about v13 and later, though. Ugh, I had missed these =/ Thanks for the heads up! > No idea what the cause may be, but perhaps the right answer > is just to revert this and not try to fix the bug in v12? I'll have a look with fresh eyes in the morning, and hopefully it will be fixable. If it get's messy it's quite possible that skipping v12 is the best option. -- Daniel Gustafsson
> On 19 Nov 2023, at 21:08, Daniel Gustafsson <daniel@yesql.se> wrote: > I'll have a look with fresh eyes in the morning, and hopefully it will be > fixable. If it get's messy it's quite possible that skipping v12 is the best > option. As an update, discussing with Andres off-list I have a potential patch for this, need to rebuild LLVM debug builds tomorrow to test further but there should be either a fix or a revert soon. -- Daniel Gustafsson
> On 20 Nov 2023, at 22:53, Daniel Gustafsson <daniel@yesql.se> wrote: > >> On 19 Nov 2023, at 21:08, Daniel Gustafsson <daniel@yesql.se> wrote: > >> I'll have a look with fresh eyes in the morning, and hopefully it will be >> fixable. If it get's messy it's quite possible that skipping v12 is the best >> option. > > As an update, discussing with Andres off-list I have a potential patch for > this, need to rebuild LLVM debug builds tomorrow to test further but there > should be either a fix or a revert soon. After some more work with Andres, the bug was found to be a void pointer not created in the referenced context. The attached diff fixes the issue (which is related to code refactored in v13 hence why it only affects v12). I will apply that to v12 shortly after a little more testing. -- Daniel Gustafsson
Вложения
> On 8 Dec 2023, at 14:38, Daniel Gustafsson <daniel@yesql.se> wrote: > I will apply that to v12 shortly after a little more testing. Done, and canebrake/urutu has since turned green. -- Daniel Gustafsson