Craig>That moves work further away from the DB, which has its own costs, and isn't something you're likely to be happy with if you're looking at things like optimising PL/PgSQL with a bytecode compiler. But it's the best we have right now.
What if JVM was started within a background worker?
Then JVM can spawn several threads that serve PL requests on a "thread per backend" basis.
You can't really execute the plpgsql-compiled-to-bytecode outside the user session, so you need a JVM in it anyway.
You probably could have a bgworker or pool of bgworkers doing your plpgsql compilation and caching. But because your plpgsql might reference uncommitted catalog entries in the local backend, you'd the bgworker to join an exported snapshot from the backend you're compiling the plpgsql for. If it doesn't let you avoid having a jvm in each backend it's not likely to be too useful.
Craig>You may be able to greatly reduce that cost if you can store your cached compiled data in a shared memory segment created by your extension.
Craig>This will get a bit easier with the new dynamic shared memory infrastructure, but it's going to be no fun at all to make that play with the JVM. You'll probably need a lot of JNI.