It's a bit unfortunate that we're doing the pending list flush while the vacuum memory is allocated at all. Is there any reason other than the way the callbacks are defined that gin doesn't do the pending list flush before vacuum does the heap scan and before it allocates any memory using maintenance_work_mem?
(I'm guessing doing it after vacuum is finished would have different problems with tuples in the pending queue not getting vacuumed?)