In the b-tree code, we solved that problem back in 2006, so it can be done but requires a bit more code. In b-tree, we solved it with a "vacuum cycle ID" number that's set on the page halves when a page is split. That allows VACUUM to identify pages that have been split concurrently sees them, and "jump back" to vacuum them too. See commit http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h= 5749f6ef0cc1c67ef9c9ad2108b3d97b82555c80. It should be possible to do something similar in GiST, and in fact you might be able to reuse the NSN field that's already set on the page halves on split, instead of adding a new "vacuum cycle ID".
...
Another note. Assuming we have NSN which can play the role of "vacuum cycle ID", can we implement sequential (with possible "jump back") index scan for GiST?
Yeah, I think it would work. It's pretty straightforward, the page split code already sets the NSN, just when we need it. Vacuum needs to memorize the current NSN when it begins, and whenever it sees a page with a higher NSN (or the FOLLOW_RIGHT flag is set), follow the right-link if it points to lower-numbered page.
I mean "full index scan" feature for SELECT queries might be implemented as well as sequential VACUUM.