It doesn't seem that unlikely that somebody does an ALTER SERVER OPTIONS
SET .. to add an extension to be shippable while connections are already
using the fdw. It'll be confusing if some clients are fast and some
others are really slow.
This seems more likely than anyone mucking around with extension stuff (adding new functions (and working with FDW in production at the same time?)) or adding/dropping whole extensions (you’ll have more problems than a stale cache, whole columns will disappear if you "DROP EXTENSION postgis CASCADE"), and has the added benefit of not needing me to muck into core stuff for my silly feature.
I’ll have a look at doing invalidation for the case of changes to the FDW wrappers and servers.
P.