I've attached an updated version of the patch which fixes up some incorrect logic in the foreign key matching code, plus various comment improvements.
I've made a few updated to the patch to simplify some logic in the code which analyses the join condition. The result is slightly faster code for detecting either successful or unsuccessful join removal.
I've also been doing a little benchmarking of the overhead that this adds to planning time for a handful of different queries.
With the queries I tested the overhead was between ~20 and ~423 nanoseconds per SEMI or ANTI join, the 20 was for the earliest fast path out on an unsuccessful removal and the 423 was for a successful removal. (tests done on a 4 year old intel i5 laptop). This accounted for between 0.01% and 0.2% of planning time for the tested queries. I was quite happy with this, but I did manage to knock it down a little more with the bms_get_singleton_v1.patch, which I've attached. This reduces the range to between ~15 and ~409 nanoseconds, but probably this is going into micro benchmark territory... so perhaps not worth the extra code...
With the benchmarks I just put semiorantijoin_is_removable() in a tight 1 million iteration loop and grabbed the total planning time for that, I then compared this to an unpatched master's planning time after dividing the time reported for the 1 million removals version by 1 million.
I didn't really find a good way to measure the extra overhead in actually loading the foreign key constraints in get_relation_info()