I think ultimately we'll need to measure the false positive rate, so that we can use it to dynamically disable the bloom filter if it gets inefficient. Also maybe put some of that into EXPLAIN ANALYZE.
I'm not so convinced that will be a good idea. What if the filter does not help much to start with, we disable it because of that, then we get some different probe values later in the scan which the bloom filter would have helped to eliminate earlier.
Maybe it would be better to, once the filter is built, simply count the number of 1 bits and only use the filter if there's less than <threshold> 1 bits compared to the size of the filter in bits. There's functionality in bms_num_members() to do this, and there's also __builtin_popcount() in newer version of GCC, which we could have some wrapper around, perhaps.