Maybe PostgreSQL can't find a way to calculate having estimates?
I wasn't even thinking of the HAVING estimates I was thinking of just the raw aggregates. It can't implement the HAVING until has the raw aggregate in hand. But, what is the actual row count without the HAVING? Well, I notice now this line:
Rows Removed by Filter: 6787359
So the row count of rows=86 is mostly due to the HAVING, not due to the raw aggregation, a point I overlooked initially. So the planner is not mistaken in thinking that a huge number of rows need to be passed up--it is correct in thinking that.