The query is derived from a view. I want to select only one column from a view with many columns. I figured the optimizer would be smart enough in this case not to join all of the tables needed for the columns I wasn't referencing at all.
I thought it might be that postgres couldn't optimize the query due to the fact that it was a view, so that's why I pulled out this portion to see if it would speed up. But it didn't.