You are right, postgresql Count() function is slow, because; It's physically count the rows one by one.
Other database systems using indexes for counting, but postgresql walk through all rows in multiple transactions with different row states for calculating the real row count. This is about architecture of postgresql.
If you use WHERE condition on indexed column in your query, this will be much faster.