index speed and failed expectations?

sol=> \d stats;
                  Table "public.stats"
     Column    |              Type              | Modifiers
--------------+--------------------------------+-----------
  id           | integer                        | not null
  start_time   | timestamp(0) without time zone | not null
...
Indexes:
     "stats_start_time_idx" btree (start_time)
     "stats_id_key" btree (id)

There are roughly half a million rows.

This query from the console:

select * from stats order by start_time;

takes 8 seconds before starting its output. Am I wrong in assuming that
the index on start_time should make ORDER BY orders of magnitude faster?
Or is this already fast enough? Or should I max up some memory (buffer)
setting to achieve greater speeds? Not that the speed is crucial, just
curious.

TIA.