Limit postgres user logins

My hba file has:

local   all         postgres                                    md5
host    all         admin         127.0.0.1/32          md5
host    all         dbuser        127.0.0.1/32          md5

But that still allows users other than postgres to attempt to login as postgres. Is there any way I can limit access to
thepostgres user specified as being logged in as postgres? 

gvim