Allow connections by IP address?

PG 17.latest

My server has two IP addresses:

10.1.2.3.4

10.1.2.3.5 (a VIP)

Some connections should only come in through the VIP, while others (like replication) must come in through .4 and others (f.e. administrators, can come in from .4 or .5).

Is there any way to restrict that?  I don't see anything in https://www.postgresql.org/docs/17/auth-pg-hba-conf.html but may be overlooking something.

(Why don't we use a connection pooler?  The 3rd party application has only been validated against direct connections to PG. Bugs in PgPool caused problems in prod.)