If you didn't turn it off, you have parallel workers on by default with v12. If work_mem is set high, memory use may be much higher as each node in a complex plan could end up executing in parallel.
Also, do you use a connection pooler such as pgbouncer or pgpool? What is max_connections set to?