Perhaps a bunch of processes waiting on the access exclusive lock on the materialized view being released?
log_lock_waits might help you if the MV takes more than a second to refresh, otherwise, you might need to have a look at ungranted locks in pg_locks and see if the number of locks spikes during the refresh.
I think David's got the right idea here. Like he said, investigate pg_locks, if it is the refresh materialized view, you can avoid the problem by doing 'REFRESH MATERIALIZED VIEW CONCURRENTLY'. You will need at least one unique index on the table.