I have found Barman to be adequate for my requirements (though still looking for alternatives). It does what it says on the box and thats fine. At the moment I have ~25 clusters, each with ~25 databases containing 200 schemas (multitenancy thing). The clusters run off of a synchronously replicated SAN, but I dont think that will affect read ops. I do have the option to put the dumps on a non-replicated LUN.
What implications does a call to pg_dumpall have on my databases? Is there even a remote chance that a lock could be started by pg_dumpall?
If we consider that I for some reason or the other can't use pg_dumpall against the production clusters, what are my options? One idea is that of using asynchronous replication and pull the dumps off of them. Are there any pitfalls related to the replication? During normal operations, will postgres ensure that the state on the slaves always reflect the state of the masters? In effect it would work similar to if I did the dump on the production servers? I recall reading something about asynchronous replication, but I'm unsure of what it was exactly or if it affects backups.