Per discussionat the developer meeting in Brussels, here's a patch that makes some updates to the backup APIs, to support non-exclusive backups without using pg_basebackup. The idea is to fix at least three main issues that are there today -- that you cannot run concurrent backups, that the backup_label file is created in the data directory which makes it impossible to distinguish between a cluster restored from backup and one that crashed while a backup was running, and a cluster can get "stuck" in backup mode if the backup script/software crashes.
To make this work, this patch:
* Introduces a new argument for pg_start_backup(), for "exclusive". It defaults to true, which means that just calling pg_start_backup() like before will behave exactly like before. If it's called with exclusive='f', a non-exclusive backup is started, and the backup label is collected in memory instead of in the backup_label file.
* If the client disconnects with a non-exclusive backup running, the backup is automatically aborted. This is the same thing that pg_basebackup does. To use these non-exclusive backups the backup software will have to maintain a persistent connection to the database -- something that should not be a problem for any of the modern ones out there (but would've been slightly trickier back in the days when we suggested shellscripts)
* A new version of pg_stop_backup is created, taking an argument specifying if it's exclusive or not. The current version of pg_stop_backup() continues to work for exclusive backups, with no changes to behavior. The new pg_stop_backup will return a record of three columns instead of just the value -- the LSN (pglsn), the backup label file (text) and the tablespace map file (text). If used to cancel an exclusive backup, backup label file and tablespace map will be NULL. At this point it's up to the backup software to write down the files in the location of the backup.
Attached patch currently doesn't include full documentation, since Bruce was going to restructure that section of the docs (also based on the devmeeting). I will write up the docs once that is done (I assume it will be soon enough, or I'll go do it regardless), but I wanted to get some review in on the code while waiting.