Just for funsies, today I decided to try and serve postgres data out of AFS (just a tablespace, not an entire database cluster). For those of you not in the know, AFS is a distributed, networked file system. It's like NFS, but with differences in structure, permissions, quotas, administration, and many other things that are beyond the scope of this list (you can find details yourself if you're so inclined). I wanted to see if there were any potential benefits, how it compared in terms of performance to a standard local-disk cluster, and what hacks had to be put into place for things to work.
Does anyone else have experience trying to serve clusters or data from shared file systems? I'd like to see how your experiences compare to mine.
- AFS uses Kerberos for authentication, and you need to have all the Kerberos bits and tickets set up for the postgres system user in order to have proper authentication for AFS. If you don't, you'd have to give system:anyuser (the AFS equivalent of global access) write access to the appropriate directories, which is bad for many reasons.
- As a result of the previous point, you can't use pg_ctl to start the server. pg_ctl doesn't pass Kerberos credentials. I had to start the server using the postgres binary and precede it with a ticket passing mechanism (mine is called runauth).
- AFS was about 8% slower in both reading and writing data (both the local disk and the AFS file server had 80 GB, 7200 RPM disks in RAID 1, and the logs were on a separate disk on the local file-system, so this is a fairly scientific metric).
- You can't take advantage of the shared file system because you can't share tablespaces among clusters or servers. You'd either get an error for trying to initialize a non-empty directory, or if you hack it to try and fool the clusters you could wind up with data corruption and race conditions since the clusters don't play nice.
- You can't take advantage of AFS replication (where you can put read-only copies of files in one tree and read-write copies in another tree) since trying to point a cluster at a read-only tablespace requires you to write to that tree, which you can't do since it's read-only. You're better off using something else for database replication.
All in all, while the prospect of using existing shared disk space was intriguing, I couldn't take advantage of any features that would make it truly worthwhile, there was a non-trivial performance hit, and it required a few small hacks to get working.
But what about all of you, what are your thoughts and experiences with trying to serve databases out of shared file systems?
Peter