Обсуждение: Running on an NFS Mounted Directory
I was wondering if there were any performance issues with having a data directory that was an nfs mounted drive? Say like a SAN or NAS device? Has anyone done this before?
On Wed, Apr 26, 2006 at 10:06:58PM -0400, Ketema Harris wrote: > I was wondering if there were any performance issues with having a data > directory that was an nfs mounted drive? Say like a SAN or NAS device? Has > anyone done this before? My understanding is that NFS is pretty poor in performance in general, so I would expect it to be particularly bad for a DB. You might run some (non-DB) performance tests to get a feel for how bad it might me. (Someone once told me that NFS topped out at around 12MB/s, but I don't know if that's really true [they were trying to sell a competitive networked filesystem]). In any event, you're at least limited by ethernet speeds, if not more. -- Steve Wampler -- swampler@noao.edu The gods that smiled on your birth are now laughing out loud.
On Wed, Apr 26, 2006 at 07:35:42PM -0700, Steve Wampler wrote: > On Wed, Apr 26, 2006 at 10:06:58PM -0400, Ketema Harris wrote: > > I was wondering if there were any performance issues with having a data > > directory that was an nfs mounted drive? Say like a SAN or NAS device? Has > > anyone done this before? > > My understanding is that NFS is pretty poor in performance in general, > so I would expect it to be particularly bad for a DB. You might run > some (non-DB) performance tests to get a feel for how bad it might me. > (Someone once told me that NFS topped out at around 12MB/s, but I don't > know if that's really true [they were trying to sell a competitive > networked filesystem]). > > In any event, you're at least limited by ethernet speeds, if not more. More importantly, the latency involved will kill commit performance. If it doesn't then it's likely that fsync isn't being obeyed, which means 0 data integrity. -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
We have gotten very good performance from netapp and postgres 7.4.11 . I was able to push about 100MB/s over gigE, but that was limited by our netapp. DAS will generally always be faster, but if for example you have 2 disks vs. 100 NFS mounted ,NFS will be faster. NFS is very reliable and I would stay away from iscsi. Regards, Dan Gorman On Apr 26, 2006, at 7:35 PM, Steve Wampler wrote: > On Wed, Apr 26, 2006 at 10:06:58PM -0400, Ketema Harris wrote: >> I was wondering if there were any performance issues with having a >> data >> directory that was an nfs mounted drive? Say like a SAN or NAS >> device? Has >> anyone done this before? > > My understanding is that NFS is pretty poor in performance in general, > so I would expect it to be particularly bad for a DB. You might run > some (non-DB) performance tests to get a feel for how bad it might me. > (Someone once told me that NFS topped out at around 12MB/s, but I > don't > know if that's really true [they were trying to sell a competitive > networked filesystem]). > > In any event, you're at least limited by ethernet speeds, if not more. > > -- > Steve Wampler -- swampler@noao.edu > The gods that smiled on your birth are now laughing out loud. > > ---------------------------(end of > broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster
I am looking for the best solution to have a large amount of disk storage attached to my PostgreSQL 8.1 server. I was thinking of having a san or nas attached device be mounted by the pg server over nfs, hence the question about nfs performance. What other options/protocols are there to get high performance and data integrity while having the benefit of not having the physical storage attached to the db server? On 4/27/06 12:55 AM, "Jim C. Nasby" <jnasby@pervasive.com> wrote: > On Wed, Apr 26, 2006 at 07:35:42PM -0700, Steve Wampler wrote: >> On Wed, Apr 26, 2006 at 10:06:58PM -0400, Ketema Harris wrote: >>> I was wondering if there were any performance issues with having a data >>> directory that was an nfs mounted drive? Say like a SAN or NAS device? Has >>> anyone done this before? >> >> My understanding is that NFS is pretty poor in performance in general, >> so I would expect it to be particularly bad for a DB. You might run >> some (non-DB) performance tests to get a feel for how bad it might me. >> (Someone once told me that NFS topped out at around 12MB/s, but I don't >> know if that's really true [they were trying to sell a competitive >> networked filesystem]). >> >> In any event, you're at least limited by ethernet speeds, if not more. > > More importantly, the latency involved will kill commit performance. If > it doesn't then it's likely that fsync isn't being obeyed, which means 0 > data integrity.
On Thu, Apr 27, 2006 at 08:38:55AM -0400, Ketema Harris wrote: >I am looking for the best solution to have a large amount of disk storage >attached to my PostgreSQL 8.1 server. >What other options/protocols are there to get high performance and data >integrity while having the benefit of not having the physical storage >attached to the db server? These are two distinct requirements. Are both really requirements or is one "nice to have"? The "best" solution for "a large amount of disk storage" isn't "not having the physical storage attached to the db server". If you use non-local storage it will be slower and more expensive, quite likely by a large margin. There may be other advantages to doing so, but you haven't mentioned any of those as requirements. Mike Stone
OK. My thought process was that having non local storage as say a big raid 5 san ( I am talking 5 TB with expansion capability up to 10 ) would allow me to have redundancy, expandability, and hopefully still retain decent performance from the db. I also would hopefully then not have to do periodic backups from the db server to some other type of storage. Is this not a good idea? How bad of a performance hit are we talking about? Also, in regards to the commit data integrity, as far as the db is concerned once the data is sent to the san or nas isn't it "written"? The storage may have that write in cache, but from my reading and understanding of how these various storage devices work that is how they keep up performance. I would expect my bottleneck if any to be the actual Ethernet transfer to the storage, and I am going to try and compensate for that with a full gigabit backbone. On 4/27/06 8:44 AM, "Michael Stone" <mstone+postgres@mathom.us> wrote: > On Thu, Apr 27, 2006 at 08:38:55AM -0400, Ketema Harris wrote: >> I am looking for the best solution to have a large amount of disk storage >> attached to my PostgreSQL 8.1 server. > >> What other options/protocols are there to get high performance and data >> integrity while having the benefit of not having the physical storage >> attached to the db server? > > These are two distinct requirements. Are both really requirements or is > one "nice to have"? The "best" solution for "a large amount of disk > storage" isn't "not having the physical storage attached to the db > server". If you use non-local storage it will be slower and more > expensive, quite likely by a large margin. There may be other advantages > to doing so, but you haven't mentioned any of those as requirements. > > Mike Stone
On Thu, Apr 27, 2006 at 08:57:51 -0400, Ketema Harris <ketema@gmail.com> wrote: > performance from the db. I also would hopefully then not have to do > periodic backups from the db server to some other type of storage. Is this > not a good idea? How bad of a performance hit are we talking about? Also, You always need to do backups if you care about your data. What if someone accidental deletes a lot of data? What if someone blows up your data center (or there is a flood)?
Yes, your right, I meant not have to do the backups from the db server itself. I can do that within the storage device now, by allocating space for it, and letting the device copy the data files on some periodic basis. On 4/27/06 9:05 AM, "Bruno Wolff III" <bruno@wolff.to> wrote: > On Thu, Apr 27, 2006 at 08:57:51 -0400, > Ketema Harris <ketema@gmail.com> wrote: >> performance from the db. I also would hopefully then not have to do >> periodic backups from the db server to some other type of storage. Is this >> not a good idea? How bad of a performance hit are we talking about? Also, > > You always need to do backups if you care about your data. What if someone > accidental deletes a lot of data? What if someone blows up your data > center (or there is a flood)?
On Thu, Apr 27, 2006 at 08:57:51AM -0400, Ketema Harris wrote: > OK. My thought process was that having non local storage as say a big raid > 5 san ( I am talking 5 TB with expansion capability up to 10 ) would allow > me to have redundancy, expandability, and hopefully still retain decent > performance from the db. I also would hopefully then not have to do > periodic backups from the db server to some other type of storage. Is this > not a good idea? How bad of a performance hit are we talking about? Also, > in regards to the commit data integrity, as far as the db is concerned once > the data is sent to the san or nas isn't it "written"? The storage may have > that write in cache, but from my reading and understanding of how these > various storage devices work that is how they keep up performance. I would > expect my bottleneck if any to be the actual Ethernet transfer to the > storage, and I am going to try and compensate for that with a full gigabit > backbone. Well, if you have to have both the best performance and remote attach storage, I think you'll find that a fibre-channel SAN is still the king of the hill. 4Gb FC switches are common now, though finding a 4Gb HBA for your computer might be a trick. 2Gb HBAs are everywhere in FC land. That's a premium price solution, however, and I don't know anything about how well PG would perform with a FC SAN. We use our SAN for bulk science data and leave the PGDB on a separate machine with local disk. -- Steve Wampler -- swampler@noao.edu The gods that smiled on your birth are now laughing out loud.
On Thu, Apr 27, 2006 at 08:57:51AM -0400, Ketema Harris wrote: >OK. My thought process was that having non local storage as say a big raid >5 san ( I am talking 5 TB with expansion capability up to 10 ) That's two disk trays for a cheap slow array. (Versus a more expensive solution with more spindles and better seek performance.) >would allow >me to have redundancy, expandability, and hopefully still retain decent >performance from the db. I also would hopefully then not have to do >periodic backups from the db server to some other type of storage. No, backups are completely unrelated to your storage type; you need them either way. On a SAN you can use a SAN backup solution to back multiple systems up with a single backup unit without involving the host CPUs. This is fairly useless if you aren't amortizing the cost over a large environment. >Is this not a good idea? It really depends on what you're hoping to get. As described, it's not clear. (I don't know what you mean by "redundancy, expandability" or "decent performance".) >How bad of a performance hit are we talking about? Way too many factors for an easy answer. Consider the case of NAS vs SCSI direct attach storage. You're probably in that case comparing a single 125MB/s (peak) gigabit ethernet channel to (potentially several) 320MB/s (peak) SCSI channels. With a high-end NAS you might get 120MB/s off that GBE. With a (more realistic) mid-range unit you're more likely to get 40-60MB/s. Getting 200MB/s off the SCSI channel isn't a stretch, and you can fairly easily stripe across multiple SCSI channels. (You can also bond multiple GBEs, but then your cost & complexity start going way up, and you're never going to scale as well.) If you have an environment where you're doing a lot of sequential scans it isn't even a contest. You can also substitute SATA for SCSI, etc. For a FC SAN the peformance numbers are a lot better, but the costs & complexity are a lot higher. An iSCSI SAN is somewhere in the middle. >Also, in regards to the commit data integrity, as far as the db is >concerned once the data is sent to the san or nas isn't it "written"? >The storage may have that write in cache, but from my reading and >understanding of how these various storage devices work that is how >they keep up performance. Depends on the configuration, but yes, most should be able to report back a "write" once the data is in a non-volatile cache. You can do the same with a direct-attached array and eliminate the latency inherent in accessing the remote storage. >I would expect my bottleneck if any to be the actual Ethernet transfer >to the storage, and I am going to try and compensate for that with a >full gigabit backbone. see above. The advantages of a NAS or SAN are in things you haven't really touched on. Is the filesystem going to be accessed by several systems? Do you need the ability to do snapshots? (You may be able to do this with direct-attach also, but doing it on a remote storage device tends to be simpler.) Do you want to share one big, expensive, reliable unit between multiple systems? Will you be doing failover? (Note that failover requires software to go with the hardware, and can be done in a different way with local storage also.) In some environments the answers to those questions are yes, and the price premium & performance implications are well worth it. For a single DB server the answer is almost certainly "no". Mike Stone
On Thu, Apr 27, 2006 at 09:06:48 -0400, Ketema Harris <ketema@gmail.com> wrote: > Yes, your right, I meant not have to do the backups from the db server > itself. I can do that within the storage device now, by allocating space > for it, and letting the device copy the data files on some periodic basis. Only if the database server isn't running or your SAN provides a way to provide a snapshot of the data at a particular instant in time.
First, I appreciate all of your input. >No, backups are completely unrelated to your storage type; you need them > either way. Please another post. I meant the storage would do the back ups. >redundancy, expandability What I mean by these stupid flavor words is: Redundancy : raid 5. Expandability : the ability to stick another drive in my array and get more storage and not have to turn of the db. >Do you > need the ability to do snapshots? Yes. >Do you want to share one big, expensive, reliable unit between > multiple systems? Will you be doing failover? Yes, and Yes. Really on one other system, a phone system, but it is the crux of my business and will be writing a lot of recorded phone calls. I am working with a storage company now to set up the failover, I want the db and phone systems to never no if the storage switched over. You have given me a lot to think about. The performance concerns me and I will have to find some way to test. Perhaps spending a little less on the storage system and more on the actual servers is the way to go? Then utilize some combination off pg_backup, and the archive_command directive with a periodic script. Thank You all. I will keep researching this and the more input the better. Thank You. On 4/27/06 9:24 AM, "Michael Stone" <mstone+postgres@mathom.us> wrote: > On Thu, Apr 27, 2006 at 08:57:51AM -0400, Ketema Harris wrote: >> OK. My thought process was that having non local storage as say a big raid >> 5 san ( I am talking 5 TB with expansion capability up to 10 ) > > That's two disk trays for a cheap slow array. (Versus a more expensive > solution with more spindles and better seek performance.) > >> would allow >> me to have redundancy, expandability, and hopefully still retain decent >> performance from the db. I also would hopefully then not have to do >> periodic backups from the db server to some other type of storage. > > No, backups are completely unrelated to your storage type; you need them > either way. On a SAN you can use a SAN backup solution to back multiple > systems up with a single backup unit without involving the host CPUs. > This is fairly useless if you aren't amortizing the cost over a large > environment. > >> Is this not a good idea? > > It really depends on what you're hoping to get. As described, it's not > clear. (I don't know what you mean by "redundancy, expandability" or > "decent performance".) > >> How bad of a performance hit are we talking about? > > Way too many factors for an easy answer. Consider the case of NAS vs > SCSI direct attach storage. You're probably in that case comparing a > single 125MB/s (peak) gigabit ethernet channel to (potentially several) > 320MB/s (peak) SCSI channels. With a high-end NAS you might get 120MB/s > off that GBE. With a (more realistic) mid-range unit you're more likely > to get 40-60MB/s. Getting 200MB/s off the SCSI channel isn't a stretch, > and you can fairly easily stripe across multiple SCSI channels. (You can > also bond multiple GBEs, but then your cost & complexity start going way > up, and you're never going to scale as well.) If you have an environment > where you're doing a lot of sequential scans it isn't even a contest. > You can also substitute SATA for SCSI, etc. > > For a FC SAN the peformance numbers are a lot better, but the costs & > complexity are a lot higher. An iSCSI SAN is somewhere in the middle. > >> Also, in regards to the commit data integrity, as far as the db is >> concerned once the data is sent to the san or nas isn't it "written"? >> The storage may have that write in cache, but from my reading and >> understanding of how these various storage devices work that is how >> they keep up performance. > > Depends on the configuration, but yes, most should be able to report > back a "write" once the data is in a non-volatile cache. You can do the > same with a direct-attached array and eliminate the latency inherent in > accessing the remote storage. > >> I would expect my bottleneck if any to be the actual Ethernet transfer >> to the storage, and I am going to try and compensate for that with a >> full gigabit backbone. > > see above. > > The advantages of a NAS or SAN are in things you haven't really touched > on. Is the filesystem going to be accessed by several systems? Do you > need the ability to do snapshots? (You may be able to do this with > direct-attach also, but doing it on a remote storage device tends to be > simpler.) Do you want to share one big, expensive, reliable unit between > multiple systems? Will you be doing failover? (Note that failover > requires software to go with the hardware, and can be done in a > different way with local storage also.) In some environments the answers > to those questions are yes, and the price premium & performance > implications are well worth it. For a single DB server the answer is > almost certainly "no". > > Mike Stone
The SAN has the snapshot capability. On 4/27/06 9:31 AM, "Bruno Wolff III" <bruno@wolff.to> wrote: > On Thu, Apr 27, 2006 at 09:06:48 -0400, > Ketema Harris <ketema@gmail.com> wrote: >> Yes, your right, I meant not have to do the backups from the db server >> itself. I can do that within the storage device now, by allocating space >> for it, and letting the device copy the data files on some periodic basis. > > Only if the database server isn't running or your SAN provides a way to > provide a snapshot of the data at a particular instant in time.
On Thu, Apr 27, 2006 at 09:41:21AM -0400, Ketema Harris wrote: >>No, backups are completely unrelated to your storage type; you need them >> either way. >Please another post. I meant the storage would do the back ups. Which isn't a backup. Even expensive storage arrays can break or burn down. >>redundancy, expandability >What I mean by these stupid flavor words is: >Redundancy : raid 5. You can get that without external storage. >Expandability : the ability to stick another drive in my array and get more >storage and not have to turn of the db. You can also get that without external storage assuming you choose a platform with a volume manager. >>Do you >> need the ability to do snapshots? >Yes. If that's a hard requirement you'll have to eat the cost & performance problems of an external solution or choose a platform that will let you do that with direct-attach storage. (Something with a volume manager.) >>Do you want to share one big, expensive, reliable unit between >> multiple systems? Will you be doing failover? >Yes, and Yes. Really on one other system, a phone system, but it is the >crux of my business and will be writing a lot of recorded phone calls. I am >working with a storage company now to set up the failover, I want the db and >phone systems to never no if the storage switched over. If you actually have a couple of systems you're trying to fail over, a FC SAN may be a reasonable solution. Depending on your reliability requirement you can have multiple interfaces & FC switches to get redundant paths and a much higher level of storage reliability than you could get with direct attach storage. OTOH, if the DB server itself breaks you're still out of luck. :) You might compare that sort of solution with a solution that has redundant servers and implements the failover in software instead of hardware. Mike Stone
On Thu, Apr 27, 2006 at 10:04:19AM -0400, Michael Stone wrote: > >>redundancy, expandability > >What I mean by these stupid flavor words is: > >Redundancy : raid 5. > > You can get that without external storage. Yes, but some dedicated storage devices actually provide good performance with RAID5. Most simpler solutions give pretty abysmal write performance. > >>Do you > >>need the ability to do snapshots? > >Yes. > > If that's a hard requirement you'll have to eat the cost & performance > problems of an external solution or choose a platform that will let you > do that with direct-attach storage. (Something with a volume manager.) I'm wondering if PITR would suffice. Or maybe even Slony. > >>Do you want to share one big, expensive, reliable unit between > >>multiple systems? Will you be doing failover? > >Yes, and Yes. Really on one other system, a phone system, but it is the > >crux of my business and will be writing a lot of recorded phone calls. I am > >working with a storage company now to set up the failover, I want the db > >and > >phone systems to never no if the storage switched over. > > If you actually have a couple of systems you're trying to fail over, a > FC SAN may be a reasonable solution. Depending on your reliability > requirement you can have multiple interfaces & FC switches to get > redundant paths and a much higher level of storage reliability than you > could get with direct attach storage. OTOH, if the DB server itself > breaks you're still out of luck. :) You might compare that sort of > solution with a solution that has redundant servers and implements the > failover in software instead of hardware. BTW, I know a company here in Austin that does capacity planning for complex systems like this; contact me off-list if you're interested in talking to them. -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
On Thu, Apr 27, 2006 at 12:50:16PM -0500, Jim C. Nasby wrote: >Yes, but some dedicated storage devices actually provide good >performance with RAID5. Most simpler solutions give pretty abysmal write >performance. dedicated storage device != SAN != NAS. You can get good performance in a dedicated direct-attach device without paying for the SAN/NAS infrastructure if you don't need it; you don't have to go right from EMC to PERC with nothing in the middle. Mike Stone
So do NAS's Dan On Apr 27, 2006, at 6:42 AM, Ketema Harris wrote: > The SAN has the snapshot capability. > > > On 4/27/06 9:31 AM, "Bruno Wolff III" <bruno@wolff.to> wrote: > >> On Thu, Apr 27, 2006 at 09:06:48 -0400, >> Ketema Harris <ketema@gmail.com> wrote: >>> Yes, your right, I meant not have to do the backups from the db >>> server >>> itself. I can do that within the storage device now, by >>> allocating space >>> for it, and letting the device copy the data files on some >>> periodic basis. >> >> Only if the database server isn't running or your SAN provides a >> way to >> provide a snapshot of the data at a particular instant in time. > > > > ---------------------------(end of > broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org
On Wed, 26 Apr 2006 23:55:24 -0500, Jim C. Nasby wrote: > On Wed, Apr 26, 2006 at 07:35:42PM -0700, Steve Wampler wrote: >> On Wed, Apr 26, 2006 at 10:06:58PM -0400, Ketema Harris wrote: >> > I was wondering if there were any performance issues with having a data >> > directory that was an nfs mounted drive? Say like a SAN or NAS device? Has >> > anyone done this before? >> >> My understanding is that NFS is pretty poor in performance in general, NFS is not a good choice for several reasons. First, NFS takes priority in the system kernel, and will slow down all other operations. Your best choice, as pointed out by others, is a DAS solutions. If you must use NFS, you should consider putting it on a fast dedicated network by itself.