Обсуждение: Postgres over Linux NBD or NFS

От:
Andrew Hammond
Дата:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We're looking for an alternative to fiber-channel disk arrays for mass
storage. One of the ideas that we're exploring would involve having the
cluster on an NFS mounted filesystem. Another technology we're looking
at is the Linux NBD (Network Block Device).

Has anyone had any experience with running postgres over either of these
technologies? What issues do we need to know about / pay attention to?

- --
Andrew Hammond    416-673-4138    
Database Administrator, Afilias Canada Corp.
CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA1yK6gfzn5SevSpoRAtYAAKCrghfAKV5kVuiTd/2TOwEbr4Q7hACgr3rT
mEvFi8AOHX9I43T45fH1e0U=
=1Cs9
-----END PGP SIGNATURE-----

От:
Tom Lane
Дата:

Andrew Hammond <> writes:
> We're looking for an alternative to fiber-channel disk arrays for mass
> storage. One of the ideas that we're exploring would involve having the
> cluster on an NFS mounted filesystem. Another technology we're looking
> at is the Linux NBD (Network Block Device).

There are a lot of horror stories concerning running databases (not only
Postgres) over NFS.  I wouldn't recommend it.  Dunno anything about NBD
though.

            regards, tom lane

От:
Andrew Rawnsley
Дата:

On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> We're looking for an alternative to fiber-channel disk arrays for mass
> storage. One of the ideas that we're exploring would involve having the
> cluster on an NFS mounted filesystem. Another technology we're looking
> at is the Linux NBD (Network Block Device).
>

No idea about NBDs, but its generally accepted that running over NFS
would significantly
decrease reliability and performance, i.e. it would be a Bad Move (tm).
Not sure what you
think to gain. I sure wouldn't trust NFS with a production database.

What exactly are you trying to gain, avoid, or do?

> Has anyone had any experience with running postgres over either of
> these
> technologies? What issues do we need to know about / pay attention to?
>
> - --
> Andrew Hammond    416-673-4138    
> Database Administrator, Afilias Canada Corp.
> CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFA1yK6gfzn5SevSpoRAtYAAKCrghfAKV5kVuiTd/2TOwEbr4Q7hACgr3rT
> mEvFi8AOHX9I43T45fH1e0U=
> =1Cs9
> -----END PGP SIGNATURE-----
> <ahammond.vcf>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to
> )
>
--------------------

Andrew Rawnsley
President
The Ravensfield Digital Resource Group, Ltd.
(740) 587-0114
www.ravensfield.com


От:
"Scott Marlowe"
Дата:

On Mon, 2004-06-21 at 20:46, Andrew Rawnsley wrote:
> On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > We're looking for an alternative to fiber-channel disk arrays for mass
> > storage. One of the ideas that we're exploring would involve having the
> > cluster on an NFS mounted filesystem. Another technology we're looking
> > at is the Linux NBD (Network Block Device).
> >
>
> No idea about NBDs, but its generally accepted that running over NFS
> would significantly
> decrease reliability and performance, i.e. it would be a Bad Move (tm).
> Not sure what you
> think to gain. I sure wouldn't trust NFS with a production database.
>
> What exactly are you trying to gain, avoid, or do?

I've gotten good performance over NFS using switched 100, then later
gigabit.  But I wouldn't trust it for diddly.


От:
Christopher Browne
Дата:

In an attempt to throw the authorities off his trail,  (Andrew Rawnsley) transmitted:
> On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:
>> We're looking for an alternative to fiber-channel disk arrays for mass
>> storage. One of the ideas that we're exploring would involve having the
>> cluster on an NFS mounted filesystem. Another technology we're looking
>> at is the Linux NBD (Network Block Device).
>
> No idea about NBDs, but its generally accepted that running over NFS
> would significantly decrease reliability and performance, i.e. it
> would be a Bad Move (tm). Not sure what you think to gain. I sure
> wouldn't trust NFS with a production database.
>
> What exactly are you trying to gain, avoid, or do?

The point of the exercise is to try to come up with something that is
a near-substitute for a SAN.

With a SAN, you have a box with a whole lot of disk in it, and then
your database servers connect to that box, typically via something
like fibrechannel.

One of the goals is for this to allow trying out Opterons at low risk.
Should performance turn out to suck or there be some other
disqualification, it's simple to hook the disk up to something else
instead.

The other goal is to be able to stick LOTS of disk into one box, and
dole it out to multiple servers.  It's more expensive to set up and
manage 3 RAID arrays than it is to set up and manage just 1, because
you have to manage 3 sets of disk hardware rather than 1.

But I'm getting convinced that the attempt to get this clever about it
is counterproductive unless you have outrageous amounts of money to
throw at it.

- NFS may well be acceptable if you buy into something with potent FS
  semantics, as with NetApp boxes.  But they're REALLY expensive.

- FibreChannel offers interesting options in conjunction with a fairly
  smart SAN box and Veritas, where you could have 5TB of storage in
  one box, and then assign 2TB apiece to two servers, and the other
  1TB to a third.  But the pricing premium again leaps out at ya.

The "poor man's approach" involves trying to fake this by building a
"disk box" running Linux that exports the storage either as a
filesystem (using NFS) or as disk blocks (NBD).  NFS clearly doesn't
provide the filesystem semantics needed to get decent reliability;
with NBD, it's not clear what happens :-(.

Barring that, it means building a separate RAID array for each server,
and living with the limitation that a full set of disk hardware has to
be devoted to each DB server.
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','acm.org').
http://www3.sympatico.ca/cbbrowne/
Rules of the  Evil Overlord #46. "If an advisor says  to me "My liege,
he  is but  one man.  What can  one man  possibly do?",  I  will reply
"This." and kill the advisor." <http://www.eviloverlord.com/>

От:
Andrew Rawnsley
Дата:

There are some less expensive baby-SAN options coming out now - Dell
has an
rebranded EMC baby SAN (which of course doesn't work with any other EMC
system...) that starts at about $6000 or so. Just read the announcement
- don't know
anything else. While there have been some reports of undewhelming
performance for
database applications, the Apple XRaid has a sweet price point,
particularly if you're
in an industry that they want some exposure in (we're in financial
services, they almost
gave it to us...$7000 for 2TB, batteries, accessory kits, etc), and a
decent feature
set. It works with non-Apple stuff..

The baby-SANs don't necessarily do many of the things that you can get
out of a full-blown
EMC/NetApp rig, but then again, you're not paying for it either.

There are a lot of lower-cost storage options popping up now, as
IDE/SATA disks arrays
proliferate. You can get external RAID boxes that talk SCSI or fiber
with IDE disks for
dirt these days. Small, too. Portable. I'm see little need to buy
massive boxes with
internal storage arrays anymore.


On Jun 22, 2004, at 7:50 AM, Christopher Browne wrote:

> In an attempt to throw the authorities off his trail,
>  (Andrew Rawnsley) transmitted:
>> On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:
>>> We're looking for an alternative to fiber-channel disk arrays for
>>> mass
>>> storage. One of the ideas that we're exploring would involve having
>>> the
>>> cluster on an NFS mounted filesystem. Another technology we're
>>> looking
>>> at is the Linux NBD (Network Block Device).
>>
>> No idea about NBDs, but its generally accepted that running over NFS
>> would significantly decrease reliability and performance, i.e. it
>> would be a Bad Move (tm). Not sure what you think to gain. I sure
>> wouldn't trust NFS with a production database.
>>
>> What exactly are you trying to gain, avoid, or do?
>
> The point of the exercise is to try to come up with something that is
> a near-substitute for a SAN.
>
> With a SAN, you have a box with a whole lot of disk in it, and then
> your database servers connect to that box, typically via something
> like fibrechannel.
>
> One of the goals is for this to allow trying out Opterons at low risk.
> Should performance turn out to suck or there be some other
> disqualification, it's simple to hook the disk up to something else
> instead.
>
> The other goal is to be able to stick LOTS of disk into one box, and
> dole it out to multiple servers.  It's more expensive to set up and
> manage 3 RAID arrays than it is to set up and manage just 1, because
> you have to manage 3 sets of disk hardware rather than 1.
>
> But I'm getting convinced that the attempt to get this clever about it
> is counterproductive unless you have outrageous amounts of money to
> throw at it.
>
> - NFS may well be acceptable if you buy into something with potent FS
>   semantics, as with NetApp boxes.  But they're REALLY expensive.
>
> - FibreChannel offers interesting options in conjunction with a fairly
>   smart SAN box and Veritas, where you could have 5TB of storage in
>   one box, and then assign 2TB apiece to two servers, and the other
>   1TB to a third.  But the pricing premium again leaps out at ya.
>
> The "poor man's approach" involves trying to fake this by building a
> "disk box" running Linux that exports the storage either as a
> filesystem (using NFS) or as disk blocks (NBD).  NFS clearly doesn't
> provide the filesystem semantics needed to get decent reliability;
> with NBD, it's not clear what happens :-(.
>
> Barring that, it means building a separate RAID array for each server,
> and living with the limitation that a full set of disk hardware has to
> be devoted to each DB server.
> --
> wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','acm.org').
> http://www3.sympatico.ca/cbbrowne/
> Rules of the  Evil Overlord #46. "If an advisor says  to me "My liege,
> he  is but  one man.  What can  one man  possibly do?",  I  will reply
> "This." and kill the advisor." <http://www.eviloverlord.com/>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faqs/FAQ.html
>
--------------------

Andrew Rawnsley
President
The Ravensfield Digital Resource Group, Ltd.
(740) 587-0114
www.ravensfield.com


От:
"Matt Clark"
Дата:

How about iSCSI?  This is exactly what it's for - presenting a bunch of
remote SCSI hardware as if it were local.

There are several reference implementations on SourceForge from Intel, Cisco
& others.

I've never tried it myself, but I would if I had the need.  And let's face
it there are some very big players selling very pricey kit that uses it, so
you should have pretty high confidence that the fundamentals are strong.

M



> The other goal is to be able to stick LOTS of disk into one
> box, and dole it out to multiple servers.  It's more
> expensive to set up and manage 3 RAID arrays than it is to
> set up and manage just 1, because you have to manage 3 sets
> of disk hardware rather than 1.
[snip]
> The "poor man's approach" involves trying to fake this by
> building a "disk box" running Linux that exports the storage
> either as a filesystem (using NFS) or as disk blocks (NBD).
> NFS clearly doesn't provide the filesystem semantics needed
> to get decent reliability; with NBD, it's not clear what happens :-(.



От:
Andrew Hammond
Дата:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

|> What exactly are you trying to gain, avoid, or do?

Gain: seperate database storage from processing. This lets me move
clusters from one server to another easily. Just stop the postgres
instance on server A and unmount it's filesystem. Then mount it on
server B and start postgres instance on server B. It gives me some
fail-over capability as well as scalability and a lot of flexibility in
balancing load over multiple servers.

Avoid: paying for brutally expensive FC gear.

- --
Andrew Hammond    416-673-4138    
Database Administrator, Afilias Canada Corp.
CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFA2Djagfzn5SevSpoRAj+bAKDFFgrhX+G1gkZRrydow3j/j35VaACbBN3Y
C/0nWmqcwo/UlqvYpng06Ks=
=k2vg
-----END PGP SIGNATURE-----

От:
"Carlos Eduardo Smanioto"
Дата:

Anselmo bom dia!

Não é custoso monstar um Cluster (Storage caseiro) em PostgreSQL, Está em
discussão no forum da PostgreSQL a possíbilidade de usar o NFS (Network file
system) ou o NBD (Network Block Device), ambos consistem em "Mapear"  a
partição de dados do PostgreSQL em uma OUTRA máquina com PostgreSQL a fim de
que os as duas máquinas trabalhem com a mesma base de dados.

Carlos Eduardo Smanioto
Infra Estrutura - Servidores e Segurança
Planae - Tecnologia da Informação
Fone/Fax +55 14 3224-3066 Ramal 207
www.planae.com.br

----- Original Message -----
From: "Scott Marlowe" <>
To: "Andrew Rawnsley" <>
Cc: "Andrew Hammond" <>;
<>
Sent: Tuesday, June 22, 2004 12:46 AM
Subject: Re: [PERFORM] Postgres over Linux NBD or NFS


> On Mon, 2004-06-21 at 20:46, Andrew Rawnsley wrote:
> > On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:
> >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > We're looking for an alternative to fiber-channel disk arrays for mass
> > > storage. One of the ideas that we're exploring would involve having
the
> > > cluster on an NFS mounted filesystem. Another technology we're looking
> > > at is the Linux NBD (Network Block Device).
> > >
> >
> > No idea about NBDs, but its generally accepted that running over NFS
> > would significantly
> > decrease reliability and performance, i.e. it would be a Bad Move (tm).
> > Not sure what you
> > think to gain. I sure wouldn't trust NFS with a production database.
> >
> > What exactly are you trying to gain, avoid, or do?
>
> I've gotten good performance over NFS using switched 100, then later
> gigabit.  But I wouldn't trust it for diddly.
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings


От:
"Matthew Nuzum"
Дата:

I just got the BigAdmin newsletter from Sun today... interestingly enough it
had a link to an article described as:

> Database Performance with NAS: Optimizing Oracle on NFS
> This paper discusses the operation of relational databases with network
> attached storage (NAS). Management implications and performance
> expectations for databases using NAS are presented.
The link points to: http://www.sun.com/bigadmin/content/nas/

I read just enough to see if it is relevant.  Here is the first part of the
summary:

> IT departments are increasingly utilizing Network Attached Storage (NAS)
> and the Network File System (NFS) to meet the storage needs of mission-
> critical relational databases. Reasons for this adoption include improved
> storage virtualization, ease of storage deployment, decreased complexity,
> and decreased total cost of ownership. This paper directly examines the
> performance of databases with NAS. In laboratory tests comparing NFS with
> local storage, NFS is shown capable of sustaining the same workload level
> as local storage. Under similar workload conditions, NFS does consume an
> increased number of CPU cycles; however, the proven benefits of NFS and
> NAS outweigh this penalty in most production environments.

Matthew Nuzum        | ISPs: Make $200 - $5,000 per referral by
www.followers.net        | recomending Elite CMS to your customers!
    | http://www.followers.net/isp