Обсуждение: DRBD and Postgres: how to improve the perfomance?
Hello.
We have made some performance tests with DRBD and Postgresql 8.2.3. We
have two identical servers in a cluster (Dell 2950) with a partition of
100 GB managed by DRBD: once we checked Postgres keeping his data folder
in a local partition, the second time we moved the data folder in the
shared partition. The two servers are connected point to point using a
cross cable to reduce their latency.
The partition is mounted with the option noatime in order to not update
the inode access time in case of read access.
We used pgbench for the testings, creating a dabase of about 3GB with a
scale of 200. After we perfomed 10 tests for each configuration,
simulating the usage of 100 clients with 500 transactions each.
DRBD configuration:
--------------------------------------------------------------------------------------------
resource drbd0 {
protocol C;
incon-degr-cmd "halt -f";
on db-node1 {
device /dev/drbd0;
disk /dev/sda2;
address 10.0.0.201:7788;
meta-disk internal;
}
on db-node2 {
device /dev/drbd0;
disk /dev/sda2;
address 10.0.0.202:7788;
meta-disk internal;
}
syncer {
rate 700000K;
}
}
--------------------------------------------------------------------------------------------
Pgbench
--------------------------------------------------------------------------------------------
pgbench -i pgbench -s 200
pgbench -c 100 -t 500 pgbench
--------------------------------------------------------------------------------------------
The results were that the TPS (transaction per second) with Postgres
running in the local partition is almost double than the one with the DRDB:
Postgres in shared DRBD partition: 60.863324 TPS
Postgres in local partition: 122.016138 TPS
Obviously, working with the database in DRBD, we had two writes instead
of only one but we are a bit disappointed about the low results. We
would like to know if there is any way to improve the performance in
order to have a 3/4 rate instead of the 1/2 one.
We would really appreciate it if you could give us some feedback.
Thank you in advance,
Maila Fatticcioni
--
______________________________________________________________
Maila Fatticcioni
______________________________________________________________
Mediterranean Broadband Infrastructure s.r.l.
ITALY
______________________________________________________________
Вложения
Maila Fatticcioni wrote:
> Hello.
> We have made some performance tests with DRBD and Postgresql 8.2.3. We
> have two identical servers in a cluster (Dell 2950) with a partition of
> 100 GB managed by DRBD: once we checked Postgres keeping his data folder
> in a local partition, the second time we moved the data folder in the
> shared partition. The two servers are connected point to point using a
> cross cable to reduce their latency.
> The partition is mounted with the option noatime in order to not update
> the inode access time in case of read access.
> We used pgbench for the testings, creating a dabase of about 3GB with a
> scale of 200. After we perfomed 10 tests for each configuration,
> simulating the usage of 100 clients with 500 transactions each.
>
> DRBD configuration:
> --------------------------------------------------------------------------------------------
> resource drbd0 {
>
> protocol C;
> incon-degr-cmd "halt -f";
>
> on db-node1 {
> device /dev/drbd0;
> disk /dev/sda2;
> address 10.0.0.201:7788;
> meta-disk internal;
> }
>
> on db-node2 {
> device /dev/drbd0;
> disk /dev/sda2;
> address 10.0.0.202:7788;
> meta-disk internal;
> }
> syncer {
> rate 700000K;
> }
> }
> --------------------------------------------------------------------------------------------
>
> Pgbench
>
> --------------------------------------------------------------------------------------------
> pgbench -i pgbench -s 200
> pgbench -c 100 -t 500 pgbench
> --------------------------------------------------------------------------------------------
>
> The results were that the TPS (transaction per second) with Postgres
> running in the local partition is almost double than the one with the DRDB:
>
> Postgres in shared DRBD partition: 60.863324 TPS
> Postgres in local partition: 122.016138 TPS
>
> Obviously, working with the database in DRBD, we had two writes instead
> of only one but we are a bit disappointed about the low results. We
> would like to know if there is any way to improve the performance in
> order to have a 3/4 rate instead of the 1/2 one.
You seem to be limited by the speed you can fsync the WAL to the DRBD
device. Using a RAID controller with a battery-backed up cache in both
servers should help, with and without DRBD. You might find that the
difference between local and shared partition just gets bigger, but you
should get better numbers.
In 8.3, you could turn synchronous_commit=off, if you can accept the
loss of recently committed transactions in case of a crash.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On 9/7/07, Maila Fatticcioni <mfatticcioni@mbigroup.it> wrote: > Obviously, working with the database in DRBD, we had two writes instead > of only one but we are a bit disappointed about the low results. We > would like to know if there is any way to improve the performance in > order to have a 3/4 rate instead of the 1/2 one. Have you considered warm standby PITR? It achieves essentially the same thing with very little overhead on the master. The only downside relative to DRDB is you have to think about the small gap between WAL file rotations. From what I understand, there is some new stuff (check out skype skytools) that may help minimize this problem. merlin
On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote: > protocol C; Try protocol B instead. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
--On Freitag, September 07, 2007 20:00:16 +0100 Simon Riggs
<simon@2ndquadrant.com> wrote:
> On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote:
>
>> protocol C;
>
> Try protocol B instead.
But that would have an impact on transaction safety, wouldn't it? It will
return immediately after reaching the remote buffer cache and you can't be
sure your data hits the remote disk.
It's a while ago i've played with such a setup, but it could be worth to
play around with max_buffers, al-extends, snd-bufsize. Oh and i think
Maila's 'rate' setting is too high: i've found rate settings
counterproductive when set too high (try a value slightly above your max
bandwidth of your connection). But i second Heikki, you should take care on
your disk setup as well.
--
Thanks
Bernd
On Fri, 2007-09-07 at 23:54 +0200, Bernd Helmle wrote: > --On Freitag, September 07, 2007 20:00:16 +0100 Simon Riggs > <simon@2ndquadrant.com> wrote: > > > On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote: > > > >> protocol C; > > > > Try protocol B instead. > > But that would have an impact on transaction safety, wouldn't it? It will > return immediately after reaching the remote buffer cache and you can't be > sure your data hits the remote disk. You're right, but the distinction is a small one. What are the chances of losing two independent servers within a few milliseconds of each other? If performance is an issue it is a particularly important distinction. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
"Simon Riggs" <simon@2ndquadrant.com> writes: > You're right, but the distinction is a small one. What are the chances > of losing two independent servers within a few milliseconds of each > other? If they're on the same power bus? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gregory Stark wrote:
> "Simon Riggs" <simon@2ndquadrant.com> writes:
>
>> You're right, but the distinction is a small one. What are the chances
>> of losing two independent servers within a few milliseconds of each
>> other?
>
> If they're on the same power bus?
That chance is minuscule or at least should be. Of course we are
assuming some level of conditioned power that is independent of the
power bus, e.g; a UPS.
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG4sqvATb/zqfZUUQRAq/qAKCkkFX/hTddRJriMGMYhjy04REwvgCfUoY5
pzcyvahVvsaAL8qlkJVtbX0=
=nzIH
-----END PGP SIGNATURE-----
Joshua D. Drake wrote: > Gregory Stark wrote: >> "Simon Riggs" <simon@2ndquadrant.com> writes: > >>> You're right, but the distinction is a small one. What are the chances >>> of losing two independent servers within a few milliseconds of each >>> other? >> If they're on the same power bus? > > That chance is minuscule or at least should be. Of course we are > assuming some level of conditioned power that is independent of the > power bus, e.g; a UPS. how is that making it different in practise ? - if both are on the same UPS they are affectively on the same power bus ... If the UPS fails (or the generator is not kicking in which happens way more often than people would believe) they could still fail at the very same time .... Stefan
"Joshua D. Drake" <jd@commandprompt.com> writes:
> Gregory Stark wrote:
>> "Simon Riggs" <simon@2ndquadrant.com> writes:
>>> You're right, but the distinction is a small one. What are the chances
>>> of losing two independent servers within a few milliseconds of each
>>> other?
>>
>> If they're on the same power bus?
> That chance is minuscule or at least should be.
It seems a bit silly to be doing replication to a slave server that has
any common point of failure with the master.
However, it seems like the point here is not so much "can you recover
your data" as what a commit means. Do you want a commit reported to the
client to mean the data is safely down to disk in both places, or only
one?
regards, tom lane
"Joshua D. Drake" <jd@commandprompt.com> writes: > That chance is minuscule or at least should be. Of course we are > assuming some level of conditioned power that is independent of the > power bus, e.g; a UPS. I find your faith in UPSes charmingly quaint. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Stefan Kaltenbrunner wrote: > Joshua D. Drake wrote: >> Gregory Stark wrote: >>> "Simon Riggs" <simon@2ndquadrant.com> writes: >>>> You're right, but the distinction is a small one. What are the chances >>>> of losing two independent servers within a few milliseconds of each >>>> other? >>> If they're on the same power bus? >> That chance is minuscule or at least should be. Of course we are >> assuming some level of conditioned power that is independent of the >> power bus, e.g; a UPS. > > how is that making it different in practise ? - if both are on the same > UPS they are affectively on the same power bus ... Well I was thinking the bus that is in the wall. I would assume that people were smart enough to have independent UPS systems for each server. city power->line conditioning generator->panel->plug->UPS->server wash, rinse repeat. > If the UPS fails (or the generator is not kicking in which happens way > more often than people would believe) they could still fail at the very > same time .... Sincerely, Joshua D. Drake > > > Stefan > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org > - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG4tOKATb/zqfZUUQRAiSTAJ4pqQqsP7aH9GPJYjY3hZDvKzU8cACeKKJ3 wAae0tl2XswsjgEncIsOBlw= =xsGZ -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gregory Stark wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
>
>> That chance is minuscule or at least should be. Of course we are
>> assuming some level of conditioned power that is independent of the
>> power bus, e.g; a UPS.
>
> I find your faith in UPSes charmingly quaint.
It isn't my faith in a UPS. It is my real world knowledge.
Further I will exert what I already replied to Stefan:
city power->line conditioning generator->panel->plug->UPS->server
You would have to have lightning handed by God to your server to have a
total power failure without proper shutdown in the above scenario.
Sincerely,
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG4tWpATb/zqfZUUQRAl00AJ4jC/CWkqrxeUjT0REedQAG3cvPPgCcCKkU
zbCu41UT25PnL7f7bT7dfXQ=
=tV5r
-----END PGP SIGNATURE-----
Joshua D. Drake wrote: > Stefan Kaltenbrunner wrote: >> Joshua D. Drake wrote: >>> Gregory Stark wrote: >>>> "Simon Riggs" <simon@2ndquadrant.com> writes: >>>>> You're right, but the distinction is a small one. What are the chances >>>>> of losing two independent servers within a few milliseconds of each >>>>> other? >>>> If they're on the same power bus? >>> That chance is minuscule or at least should be. Of course we are >>> assuming some level of conditioned power that is independent of the >>> power bus, e.g; a UPS. >> how is that making it different in practise ? - if both are on the same >> UPS they are affectively on the same power bus ... > > Well I was thinking the bus that is in the wall. I would assume that > people were smart enough to have independent UPS systems for each server. > > city power->line conditioning generator->panel->plug->UPS->server > > wash, rinse repeat. the typical datacenter version of this is actually more like: city power->UPS (with generator in parallel)->panel->plug or city power->flywheel->(maybe UPS)->panel->plug it is not really that common to have say two different UPS feeds in your rack (at least not for normal housing or the average corporate datacenter) - mostly you get two feeds from different power distribution panels (so different breakers) but that's about it. Having a local UPS attached is usually not really that helpful either because those have limited capacity need space and are an additional thing that can (and will) fail. Stefan
"Joshua D. Drake" <jd@commandprompt.com> writes: > It isn't my faith in a UPS. It is my real world knowledge. > > Further I will exert what I already replied to Stefan: > > city power->line conditioning generator->panel->plug->UPS->server > > You would have to have lightning handed by God to your server to have a > total power failure without proper shutdown in the above scenario. Which happens a couple times a year to various trusting souls. I suppose you're not a regular reader of Risks? Or a regular user of Livejournal for that matter? Analog is hard, let's go shopping. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
--On Samstag, September 08, 2007 12:39:37 -0400 Tom Lane
<tgl@sss.pgh.pa.us> wrote:
> However, it seems like the point here is not so much "can you recover
> your data" as what a commit means. Do you want a commit reported to the
> client to mean the data is safely down to disk in both places, or only
> one?
Yeah, that's what i meant to say. DRBD provides a handful other tweaks
besides changing the sync protocol, i'd start with them first. You can get
back experimenting with the sync protocol if there are still performance
issues then. I don't hesitate changing to B as long as I'm aware that it
changed semantics and I can deal with them.
--
Thanks
Bernd
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Stefan Kaltenbrunner wrote:
> Joshua D. Drake wrote:
>> Stefan Kaltenbrunner wrote:
>>> how is that making it different in practise ? - if both are on the same
>>> UPS they are affectively on the same power bus ...
>> Well I was thinking the bus that is in the wall. I would assume that
>> people were smart enough to have independent UPS systems for each server.
>>
>> city power->line conditioning generator->panel->plug->UPS->server
>>
>> wash, rinse repeat.
>
> the typical datacenter version of this is actually more like:
>
> city power->UPS (with generator in parallel)->panel->plug
Right, this is what we have at our colo except that I add a UPS where
appropriate in between panel and plug.
> city power->flywheel->(maybe UPS)->panel->plug
>
> it is not really that common to have say two different UPS feeds in your
> rack (at least not for normal housing or the average corporate
> datacenter) - mostly you get two feeds from different power distribution
> panels (so different breakers) but that's about it.
> Having a local UPS attached is usually not really that helpful either
> because those have limited capacity need space and are an additional
> thing that can (and will) fail.
We don't have the capacity issue. We use the UPS explicitly for specific
cases (like the one mentioned at the beginning of the thread). The whole
idea is to insure clean shutdown in case of "total" power failure.
Sincerely,
Joshua D. Drake
>
>
> Stefan
>
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG4xM5ATb/zqfZUUQRAszhAJ4qmwJQFHd/O5/alOSg1exrYEDe0wCeN6na
8BgWWO1aGELPOuX3xivEBVU=
=ETwV
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Joshua D. Drake wrote:
> Stefan Kaltenbrunner wrote:
>> Joshua D. Drake wrote:
>>> Stefan Kaltenbrunner wrote:
>
>>>> how is that making it different in practise ? - if both are on the same
>>>> UPS they are affectively on the same power bus ...
>>> Well I was thinking the bus that is in the wall. I would assume that
>>> people were smart enough to have independent UPS systems for each server.
>>>
>>> city power->line conditioning generator->panel->plug->UPS->server
>>>
>>> wash, rinse repeat.
>> the typical datacenter version of this is actually more like:
>
>> city power->UPS (with generator in parallel)->panel->plug
>
> Right, this is what we have at our colo except that I add a UPS where
> appropriate in between panel and plug.
Err plug and machine.
Joshua D. Drake
- ---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG4xSTATb/zqfZUUQRAkgyAJwKLLz0Jywex/b3d6hk8L2gHsZaXQCfYCyH
6Z/mdtOvnXc4MixgxchrxNY=
=kv8v
-----END PGP SIGNATURE-----
On Sat, 8 Sep 2007, Joshua D. Drake wrote: > You would have to have lightning handed by God to your server to have a > total power failure without proper shutdown in the above scenario. Do you live somewhere without thunderstorms? This is a regular event in this part of the world during the summer. It happened to me once this year and once last; lost count for previous ones. In both of the recent cases it's believed the servers were burned from the Ethernet side because somewhere in the network was a poor switch that wasn't isolated well enough from the grid when the building was hit. Lightning is tricky that way; cable TV and satellite wiring are also weak links that way. I didn't feel too bad about last year's because the building next door burned to the ground after being hit within a few seconds of mine, so the fact that I had some server repair didn't seem like too much of a hardship in comparison. The system was down until the fire department had put out the blaze and I was allowed back into the building though. Good thing my cell phone works in the server room; that phone call asking "are you aware the building is being evacuated?" is always a fun one. I'm not saying God hates you as much as me, because the fact that you've made this statement says it's clearly not the case, just that a plan presuming he's your buddy is a bad one. The way you're characterizing this risk reminds me of when people quote the odds that they'll lose two disks within seconds of one another in a disk array as if it's impossible, something else which of course has also happened to me many times during my career. -- * Greg "Lucky" Smith gsmith@gregsmith.com Baltimore, MD
Greg Smith <gsmith@gregsmith.com> writes:
> On Sat, 8 Sep 2007, Joshua D. Drake wrote:
>> You would have to have lightning handed by God to your server to have a
>> total power failure without proper shutdown in the above scenario.
> Do you live somewhere without thunderstorms? This is a regular event in
> this part of the world during the summer. It happened to me once this
> year and once last; lost count for previous ones. In both of the recent
> cases it's believed the servers were burned from the Ethernet side because
> somewhere in the network was a poor switch that wasn't isolated well
> enough from the grid when the building was hit. Lightning is tricky that
> way; cable TV and satellite wiring are also weak links that way.
Yeah. I've lost half a dozen modems of varying generations, a server
motherboard, a TiVo, a couple of VCRs, and miscellaneous other equipment
from strikes near my house --- none closer than a couple blocks away.
I don't really care to think about what would still work after a direct
hit, despite the whole-house surge suppressor at the meter and the local
suppressor on each circuit and the allegedly surge-proof UPSes powering
all the valuable stuff. I've also moved heavily into wireless local
net to eliminate any direct electrical connections between machines that
are not on the same power circuit (the aforesaid burned motherboard
taught me that particular lesson). And yet I still fear every time a
thunderstorm passes over.
Then of course there are the *other* risks, such as the place burning to
the ground, or getting drowned by a break in the city reservoir that's
a couple hundred yards up the hill (but at least I needn't worry about
any nearby rivers rising, as I'm well above them). Or maybe being
burgled by Oracle employees who are specifically after my backup tapes.
If you ain't got a backup plan, you *will* lose data. Imagining that
there is one perfect technological solution to this problem is the very
fastest way to lose.
regards, tom lane
Thank you very much for your ideas. I've tried to change the protocol
from C to B and I got an increase in the number of TPS: 64.555763.
Now I would like to follow the advice of Mr. Bernd Helmle and change the
value of snd-bufsize.
The servers are cross connected with a common 100 Mbit/sec Ethernet so I
think they have a bandwidth around 80 Mbit/sec (even if I haven't yet
done any test on it). A rate of 70Mb seems reasonable to me.
The two servers are in two different racks (next to each other) and they
have two power supplies connected to two different sets of UPS.
Unfortunately we cannot accept a loss of recently committed transactions
so we cannot put the synchronous_commit to off.
Regards,
Maila Fatticcioni
Simon Riggs wrote:
> On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote:
>
>> protocol C;
>
> Try protocol B instead.
>
--
______________________________________________________________
Maila Fatticcioni
______________________________________________________________
Mediterranean Broadband Infrastructure s.r.l.
ITALY
______________________________________________________________
Вложения
On Tue, Sep 11, 2007 at 04:47:40PM +0200, Maila Fatticcioni wrote: > The servers are cross connected with a common 100 Mbit/sec Ethernet so I > think they have a bandwidth around 80 Mbit/sec (even if I haven't yet > done any test on it). A rate of 70Mb seems reasonable to me. Umm, seriously? Unless that was a typo, you should consider very seriously to go to gigabit; it's cheap these days, and should provide you with a very decent speed boost if the network bandwidth is the bottleneck. /* Steinar */ -- Homepage: http://www.sesse.net/
Simon Riggs schrieb: > On Fri, 2007-09-07 at 11:37 +0200, Maila Fatticcioni wrote: > >> protocol C; > > Try protocol B instead. > Sure? I've always heard that there has yet to be a case found, where B is better than C. We use DRBD with protocol C, and are quite happy with it.
On Mon, Sep 10, 2007 at 12:06:40AM -0400, Greg Smith wrote: > On Sat, 8 Sep 2007, Joshua D. Drake wrote: > > >You would have to have lightning handed by God to your server to have a > >total power failure without proper shutdown in the above scenario. > > Do you live somewhere without thunderstorms? This is a regular event in Actually, he does. :) Or at least I don't think Portland gets a lot of t-storms, just rain by the bucketful. -- Decibel!, aka Jim Nasby decibel@decibel.org EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
Вложения
On Mon, Sep 10, 2007 at 12:54:37AM -0400, Tom Lane wrote: > Greg Smith <gsmith@gregsmith.com> writes: > > On Sat, 8 Sep 2007, Joshua D. Drake wrote: > >> You would have to have lightning handed by God to your server to have a > >> total power failure without proper shutdown in the above scenario. > > > Do you live somewhere without thunderstorms? This is a regular event in > > this part of the world during the summer. It happened to me once this > > year and once last; lost count for previous ones. In both of the recent > > cases it's believed the servers were burned from the Ethernet side because > > somewhere in the network was a poor switch that wasn't isolated well > > enough from the grid when the building was hit. Lightning is tricky that > > way; cable TV and satellite wiring are also weak links that way. > > Yeah. I've lost half a dozen modems of varying generations, a server > motherboard, a TiVo, a couple of VCRs, and miscellaneous other equipment > from strikes near my house --- none closer than a couple blocks away. > I don't really care to think about what would still work after a direct > hit, despite the whole-house surge suppressor at the meter and the local > suppressor on each circuit and the allegedly surge-proof UPSes powering > all the valuable stuff. I've also moved heavily into wireless local <dons EE hat> Pretty much every surge supressor out there is a POS... 99.9% of them just wire a varistor across the line; like a $0.02 part is going to stop a 10,00+ amp discharge. The only use I have for those things is if they come with an equipment guarantee, though I have to wonder how much those are still honored, since as you mention it's very easy for equipment to be fried via other means (ethernet, monitor, etc). > net to eliminate any direct electrical connections between machines that > are not on the same power circuit (the aforesaid burned motherboard > taught me that particular lesson). And yet I still fear every time a > thunderstorm passes over. Wired is safe as long as everything's on the same circuit. My house is wired for ethernet with a single switch running what's going to every room, but in each room I have a second switch on the same power as whatever's in that room; so if there is a strike it's far more likely that I'll lose switches and not hardware. > Then of course there are the *other* risks, such as the place burning to > the ground, or getting drowned by a break in the city reservoir that's > a couple hundred yards up the hill (but at least I needn't worry about Invest in sponges. Lots of them. :) -- Decibel!, aka Jim Nasby decibel@decibel.org EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
Вложения
On Tue, Sep 11, 2007 at 04:57:24PM +0200, Steinar H. Gunderson wrote: > On Tue, Sep 11, 2007 at 04:47:40PM +0200, Maila Fatticcioni wrote: > > The servers are cross connected with a common 100 Mbit/sec Ethernet so I > > think they have a bandwidth around 80 Mbit/sec (even if I haven't yet > > done any test on it). A rate of 70Mb seems reasonable to me. > > Umm, seriously? Unless that was a typo, you should consider very seriously to > go to gigabit; it's cheap these days, and should provide you with a very > decent speed boost if the network bandwidth is the bottleneck. Actually, in this case, I suspect that latency will be far more critical than overall bandwidth. I don't know if it's inherent to Gig-E, but my limited experience has been that Gig-E has higher latency than 100mb. -- Decibel!, aka Jim Nasby decibel@decibel.org EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
Вложения
Decibel! wrote: > <dons EE hat> > Pretty much every surge supressor out there is a POS... 99.9% of them > just wire a varistor across the line; like a $0.02 part is going to stop > a 10,00+ amp discharge. > > The only use I have for those things is if they come with an equipment > guarantee, though I have to wonder how much those are still honored, > since as you mention it's very easy for equipment to be fried via other > means (ethernet, monitor, etc). My UPSs, from American Power Conversion, have one of those impressive guarantees. It specifies that all connections to my computer must be protected: power, modem, ethernet, etc. It further specifies that everything must be UL or CSA approved, and so on and so forth. Well, that is what I have. > >> net to eliminate any direct electrical connections between machines that >> are not on the same power circuit (the aforesaid burned motherboard >> taught me that particular lesson). And yet I still fear every time a >> thunderstorm passes over. > > Wired is safe as long as everything's on the same circuit. My house is > wired for ethernet with a single switch running what's going to every > room, but in each room I have a second switch on the same power as > whatever's in that room; so if there is a strike it's far more likely > that I'll lose switches and not hardware. My systems are all in the same room. The UPS for the main system has a single outlet on a circuit all its own all the way back to the power panel at the building entrance. The UPS for my other system also has a outlet on a circuit all its own all the way back to the power panel at the building entrance -- on the other side of my 240 volt service so they sorta-kinda balance out. The only other UPS is a little 620 VA one for the power to the Verizon FiOS leading into my house. That is fibre-optic all the way to the pole. I will probably get less lightning coming in that way than when I used to be on copper dial-up. ;-) > >> Then of course there are the *other* risks, such as the place burning to >> the ground, or getting drowned by a break in the city reservoir that's >> a couple hundred yards up the hill (but at least I needn't worry about > > Invest in sponges. Lots of them. :) -- .~. Jean-David Beyer Registered Linux User 85642. /V\ PGP-Key: 9A2FC99A Registered Machine 241939. /( )\ Shrewsbury, New Jersey http://counter.li.org ^^-^^ 22:00:01 up 34 days, 1:22, 5 users, load average: 4.05, 4.22, 4.25
Hi,
Decibel! wrote:
> Actually, in this case, I suspect that latency will be far more critical
> than overall bandwidth. I don't know if it's inherent to Gig-E, but my
> limited experience has been that Gig-E has higher latency than 100mb.
I've been looking for some benchmarks, but it's rather hard to find. It
looks like people are much more concerned about throughput ?!?
However, I'd like to share some of the sites I've found, especially
regarding Fast Ethernet vs. Gigabit Ethernet:
- Ashford Computer Consulting Service benchmarked five different
gigabit ethernet adapters [1], back in 2004. For most cards they
measured between ca. 100 - 150 microseconds for a UDP round trip of a
token, a so called hot potato benchmark. Unfortunately they didn't
compare with Fast Ethernet.
- The NetPIPE project has some of it's measurements at the very bottom
of it's website [2]. Mostly for high speed and low latency links. Again,
Fast Ethernet is missing. The diagram tells the following latencies (in
microseconds):
75 10 Gigabit Ethernet
62 Gigabit Ethernet
8 Myrinet
7.5 Infini Band
4.7 Atoll
4.2 SCI
I've no explanation for the significantly better measure for gigabit
ethernet compared with the above benchmark. From their description I'm
concluding that they also measured a round-trip, but not via UDP.
The bad value for 10 Gigabit Ethernet is due to a poor Intel adapter,
which also has poor throughput. They claim that newer adapters are better.
- Finally, I've found a latency comparison between Fast vs Gigabit
Ethernet, here [3]. Figure 6, in the second third of the page shows a
NetPIPE latency benchmark between Ethernet, Fast Ethernet and Gigabit
Ethernet (additionally ATM and FDDI). It looks like Gigabit Ethernet
features slightly better latency.
From these findings I'm concluding, that commodity Ethernet hardware
has quite similar latencies, no matter if you are using Fast, Gigabit or
10 Gigabit Ethernet. If you really want to have a low latency
interconnect, you need to pay the extra bucks for specialized, low
latency networking hardware (which may still be based on 10GE, see
Myrinet's 10GE adapter).
If you know other resources, I'd be curious to know.
Regards
Markus
[1]: Ashford Computer Consulting Service, GigE benchmarks:
http://www.accs.com/p_and_p/GigaBit/conclusion.html
[2]: NetPIPE website:
http://www.scl.ameslab.gov/netpipe/
[3]: Gigabit Ethernet and Low-Cost Supercomputing
http://www.scl.ameslab.gov/Publications/Gigabit/tr5126.html