Re: New system recommendations

I'm putting both the private email and the thread back on the list, as
there's interesting data in here people could use.  I don't think I'm
betraying any trust here, but if I am, please, let me know...

On Thu, 2006-04-27 at 17:37, Benjamin Krajmalnik wrote:
> Thanks for the feedback.
> I wonder if the Dell backplane will take a stabdard RAID controller
> instead of using the ROMB.
> I may investigate just getting 146GB drives for the DL360 and using that
> instead, and maybe setting up the dell as a backup server, creating
> backups and moving the files over.

If you've got some spare smaller drives laying about, you can always set
up both and benchmark them against each other, plus that gives you a
ready to go backup machine.

> Quick question - I do not have a transaction frame explicitly declared
> in m stored procedure (which is used to handle all of the inserts).
> In plpgsql, how would I wrap the code inside a transaction frame?

All stored procs run as a single transaction.  I.e. the simple act of
making it a user defined function has made it a single transaction, with
all the benefits that entails.

> This is such a dilemma - we are launching a new service, but until we
> sign up clients on it I don't have the budget to go crazy on hardware.
> I must make do with what I have.  What I was thinking was mybe setting
> up the Dell to be the web server and the backup database server, and
> having the production database run on the DL 360, provided I can get
> better database performance.  If I can get significant better
> performance on FreeBSD I will go to it, otherwise I will stay with
> Windows.

I'd test the two.  I'm guessing that at least for the initial ramp up,
windows will be ok.  The real performance issue for windows is opening
connections is much slower.  If the majority of your machine's time is
spent on the query, the overhead of opening connections will be lost in
the noise.

> The stored procedure is a plpgsql function which gets passed parameters
> from the monitoring agent.
> It then dynamically creates a device record for the monitored device if
> one does not yet exist.
> Once that is done it creates a test record for the particular test if
> one does not exist.

Up to now, fine.  Performance on any machine should be ok.

> Once that is done, it aggregates dynamically the data into 4 tables - a
> daily snapshot, a weekly snapshot, and a monthly snapshot, and a
> dashboard snapshot.

Are you running aggregate functions against the whole table to do this?
If so, this isn't going to scale.  If you're just taking the data
entered in this stored proc and adding it to a table that contains that
data, then maybe it's ok.  But if it's updating based on a huge amount
of data, then that's going to be slow.

> Once all of that has been accomplished, it creates a raw log entry
> (which as of next week will go to a given partition).  This data goes
> into the partitioned table to facilitate purging of retention periods
> without hammering at the database and having to rebalance indices (I
> just truncate the partition once it is no longer needed).

Sounds reasonable.