Re: Pros/cons of big databases vs smaller databases and RDS

On 10/5/21 5:06 PM, Wells Oliver wrote:

Hi-- to keep it short, I feel like I've generally heard the larger your DB is, the less efficient it might run, likely due to disk I/O. Maybe I'm terribly mistaken in this perception.

Less *efficient*???

Of course ceteris paribus it takes longer to do "whole database" activities on a larger database because... there's more data, but that should be obvious to all.

So you scale up the IO (SAN, RAID 50, faster disks, etc)  or partition the data or both.

We have two DBs, one primarily accessed by humans and systems, which is ~1TB in size, and aggregates most of what we store in raw, longer format on a second DB that is about ~6TB in size.

As we consider plans to migrate to RDS, we've talked a lot about combining the two as more and more the case is querying the larger DB and wanting data only available in the smaller DB.

Of course, we can solve this by copying things back and forth, but we're also thinking: why not just one big DB?

Anyone have any experiences with a similar project, and especially any technical configurations that might be beneficial in using RDS?

Nothing like your scenario, but we recently migrated an 8TB Oracle database hosted on a VM + (big fancy) SAN to a 6TB RDS Postgresql database.  Reads are faster, and writes seem a bit slower, but enough software changed that comparisons are dubious.

One thing I can say is, "watch your costs!!"  Amazon does an excellent job of obfuscating that, hiding it from everyone but the accountants.  It's dreadfully easy for technical people to unknowingly spend tens or hundreds of thousands of dollars in a month when testing things.