Re: Uber migrated from Postgres to MySQL

On 07/28/2016 03:16 PM, Bruce Momjian wrote:
> On Thu, Jul 28, 2016 at 12:35:23AM -0700, Jeff Janes wrote:
>> On Wed, Jul 27, 2016 at 9:48 PM, John R Pierce  wrote:
>>> On 7/27/2016 9:39 PM, Jeff Janes wrote:
>>>>
>>>> That depends on how how many objects there are consuming that 1 TB.
>>>> With millions of small objects, you will have problems.  Not as many
>>>> in 9.5 as there were in 9.1, but still it does not scale linearly in
>>>> the number of objects.  If you only have thousands of objects, then as
>>>> far as I know -k works like a charm.
>>>
>>>
>>> millions of tables?
>>
>> Well, it was a problem at much smaller values, until we fixed many of
>> them.  But the perversity is, if you are stuck on a version before the
>> fixes, the problems prevent you from getting to a version on which it
>> is not a problem any more.
> 
> Uh, that is only true if the slowness was in _dumping_ many objects. 
> Most of the fixes have been for _restoring_ many objects, and that is
> done in the new cluster, so they should be OK.


Not really true. I ran into two separate cases where on older (pre 9.3 I
believe) Postgres if you had hundreds of thousands of tables (in the
case I remember well, it was about 500k tables) the schema dump from the
old cluster basically never finished (ok, was killed after about a
week). I had to find the patch that fixed a good bit of the slowness and
backport it to the older version so we could successfully run pg_upgrade
(in something like 14 hours instead of 7+ days).

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development