Re: [GENERAL] Logical decoding CPU-bound w/ large number of tables

Поиск
Список
Период
Сортировка
От Andres Freund
Тема Re: [GENERAL] Logical decoding CPU-bound w/ large number of tables
Дата
Msg-id 20170506013810.2og6vax6a2u3rhjc@alap3.anarazel.de
обсуждение исходный текст
Ответ на Re: [GENERAL] Logical decoding CPU-bound w/ large number of tables  (Tom Lane <tgl@sss.pgh.pa.us>)
Ответы Re: [GENERAL] Logical decoding CPU-bound w/ large number of tables  (Mathieu Fenniak <mathieu.fenniak@replicon.com>)
Список pgsql-general
On 2017-05-05 21:32:27 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2017-05-05 14:24:07 -0600, Mathieu Fenniak wrote:
> >> It appears that most of the time is spent in the
> >> RelfilenodeMapInvalidateCallback and CatalogCacheIdInvalidate cache
> >> invalidation callbacks, both of which appear to be invalidating caches
> >> based upon the cache value.
>
> > I think optimizing those has some value (and I see Tom is looking at
> > that aspect, but the bigger thing would probably be to do fewer lookups.
>
> I'm confused --- the lookup side of things is down in the noise in
> Mathieu's trace.

Err, sorry. Completely mangled that sentence.  Executing fewer
invalidations.  We currently are likely re-executing the same set of
invalidations constantly in Mathieu's case.

Background: When decoding a transaction during logical decoding we're
currently re-executing *all* a transaction's own cache invalidations, if
it has any, at every new command-id observed in the WAL stream.  That's
because currently invalidations are only sent at commit, so we don't
know from "when" they are.  But I think there's some very low-hanging
fruits reducing the frequency at which those are executed.

In many cases where there's just a few schema changes in a transaction,
this doesn't hurt badly.  But if you have a transaction that does a
bootload of schema changes *and* a has a lot of other changes, it gets
expensive.

Mathieu: The above also indicates a possible workaround, you can try
separating larger amounts of data manipulations from schema changes,
into separate transactions.

Greetings,

Andres Freund


В списке pgsql-general по дате отправления:

Предыдущее
От: Tom Lane
Дата:
Сообщение: Re: [GENERAL] Logical decoding CPU-bound w/ large number of tables
Следующее
От: Igor Korot
Дата:
Сообщение: [GENERAL] Where is the error?