Re: Multiple COPY statements for one table vs one for ~half a billion records

Поиск
Список
Период
Сортировка
От Ron Johnson
Тема Re: Multiple COPY statements for one table vs one for ~half a billion records
Дата
Msg-id CANzqJaCBmGpbqc6RS-BubJpeW3aUNiM3knQL0KWOmCFdoeOQvg@mail.gmail.com
обсуждение исходный текст
Ответ на [MASSMAIL] Multiple COPY statements for one table vs one for ~half a billion records  (Carl L <cllewellyno@gmail.com>)
Список pgsql-general
Дерево обсуждения
On Thu, Apr 4, 2024 at 2:04 PM Carl L <cllewellyno@gmail.com> wrote:
Hi there,

I have around half a billion records that are being generated from a back end that are split into 80 threads (one per core) and I'm performing a copy from memory ( from stdin binary) into Postgres from each of these threads - i.e. there are 80 COPY statements being generated for one table that are running concurrently. I can see each of the Postgres processes sitting at around 15% CPU usage.

Is the target table partitioned in the same way that the input data is split?

That would make things faster...
 
These are all also in the same transaction - I am the only one connected, so it's not an issue to hold a big transaction.

Unless it fills up your WAL partition.

В списке pgsql-general по дате отправления:

Предыдущее
От: Carl L
Дата:
Сообщение: [MASSMAIL] Multiple COPY statements for one table vs one for ~half a billion records
Следующее
От: Tom Lane
Дата:
Сообщение: Re: What permissions are required for e.g. EXPLAIN UPDATE ...