Обсуждение: Bulk Insert into PostgreSQL

Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.

From: Srinivas Karthik V [mailto:skarthikv.iitb@gmail.com]
> I was using copy command to load. Removing the primary key constraint on
> the table and then loading it helps a lot. In fact, a 400GB table was loaded
> and the primary constraint was added in around 15 hours.  Thanks for the
> wonderful suggestions.

400 GB / 15 hours = 7.6 MB/s

That looks too slow.  I experienced a similar slowness.  While our user tried to INSERT (not COPY) a billion record,
theyreported INSERTs slowed down by 10 times or so after inserting about 500 million records.  Periodic pstack runs on
Linuxshowed that the backend was busy in btree operations.  I didn't pursue the cause due to other businesses, but
theremight be something to be improved.
 


Regards
Takayuki Tsunakawa

On Sun, Jul 1, 2018 at 5:19 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:
> 400 GB / 15 hours = 7.6 MB/s
>
> That looks too slow.  I experienced a similar slowness.  While our user tried to INSERT (not COPY) a billion record,
theyreported INSERTs slowed down by 10 times or so after inserting about 500 million records.  Periodic pstack runs on
Linuxshowed that the backend was busy in btree operations.  I didn't pursue the cause due to other businesses, but
theremight be something to be improved. 

What kind of data was indexed? Was it a bigserial primary key, or
something else?

--
Peter Geoghegan

From: Peter Geoghegan [mailto:pg@bowt.ie]
> What kind of data was indexed? Was it a bigserial primary key, or
> something else?

Sorry, I don't remember it.  But the table was for storing some machine usage data, and I don't think any sequence was
usedin the index.
 

According to my faint memory, iostat showed many reads on the database storage, and correspondingly pstack showed
ReadBufferExtendedduring the btree operations.  shared_buffers was multiple GBs.  I wondered why btree operations
didn'tbenefit from the caching of non-leaf nodes.
 

Regards
Takayuki Tsunakawa

On Tue, Jul 3, 2018 at 4:34 PM, Srinivas Karthik V
<skarthikv.iitb@gmail.com> wrote:
> @Peter: I was indexing the primary key of all the tables in tpc-ds. Some of
> the fact tables has multiple columns as part of the primary key. Also, most
> of them are numeric type.

Please see my mail to -hackers on suffix truncation:
https://postgr.es/m/CAH2-Wzn5XbCzk6u0GL+uPnCp1tbrp2pJHJ=3bYT4yQ0_zzHxmw@mail.gmail.com

Perhaps this is related in some way, since in both cases we're talking
about a composite index on varlena-type columns, where the types have
expensive comparisons.

-- 
Peter Geoghegan