RE: Loading 500m json files to database

Поиск
Список
Период
Сортировка
От Kevin Brannen
Тема RE: Loading 500m json files to database
Дата
Msg-id SA0PR19MB42555877CA8D229BA3F405ACA4F10@SA0PR19MB4255.namprd19.prod.outlook.com
обсуждение исходный текст
Ответ на Re: Loading 500m json files to database  (pinker <pinker@onet.eu>)
Ответы Re: Loading 500m json files to database
Список pgsql-general
From: pinker <pinker@onet.eu>

> it's a cloud and no plpythonu extension avaiable unfortunately


You're misunderstanding him. See David's post for an example, but the point was that you can control all of this from
an*external* Perl, Python, Bash, whatever program on the command line at the shell. 

In pseudo-code, probably fed by a "find" command piping filenames to it:

while more files
do { read in a file name & add to list } while (list.length < 1000);
process entire list with \copy commands to 1 psql command

I've left all kinds of checks out of that, but that's the basic thing that you need, implement in whatever scripting
languageyou're comfortable with. 

HTH,
Kevin
This e-mail transmission, and any documents, files or previous e-mail messages attached to it, may contain confidential
information.If you are not the intended recipient, or a person responsible for delivering it to the intended recipient,
youare hereby notified that any disclosure, distribution, review, copy or use of any of the information contained in or
attachedto this message is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately
notifyus by reply e-mail, and destroy the original transmission and its attachments without reading them or saving them
todisk. Thank you. 



В списке pgsql-general по дате отправления:

Предыдущее
От: Jerry Sievers
Дата:
Сообщение: Re: avoid WAL for refresh of materialized view
Следующее
От: Rob Sargent
Дата:
Сообщение: Re: Loading 500m json files to database