Re: speed up full table scan using psql

Поиск
Список
Период
Сортировка
От Lian Jiang
Тема Re: speed up full table scan using psql
Дата
Msg-id CA+aY8X6+ENwVh0megxbHczoX-W-Yd6BmHKRUYS3MJNLGLJDz_w@mail.gmail.com
обсуждение исходный текст
Ответ на Re: speed up full table scan using psql  (Adrian Klaver <adrian.klaver@aklaver.com>)
Ответы Re: speed up full table scan using psql
Список pgsql-general
The whole command is:

psql %(pg_uri)s -c %(sql)s | %(sed)s | %(pv)s | %(split)s) 2>&1 | %(tr)s

where:
sql is "copy (select row_to_json(x_tmp_uniq) from public.mytable x_tmp_uniq) to stdout"
sed, pv, split, tr together format and split the stdout into jsonl files.

Hope this helps.


On Wed, May 31, 2023 at 9:16 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 5/31/23 13:57, Lian Jiang wrote:
> The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq)
> from public.mytable x_tmp_uniq) to stdout"

What is taking the stdout and what it is it doing?

> postgres version:  14.7
> Does this mean COPY and java CopyManager may not help since my psql
> command already uses copy?
>
> Regarding pg_dump, it does not support json format which means extra
> work is needed to convert the supported format to jsonl (or parquet) so
> that they can be imported into snowflake. Still exploring but want to
> call it out early. Maybe 'custom' format can be parquet?
>
>
> Thanks
> Lian

--
Adrian Klaver
adrian.klaver@aklaver.com



--

В списке pgsql-general по дате отправления:

Предыдущее
От: Adrian Klaver
Дата:
Сообщение: Re: speed up full table scan using psql
Следующее
От: Jim Vanns
Дата:
Сообщение: Dynamic creation of list partitions in highly concurrent write environment