Re: COPY TO STDOUT Apache Arrow support

Поиск
Список
Период
Сортировка
От Adam Lippai
Тема Re: COPY TO STDOUT Apache Arrow support
Дата
Msg-id CAGrfaBXhi_mNzVQR=ZirOCm1AFFw2ntH_Uk=7CKFNus-c4ajHg@mail.gmail.com
обсуждение исходный текст
Ответ на COPY TO STDOUT Apache Arrow support  (Adam Lippai <adam@rigo.sk>)
Ответы Re: COPY TO STDOUT Apache Arrow support  (Adam Lippai <adam@rigo.sk>)
Список pgsql-hackers
Дерево обсуждения
Hi,

There are two bigger developments in this topic:
  1. Pandas 2.0 is released and it can use Apache Arrow as a backend
  2. Apache Arrow ADBC is released which standardizes the client API. Currently it uses the postgresql wire protocol underneath
Best regards,
Adam Lippai

On Thu, Apr 21, 2022 at 10:41 AM Adam Lippai <adam@rigo.sk> wrote:
Hi,

would it be possible to add Apache Arrow streaming format to the copy backend + frontend?
The use case is fetching (or storing) tens or hundreds of millions of rows for client side data science purposes (Pandas, Apache Arrow compute kernels, Parquet conversion etc). It looks like the serialization overhead when using the postgresql wire format can be significant.

Best regards,
Adam Lippai

В списке pgsql-hackers по дате отправления:

Предыдущее
От: Fujii Masao
Дата:
Сообщение: Re: Issue in postgres_fdw causing unnecessary wait for cancel request reply
Следующее
От: sirisha chamarthi
Дата:
Сообщение: Fix documentation for max_wal_size and min_wal_size