fast read of binary data

Поиск
Список
Период
Сортировка
От Eildert Groeneveld
Тема fast read of binary data
Дата
Msg-id 1352717148.3684.18.camel@eno.tzv.local
обсуждение исходный текст
Ответы Re: fast read of binary data  ("Albe Laurenz" <laurenz.albe@wien.gv.at>)
Re: fast read of binary data  (Arjen van der Meijden <acmmailing@tweakers.net>)
Re: fast read of binary data  (Merlin Moncure <mmoncure@gmail.com>)
Список pgsql-performance
Dear All

I am currently implementing using a compressed binary storage scheme
genotyping data. These are basically vectors of binary data which may be
megabytes in size.

Our current implementation uses the data type bit varying.

What we want to do is very simple: we want to retrieve such records from
the database and transfer it unaltered to the client which will do
something (uncompressing) with it. As massive amounts of data are to be
moved, speed is of great importance, precluding any to and fro
conversions.

Our current implementation uses Perl DBI; we can retrieve the data ok,
but apparently there is some converting going on.

Further, we would like to use ODBC from Fortran90 (wrapping the
C-library)  for such transfers. However, all sorts funny things happen
here which look like conversion issues.

In old fashioned network database some decade ago (in pre SQL times)
this was no problem. Maybe there is someone here who knows the PG
internals sufficiently well to give advice on how big blocks of memory
(i.e. bit varying records) can between transferred UNALTERED between
backend and clients.

looking forward to you response.

greetings

Eildert



--
Eildert Groeneveld
===================================================
Institute of Farm Animal Genetics (FLI)
Mariensee 31535 Neustadt Germany
Tel : (+49)(0)5034 871155 Fax : (+49)(0)5034 871143
e-mail: eildert.groeneveld@fli.bund.de
web:    http://vce.tzv.fal.de
==================================================



В списке pgsql-performance по дате отправления:

Предыдущее
От: "Albe Laurenz"
Дата:
Сообщение: Re: Index is not using
Следующее
От: "Albe Laurenz"
Дата:
Сообщение: Re: fast read of binary data