On 1/15/17 3:04 PM, Adrian Klaver wrote:
> On 01/15/2017 12:32 PM, Jim Nasby wrote:
>> Interesting... https://github.com/blue-yonder/turbodbc
>
> Yes, interesting but you are still dependent on the underlying ODBC
> implementation. Not sure how that impacts performance versus a method
> with fewer hops?
Oh, I'd hope that a native libpq implementation would be faster than
going through ODBC. But, there's presumably useful info that can be
picked up here; the bit about buffering was certainly interesting.
BTW, the person that brought this to my attention had mentioned that a
lot of people doing data science with data living in Postgres feel the
need to extract the data from Postgres into something like HDFS before
they can do anything useful, because apparently data access through HDFS
is 3x faster than through Postgres. My impression is that at least part
of that is due to using Pandas from_sql functionality (which AIUI
marshals everything through SQL Alchemy), but anything that can be done
on the psycopg2 side would help.
I'm also looking into speeding up SPI access through plpython; depending
on how you want to measure I've gotten a 30-600% improvement by removing
the buffering that SPI does by default.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)