Обсуждение: QSoC proposal: Rewrite pg_dump and pg_restore

Hello!<br /> Here is the text of my proposal which I've applied to GSoC.<br /> (and link <a
class="moz-txt-link-freetext"
href="https://docs.google.com/document/d/1s-Q4rzEysPxo-dINsk_eKFJOBoVjNYDrQ-Oh75gtYEM/edit?usp=sharing">https://docs.google.com/document/d/1s-Q4rzEysPxo-dINsk_eKFJOBoVjNYDrQ-Oh75gtYEM/edit?usp=sharing</a>)<br
/>Any suggestions and comments are welcome.<br /><b id="docs-internal-guid-cbd5f555-d7c4-35b8-0764-622e27f77da0"
style="font-weight:normal;"><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
center;"><spanstyle="font-size:19px;font-family:'Times New
 

Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">PostgreSQL
GSoC2014 proposal</span><br /><span style="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><br
/><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Project
name</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Rewrite
(add)pg_dump and pg_restore utilities as libraries (.so, .dll & .dylib)</span><br /><span
style="font-size:11px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><br
/><spanstyle="font-size:11px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Short
description</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">pg_dump
isa utility for backing up a PostgreSQL database. It makes consistent backups even if the database is being used
concurrently.pg_restore is a utility for restoring a PostgreSQL database from an archive created by pg_dump in one of
thenon-plain-text formats. I think it will be more comforatable to use these tools as libraries.</span><br /><span
style="font-size:11px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Name</span><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">:
AlexanderShvidchenko</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">E-mail</span><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">:
</span><ahref="mailto:askellio@gmail.com" style="text-decoration:none;"><span style="font-size:16px;font-family:'Times
New

Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">askellio@gmail.com</span></a><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Location</span><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">:
Rostov-on-Don,Russia (UTC +04.00)</span><br /><span style="font-size:11px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Benefits
tothe PostgreSQL Community</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><span style="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">This
featurewill expand opportunities to work with backups of databases. Especially, it’s important for third-party
developers.It will be easier to use the functionality of the tools in applications when they will be
libraries.</span><br/><span style="font-size:11px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Quantifiable
results</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Static
anddynamic libraries with the functionality of pg_dump and pg_restore tools (.so, .dll and .dylib files)</span><br
/><spanstyle="font-size:11px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Project
Schedule</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">until
May31 </span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Make
codereview and solve architecture questions with help of community</span><p dir="ltr"
style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:      justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1
June– 30 June</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Detailed
implementationof libraries.</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
 justify;"><span style="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1
July– 31 July</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Finish
Implementationof libraires and begin testing. </span><p dir="ltr"
style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:      justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1
August-15 August</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Final
refactoring,testing and commit.</span><br /><span style="font-size:11px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Academic
experience</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">I
enteredthe university in 2013. Before entering the university I finished the college in 2012. My graduate work in the
collegewas the client-server application. It was a realization of XMPP. The client was realized in Qt. The client
workedwith SQLite database and the server worked with MySQL database.</span><br /><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Why
isPostgreSQL?</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">-
</span><spanstyle="font-size:16px;font-family:'Times New
 

Roman';color:#444444;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">I’m
interetedin this idea and believe this project would be useful for the community;</span><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">-
PostgreSQLis a very respected community. I would be proud to be a part of it;</span><p dir="ltr"
style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:      justify;"><span
style="font-size:16px;font-family:'TimesNew 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">-
PostgreSQLis one of the best DBMS and I would like to make it better.</span><br /><span
style="font-size:11px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Links</span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1)
PostgreSQL9.3.3 Documentation, pg_dump</span><p dir="ltr"
style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:      justify;"><a
href="http://www.postgresql.org/docs/9.3/static/app-pgdump.html"style="text-decoration:none;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://www.postgresql.org/docs/9.3/static/app-pgdump.html</span></a><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">2)
PostgreSQL9.3.3 Documentation, pg_restore</span><p dir="ltr"
style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:      justify;"><a
href="http://www.postgresql.org/docs/9.3/static/app-pgrestore.html"style="text-decoration:none;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://www.postgresql.org/docs/9.3/static/app-pgrestore.html</span></a><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">3)
Staticlibrary</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><ahref="http://en.wikipedia.org/wiki/Static_library" style="text-decoration:none;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://en.wikipedia.org/wiki/Static_library</span></a><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p
dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:       justify;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">4)
Dynamiclibrary</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:
justify;"><ahref="http://en.wikipedia.org/wiki/Dynamic-link_library" style="text-decoration:none;"><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://en.wikipedia.org/wiki/Dynamic-link_library</span></a><span
style="font-size:16px;font-family:'TimesNew
 

Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span></b><br
class="Apple-interchange-newline"/><br /> With best wishes,<br /> Alexander S.<br /> 

On Tue, Mar 18, 2014 at 8:41 PM, Alexandr <askellio@gmail.com> wrote:
> Rewrite (add) pg_dump and pg_restore utilities as libraries (.so, .dll &
> .dylib)

This strikes me as (1) pretty vague and (2) probably too hard for a
summer project.

I mean, getting the existing binaries to build libraries that you can
call with some trivial interface that mimics the existing command-line
functionality of pg_dump might be doable, but that's not all that
interesting.  What people are really going to want is a library with a
sophisticated API that lets you do interesting things
programmatically.  But that's going to be hard.  AFAIK, nobody's even
tried to figure out what that API should look like.  Even if we had
that worked out, a non-trivial task, the pg_dump source code is a
mess, so refactoring it to provide such an API is likely to be a job
and a half.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 03/21/2014 09:28 AM, Robert Haas wrote:
> On Tue, Mar 18, 2014 at 8:41 PM, Alexandr <askellio@gmail.com> wrote:
>> Rewrite (add) pg_dump and pg_restore utilities as libraries (.so, .dll &
>> .dylib)
> 
> This strikes me as (1) pretty vague and (2) probably too hard for a
> summer project.
> 
> I mean, getting the existing binaries to build libraries that you can
> call with some trivial interface that mimics the existing command-line
> functionality of pg_dump might be doable, but that's not all that
> interesting.  What people are really going to want is a library with a
> sophisticated API that lets you do interesting things
> programmatically.  But that's going to be hard.  AFAIK, nobody's even
> tried to figure out what that API should look like.  Even if we had
> that worked out, a non-trivial task, the pg_dump source code is a
> mess, so refactoring it to provide such an API is likely to be a job
> and a half.

... and still wouldn't solve one of the most frequently requested things
for pg_dump / pg_restore, which is the ability to use them *server-side*
over a regular PostgreSQL connection. It'd be useful progress toward
that, though.

Right now, we can't even get the PostgreSQL server to emit DDL for a
table, let alone do anything more sophisticated.

Here's how I think it needs to look:

- Design a useful API for pg_dump and pg_restore that is practical to use for pg_dump and pg_restore's current tasks
(fastdatabase dump/restore) and also useful for extracting specific objects from the database. When designing, consider
thatwe'll want to expose this API or functions that use it over SQL later.
 

- Create a new "libpqdump" library.

- Implement the designed API in the new library, moving and adjusting code from pg_dump / pg_restore where possible,
writingnew code where not.
 

- Refactor (closer to rewrite) pg_dump and pg_restore to use libpqdump, removing as much knowledge of the system
catalogsetc as possible from them.
 

- Make sure the result still performs OK

THEN, once that's settled in:

- Modify libpqdump to support compilation as a backend extension, with use of the SPI for queries and use of syscaches
ordirect scans where possible.
 

- Write a "pg_dump" extension that uses libpqdump in SPI mode to expose its API over SQL, or at least uses it to
provideSQL functions to describe database objects. So you can dump a DB, or a subset of it, over SQL.
 

After all, a "libpgdump" won't do much good for the large proportion of
PostgreSQL users who use Java/JDBC, who can't use a native library
(without hideous hacks with JNI). For the very large group who use libpq
via language-specific client interfaces like the Pg gem for Ruby,
psycopg2 for Python, DBD::Pg for Perl, etc, it'll require a lot of work
to wrap the API and maintain it. Wheras a server-side SQL-callable
interface would be useful and immediately usable for all of them.

-- Craig Ringer                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services

Craig Ringer <craig@2ndquadrant.com> writes:
> Here's how I think it needs to look:
> [ move all the functionality to the backend ]

Of course, after you've done all that work, you've got something that is
of exactly zero use to its supposed principal use-case, pg_dump.  pg_dump
will still have to support server versions that predate all these fancy
new dump functions, and that pretty much ensures that most of pg_dump's
core functionality will still be on the client side.  Or, if you try to
finesse that problem by making sure the new server APIs correspond to
easily-identified pieces of pg_dump code, you'll probably end up with APIs
that nobody else wants to use :-(.

In any case, I quite agree with the sentiment that this is not a suitable
problem for a GSOC project.
        regards, tom lane

On 03/21/2014 11:09 AM, Tom Lane wrote:
> Craig Ringer <craig@2ndquadrant.com> writes:
>> Here's how I think it needs to look:
>> [ move all the functionality to the backend ]
> 
> Of course, after you've done all that work, you've got something that is
> of exactly zero use to its supposed principal use-case, pg_dump.  pg_dump
> will still have to support server versions that predate all these fancy
> new dump functions, and that pretty much ensures that most of pg_dump's
> core functionality will still be on the client side.  Or, if you try to
> finesse that problem by making sure the new server APIs correspond to
> easily-identified pieces of pg_dump code, you'll probably end up with APIs
> that nobody else wants to use :-(.

Yeah, that's why it's necessary to create a "libpqdump" that's usable
client-side even if you want server-side dump support.

So it's "allow the functionality to be used from the backend as well",
not just "move all the functionality to the backend".

-- Craig Ringer                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Mar 20, 2014 at 11:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Craig Ringer <craig@2ndquadrant.com> writes:
>> Here's how I think it needs to look:
>> [ move all the functionality to the backend ]
>
> Of course, after you've done all that work, you've got something that is
> of exactly zero use to its supposed principal use-case, pg_dump.  pg_dump
> will still have to support server versions that predate all these fancy
> new dump functions, and that pretty much ensures that most of pg_dump's
> core functionality will still be on the client side.  Or, if you try to
> finesse that problem by making sure the new server APIs correspond to
> easily-identified pieces of pg_dump code, you'll probably end up with APIs
> that nobody else wants to use :-(.

It's worse than that.  If you put all the logic in the server, then a
dump taken on an older version won't be able to quote keywords added
in the newer version.  Go directly to fail.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 03/21/2014 09:38 AM, Robert Haas wrote:
> On Thu, Mar 20, 2014 at 11:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Craig Ringer <craig@2ndquadrant.com> writes:
>>> Here's how I think it needs to look:
>>> [ move all the functionality to the backend ]
>> Of course, after you've done all that work, you've got something that is
>> of exactly zero use to its supposed principal use-case, pg_dump.  pg_dump
>> will still have to support server versions that predate all these fancy
>> new dump functions, and that pretty much ensures that most of pg_dump's
>> core functionality will still be on the client side.  Or, if you try to
>> finesse that problem by making sure the new server APIs correspond to
>> easily-identified pieces of pg_dump code, you'll probably end up with APIs
>> that nobody else wants to use :-(.
> It's worse than that.  If you put all the logic in the server, then a
> dump taken on an older version won't be able to quote keywords added
> in the newer version.  Go directly to fail.
>


Yeah. This tantalizing project has been looked at several times and 
found to be a viper's nest.

What would be useful for many purposes, and is a long-standing project 
of mine that I still haven't found time to make progress on, is that the 
server should contain functions to produce the creation SQL for all its 
own objects, free of the locks that pg_dump requires for consistency.

That would be a great SoC project, incidentally. I'd even volunteer to 
mentor that one.


cheers

andrew

Andrew Dunstan escribió:

> What would be useful for many purposes, and is a long-standing
> project of mine that I still haven't found time to make progress on,
> is that the server should contain functions to produce the creation
> SQL for all its own objects, free of the locks that pg_dump requires
> for consistency.

Maybe you'd like my "DDL deparse" project, then.  Right now it's only of
use for event triggers during DDL execution, but I don't see any strong
reason it cannot be used to reconstruct object creation commands from
only their identifying OID.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services