Обсуждение: QSoC proposal: Rewrite pg_dump and pg_restore
Hello!<br /> Here is the text of my proposal which I've applied to GSoC.<br /> (and link <a class="moz-txt-link-freetext" href="https://docs.google.com/document/d/1s-Q4rzEysPxo-dINsk_eKFJOBoVjNYDrQ-Oh75gtYEM/edit?usp=sharing">https://docs.google.com/document/d/1s-Q4rzEysPxo-dINsk_eKFJOBoVjNYDrQ-Oh75gtYEM/edit?usp=sharing</a>)<br />Any suggestions and comments are welcome.<br /><b id="docs-internal-guid-cbd5f555-d7c4-35b8-0764-622e27f77da0" style="font-weight:normal;"><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: center;"><spanstyle="font-size:19px;font-family:'Times New Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">PostgreSQL GSoC2014 proposal</span><br /><span style="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><br /><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Project name</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Rewrite (add)pg_dump and pg_restore utilities as libraries (.so, .dll & .dylib)</span><br /><span style="font-size:11px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><br /><spanstyle="font-size:11px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Short description</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">pg_dump isa utility for backing up a PostgreSQL database. It makes consistent backups even if the database is being used concurrently.pg_restore is a utility for restoring a PostgreSQL database from an archive created by pg_dump in one of thenon-plain-text formats. I think it will be more comforatable to use these tools as libraries.</span><br /><span style="font-size:11px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Name</span><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">: AlexanderShvidchenko</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">E-mail</span><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">: </span><ahref="mailto:askellio@gmail.com" style="text-decoration:none;"><span style="font-size:16px;font-family:'Times New Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">askellio@gmail.com</span></a><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Location</span><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">: Rostov-on-Don,Russia (UTC +04.00)</span><br /><span style="font-size:11px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Benefits tothe PostgreSQL Community</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">This featurewill expand opportunities to work with backups of databases. Especially, it’s important for third-party developers.It will be easier to use the functionality of the tools in applications when they will be libraries.</span><br/><span style="font-size:11px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Quantifiable results</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Static anddynamic libraries with the functionality of pg_dump and pg_restore tools (.so, .dll and .dylib files)</span><br /><spanstyle="font-size:11px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Project Schedule</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">until May31 </span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Make codereview and solve architecture questions with help of community</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1 June– 30 June</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Detailed implementationof libraries.</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1 July– 31 July</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Finish Implementationof libraires and begin testing. </span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1 August-15 August</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Final refactoring,testing and commit.</span><br /><span style="font-size:11px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Academic experience</span><pdir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">I enteredthe university in 2013. Before entering the university I finished the college in 2012. My graduate work in the collegewas the client-server application. It was a realization of XMPP. The client was realized in Qt. The client workedwith SQLite database and the server worked with MySQL database.</span><br /><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Why isPostgreSQL?</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><spanstyle="font-size:16px;font-family:'Times New Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">- </span><spanstyle="font-size:16px;font-family:'Times New Roman';color:#444444;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">I’m interetedin this idea and believe this project would be useful for the community;</span><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">- PostgreSQLis a very respected community. I would be proud to be a part of it;</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">- PostgreSQLis one of the best DBMS and I would like to make it better.</span><br /><span style="font-size:11px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Links</span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">1) PostgreSQL9.3.3 Documentation, pg_dump</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><a href="http://www.postgresql.org/docs/9.3/static/app-pgdump.html"style="text-decoration:none;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://www.postgresql.org/docs/9.3/static/app-pgdump.html</span></a><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">2) PostgreSQL9.3.3 Documentation, pg_restore</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><a href="http://www.postgresql.org/docs/9.3/static/app-pgrestore.html"style="text-decoration:none;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://www.postgresql.org/docs/9.3/static/app-pgrestore.html</span></a><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">3) Staticlibrary</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><ahref="http://en.wikipedia.org/wiki/Static_library" style="text-decoration:none;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://en.wikipedia.org/wiki/Static_library</span></a><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span><p dir="ltr"style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">4) Dynamiclibrary</span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align: justify;"><ahref="http://en.wikipedia.org/wiki/Dynamic-link_library" style="text-decoration:none;"><span style="font-size:16px;font-family:'TimesNew Roman';color:#1155cc;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap;">http://en.wikipedia.org/wiki/Dynamic-link_library</span></a><span style="font-size:16px;font-family:'TimesNew Roman';color:#000000;background-color:#ffffff;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span></b><br class="Apple-interchange-newline"/><br /> With best wishes,<br /> Alexander S.<br />
On Tue, Mar 18, 2014 at 8:41 PM, Alexandr <askellio@gmail.com> wrote: > Rewrite (add) pg_dump and pg_restore utilities as libraries (.so, .dll & > .dylib) This strikes me as (1) pretty vague and (2) probably too hard for a summer project. I mean, getting the existing binaries to build libraries that you can call with some trivial interface that mimics the existing command-line functionality of pg_dump might be doable, but that's not all that interesting. What people are really going to want is a library with a sophisticated API that lets you do interesting things programmatically. But that's going to be hard. AFAIK, nobody's even tried to figure out what that API should look like. Even if we had that worked out, a non-trivial task, the pg_dump source code is a mess, so refactoring it to provide such an API is likely to be a job and a half. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 03/21/2014 09:28 AM, Robert Haas wrote: > On Tue, Mar 18, 2014 at 8:41 PM, Alexandr <askellio@gmail.com> wrote: >> Rewrite (add) pg_dump and pg_restore utilities as libraries (.so, .dll & >> .dylib) > > This strikes me as (1) pretty vague and (2) probably too hard for a > summer project. > > I mean, getting the existing binaries to build libraries that you can > call with some trivial interface that mimics the existing command-line > functionality of pg_dump might be doable, but that's not all that > interesting. What people are really going to want is a library with a > sophisticated API that lets you do interesting things > programmatically. But that's going to be hard. AFAIK, nobody's even > tried to figure out what that API should look like. Even if we had > that worked out, a non-trivial task, the pg_dump source code is a > mess, so refactoring it to provide such an API is likely to be a job > and a half. ... and still wouldn't solve one of the most frequently requested things for pg_dump / pg_restore, which is the ability to use them *server-side* over a regular PostgreSQL connection. It'd be useful progress toward that, though. Right now, we can't even get the PostgreSQL server to emit DDL for a table, let alone do anything more sophisticated. Here's how I think it needs to look: - Design a useful API for pg_dump and pg_restore that is practical to use for pg_dump and pg_restore's current tasks (fastdatabase dump/restore) and also useful for extracting specific objects from the database. When designing, consider thatwe'll want to expose this API or functions that use it over SQL later. - Create a new "libpqdump" library. - Implement the designed API in the new library, moving and adjusting code from pg_dump / pg_restore where possible, writingnew code where not. - Refactor (closer to rewrite) pg_dump and pg_restore to use libpqdump, removing as much knowledge of the system catalogsetc as possible from them. - Make sure the result still performs OK THEN, once that's settled in: - Modify libpqdump to support compilation as a backend extension, with use of the SPI for queries and use of syscaches ordirect scans where possible. - Write a "pg_dump" extension that uses libpqdump in SPI mode to expose its API over SQL, or at least uses it to provideSQL functions to describe database objects. So you can dump a DB, or a subset of it, over SQL. After all, a "libpgdump" won't do much good for the large proportion of PostgreSQL users who use Java/JDBC, who can't use a native library (without hideous hacks with JNI). For the very large group who use libpq via language-specific client interfaces like the Pg gem for Ruby, psycopg2 for Python, DBD::Pg for Perl, etc, it'll require a lot of work to wrap the API and maintain it. Wheras a server-side SQL-callable interface would be useful and immediately usable for all of them. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Craig Ringer <craig@2ndquadrant.com> writes: > Here's how I think it needs to look: > [ move all the functionality to the backend ] Of course, after you've done all that work, you've got something that is of exactly zero use to its supposed principal use-case, pg_dump. pg_dump will still have to support server versions that predate all these fancy new dump functions, and that pretty much ensures that most of pg_dump's core functionality will still be on the client side. Or, if you try to finesse that problem by making sure the new server APIs correspond to easily-identified pieces of pg_dump code, you'll probably end up with APIs that nobody else wants to use :-(. In any case, I quite agree with the sentiment that this is not a suitable problem for a GSOC project. regards, tom lane
On 03/21/2014 11:09 AM, Tom Lane wrote: > Craig Ringer <craig@2ndquadrant.com> writes: >> Here's how I think it needs to look: >> [ move all the functionality to the backend ] > > Of course, after you've done all that work, you've got something that is > of exactly zero use to its supposed principal use-case, pg_dump. pg_dump > will still have to support server versions that predate all these fancy > new dump functions, and that pretty much ensures that most of pg_dump's > core functionality will still be on the client side. Or, if you try to > finesse that problem by making sure the new server APIs correspond to > easily-identified pieces of pg_dump code, you'll probably end up with APIs > that nobody else wants to use :-(. Yeah, that's why it's necessary to create a "libpqdump" that's usable client-side even if you want server-side dump support. So it's "allow the functionality to be used from the backend as well", not just "move all the functionality to the backend". -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Fri, Mar 21, 2014 at 4:09 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> [ move all the functionality to the backend ]
Of course, after you've done all that work, you've got something that is
of exactly zero use to its supposed principal use-case, pg_dump. pg_dump
will still have to support server versions that predate all these fancy
new dump functions, and that pretty much ensures that most of pg_dump's
core functionality will still be on the client side. Or, if you try to
finesse that problem by making sure the new server APIs correspond to
easily-identified pieces of pg_dump code, you'll probably end up with APIs
that nobody else wants to use :-(.
Or you should mandate that new server versions should be able to consume _old_ pg_dump version output. This would change the recommended "when upgrading, dump using the new pg_dump" to ""when upgrading, dump using the old pg_dump".
This would be necessary policy going forward anyway, if most of the pg_dump functionality was server-side, because it would be generating dumps in the server-version dump format, not the client-version format.
'Regards
Marcin Mańk
(goes back to lurker cave...)
On Thu, Mar 20, 2014 at 11:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Craig Ringer <craig@2ndquadrant.com> writes: >> Here's how I think it needs to look: >> [ move all the functionality to the backend ] > > Of course, after you've done all that work, you've got something that is > of exactly zero use to its supposed principal use-case, pg_dump. pg_dump > will still have to support server versions that predate all these fancy > new dump functions, and that pretty much ensures that most of pg_dump's > core functionality will still be on the client side. Or, if you try to > finesse that problem by making sure the new server APIs correspond to > easily-identified pieces of pg_dump code, you'll probably end up with APIs > that nobody else wants to use :-(. It's worse than that. If you put all the logic in the server, then a dump taken on an older version won't be able to quote keywords added in the newer version. Go directly to fail. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 03/21/2014 09:38 AM, Robert Haas wrote: > On Thu, Mar 20, 2014 at 11:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Craig Ringer <craig@2ndquadrant.com> writes: >>> Here's how I think it needs to look: >>> [ move all the functionality to the backend ] >> Of course, after you've done all that work, you've got something that is >> of exactly zero use to its supposed principal use-case, pg_dump. pg_dump >> will still have to support server versions that predate all these fancy >> new dump functions, and that pretty much ensures that most of pg_dump's >> core functionality will still be on the client side. Or, if you try to >> finesse that problem by making sure the new server APIs correspond to >> easily-identified pieces of pg_dump code, you'll probably end up with APIs >> that nobody else wants to use :-(. > It's worse than that. If you put all the logic in the server, then a > dump taken on an older version won't be able to quote keywords added > in the newer version. Go directly to fail. > Yeah. This tantalizing project has been looked at several times and found to be a viper's nest. What would be useful for many purposes, and is a long-standing project of mine that I still haven't found time to make progress on, is that the server should contain functions to produce the creation SQL for all its own objects, free of the locks that pg_dump requires for consistency. That would be a great SoC project, incidentally. I'd even volunteer to mentor that one. cheers andrew
Andrew Dunstan escribió: > What would be useful for many purposes, and is a long-standing > project of mine that I still haven't found time to make progress on, > is that the server should contain functions to produce the creation > SQL for all its own objects, free of the locks that pg_dump requires > for consistency. Maybe you'd like my "DDL deparse" project, then. Right now it's only of use for event triggers during DDL execution, but I don't see any strong reason it cannot be used to reconstruct object creation commands from only their identifying OID. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services