Re: (A) native Windows port

On Tuesday 09 July 2002 04:17 pm, Hannu Krosing wrote:
> On Tue, 2002-07-09 at 22:10, Lamar Owen wrote:
> > The pre-upgrade script is run in an environment that isn't robust enough
> > to handle that.  What if you run out of disk space during the dump?

> You can either check beforehand or abort and delete the offending
> dumpfile.

And what if you have enough disk space to do the dump, but then that causes 
the OS upgrade to abort because there wasn't enough space left to finish 
upgrading (larger packages, perhaps)?  The system's hosed, and it's our 
fault.

> > What if a postmaster is running -- and many people stop their postmaster
> > before upgrading their version of PostgreSQL?

> It is quite easy to both check for a running postmaster and start/stop
> one.

Not when there is no ps in your path.  Or pg_ctl for that matter.  Nor is 
there necessarily a /proc tree waiting to be exploited.  We're talking the 
anaconda environment, which is tailored for OS installation and upgrading.  
You cannot start a postmaster; you cannot check to see if one is running -- 
you can't even check to see if you're in the anaconda chroot or not, so that 
you can use more tools if not in the OS installation mode.  Again -- the 
total OS upgrade path is a big part of this scenario, as far as the RPM's are 
concerned.  The Debian package may or may not have as grievous a structure.

The only tool you can really use under the anaconda chroot is busybox, and it 
may not do what you want it to.

> > Besides, at least in the case of the RPM, during OS upgrade time the %pre
> > scriptlet (the one you allude to) isn't running in a system with all the
> > normal tools available.

> I don't think that postmaster needs very many normal tools - it should
> be quite independent, except for compat  libs for larger version
> upgrades

The problem there is that you really have no way to tell the system which sets 
of libraries you want.  More to the point: RPM dependencies cannot take 
conditionals and have no concept of if..then.  Nor can you tell the system to 
_install_ the new postgresql instead of _upgrade_ (incidentally, in the RPM 
context an upgrade is an install of the new version followed by an uninstall 
of the old one -- if the new one overwrote files their traces are just wiped 
from the RPM database, if they weren't overwritten, the files get wiped along 
with their respective database entries).  If I could _force_ no upgrades, it 
would be much easier -- but I can't.  Nor can I be sure the %pre scriptlet 
will be run -- some people are so paranoid that they use rpm -U --no-scripts 
religiously.

Thus, when the old postgresql rpm's database entries (in practice virtually 
every old executable gets overwritten) are removed, its dependency 
information is also removed.  As the install/upgrade path builds a complete 
dependency tree of the final installation as part of the process, it knows 
whether the compat libs are needed or not.  If no other program needs them, 
you don't get them, even if you kept an old backend around that does need 
them.  But you really can't make the -server subpackage Require the compat 
packages, because you don't necessarily know what they will be named, or 
anything else they will provide.  If compat libs are even available for the 
version you're upgrading from.

> > Nor is there a postmaster running.  Due to a largish
> > RAMdisk, a postmaster running might cause all manners of problems.

> I don't know anything about the largish RAMdisk,what I meant was that
> postmaster (a 2.7 MB program with ~4 MB RAM footprint) could include the
> functionality of pg_dump and be runnable in single-user mode for dumping
> old databases.

If a standalone backend could reliably dump the database without needing 
networking and many of the other things we take for granted (the install mode 
is a cut-down single-user mode of sorts, running in a chroot of a sort), then 
it might be worth looking at.

> > And an error in the scriptlet could potentially cause the OS upgrade to
> > abort in midstream -- not a nice thing to do to users, having a package

> But is it not the same with _every_ package ? Is there any actual
> upgrading done in the pre/post scripts or are they generally not to be
> trusted ?

No other package is so *different* to require such a complicated upgrade 
process.  Some packages do more with their scriptlets than others, but no 
package does anything near as complicated as dumping a database.  

> We already do a pretty good job with pg_dump, but I would still not
> trust it to do everything automatically and erase the originals.

And that's a big problem.  We shouldn't have that ambivalence.  IOW, I think 
we need more upgrade testing.  I don't think I've seen a cycle yet that 
didn't have upgrade problems.

> If we start claiming that postgresql can do automatic "binary" upgrades
> there will be much fun with people who have some application that runs
> fine on 7.0.3 but barfs on 7.1.2, even if it is due to stricter
> adherence to SQL99 and the SQL is completely out of control or rpm/apt.

That's just us not being backward compatible.  I'm impacted by those things, 
being that I'm running OpenACS here on 7.2.1, when OACS is optimized for 7.1.  
Certain things are very broken.

> There may be even some lazy people who will think that now is the time
> to auto-upgrade from 6.x ;/

And why not?  If Red Hat Linux can upgrade a whole operating environment from 
version 2.0 all the way up to 7.3 (which they claim), why can't we?  If we 
can just give people the tools to deal with potential problems after the 
upgrade, then I think we can do it.  Such a tool as a old-version dumper 
would be a lifesaver to people, I believe.

> Ok. But would it be impossible to move the old postmaster to some other
> place, or is the environment too fragile even for that ?

That is what I have done in the past -- the old backend got copied over (the 
executable), then a special script was run (after upgrade, manually, by the 
user) that tried to pull a dump using the old backend.  It wasn't reliable.  
The biggest problem is that I have no way of insuring that the old backend's 
dependencies stay satisfied -- 'satisfied' meaning that the old glibc stays 
installed for compatibility.  Glibc, after all, is being upgraded out from 
under us, and I can't stop it or even slow it down.

And this could even be that most pathological of cases, where an a.out based 
system is being upgraded to an elf system without a.out kernel support.  
(point of note: PostgreSQL first appeared in official Red Hat Linux as 
version 6.2.1, released with Red Hat Linux 5.0, which was ELF/glibc 
(contrasted to 3.0.3 which was a.out/libc4 and 4.x which was ELF/libc5) -- 
but I don't know about the Debian situation and its pathology.)

> If we move the old postmaster instead of copying then there will be a
> lot less issues about running out of disk space :)

The disk space issue is with the ASCII dump file itself.  Furthermore, what 
happens if the dumpfile is greater than MAXFILESIZE?  Again, wc isn't in the 
path (because it too is being upgraded out from under us -- nothing is left 
untouched by the upgrade EXCEPT that install image RAMdisk, which has a very 
limited set of tools (and a nonstandard kernel to boot).  Networking might or 
might not be available.  Unix domain sockets might or might not be available.

But the crux is that the OS upgrade environment is designed to do one thing 
and one thing alone -- get the OS installed and/or upgraded.  General purpose 
tools just take up space on the install media, a place where space is at a 
very high premium.

> What we are facing here is a problem similar to trying upgrade all users
> C programs when upgrading gcc. While it would be a good thing, nobody
> actually tries to do it - we require them to have source code and to do
> the "upgrade" manually.

Is that directly comparable?  If you have alot of user functions written in C 
then possibly.  But I'm not interested in pathological cases -- I'm
interested in something that works OK for the majority of users.  As long as 
it works properly for users who aren't sophisticated enough to need the 
pathological cases handled, then it should be available.  Besides, one can 
always dump and restore if one wants to.  And just how well does the 
venerable dump/restore cycle work in the presence of these pathological 
cases?

Red Hat Linux doesn't claim upgradability in the presence of highly 
pathological cases (such as rogue software installed from non-RPM sources, or 
non-Red Hat RPM's installed (particularly Ximian Gnome).  So you have to go 
through a process with that.  But it is something you can recover from after 
the upgrade is complete.  That's what I'm after.  I don't hold out hope for a 
fully automatic upgrade -- it would be nice, but we are too extensible for it 
to be practical.  No -- I want tools to be able to recover my old data 
without the old version backend held-over from the previous install.  And I 
think this is a very resonable expectation.

> That's what I propose - dump all databases in pre-upgrade (if you are
> concerned about disk usage, run it twice, first to | wc and then to a
> file) and try to load in post-upgrade.

The wc utility isn't in the path in an OS install situation.  The df utility 
isn't in the path, either.  You can use python, though. :-)  Not that that 
would be a good thing in this context, however.

> There will still be some things that are be impossible to "upgrade" like
> upgrading a.out "C" functions to elf format backend.

If a user is sophisticated enough to write such, that user is sophisticated 
enough to take responsibility for the upgrade.  I'm not talking about users 
of that level here.  But even then, it would be nice to at least get the data 
back out -- the function can then be rebuilt easily enough from source.

> Perhaps we will be able to detect what we can actually upgrade and bail
> out if we find something unupgradable ?

All is alleviated if I can run a utility after the fact to read in my old 
data, without requiring the old packaged binaries.  I don't have to 
workaround ANYTHING.

Again I say -- would such a data dumper not be useful in cases of system 
catalog corruption that prevents a postmaster from starting?  I'm talking 
about a multipurpose utility here, not just something to make my life as RPM 
maintainer easy.

The pg_fsck program is a good beginning to such a program.
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11