Re: [HACKERS] PG_UPGRADE status?

Tom Lane wrote:
> Lamar Owen <lamar.owen@wgcr.org> writes:
> > Pipe or file, both versions have to be installed at the same time, so,
> > either way, it's messy.
> 
> Er, no, that's the whole point.  The easy way to attack this is
>         (1) While running old installation, pg_dumpall into a file.
>         (2) Shut down old postmaster, blow away old database files.
>         (3) Install new version, initdb, start new postmaster.
>         (4) Restore from pg_dump output file.

Would to God it were that easy!  During an RPM upgrade, I have to
observer the following:
1.)    The user types rpm -Uvh postgresql*.i386.rpm, or executes an upgrade
from an older RedHat version to a newer RedHat version.

2.)    The first rpm's preinstall script starts running.  The old version
of that rpm is still installed at this point, BUT I CAN'T EXECUTE ANY
DAEMONS -- the upgrade MIGHT be running in the wicked chroot environment
of the RedHat installer, with its restrictive set of commands.  So, I
CANNOT start a postmaster, nor can I be assured that a postmaster is
running -- according to RedHat, since it could be running in the chroot
installer, I can't even run a ps to SEE if postmaster is running
(problems with a chrooted /proc...).  Therefore, the preinstall script
CANNOT execute pg_dumpall.  I can't even run a standalone backend --
postmaster MIGHT be running.... And, I can't test to see if I'm running
in the installer or not... ;-( The only thing I CAN do is check /tmp for
the lock file.

3.)    Once the preinstall script is finished, rpm blows in the first rpm's
files.  This of course overwrites the previous version.

4.)    Once all files are blown in, the postinstall script can run.  It has
the same restrictions that the preinstall script does, since the rpm
COULD be running in the chroot installer.

5.)    Repeat 2-4 for the remainder of the rpms.

If it weren't for the restrictions, it wouldn't be too hard.  I think I
have it mostly solved -- I just have to clean up some code and do
testing.  I'm using a two-stage plan -- the preinstall of the main
package (which only contains clients, client libraries, and
documentation) detects whether an old version of PGDATA is there or
not.  If it is, a backup of the PGDATA tree is performed.  The hard part
there is making sure a backend isn't running -- I haven't figured out
how to reliably detect a running postmaster without /proc or ps.  The
lock file would seem to be a reliable flag -- but, what if the last
invocation of postmaster crashed for some reason, left the lockfile, and
the user, on the next boot, decides to upgrade versions of RedHat....

Stage two is performed in the server package's startup script
(/etc/rc.d/init.d/postgresql) -- it detects the backup, cleans up
PGDATA, initdb's, dumps the data from the old PGDATA (with the old
binaries), and restores the data with the new binaries.

> > convert to the latest and greatest without a backend running?  All of
> > the code to deal with any version is out there in CVS already.
> 
> Go for it ;-).

For some reason, I just KNEW you'd say that :-).  Given six months of
spare time, I probably could.  But, in the meantime, people's databases
are getting farkled by rpm upgrades, so I have to solve the problem.

> > the latest version of a tuple anyway.  Was this the issue with
> > pg_upgrade and MVCC, or am I misunderstanding it?
> 
> The issue with MVCC is that the state of a tuple isn't solely determined
> by what is in the disk file for its table; you have to also consult
> pg_log to see whether recent transactions have been committed or not.
> pg_upgrade doesn't import the old pg_log into the new database (and
> can't very easily, since the new database will have its own), so there's
> a problem with recent tuples possibly getting lost.

The behavior I'm describing for pg_upgrade (let me name my program
something different, for clarity, pg_data_uprev) is to take an old
PGDATA tree, and convert it to new format into a blank, non-initdbed
tree, and get a consistent new format PGDATA tree.  Thus, there are no
existing files at all to worry with.  Visualize a filter -- old-PGDATA
-> pg_data_uprev -> new-PGDATA, with no backends involved at all.

Lamar Owen
WGCR Internet Radio