Packaging automation, testing and work-reduction

Hi all

I'm interested in getting more involved in the RPM packaging process for
PostgreSQL.  I know I'm new to the scene, so the following might be
jumping the gun a bit, but I'm also aware that it's hard to find time
for PGDG RPM work and more contribution might be valued.

I've produced new spec files for PostgreSQL 9.4 that unify all of
EL-5/6/7 and Fedora 19/20/21 into a single spec that can be rebuilt for
each platform, and I think that's ready for wider testing now. Builds
for all platforms work correctly using 'mock' on a Fedora 20 build-host,
and they test-install into the target platform correctly, and
interoperate with existing PGDG RPMs.

I've fixed a number of packaging bugs in the process. I'll attach the
spec and the rest of the RPM sources in a followup mail to the "unifying
the spec files" thread.

I'm looking at future steps now, and I'd like to outline some ideas that
might make PGDG RPMs easier to maintain, more friendly toward
contribution from others, more reliable, easier to test, and more
inter-operable with other repositories.

My personal motivation - just so I'm upfront about this - is that what
I'm talking about would make it a *lot* easier for me to produce
"hotfix" RPMs for 2ndQuadrant customers who want fixes backported before
a new minor release, produce backport RPMs where they want something
backported into 9.2 or whatever, produce RPMs for customers with
perf/bug fixes that haven't been accepted by upstream yet, and make
packaging BDR a lot easier too.

From my understanding of how things currently work I think it'd also
make it easier to maintain PGDG, but of course I don't have much history
here or the experience with the current process to really back that.


I'd like to (short version):

* Depend on EPEL
  (some packages already do, it's just not declared)
* Drop packages from PGDG that're already in EPEL
* Unify spec files for different dists
* Build with Mock
* Use Koji
* Test with Mock
* Let Koji handle createrepo etc
* Move package management to git


Details of what I'm suggesting:

- Depend on EPEL for RHEL/CentOS/SciLinux and remove all packages
  provided by EPEL from PGDG to reduce duplication and potential for
  conflicts. (This might involve applying for EPEL dev status
  eventually, but shouldn't to start with).

  As many packages would work w/o EPEL I doubt it'd have to be a hard
  dependency. Just say that EPEL is recommended and will be required
  for some packages.


- Progressively unify spec files for other packages to remove the need
  for individual specs for each target OS, like I've already done
  for postgresql94. (Done for Pg 9.4, not prior, but see next item).


- For PostgreSQL packages also make the major versions supported in the
  same define, just by changing the macros. So packaging bug fixes are
  easier. (I haven't done this, but it doesn't look too hard).


- Always build packages using mock, so issues with build-depends and
  undeclared runtime dependencies like those I've recently reported are
  identified at build time. (This works with the packages I've
  rewritten).

  Use mock target definition files that include EPEL repositories for
  EL targets.


- Test-install packages in another mock chroot after building, and
  run some basic tests - initdb, etc, at least. Trivial to automate,
  just run a script within the mock chroot. (I've done part of this
  and it's not hard to finish off).


- Automate package building with mock via Koji by setting up a Koji
  server for PGDG. This is fairly well documented, and really just
  needs a host with a pile of disk space, preferably running a new
  Fedora release. Builds can then be submitted to Koji, either as SRPMs
  or via pushes to git. The latter involves tarballs in git, but
  that's what Fedora does.

  Koji can pull the packages from Fedora and EPEL to use as a base
  for builds, but (using Mock) will only install and enable those
  declared for dependencies or build-depends in any individual build.

  I haven't set a Koji up, but I'd very much like to. If PGDG
  wants to go this way I'll be happy to do it for PGDG, rather than
  creating a private Koji.

  Check out:

  http://koji.fedoraproject.org/koji/
  https://fedorahosted.org/koji/

  'mash' is used to manage koji -> yum repo work.


- Let Koji take care of running createrepo, repoview, etc as updates
  are pushed, rather than running the manually;


- Test repositories with repoclosure and complain if we find
  dependencies that can't be satisfied, e.g.

    repoclosure -l fedora -l pgdg94

  or

    repoclosure -l el5 -l epel5 -l pgdg91

  (with a corresponding yum configuration).

  This can be done within 'mock', or with individual yum config files
  rather than using the system ones.



Thoughts?

It probably makes sense to fire up a Koji instance first. Then start
progressively getting packages building with corrected build-depends and
depends, using mock, and where possible unified to build the same spec
on all dists, moving them into Koji-friendly git management as that happens.

As packages are updated, the copy in svn would want to be updated to
match so there's no drift, marking the spec in git as the authoritative one.

Once everything's building happily in Koji its repos could become the
authorative ones.

Crazy talk?
--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services