Re: Packaging automation, testing and work-reduction

This all sounds exciting, but since Devrim is out, let's table the discussion till he gets back (or at least gets near
akeyboard). 


On Aug 19, 2014, at 1:45 AM, Craig Ringer <craig@2ndQuadrant.com> wrote:

> Hi all
>
> I'm interested in getting more involved in the RPM packaging process for
> PostgreSQL.  I know I'm new to the scene, so the following might be
> jumping the gun a bit, but I'm also aware that it's hard to find time
> for PGDG RPM work and more contribution might be valued.
>
> I've produced new spec files for PostgreSQL 9.4 that unify all of
> EL-5/6/7 and Fedora 19/20/21 into a single spec that can be rebuilt for
> each platform, and I think that's ready for wider testing now. Builds
> for all platforms work correctly using 'mock' on a Fedora 20 build-host,
> and they test-install into the target platform correctly, and
> interoperate with existing PGDG RPMs.
>
> I've fixed a number of packaging bugs in the process. I'll attach the
> spec and the rest of the RPM sources in a followup mail to the "unifying
> the spec files" thread.
>
> I'm looking at future steps now, and I'd like to outline some ideas that
> might make PGDG RPMs easier to maintain, more friendly toward
> contribution from others, more reliable, easier to test, and more
> inter-operable with other repositories.
>
> My personal motivation - just so I'm upfront about this - is that what
> I'm talking about would make it a *lot* easier for me to produce
> "hotfix" RPMs for 2ndQuadrant customers who want fixes backported before
> a new minor release, produce backport RPMs where they want something
> backported into 9.2 or whatever, produce RPMs for customers with
> perf/bug fixes that haven't been accepted by upstream yet, and make
> packaging BDR a lot easier too.
>
> From my understanding of how things currently work I think it'd also
> make it easier to maintain PGDG, but of course I don't have much history
> here or the experience with the current process to really back that.
>
>
> I'd like to (short version):
>
> * Depend on EPEL
>  (some packages already do, it's just not declared)
> * Drop packages from PGDG that're already in EPEL
> * Unify spec files for different dists
> * Build with Mock
> * Use Koji
> * Test with Mock
> * Let Koji handle createrepo etc
> * Move package management to git
>
>
> Details of what I'm suggesting:
>
> - Depend on EPEL for RHEL/CentOS/SciLinux and remove all packages
>  provided by EPEL from PGDG to reduce duplication and potential for
>  conflicts. (This might involve applying for EPEL dev status
>  eventually, but shouldn't to start with).
>
>  As many packages would work w/o EPEL I doubt it'd have to be a hard
>  dependency. Just say that EPEL is recommended and will be required
>  for some packages.
>
>
> - Progressively unify spec files for other packages to remove the need
>  for individual specs for each target OS, like I've already done
>  for postgresql94. (Done for Pg 9.4, not prior, but see next item).
>
>
> - For PostgreSQL packages also make the major versions supported in the
>  same define, just by changing the macros. So packaging bug fixes are
>  easier. (I haven't done this, but it doesn't look too hard).
>
>
> - Always build packages using mock, so issues with build-depends and
>  undeclared runtime dependencies like those I've recently reported are
>  identified at build time. (This works with the packages I've
>  rewritten).
>
>  Use mock target definition files that include EPEL repositories for
>  EL targets.
>
>
> - Test-install packages in another mock chroot after building, and
>  run some basic tests - initdb, etc, at least. Trivial to automate,
>  just run a script within the mock chroot. (I've done part of this
>  and it's not hard to finish off).
>
>
> - Automate package building with mock via Koji by setting up a Koji
>  server for PGDG. This is fairly well documented, and really just
>  needs a host with a pile of disk space, preferably running a new
>  Fedora release. Builds can then be submitted to Koji, either as SRPMs
>  or via pushes to git. The latter involves tarballs in git, but
>  that's what Fedora does.
>
>  Koji can pull the packages from Fedora and EPEL to use as a base
>  for builds, but (using Mock) will only install and enable those
>  declared for dependencies or build-depends in any individual build.
>
>  I haven't set a Koji up, but I'd very much like to. If PGDG
>  wants to go this way I'll be happy to do it for PGDG, rather than
>  creating a private Koji.
>
>  Check out:
>
>  http://koji.fedoraproject.org/koji/
>  https://fedorahosted.org/koji/
>
>  'mash' is used to manage koji -> yum repo work.
>
>
> - Let Koji take care of running createrepo, repoview, etc as updates
>  are pushed, rather than running the manually;
>
>
> - Test repositories with repoclosure and complain if we find
>  dependencies that can't be satisfied, e.g.
>
>    repoclosure -l fedora -l pgdg94
>
>  or
>
>    repoclosure -l el5 -l epel5 -l pgdg91
>
>  (with a corresponding yum configuration).
>
>  This can be done within 'mock', or with individual yum config files
>  rather than using the system ones.
>
>
>
> Thoughts?
>
> It probably makes sense to fire up a Koji instance first. Then start
> progressively getting packages building with corrected build-depends and
> depends, using mock, and where possible unified to build the same spec
> on all dists, moving them into Koji-friendly git management as that happens.
>
> As packages are updated, the copy in svn would want to be updated to
> match so there's no drift, marking the spec in git as the authoritative one.
>
> Once everything's building happily in Koji its repos could become the
> authorative ones.
>
> Crazy talk?
> --
> Craig Ringer                   http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>
>
> --
> Sent via pgsql-pkg-yum mailing list (pgsql-pkg-yum@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-pkg-yum

---
Jeff Frost <jeff@pgexperts.com>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
http://www.pgexperts.com/