Обсуждение: Hackathon news

Поиск
Список
Период
Сортировка

Hackathon news

От
Christoph Berg
Дата:
Hi,

before too much time has passed, let's get activity back to this
mailing list.

Dimitri, Magnus and I had some semi-secret meeting at my house, and
worked out the blueprint for getting things flying. I'll just post our
notes and my blog post here as well so people have a chance to catch
up and don't have to dig into various places.

https://github.com/dimitri/apt.postgresql.org/blob/master/hackaton-20120821.md

-----------

# apt.postgresql.org hackathon

The hackathon happened in Credativ place and Christoph Berg and included
Magnus Hagander and Dimitri Fontaine. It took place 2012-08-21 and 22.

That document is both a report of what we managed to do in the hackathon,
including some decisions we took, and a TODO list for later.

## Hackathon goal

We want to build all PostgreSQL stable releases for all supported debian and
ubuntu distributions for i386 and amd64. We already have a Jenkins setup
allowing us to manage a build queue, thanks to Christoph.

## Building pl/sh for squeeze

We need to:

 1. get our own postgresql-common package with the right supported-versions script
 2. change the cowbuilder image source list to inclure our packages
 3. build plsh in the build environment
 4. get plsh packages for squeeze for all those PG versions
 5. publish them in a debian repository in localhost

### postgresql-common

Editing supported-versions is enough, `postgresql-server-dev-all`
dependencies are taken from that at build time.

### apt preferences

We want that the `~pgdg` packages take priority over the debian provided
packages for the same versions, so we need to do some APT pinning magic here
to make that happen, and include that setup in the repository and in the
installation instructions.

Default policy from the repository is a backports like policy.

We also provide a package that install a
`/etc/apt/preferences.d/apt.postgresql.org` file with the right pinning for
a full pgdg experience:

    Package: *
    Pin: origin apt.postgresql.org
    Pin-Priority: 600

### libpq

We ship latest stable PostgreSQL version of libpq in all stable
distributions (that's libpq from 9.1 for squeeze and wheezy), and we ship
the latest-including-beta in sid (libpq from 9.2 nowadays).

We need to build extensions with that libpq version too, of course.

### distributions and release policy

We publish maintained PostgreSQL versions to current debian and ubuntu
releases. We also publish old PostgreSQL versions to current and old debian
and ubuntu versions.

Distributions examples with wheezy:

 - wheezy-pgdg               (only PG supported versions, 8.3 -> 9.1)
 - wheezy-pgdg-testing       (including 9.2)
 - wheezy-pgdg-deprecated    (including 8.1)

Same thing with lenny, squeeze, jessy(?), sid, and ubuntu releases lucid,
precise and some others.

# Problems to solve later

## Automatic build

Several sources are available for the packaging. We can either build from
the current `apt-get source` packaging found in sid or from the source code
repository on `alioth` (svn and git are in use over there).

One idea is to build *testing* packages from their source code repositories
and *production* package directely from the `sid` distribution. That means
we're doing both Quality Assurance and Backports, but that might be a little
too much for the first version of this build system.

## Build Images Optimisation

It should be possible to provide for build images (pbuilder, cowbuilder)
including PostgreSQL build dependancies and our own packages pre-installed.
That means we need to rebuild those image at each PostgreSQL Minor release.

We might need a Jenkins job to do that.

## apt.postgresql.org website

We need some templates and build the website with them, or just use django
here. The main installation instruction should go to the main PostgreSQL
website. We agree that Magnus will do the django parts.

## Where to get the sources from

The current Jenkins install is fetching package sources from `bzr`, `git`
and `svn` repositories, we could do it the *backports* way and fetch debian
sid sources instead with `apt-get source`.

## pg_buildext

`debian/pgversions` should include a `*` facility so that we can build
extensions against new PostgreSQL releases without any editing. Worst case
it doesn't work and we report a bug.

## publishing policy

Generally we want to only include packages that come from Debian unstable,
usually in the version from there. Exceptions should be rare so we don't run
out of sync and lose track.

We need to publish a policy about Software Licences and content (postgresql,
extensions, some special cases for clients, but not any client software). We
only publish backend software with an OSI approved licence.

# Build Environment

## Sources

- PostgreSQL common packaging is on bzr
  http://anonscm.debian.org/bzr/pkg-postgresql/

- Debian Packaging for Extensions is on alioth SVN
  https://alioth.debian.org/scm/browser.php?group_id=30209

- Our own build infrastructure is on github
  https://github.com/dimitri/apt.postgresql.org

## Jenkins

- we want the configuration in git if possible: SCM Sync configuration plugin

-----------

http://www.df7cb.de/blog/2012/PostgreSQL_in_Debian_Hackathon.html

-----------

Almost a year has passed since my talk at pgconf.eu 2011 in Amsterdam on
<a
href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/166-connecting-the-debian-and-postgresql-worlds/">Connecting
theDebian and PostgreSQL worlds</a>, 
and unfortunately little has happened on that front, mostly due to my limited
spare time between family and job.
<a href="http://pgapt.debian.net/">pgapt.debian.net</a> is up and running, but
got few updates and is lagging behind on PostgreSQL releases.

Luckily, we got the project moving. <a href="http://tapoueh.org/blog/index.html">Dimitri Fontaine</a>
and <a href="http://blog.hagander.net/">Magnus Hagander</a> suggested to do a
face-to-face meeting, so we got together at my house for two days last week and
discussed ideas, repository layouts, build scripts, and whatnot to get all of
us aligned for pushing the project ahead. My
<a href="http://www.credativ.de/">employer</a> sponsored my time off work for
that. We almost finished moving the repository to postgresql.org
infrastructure, barring some questions of how to hook the repository into the
existing mirror infrastructure; this should get resolved this week.

The build server running Jenkins is still located on my laptop, but moving this
to a proper host will also happen really soon now. We are using
<a href="http://michael-prokop.at/blog/">Mika Prokop</a>'s
<a href="http://jenkins-debian-glue.org/">jenkins-debian-glue</a> scripts for
driving the package build from Jenkins. The big plus point about Jenkins is
that it makes executing jobs on different distributions and architectures in
parallel much easier than a bunch of homemade shell scripts could get us with
reasonable effort.

Here's a list of random points we discussed:

 * We decided to go for "pgdg" in version numbers and distribution names, i.e.
   packages will have version numbers like 9.1.5-1.pgdg+1, with distributions
   wheezy-pgdg, squeeze-pgdg, and so on.
 * There will be Debian-testing-style distributions called like
   wheezy-pgdg-testing that packages go into for some time before they get
   promoted to the "live" distributions.
 * PostgreSQL versions out of support (8.2 and below) will not be removed from
   the repository, but will be moved to distributions called like
   wheezy-pgdg-deprecated. People will still be able to use them, but the
   naming should make it clear that they should really be upgrading.
 * We have a slightly modified (compared to Debian unstable) postgresql-common
   package that sets the "supported-versions" to all versions supported by the
   PostgreSQL project. That will make the postgresql-server-dev-all package
   pull in build-dependencies for all server versions, and make extension
   module packages compile for all of them automatically. (Provided they are
   using pg_buildext.)
 * There's no Ubuntu support in there yet, but that's mostly only a matter of
   adding more cowbuilder chroots to the build jobs. TBD soon.

We really aim at using unmodified packages from Debian as much as possible, and
in fact this project doesn't mean to replace Debian's PostgreSQL packaging
work, but to extend it beyond the number of server versions (and Debian and
Ubuntu versions covered) supported. The people behind the Debian and Ubuntu
packages, and this repository are mostly the same, so we will claim that "our"
packages will be the same quality as the "original" ones. Big thanks go to
<a href="http://www.piware.de/">Martin Pitt</a> for maintaining the
postgresql-common testsuite that really covers every aspect of running
PostgreSQL servers on Debian/Ubuntu systems.

Stay tuned for updates! :)

[[!tag debian postgresql]]

-----------

There is code for webpages using the postgresql.org template in there,
but I haven't updated any content yet:
https://github.com/dimitri/apt.postgresql.org/tree/master/web
Some of it is already visible at pgapt.debian.net, but that needs more
work. (And that's the old repo location I'm only syncing occasionally
from the new one.)

The jenkins scripts are here:
https://github.com/dimitri/apt.postgresql.org/tree/master/jenkins

generate-pgdg-source has grown the ability to put the various lib
packages from the server sources (libpq5 and friends) into the right
component in the archive, depending on the distribution targeted.

There is a new "pgdg-buildenv" package that will speed up cowbuilder
builds a bit: https://github.com/ChristophBerg/pgdg-buildenv

I'm currently updating all postgresql-x.y packages to use a debian/
directory similar to what 9.1/9.2 have. (Maybe we can even have one
that will work for all?)

The bzr URLs have (mostly) changed to end in ...pgdg instead of the
old ...sid-pgapt.

The Jenkins build host is still running on my notebook, that's one of
the next things I'm going to tackle.

Christoph
--
cb@df7cb.de | http://www.df7cb.de/

Вложения

Re: Hackathon news

От
Martin Pitt
Дата:
Hello Christoph,

thanks for the report!

Christoph Berg [2012-09-06 13:33 +0200]:
> We need to:
>
>  1. get our own postgresql-common package with the right supported-versions script

We already discussed this briefly on IRC. This seems to be the only
delta required, which is both tiny as well as bothersome as it won't
ever go away. The difference between merging to new p-common releases
and just pulling them unmodified into pgapt is considerable, so I
think we should find a way to avoid this permanent delta.

As you already plan to have a separate package which provides the apt
pinning, could we not just use the same package to divert
postgresql-client-common's
/usr/share/postgresql-common/supported-versions script ? Or we change
supported-versions to check if a file or script
/usr/share/postgresql-common/supported-versions.local exists and
return its contents/run it and return its output?

> ### distributions and release policy
>
> We publish maintained PostgreSQL versions to current debian and ubuntu
> releases. We also publish old PostgreSQL versions to current and old debian
> and ubuntu versions.

Do you plan to include running the p-common test suite in Jenkins, to
ensure that there are no mis-builds or other things we haven't thought
of (e. g. a p-common operation that does not work with 8.3)?

> Several sources are available for the packaging. We can either build from
> the current `apt-get source` packaging found in sid or from the source code
> repository on `alioth` (svn and git are in use over there).

The Debian PostgreSQL packages are in bzr. But I'd advise to only
build from released versions. To avoid having to wait for the Debian
archive processing, how about building packages from tags only,
instead of from arbitrary revisions? Then jenkins could watch out for
new tags, and immediately get going when it sees one, instead of
having to wait for half a day for the Debian archive publisher.

> One idea is to build *testing* packages from their source code repositories
> and *production* package directely from the `sid` distribution. That means
> we're doing both Quality Assurance and Backports, but that might be a little
> too much for the first version of this build system.

The testing one is interesting if we could combine it with p-common
test suite runs (but that should happen for production packages as
well). But yes, it's a matter of extra hardware resources.

> ## publishing policy
>
> Generally we want to only include packages that come from Debian unstable,
> usually in the version from there. Exceptions should be rare so we don't run
> out of sync and lose track.

Don't you also want to track experimental? That's where I put all the
betas and RCs, as well as 9.2 right now while we are in release
freeze.

Thanks!

Martin

--
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

Вложения

Re: Hackathon news

От
Christoph Berg
Дата:
Re: Martin Pitt 2012-09-07 <20120907063015.GD2625@piware.de>
> >  1. get our own postgresql-common package with the right supported-versions script
>
> We already discussed this briefly on IRC. This seems to be the only
> delta required, which is both tiny as well as bothersome as it won't
> ever go away. The difference between merging to new p-common releases
> and just pulling them unmodified into pgapt is considerable, so I
> think we should find a way to avoid this permanent delta.
>
> As you already plan to have a separate package which provides the apt
> pinning, could we not just use the same package to divert
> postgresql-client-common's
> /usr/share/postgresql-common/supported-versions script ? Or we change
> supported-versions to check if a file or script
> /usr/share/postgresql-common/supported-versions.local exists and
> return its contents/run it and return its output?

Afaict, there are two places where supported-versions really matters,
that is at p-common build time (to determine which PG version
postgresql.deb should depend on), and extension build time (to
determine which PG versions to build extensions for). The case where
the user is notified that he's running an out-of-date versions is not
so important. I'm not aware of other situations where it is used.

This seems to imply we could just make sure our build environments
have the right s-v script, and maybe even ship an unmodified version
to users, or go via some .local variant.

We need a "pgdg-repository" package on user systems for several
reasons:
* ship the repository key (which I need to refresh soonish, it's going
  to expire in October)
* possibly auto-add some /etc/apt/sources.list.d/pgdg.list file
* add some /etc/apt/preferences.d/pgdg file (these two should probably
  be debconf questions)
* Do the s-v tweaks mentioned above.

I'll try to build some poc for that later.

> > ### distributions and release policy
> >
> > We publish maintained PostgreSQL versions to current debian and ubuntu
> > releases. We also publish old PostgreSQL versions to current and old debian
> > and ubuntu versions.
>
> Do you plan to include running the p-common test suite in Jenkins, to
> ensure that there are no mis-builds or other things we haven't thought
> of (e. g. a p-common operation that does not work with 8.3)?

That's already up and running, the build pipeline in Jenkins is:
* the postgresql-x.y-source job produces postgresql_x.y-pgdgZZ+1.dsc
  where ZZ depends on the distribution (generate-pgdg-source)
* the postgreqsql-x.y-binaries job compiles the sources for all
  distributions and architectures (build-and-provide-package)
* then a common postgresql-testsuite job grabs the binaries from the
  last build and runs the testsuite for every distribution and
  architecture (postgresql-testsuite)
* finally a common dput job uploads everything to the various
  *-pgdg-testing distributions in the archive. (dput-pgdg)

If any of the first three fails, there will be no upload.

The scripts mentioned in parentheses are available at
https://github.com/dimitri/apt.postgresql.org/tree/master/jenkins
https://github.com/mika/jenkins-debian-glue/tree/master/scripts

The Jenkins setup will hopefully be publically visible soonish. I'll
also write some more documentation on how the jobs need to be
configured.

The -source script does essentially only add a changelog entry to get
the version numbers right. For the server packages (it is also used
for extension and general packages), it will also move the lib
packages (libpq5, libpq-dev, ...) to some other component in the
archive, except for the current stable PG version (beta/rc in
unstable).

IOW, this means:

postgresql-9.0's libpq will go to Section: 9.0/libs.
postgresql-9.1's libpq will go to Section: libs (this is implicitely
main/libs) for squeeze and wheezy, and to 9.1/libs for unstable
postgresql-9.2's libpq will to to Section 9.2/libs for squeeze and
wheezy, and to libs for unstable.

That way, the most recent relevant libpq is always available in a
distribution, but users can still install the others if they want to,
then just need to change:

deb http://... main

to

deb http://... main 8.3 8.4 9.2

This tweak was the reason why I put all those "sid-pgapt" branches on
bzr.debian.org, but now this tweak is done on the fly. (Which looks
surprisingly similar to the last patch Martin did on this ppa script.
I didn't look there before, though :)

As said in the last mail, the branches will be renamed to "pgdg" where
they are still useful (9.0 and below, I don't want to change the trunk
branch from the last version found in Debian), the others (9.1 and
9.2) will go away soon.

> > Several sources are available for the packaging. We can either build from
> > the current `apt-get source` packaging found in sid or from the source code
> > repository on `alioth` (svn and git are in use over there).
>
> The Debian PostgreSQL packages are in bzr. But I'd advise to only
> build from released versions. To avoid having to wait for the Debian
> archive processing, how about building packages from tags only,
> instead of from arbitrary revisions? Then jenkins could watch out for
> new tags, and immediately get going when it sees one, instead of
> having to wait for half a day for the Debian archive publisher.

That's the plan, yes. Unfortunately, the Jenkins bzr plugin doesn't
support that, but as builds are triggered manually at the moment, that
will work too.

We also experimented a bit with using "apt-get source" instead of
pulling from some VCS. It should work equally well, except that this
will need some waiting as you said.

> > One idea is to build *testing* packages from their source code repositories
> > and *production* package directely from the `sid` distribution. That means
> > we're doing both Quality Assurance and Backports, but that might be a little
> > too much for the first version of this build system.
>
> The testing one is interesting if we could combine it with p-common
> test suite runs (but that should happen for production packages as
> well). But yes, it's a matter of extra hardware resources.

We were discussing this for quite some time, but the issue isn't
really clear from the notes. The question was: do we pull extension
packages from unstable, and put them in all our distributions, or do
we pull the version from the corresponding Debian distribution? Even
if we were only rebuilding the versions Debian already had way, we
would still provide users more PG versions targetted. For example:

Debian currently has:

sid: plproxy 2.4-1 with binary plproxy-9.1
squeeze plproxy 2.1-1 with binary plproxy-8.4

Variant #1 would be

sid-pgdg plproxy 2.4-1.pgdg+1 with binaries plproxy-{8.3,8.4,9.0,9.1,9.2}
squeeze-pgdg plproxy 2.4-1.pgdg60+1 with binaries plproxy-{8.3,8.4,9.0,9.1}
                       ^

Variant #2 would be

sid-pgdg plproxy 2.4-1.pgdg+1 with binaries plproxy-{8.3,8.4,9.0,9.1,9.2}
squeeze-pgdg plproxy 2.1-1.pgdg60+1 with binaries plproxy-{8.3,8.4,9.0,9.1}
                       ^

Both have their advantages, and we could even do both at the same
time. We then decided that this would be insanely complex to handle
(and to use), and we would do #1 only.

> > ## publishing policy
> >
> > Generally we want to only include packages that come from Debian unstable,
> > usually in the version from there. Exceptions should be rare so we don't run
> > out of sync and lose track.
>
> Don't you also want to track experimental? That's where I put all the
> betas and RCs, as well as 9.2 right now while we are in release
> freeze.

The emphasis was on "from Debian". Experimental is of course also a
good place to take things from, especially during the freeze.

> Thanks!

Thanks for the feedback!

Christoph
--
cb@df7cb.de | http://www.df7cb.de/

Вложения

Re: Hackathon news

От
Martin Pitt
Дата:
Christoph Berg [2012-09-07 17:15 +0200]:
> Afaict, there are two places where supported-versions really matters,
> that is at p-common build time (to determine which PG version
> postgresql.deb should depend on), and extension build time (to
> determine which PG versions to build extensions for). The case where
> the user is notified that he's running an out-of-date versions is not
> so important.

Don't we want to tell people if upstream support for a major version
ceases, and thus apt.p.o. stops distributing packages for that major
version? I think p-common ought to warn about this.

I'm not aware of other situations where it is used.

> > Do you plan to include running the p-common test suite in Jenkins, to
> > ensure that there are no mis-builds or other things we haven't thought
> > of (e. g. a p-common operation that does not work with 8.3)?
>
> That's already up and running,

Splendid!

> The -source script does essentially only add a changelog entry to get
> the version numbers right.

Not sure how much it overlaps, but for backports to my PPA I have this
script:

  http://anonscm.debian.org/loggerhead/pkg-postgresql/postgresql-common/trunk/annotate/head:/debian/backport-ppa

there's a couple of tweaks that one has to do to the packages to make
them build properly in older releases.

> For the server packages (it is also used for extension and general
> packages), it will also move the lib packages (libpq5, libpq-dev,
> ...) to some other component in the archive, except for the current
> stable PG version (beta/rc in unstable).

Yeah, disabling the libraries for older versions is one of the things
my backport-ppa script does; you also need to fiddle with build deps
(at least for the Launchpad builders). But if your setup gets along
with the unmodified packages, so much the better of course.

> This tweak was the reason why I put all those "sid-pgapt" branches on
> bzr.debian.org, but now this tweak is done on the fly. (Which looks
> surprisingly similar to the last patch Martin did on this ppa script.
> I didn't look there before, though :)

Ah, heh :-) Sorry that I didn't point it out before, I tend to forget
about these details once they are properly scripted.

> > The Debian PostgreSQL packages are in bzr. But I'd advise to only
> > build from released versions. To avoid having to wait for the Debian
> > archive processing, how about building packages from tags only,
> > instead of from arbitrary revisions? Then jenkins could watch out for
> > new tags, and immediately get going when it sees one, instead of
> > having to wait for half a day for the Debian archive publisher.
>
> That's the plan, yes. Unfortunately, the Jenkins bzr plugin doesn't
> support that, but as builds are triggered manually at the moment, that
> will work too.

I thought Jenkins supports triggering jobs based on file/URL watches?
So you could put a watch on e. g.

  http://anonscm.debian.org/bzr/bzr/pkg-postgresql/postgresql-9.2/trunk/.bzr/branch/tags

?

> We were discussing this for quite some time, but the issue isn't
> really clear from the notes. The question was: do we pull extension
> packages from unstable, and put them in all our distributions, or do
> we pull the version from the corresponding Debian distribution?

That's a tricky one, and we might need both. E. g. squeeze's version
might support version 8.1 to 8.4, while wheezy's version supports 8.4
to 9.1, or similar. If extensions generally are backwards compatible
as much as we need them, using the unstable packages seems right and
preferrable. We definitively can't use the old packages only, as they
often need porting for newer pg releases.

Thanks,

Martin
--
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

Вложения