Обсуждение: Complicated re-distribution of pgjdbc the "open source way"

Hi all,

--
I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
opt-out if you don't care and sorry for rush).  I just want to see whether
we in Fedora are thinking a constructive way.  This is not GNU/Linux
oriented effort, but it is rather about any open source
packagers/distributors, feel free to add anybody who might be interested
in the loop.
--

_Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
several releases, mostly because there are some non-free/windows-only
related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
disallow us to build pgjdbc on free distro.

The preferred way would be to solve this upstream (making the dependencies
optional), but it is not a mandate of pgjdbc upstream to cooperate on this
-- even patches from us to support pure open source build are not wanted.
As upstream is not interested in non-maven builds, it will be most
probably even worse later.

We've done some small observation around GNU/Linux packages, and it seems
we all reinvent the very similar patches or hacks over and over again.

Because PostgreSQL connector is important part of operating system, we are
thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
should allow us to solve the issue rather sooner than later.

That project idea:

  * we should provide an _easy to use_ (documented how to build from
    source) version-ed tarball, compatible with pgjdbc

  * this tarball would be FOSS source-only, with FOSS dependencies,
    (non-free deps could be possible in future, but only as opt-in
    feature)

  * the build would be 1-step process (no need to build pgjdbc-parent-poms
    first, and others), with some obvious system dependencies

  * that tarball would allow us to 100% build-from-source _without_ tweaks

  * build from this tarball must not rely on maven repositories --
    untrusted content at distribution level

  * the testsuite should be fixed to allow us to run it easily under
    non-root user, on a local/cloud build-box

Would you be interested in having one common code-base for
open-source-distribution-model of pgjdbc, and optionally (preferably)
cooperate?  That source should be as close as possible to pgjdbc, just
limited limited set of patches to allow us to build/test/distribute
correctly and what is more important we could do the job _consistently_
with a lot _less_ packagers effort.

Just let me know if this is good/bad idea from your packackar's POV.  Some
links for discussion with upstream about issues [1,..N].

¹ By that I mean ability to build from FOSS source, _against_
  FOSS source dependencies.  By FOSS source I mean software which
  _anybody_ can read, study, copy, modify, distribute.

[1] http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
[2] http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
[3] http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
[4] http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com

Pavel

Hi Dave,

On Monday 07 of March 2016 09:31:10 Dave Cramer wrote:
> There is a far simpler way to do this.
>
> Simply create a Makefile, or ant build.xml, which I imagine you will have
> to do anyway, and do not include the bits of the code you don't want.
>
> I'd be glad to include it in the source with the assumption that you
> maintain it.

* we were rather thinking about keeping the pom.xml as original as
  possible, Makefile would be additional divergence and work when pom.xml
  works fine

* the process requires patching out some code using not-acceptable
  dependencies, as this is the cheapest way to deal with most of the
  issues for us downstream -- and we already do it to some extent

* having parent-poms packaged separately just complicates the build from
  source, so while we are on it in separate fork, we could merge it into
  pgjdbc-foss

Makefile or ant file would probably not lower the burden of
re-distribution and build from source.  Any thoughts?  But yes, we can
help with maintenance.

Pavel

> Dave Cramer
>
> davec@postgresintl.com
> www.postgresintl.com
>
> On 7 March 2016 at 09:26, Pavel Raiskup <praiskup@redhat.com> wrote:
>
> > Hi all,
> >
> > --
> > I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
> > opt-out if you don't care and sorry for rush).  I just want to see whether
> > we in Fedora are thinking a constructive way.  This is not GNU/Linux
> > oriented effort, but it is rather about any open source
> > packagers/distributors, feel free to add anybody who might be interested
> > in the loop.
> > --
> >
> > _Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
> > several releases, mostly because there are some non-free/windows-only
> > related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
> > disallow us to build pgjdbc on free distro.
> >
> > The preferred way would be to solve this upstream (making the dependencies
> > optional), but it is not a mandate of pgjdbc upstream to cooperate on this
> > -- even patches from us to support pure open source build are not wanted.
> > As upstream is not interested in non-maven builds, it will be most
> > probably even worse later.
> >
> > We've done some small observation around GNU/Linux packages, and it seems
> > we all reinvent the very similar patches or hacks over and over again.
> >
> > Because PostgreSQL connector is important part of operating system, we are
> > thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
> > should allow us to solve the issue rather sooner than later.
> >
> > That project idea:
> >
> >   * we should provide an _easy to use_ (documented how to build from
> >     source) version-ed tarball, compatible with pgjdbc
> >
> >   * this tarball would be FOSS source-only, with FOSS dependencies,
> >     (non-free deps could be possible in future, but only as opt-in
> >     feature)
> >
> >   * the build would be 1-step process (no need to build pgjdbc-parent-poms
> >     first, and others), with some obvious system dependencies
> >
> >   * that tarball would allow us to 100% build-from-source _without_ tweaks
> >
> >   * build from this tarball must not rely on maven repositories --
> >     untrusted content at distribution level
> >
> >   * the testsuite should be fixed to allow us to run it easily under
> >     non-root user, on a local/cloud build-box
> >
> > Would you be interested in having one common code-base for
> > open-source-distribution-model of pgjdbc, and optionally (preferably)
> > cooperate?  That source should be as close as possible to pgjdbc, just
> > limited limited set of patches to allow us to build/test/distribute
> > correctly and what is more important we could do the job _consistently_
> > with a lot _less_ packagers effort.
> >
> > Just let me know if this is good/bad idea from your packackar's POV.  Some
> > links for discussion with upstream about issues [1,..N].
> >
> > ¹ By that I mean ability to build from FOSS source, _against_
> >   FOSS source dependencies.  By FOSS source I mean software which
> >   _anybody_ can read, study, copy, modify, distribute.
> >
> > [1]
> > http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
> > [2]
> > http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
> > [3]
> > http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
> > [4]
> > http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com
> >
> > Pavel
> >
> >
> >
> > --
> > Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-jdbc
> >

Pavel>what is more important we could do the job _consistently_
Pavel>with a lot _less_ packagers effort.

I think "testing" is the key answer here to get consistent results.

Pavel>* the build would be 1-step process
Pavel> Any thoughts?

This all reminds me the 14 competing standards (see [1])
Do you mean "1-step" as in "1-cpu-instruction"?
Is "./build.sh" a "1-step"?

Pavel> even patches from us to support pure open source build are not wanted

I'm afraid this ^^ is misleading.

Patches are welcome provided they include tests to cover the change.
No tests -> no acceptance. It is in line with typical development
model, isn't it?

Pavel>_Open_ distribution¹
Pavel> By FOSS source I mean software which
Pavel>  _anybody_ can read, study, copy, modify, distribute

Can you tell us if org.osgi.enterprise complies with your definition
of "open distribution"?

Just in case you say "no", see [2] (download jar and note that sources
are in OSGI-OPT folder) for complete source code that is available
under Apache 2.0 license -> you can read, copy, modify and distribute
it with no problem.
Taking that into account, why do you think "org.osgi.enterprise" is
not "open distribution"?

1: https://xkcd.com/927/
2: http://mvnrepository.com/artifact/org.osgi/org.osgi.enterprise/5.0.0

Vladimir

Dave>Yes well you are only building one version. We have to build 3!

Just in case: "_Open_ distribution¹ of pgjdbc" (tm) might also want to
include all three -jre6, -jre7, -jre8 builds or at least two of them.
There are pgjdbc users that have no java 8 installed => they might
want to use pgjdbc-jre7 => package for pgjdbc-jre7 might be required.

I'm just highlighting a risk of packaging a single pgjdbc-jre8-only build.

Vladimir

Hi Dave,

On Monday 07 of March 2016 09:31:10 Dave Cramer wrote:
> There is a far simpler way to do this.
>
> Simply create a Makefile, or ant build.xml, which I imagine you will have
> to do anyway, and do not include the bits of the code you don't want.
>
> I'd be glad to include it in the source with the assumption that you
> maintain it.

* we were rather thinking about keeping the pom.xml as original as
  possible, Makefile would be additional divergence and work when pom.xml
  works fine

* the process requires patching out some code using not-acceptable
  dependencies, as this is the cheapest way to deal with most of the
  issues for us downstream -- and we already do it to some extent

* having parent-poms packaged separately just complicates the build from
  source, so while we are on it in separate fork, we could merge it into
  pgjdbc-foss

Makefile or ant file would probably not lower the burden of
re-distribution and build from source.  Any thoughts?  But yes, we can
help with maintenance.

Pavel

> Dave Cramer
>
> davec@postgresintl.com
> www.postgresintl.com
>
> On 7 March 2016 at 09:26, Pavel Raiskup <praiskup@redhat.com> wrote:
>
> > Hi all,
> >
> > --
> > I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
> > opt-out if you don't care and sorry for rush).  I just want to see whether
> > we in Fedora are thinking a constructive way.  This is not GNU/Linux
> > oriented effort, but it is rather about any open source
> > packagers/distributors, feel free to add anybody who might be interested
> > in the loop.
> > --
> >
> > _Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
> > several releases, mostly because there are some non-free/windows-only
> > related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
> > disallow us to build pgjdbc on free distro.
> >
> > The preferred way would be to solve this upstream (making the dependencies
> > optional), but it is not a mandate of pgjdbc upstream to cooperate on this
> > -- even patches from us to support pure open source build are not wanted.
> > As upstream is not interested in non-maven builds, it will be most
> > probably even worse later.
> >
> > We've done some small observation around GNU/Linux packages, and it seems
> > we all reinvent the very similar patches or hacks over and over again.
> >
> > Because PostgreSQL connector is important part of operating system, we are
> > thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
> > should allow us to solve the issue rather sooner than later.
> >
> > That project idea:
> >
> >   * we should provide an _easy to use_ (documented how to build from
> >     source) version-ed tarball, compatible with pgjdbc
> >
> >   * this tarball would be FOSS source-only, with FOSS dependencies,
> >     (non-free deps could be possible in future, but only as opt-in
> >     feature)
> >
> >   * the build would be 1-step process (no need to build pgjdbc-parent-poms
> >     first, and others), with some obvious system dependencies
> >
> >   * that tarball would allow us to 100% build-from-source _without_ tweaks
> >
> >   * build from this tarball must not rely on maven repositories --
> >     untrusted content at distribution level
> >
> >   * the testsuite should be fixed to allow us to run it easily under
> >     non-root user, on a local/cloud build-box
> >
> > Would you be interested in having one common code-base for
> > open-source-distribution-model of pgjdbc, and optionally (preferably)
> > cooperate?  That source should be as close as possible to pgjdbc, just
> > limited limited set of patches to allow us to build/test/distribute
> > correctly and what is more important we could do the job _consistently_
> > with a lot _less_ packagers effort.
> >
> > Just let me know if this is good/bad idea from your packackar's POV.  Some
> > links for discussion with upstream about issues [1,..N].
> >
> > ¹ By that I mean ability to build from FOSS source, _against_
> >   FOSS source dependencies.  By FOSS source I mean software which
> >   _anybody_ can read, study, copy, modify, distribute.
> >
> > [1]
> > http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
> > [2]
> > http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
> > [3]
> > http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
> > [4]
> > http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com
> >
> > Pavel
> >
> >
> >
> > --
> > Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-jdbc
> >

Pavel>what is more important we could do the job _consistently_
Pavel>with a lot _less_ packagers effort.

I think "testing" is the key answer here to get consistent results.

Pavel>* the build would be 1-step process
Pavel> Any thoughts?

This all reminds me the 14 competing standards (see [1])
Do you mean "1-step" as in "1-cpu-instruction"?
Is "./build.sh" a "1-step"?

Pavel> even patches from us to support pure open source build are not wanted

I'm afraid this ^^ is misleading.

Patches are welcome provided they include tests to cover the change.
No tests -> no acceptance. It is in line with typical development
model, isn't it?

Pavel>_Open_ distribution¹
Pavel> By FOSS source I mean software which
Pavel>  _anybody_ can read, study, copy, modify, distribute

Can you tell us if org.osgi.enterprise complies with your definition
of "open distribution"?

Just in case you say "no", see [2] (download jar and note that sources
are in OSGI-OPT folder) for complete source code that is available
under Apache 2.0 license -> you can read, copy, modify and distribute
it with no problem.
Taking that into account, why do you think "org.osgi.enterprise" is
not "open distribution"?

1: https://xkcd.com/927/
2: http://mvnrepository.com/artifact/org.osgi/org.osgi.enterprise/5.0.0

Vladimir

Dave>Yes well you are only building one version. We have to build 3!

Just in case: "_Open_ distribution¹ of pgjdbc" (tm) might also want to
include all three -jre6, -jre7, -jre8 builds or at least two of them.
There are pgjdbc users that have no java 8 installed => they might
want to use pgjdbc-jre7 => package for pgjdbc-jre7 might be required.

I'm just highlighting a risk of packaging a single pgjdbc-jre8-only build.

Vladimir

     Hi Pavel.

     As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
java software that relies on pgjdbc. Since we saw that this issue was
not moving forward, we investigated the issue on our own.

     While as of today the current ToroDB spec that we have is depending
on an older version of JDBC --which is highly undesirable-- we have also
built a simple mean of packaging pgjdbc for Fedora without having to
modify upstream.

     Since RPM packages allow (and some of them heavily use) patches
against upstream, this is easy to accomplish. Having a parent pom is
neither a problem --it is packaged as a subpackage (might not be ideal,
but hey, it works, and it doesn't harm). All in all, with all the
obvious advantages and disadvantages, we have working source code.
Rather than a will to patch upstream or fork pgjdbc, we have a working
spec that packages pgjdbc, removing (by patching) all the non-foss and
windows dependencies and code, and packaging the parent pom too.

     I believe iterating over this effort is better than either forking
or patching upstream.

     I don't have the code, but Matteo (cc'ed), who did this work, will
share it tomorrow.

     I'd suggest to work on this, which seems to me as a compromise
solution, and agree all on this solution, rather than plan longer-term
plans that, while ideal for some, aren't for the rest and, and above
all, will take significantly more effort. Our main goal is to deliver
packaged and foss pgjdbc to the users asap, and here we have an already
working solution.

     Matteo, please share the code whenever possible.

     Cheers,

     Álvaro

--
Álvaro Hernández Tortosa


-----------
8Kdata




On 07/03/16 15:26, Pavel Raiskup wrote:
> Hi all,
>
> --
> I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
> opt-out if you don't care and sorry for rush).  I just want to see whether
> we in Fedora are thinking a constructive way.  This is not GNU/Linux
> oriented effort, but it is rather about any open source
> packagers/distributors, feel free to add anybody who might be interested
> in the loop.
> --
>
> _Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
> several releases, mostly because there are some non-free/windows-only
> related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
> disallow us to build pgjdbc on free distro.
>
> The preferred way would be to solve this upstream (making the dependencies
> optional), but it is not a mandate of pgjdbc upstream to cooperate on this
> -- even patches from us to support pure open source build are not wanted.
> As upstream is not interested in non-maven builds, it will be most
> probably even worse later.
>
> We've done some small observation around GNU/Linux packages, and it seems
> we all reinvent the very similar patches or hacks over and over again.
>
> Because PostgreSQL connector is important part of operating system, we are
> thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
> should allow us to solve the issue rather sooner than later.
>
> That project idea:
>
>    * we should provide an _easy to use_ (documented how to build from
>      source) version-ed tarball, compatible with pgjdbc
>
>    * this tarball would be FOSS source-only, with FOSS dependencies,
>      (non-free deps could be possible in future, but only as opt-in
>      feature)
>
>    * the build would be 1-step process (no need to build pgjdbc-parent-poms
>      first, and others), with some obvious system dependencies
>
>    * that tarball would allow us to 100% build-from-source _without_ tweaks
>
>    * build from this tarball must not rely on maven repositories --
>      untrusted content at distribution level
>
>    * the testsuite should be fixed to allow us to run it easily under
>      non-root user, on a local/cloud build-box
>
> Would you be interested in having one common code-base for
> open-source-distribution-model of pgjdbc, and optionally (preferably)
> cooperate?  That source should be as close as possible to pgjdbc, just
> limited limited set of patches to allow us to build/test/distribute
> correctly and what is more important we could do the job _consistently_
> with a lot _less_ packagers effort.
>
> Just let me know if this is good/bad idea from your packackar's POV.  Some
> links for discussion with upstream about issues [1,..N].
>
> ¹ By that I mean ability to build from FOSS source, _against_
>    FOSS source dependencies.  By FOSS source I mean software which
>    _anybody_ can read, study, copy, modify, distribute.
>
> [1] http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
> [2] http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
> [3] http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
> [4] http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com
>
> Pavel
>
>
>

Hola Álvaro,

Just some observations. We've been there and done that with upstream.
Maintaining a package with patches this way can easily get to be a big
burden. At some point it becomes a habit, and the big bad surprise comes
when you want to upgrade after a period of lazyness..
I do not want to have a stand on the underlying conflict. Maybe the
healthy github pulse and the quick, accepting treatment of #252 are just
misleading me... 3:)
_If you do think that there will be things which should be permanently
patched_ , I would recommend the following to keep the gap (hence
maintenance burden) as small as possible:
- Use a fork on github.
- Have integration branches for all important upstream branches, where a
CI job continously merges upstream and tests whether it is still working.
- Open a pull request in upstream for every commit you have. Give them a
chance to sync up.
- Change only what is absolutely necessary, including not just code, but
also development practices and standards.

Side note: My understanding of maven is that you can exactly control
your direct dependencies in the pom, and you can have a list including
transitives with the dependency plugin. You can check and abort build if
an unexpected dependency comes up. So no need to have an inferior build
system just because dependencies.


On 03/07/2016 10:21 PM, Álvaro Hernández Tortosa wrote:
>
>     Hi Pavel.
>
>     As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
> java software that relies on pgjdbc. Since we saw that this issue was
> not moving forward, we investigated the issue on our own.
>
>     While as of today the current ToroDB spec that we have is
> depending on an older version of JDBC --which is highly undesirable--
> we have also built a simple mean of packaging pgjdbc for Fedora
> without having to modify upstream.
>
>     Since RPM packages allow (and some of them heavily use) patches
> against upstream, this is easy to accomplish. Having a parent pom is
> neither a problem --it is packaged as a subpackage (might not be
> ideal, but hey, it works, and it doesn't harm). All in all, with all
> the obvious advantages and disadvantages, we have working source code.
> Rather than a will to patch upstream or fork pgjdbc, we have a working
> spec that packages pgjdbc, removing (by patching) all the non-foss and
> windows dependencies and code, and packaging the parent pom too.
>
>     I believe iterating over this effort is better than either forking
> or patching upstream.
>
>     I don't have the code, but Matteo (cc'ed), who did this work, will
> share it tomorrow.
>
>     I'd suggest to work on this, which seems to me as a compromise
> solution, and agree all on this solution, rather than plan longer-term
> plans that, while ideal for some, aren't for the rest and, and above
> all, will take significantly more effort. Our main goal is to deliver
> packaged and foss pgjdbc to the users asap, and here we have an
> already working solution.
>
>     Matteo, please share the code whenever possible.
>
>     Cheers,
>
>     Álvaro
>

On 08/03/16 00:49, Árpád Magosányi wrote:
> Hola Álvaro,

     Hi Árpád, thanks for your Spanish salutation :)
>
> Just some observations. We've been there and done that with upstream.
> Maintaining a package with patches this way can easily get to be a big
> burden. At some point it becomes a habit, and the big bad surprise comes
> when you want to upgrade after a period of lazyness..

     That's surely true. However, the suggested patch to maintain
(please see the code when submitted here) is minimal. Indeed, it's not
code per se. It's just removing some sections of the code -and their
dependencies- that while useful upstream do not make sense in Fedora
(like a Windows subsystem).

     Indeed, it makes sense packager's patch mechanism. After all, you
may not need stuff that upstream *does* want to have (like, again,
Windows stuff). But obviously, upstream cannot be forced to dump that stuff.

     I doubt maintaining this kind of patches should become a burden.
They are not growing and/or diverging patches, but rather static and
straightforward to adapt if upstream modifies related files.

> I do not want to have a stand on the underlying conflict. Maybe the
> healthy github pulse and the quick, accepting treatment of #252 are just
> misleading me... 3:)

     I guess you're referring to
https://github.com/pgjdbc/pgjdbc/pull/252 That was ant, and now we're on
maven. Things may or may not be different, but as I have so far seen,
upstream is not considering removing these dependencies as of today, and
that's why a fork has been suggested.

> _If you do think that there will be things which should be permanently
> patched_ , I would recommend the following to keep the gap (hence
> maintenance burden) as small as possible:
> - Use a fork on github.
     Could be a nice way of maintaining it.
> - Have integration branches for all important upstream branches, where a
> CI job continously merges upstream and tests whether it is still working.
     Right.
> - Open a pull request in upstream for every commit you have. Give them a
> chance to sync up.

     As I mentioned, I don't think this will work. The patches that
Fedora requires are considered necessary by upstream, as they make sense
there (but not in Fedora). That's why there's all this discussion ;P

> - Change only what is absolutely necessary, including not just code, but
> also development practices and standards.

     I would keep it to the minimal, for sure.

>
> Side note: My understanding of maven is that you can exactly control
> your direct dependencies in the pom, and you can have a list including
> transitives with the dependency plugin. You can check and abort build if
> an unexpected dependency comes up. So no need to have an inferior build
> system just because dependencies.

     I'm not sure if I got what you mean here, but if you mean ant is an
inferior solution the answer is yes and pgjdbc will not move back to
and, maven is the choice and for good reasons.

     Thanks for your comments,

     Álvaro


--
Álvaro Hernández Tortosa


-----------
8Kdata



>
>
> On 03/07/2016 10:21 PM, Álvaro Hernández Tortosa wrote:
>>      Hi Pavel.
>>
>>      As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
>> java software that relies on pgjdbc. Since we saw that this issue was
>> not moving forward, we investigated the issue on our own.
>>
>>      While as of today the current ToroDB spec that we have is
>> depending on an older version of JDBC --which is highly undesirable--
>> we have also built a simple mean of packaging pgjdbc for Fedora
>> without having to modify upstream.
>>
>>      Since RPM packages allow (and some of them heavily use) patches
>> against upstream, this is easy to accomplish. Having a parent pom is
>> neither a problem --it is packaged as a subpackage (might not be
>> ideal, but hey, it works, and it doesn't harm). All in all, with all
>> the obvious advantages and disadvantages, we have working source code.
>> Rather than a will to patch upstream or fork pgjdbc, we have a working
>> spec that packages pgjdbc, removing (by patching) all the non-foss and
>> windows dependencies and code, and packaging the parent pom too.
>>
>>      I believe iterating over this effort is better than either forking
>> or patching upstream.
>>
>>      I don't have the code, but Matteo (cc'ed), who did this work, will
>> share it tomorrow.
>>
>>      I'd suggest to work on this, which seems to me as a compromise
>> solution, and agree all on this solution, rather than plan longer-term
>> plans that, while ideal for some, aren't for the rest and, and above
>> all, will take significantly more effort. Our main goal is to deliver
>> packaged and foss pgjdbc to the users asap, and here we have an
>> already working solution.
>>
>>      Matteo, please share the code whenever possible.
>>
>>      Cheers,
>>
>>      Álvaro
>>

     Hi Pavel.

     As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
java software that relies on pgjdbc. Since we saw that this issue was
not moving forward, we investigated the issue on our own.

     While as of today the current ToroDB spec that we have is depending
on an older version of JDBC --which is highly undesirable-- we have also
built a simple mean of packaging pgjdbc for Fedora without having to
modify upstream.

     Since RPM packages allow (and some of them heavily use) patches
against upstream, this is easy to accomplish. Having a parent pom is
neither a problem --it is packaged as a subpackage (might not be ideal,
but hey, it works, and it doesn't harm). All in all, with all the
obvious advantages and disadvantages, we have working source code.
Rather than a will to patch upstream or fork pgjdbc, we have a working
spec that packages pgjdbc, removing (by patching) all the non-foss and
windows dependencies and code, and packaging the parent pom too.

     I believe iterating over this effort is better than either forking
or patching upstream.

     I don't have the code, but Matteo (cc'ed), who did this work, will
share it tomorrow.

     I'd suggest to work on this, which seems to me as a compromise
solution, and agree all on this solution, rather than plan longer-term
plans that, while ideal for some, aren't for the rest and, and above
all, will take significantly more effort. Our main goal is to deliver
packaged and foss pgjdbc to the users asap, and here we have an already
working solution.

     Matteo, please share the code whenever possible.

     Cheers,

     Álvaro

--
Álvaro Hernández Tortosa


-----------
8Kdata




On 07/03/16 15:26, Pavel Raiskup wrote:
> Hi all,
>
> --
> I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
> opt-out if you don't care and sorry for rush).  I just want to see whether
> we in Fedora are thinking a constructive way.  This is not GNU/Linux
> oriented effort, but it is rather about any open source
> packagers/distributors, feel free to add anybody who might be interested
> in the loop.
> --
>
> _Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
> several releases, mostly because there are some non-free/windows-only
> related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
> disallow us to build pgjdbc on free distro.
>
> The preferred way would be to solve this upstream (making the dependencies
> optional), but it is not a mandate of pgjdbc upstream to cooperate on this
> -- even patches from us to support pure open source build are not wanted.
> As upstream is not interested in non-maven builds, it will be most
> probably even worse later.
>
> We've done some small observation around GNU/Linux packages, and it seems
> we all reinvent the very similar patches or hacks over and over again.
>
> Because PostgreSQL connector is important part of operating system, we are
> thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
> should allow us to solve the issue rather sooner than later.
>
> That project idea:
>
>    * we should provide an _easy to use_ (documented how to build from
>      source) version-ed tarball, compatible with pgjdbc
>
>    * this tarball would be FOSS source-only, with FOSS dependencies,
>      (non-free deps could be possible in future, but only as opt-in
>      feature)
>
>    * the build would be 1-step process (no need to build pgjdbc-parent-poms
>      first, and others), with some obvious system dependencies
>
>    * that tarball would allow us to 100% build-from-source _without_ tweaks
>
>    * build from this tarball must not rely on maven repositories --
>      untrusted content at distribution level
>
>    * the testsuite should be fixed to allow us to run it easily under
>      non-root user, on a local/cloud build-box
>
> Would you be interested in having one common code-base for
> open-source-distribution-model of pgjdbc, and optionally (preferably)
> cooperate?  That source should be as close as possible to pgjdbc, just
> limited limited set of patches to allow us to build/test/distribute
> correctly and what is more important we could do the job _consistently_
> with a lot _less_ packagers effort.
>
> Just let me know if this is good/bad idea from your packackar's POV.  Some
> links for discussion with upstream about issues [1,..N].
>
> ¹ By that I mean ability to build from FOSS source, _against_
>    FOSS source dependencies.  By FOSS source I mean software which
>    _anybody_ can read, study, copy, modify, distribute.
>
> [1] http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
> [2] http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
> [3] http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
> [4] http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com
>
> Pavel
>
>
>

Hola Álvaro,

Just some observations. We've been there and done that with upstream.
Maintaining a package with patches this way can easily get to be a big
burden. At some point it becomes a habit, and the big bad surprise comes
when you want to upgrade after a period of lazyness..
I do not want to have a stand on the underlying conflict. Maybe the
healthy github pulse and the quick, accepting treatment of #252 are just
misleading me... 3:)
_If you do think that there will be things which should be permanently
patched_ , I would recommend the following to keep the gap (hence
maintenance burden) as small as possible:
- Use a fork on github.
- Have integration branches for all important upstream branches, where a
CI job continously merges upstream and tests whether it is still working.
- Open a pull request in upstream for every commit you have. Give them a
chance to sync up.
- Change only what is absolutely necessary, including not just code, but
also development practices and standards.

Side note: My understanding of maven is that you can exactly control
your direct dependencies in the pom, and you can have a list including
transitives with the dependency plugin. You can check and abort build if
an unexpected dependency comes up. So no need to have an inferior build
system just because dependencies.


On 03/07/2016 10:21 PM, Álvaro Hernández Tortosa wrote:
>
>     Hi Pavel.
>
>     As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
> java software that relies on pgjdbc. Since we saw that this issue was
> not moving forward, we investigated the issue on our own.
>
>     While as of today the current ToroDB spec that we have is
> depending on an older version of JDBC --which is highly undesirable--
> we have also built a simple mean of packaging pgjdbc for Fedora
> without having to modify upstream.
>
>     Since RPM packages allow (and some of them heavily use) patches
> against upstream, this is easy to accomplish. Having a parent pom is
> neither a problem --it is packaged as a subpackage (might not be
> ideal, but hey, it works, and it doesn't harm). All in all, with all
> the obvious advantages and disadvantages, we have working source code.
> Rather than a will to patch upstream or fork pgjdbc, we have a working
> spec that packages pgjdbc, removing (by patching) all the non-foss and
> windows dependencies and code, and packaging the parent pom too.
>
>     I believe iterating over this effort is better than either forking
> or patching upstream.
>
>     I don't have the code, but Matteo (cc'ed), who did this work, will
> share it tomorrow.
>
>     I'd suggest to work on this, which seems to me as a compromise
> solution, and agree all on this solution, rather than plan longer-term
> plans that, while ideal for some, aren't for the rest and, and above
> all, will take significantly more effort. Our main goal is to deliver
> packaged and foss pgjdbc to the users asap, and here we have an
> already working solution.
>
>     Matteo, please share the code whenever possible.
>
>     Cheers,
>
>     Álvaro
>

On 08/03/16 00:49, Árpád Magosányi wrote:
> Hola Álvaro,

     Hi Árpád, thanks for your Spanish salutation :)
>
> Just some observations. We've been there and done that with upstream.
> Maintaining a package with patches this way can easily get to be a big
> burden. At some point it becomes a habit, and the big bad surprise comes
> when you want to upgrade after a period of lazyness..

     That's surely true. However, the suggested patch to maintain
(please see the code when submitted here) is minimal. Indeed, it's not
code per se. It's just removing some sections of the code -and their
dependencies- that while useful upstream do not make sense in Fedora
(like a Windows subsystem).

     Indeed, it makes sense packager's patch mechanism. After all, you
may not need stuff that upstream *does* want to have (like, again,
Windows stuff). But obviously, upstream cannot be forced to dump that stuff.

     I doubt maintaining this kind of patches should become a burden.
They are not growing and/or diverging patches, but rather static and
straightforward to adapt if upstream modifies related files.

> I do not want to have a stand on the underlying conflict. Maybe the
> healthy github pulse and the quick, accepting treatment of #252 are just
> misleading me... 3:)

     I guess you're referring to
https://github.com/pgjdbc/pgjdbc/pull/252 That was ant, and now we're on
maven. Things may or may not be different, but as I have so far seen,
upstream is not considering removing these dependencies as of today, and
that's why a fork has been suggested.

> _If you do think that there will be things which should be permanently
> patched_ , I would recommend the following to keep the gap (hence
> maintenance burden) as small as possible:
> - Use a fork on github.
     Could be a nice way of maintaining it.
> - Have integration branches for all important upstream branches, where a
> CI job continously merges upstream and tests whether it is still working.
     Right.
> - Open a pull request in upstream for every commit you have. Give them a
> chance to sync up.

     As I mentioned, I don't think this will work. The patches that
Fedora requires are considered necessary by upstream, as they make sense
there (but not in Fedora). That's why there's all this discussion ;P

> - Change only what is absolutely necessary, including not just code, but
> also development practices and standards.

     I would keep it to the minimal, for sure.

>
> Side note: My understanding of maven is that you can exactly control
> your direct dependencies in the pom, and you can have a list including
> transitives with the dependency plugin. You can check and abort build if
> an unexpected dependency comes up. So no need to have an inferior build
> system just because dependencies.

     I'm not sure if I got what you mean here, but if you mean ant is an
inferior solution the answer is yes and pgjdbc will not move back to
and, maven is the choice and for good reasons.

     Thanks for your comments,

     Álvaro


--
Álvaro Hernández Tortosa


-----------
8Kdata



>
>
> On 03/07/2016 10:21 PM, Álvaro Hernández Tortosa wrote:
>>      Hi Pavel.
>>
>>      As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
>> java software that relies on pgjdbc. Since we saw that this issue was
>> not moving forward, we investigated the issue on our own.
>>
>>      While as of today the current ToroDB spec that we have is
>> depending on an older version of JDBC --which is highly undesirable--
>> we have also built a simple mean of packaging pgjdbc for Fedora
>> without having to modify upstream.
>>
>>      Since RPM packages allow (and some of them heavily use) patches
>> against upstream, this is easy to accomplish. Having a parent pom is
>> neither a problem --it is packaged as a subpackage (might not be
>> ideal, but hey, it works, and it doesn't harm). All in all, with all
>> the obvious advantages and disadvantages, we have working source code.
>> Rather than a will to patch upstream or fork pgjdbc, we have a working
>> spec that packages pgjdbc, removing (by patching) all the non-foss and
>> windows dependencies and code, and packaging the parent pom too.
>>
>>      I believe iterating over this effort is better than either forking
>> or patching upstream.
>>
>>      I don't have the code, but Matteo (cc'ed), who did this work, will
>> share it tomorrow.
>>
>>      I'd suggest to work on this, which seems to me as a compromise
>> solution, and agree all on this solution, rather than plan longer-term
>> plans that, while ideal for some, aren't for the rest and, and above
>> all, will take significantly more effort. Our main goal is to deliver
>> packaged and foss pgjdbc to the users asap, and here we have an
>> already working solution.
>>
>>      Matteo, please share the code whenever possible.
>>
>>      Cheers,
>>
>>      Álvaro
>>

On Monday 07 of March 2016 18:31:41 Vladimir Sitnikov wrote:
> Pavel>what is more important we could do the job _consistently_
> Pavel>with a lot _less_ packagers effort.
>
> I think "testing" is the key answer here to get consistent results.

I am talking about the downstream patching effort -- that should be done
consistently in all distributions.

> Pavel>* the build would be 1-step process
> Pavel> Any thoughts?
>
> This all reminds me the 14 competing standards (see [1])
> Do you mean "1-step" as in "1-cpu-instruction"?
> Is "./build.sh" a "1-step"?

By one step I mean something like './build.sh' in terms of package
building.  That is -- no need to build parent-poms (one-file) package
before (or build stub package first, add this to repository and then build
again the full package).

> Pavel> even patches from us to support pure open source build are not wanted
>
> I'm afraid this ^^ is misleading.
>
> Patches are welcome provided they include tests to cover the change.
> No tests -> no acceptance. It is in line with typical development
> model, isn't it?

This is *your* easy excuse to not incorporate change ;).  100% coverage is
a sci-fi, as anybody must agree.  What am I going to test on Fedora if I
patch osgi out?  Am I going to test that it does not work?  If yes, I'm
fine to write the patch.

You refused attempts to post patches which would make some code optional,
at which point it is not useful to think about testing something.

Other than that, not everything is easily testable in your actual CI.

> Pavel>_Open_ distribution¹
> Pavel> By FOSS source I mean software which
> Pavel>  _anybody_ can read, study, copy, modify, distribute
>
> Can you tell us if org.osgi.enterprise complies with your definition
> of "open distribution"?

The license refuses you to modify the code, it is *clear vendor lock-in*
and:

  """
  anybody who cares a bit about open source principles should at least
  be aware of its consequences
  """

Also, your build system should allow us to build without this support.
That is how everything else works.  Can we submit patches for this?

> Just in case you say "no", see [2] (download jar and note that sources
> are in OSGI-OPT folder) for complete source code that is available
> under Apache 2.0 license -> you can read, copy, modify and distribute
> it with no problem.

But as far as I understand, those jars are not original sources -- those
are products of some build (which automatically adds some license file
there? or what is automatized?  Is it easily re-build-able?).

You are allowed to download original sources, but you can't modify them.
I stopped the observation here as that is simply not acceptable -- this
has been discussed in mentioned threads, direct reference is [1]
(reference by Vitalii).

> Taking that into account, why do you think "org.osgi.enterprise" is
> not "open distribution"?
>
> 1: thttps://xkcd.com/927/
> 2: http://mvnrepository.com/artifact/org.osgi/org.osgi.enterprise/5.0.0

:) no, this is different.  If you produce API that needs to be protected
by license, it is either wrong design, or you have bad intentions.

[1] https://lists.fedoraproject.org/pipermail/legal/2012-July/001930.html

Pavel

[ My last upload was around 2007, so I suggest to approach the Debian Java
Maintainers <pkg-java-maintainers@lists.alioth.debian.org>, however this is a
public list, only CCing one maintainer ]

On 07.03.2016 15:26, Pavel Raiskup wrote:
> Hi all,
>
> --
> I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
> opt-out if you don't care and sorry for rush).  I just want to see whether
> we in Fedora are thinking a constructive way.  This is not GNU/Linux
> oriented effort, but it is rather about any open source
> packagers/distributors, feel free to add anybody who might be interested
> in the loop.
> --
>
> _Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
> several releases, mostly because there are some non-free/windows-only
> related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
> disallow us to build pgjdbc on free distro.
>
> The preferred way would be to solve this upstream (making the dependencies
> optional), but it is not a mandate of pgjdbc upstream to cooperate on this
> -- even patches from us to support pure open source build are not wanted.
> As upstream is not interested in non-maven builds, it will be most
> probably even worse later.
>
> We've done some small observation around GNU/Linux packages, and it seems
> we all reinvent the very similar patches or hacks over and over again.
>
> Because PostgreSQL connector is important part of operating system, we are
> thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
> should allow us to solve the issue rather sooner than later.
>
> That project idea:
>
>    * we should provide an _easy to use_ (documented how to build from
>      source) version-ed tarball, compatible with pgjdbc
>
>    * this tarball would be FOSS source-only, with FOSS dependencies,
>      (non-free deps could be possible in future, but only as opt-in
>      feature)
>
>    * the build would be 1-step process (no need to build pgjdbc-parent-poms
>      first, and others), with some obvious system dependencies
>
>    * that tarball would allow us to 100% build-from-source _without_ tweaks
>
>    * build from this tarball must not rely on maven repositories --
>      untrusted content at distribution level
>
>    * the testsuite should be fixed to allow us to run it easily under
>      non-root user, on a local/cloud build-box
>
> Would you be interested in having one common code-base for
> open-source-distribution-model of pgjdbc, and optionally (preferably)
> cooperate?  That source should be as close as possible to pgjdbc, just
> limited limited set of patches to allow us to build/test/distribute
> correctly and what is more important we could do the job _consistently_
> with a lot _less_ packagers effort.
>
> Just let me know if this is good/bad idea from your packackar's POV.  Some
> links for discussion with upstream about issues [1,..N].
>
> ¹ By that I mean ability to build from FOSS source, _against_
>    FOSS source dependencies.  By FOSS source I mean software which
>    _anybody_ can read, study, copy, modify, distribute.
>
> [1] http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
> [2] http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
> [3] http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
> [4] http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com
>
> Pavel
>

Thanks for the comment.  I still think that we should fix upstream, can
we?  But ..

On Tuesday 08 of March 2016 11:05:33 Matteo Melli wrote:
> Hi all,
>
> As asked by Alvaro this is the link to the patch and instruction to remove
> JNA and Waffle from the driver and to generate a package for the
> problematic osgi interface:
> http://www.postgresql.org/message-id/CAKFrgp8Qgj32NcJiDdfPMnrPU=wY=KKR+BKX86G71N3i=2dUPg@mail.gmail.com

Thanks for the patches.  Yes, Pavel wrote similar ones few months back in
https://fedorapeople.org/cgit/pkajaba/public_git/postgresql-jdbc.git

> From there to generate a spec should be quite straightforward.

It is not completely straight-forward for other distributions.  Those
changes need to be done at one place.

> I suggest to use maven instead of ant to build project since
> modification to the pom can be handled easily by macros from
> javapackages-tools and it make easier dependency resolution thanks to
> xmvn.

Agreed.

> As Árpád mentioned this approach will require to maintain the package when
> things break. Anyway, after touching the process of migrating a package
> from Java world (or better say maven world) to distribution packaging, I
> can say that the greater effort has to be put in dependency handling.

That is why things like drivers should not _unconditionally_ depend on
some non-free, hardly useful dependencies.  Consider yourself starting
some thin-range oriented distribution from scratch -- you'll be glad to
hear that upstream allowed you to opt-out features that are redundant for
you.

> An that always bring the need to maintain the spec file. I think that if
> the patch is not so big it should be ok as an extra cost to add, and, I
> think, this is the case.

Right, but it is no need to have that patch in Gentoo, Red Hat Enterprise
Linux, Fedora, Arch, ..., *BSD, etc.  It is better to write it once and
don't push everybody in this chain reinvent the patch (or C&P is from one
package to another).

Pavel

>
> Matteo
>
> 2016-03-08 1:02 GMT+01:00 Álvaro Hernández Tortosa <aht@8kdata.com>:
>
> >
> >
> > On 08/03/16 00:49, Árpád Magosányi wrote:
> >
> >> Hola Álvaro,
> >>
> >
> >     Hi Árpád, thanks for your Spanish salutation :)
> >
> >>
> >> Just some observations. We've been there and done that with upstream.
> >> Maintaining a package with patches this way can easily get to be a big
> >> burden. At some point it becomes a habit, and the big bad surprise comes
> >> when you want to upgrade after a period of lazyness..
> >>
> >
> >     That's surely true. However, the suggested patch to maintain (please
> > see the code when submitted here) is minimal. Indeed, it's not code per se.
> > It's just removing some sections of the code -and their dependencies- that
> > while useful upstream do not make sense in Fedora (like a Windows
> > subsystem).
> >
> >     Indeed, it makes sense packager's patch mechanism. After all, you may
> > not need stuff that upstream *does* want to have (like, again, Windows
> > stuff). But obviously, upstream cannot be forced to dump that stuff.
> >
> >     I doubt maintaining this kind of patches should become a burden. They
> > are not growing and/or diverging patches, but rather static and
> > straightforward to adapt if upstream modifies related files.
> >
> > I do not want to have a stand on the underlying conflict. Maybe the
> >> healthy github pulse and the quick, accepting treatment of #252 are just
> >> misleading me... 3:)
> >>
> >
> >     I guess you're referring to https://github.com/pgjdbc/pgjdbc/pull/252
> > That was ant, and now we're on maven. Things may or may not be different,
> > but as I have so far seen, upstream is not considering removing these
> > dependencies as of today, and that's why a fork has been suggested.
> >
> > _If you do think that there will be things which should be permanently
> >> patched_ , I would recommend the following to keep the gap (hence
> >> maintenance burden) as small as possible:
> >> - Use a fork on github.
> >>
> >     Could be a nice way of maintaining it.
> >
> >> - Have integration branches for all important upstream branches, where a
> >> CI job continously merges upstream and tests whether it is still working.
> >>
> >     Right.
> >
> >> - Open a pull request in upstream for every commit you have. Give them a
> >> chance to sync up.
> >>
> >
> >     As I mentioned, I don't think this will work. The patches that Fedora
> > requires are considered necessary by upstream, as they make sense there
> > (but not in Fedora). That's why there's all this discussion ;P
> >
> > - Change only what is absolutely necessary, including not just code, but
> >> also development practices and standards.
> >>
> >
> >     I would keep it to the minimal, for sure.
> >
> >
> >> Side note: My understanding of maven is that you can exactly control
> >> your direct dependencies in the pom, and you can have a list including
> >> transitives with the dependency plugin. You can check and abort build if
> >> an unexpected dependency comes up. So no need to have an inferior build
> >> system just because dependencies.
> >>
> >
> >     I'm not sure if I got what you mean here, but if you mean ant is an
> > inferior solution the answer is yes and pgjdbc will not move back to and,
> > maven is the choice and for good reasons.
> >
> >     Thanks for your comments,
> >
> >     Álvaro
> >
> >
> > --
> > Álvaro Hernández Tortosa
> >
> >
> > -----------
> > 8Kdata
> >
> >
> >
> >
> >>
> >> On 03/07/2016 10:21 PM, Álvaro Hernández Tortosa wrote:
> >>
> >>>      Hi Pavel.
> >>>
> >>>      As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
> >>> java software that relies on pgjdbc. Since we saw that this issue was
> >>> not moving forward, we investigated the issue on our own.
> >>>
> >>>      While as of today the current ToroDB spec that we have is
> >>> depending on an older version of JDBC --which is highly undesirable--
> >>> we have also built a simple mean of packaging pgjdbc for Fedora
> >>> without having to modify upstream.
> >>>
> >>>      Since RPM packages allow (and some of them heavily use) patches
> >>> against upstream, this is easy to accomplish. Having a parent pom is
> >>> neither a problem --it is packaged as a subpackage (might not be
> >>> ideal, but hey, it works, and it doesn't harm). All in all, with all
> >>> the obvious advantages and disadvantages, we have working source code.
> >>> Rather than a will to patch upstream or fork pgjdbc, we have a working
> >>> spec that packages pgjdbc, removing (by patching) all the non-foss and
> >>> windows dependencies and code, and packaging the parent pom too.
> >>>
> >>>      I believe iterating over this effort is better than either forking
> >>> or patching upstream.
> >>>
> >>>      I don't have the code, but Matteo (cc'ed), who did this work, will
> >>> share it tomorrow.
> >>>
> >>>      I'd suggest to work on this, which seems to me as a compromise
> >>> solution, and agree all on this solution, rather than plan longer-term
> >>> plans that, while ideal for some, aren't for the rest and, and above
> >>> all, will take significantly more effort. Our main goal is to deliver
> >>> packaged and foss pgjdbc to the users asap, and here we have an
> >>> already working solution.
> >>>
> >>>      Matteo, please share the code whenever possible.
> >>>
> >>>      Cheers,
> >>>
> >>>      Álvaro
> >>>
> >>>
> >

On Tuesday 08 of March 2016 05:32:20 Dave Cramer wrote:
> We can and have accepted patches without test cases as long as an
> argument can be made why they can't be tested.

Good to hear.  I've heard something like "no test => no patch, right?!" so
far.

I our case, it is IMO no need to test the potentially opt-outed feature,
which _is_ what is all this about.  It is not needed to check in upstream
that the opt-out feature works (because everybody in downstream distro
packaging will take a look at this and do the opt-out).  You also don't
need to run CI for Gentoo specific usecases I believe (a lot of options
for many packages).  No need to install and test Fedora, we'll do that.

So, I agree -- having higher coverage is better, and if there is something
to test, we can do it.  But we need to first discuss what can be changed.

> > You refused attempts to post patches which would make some code optional,
> > at which point it is not useful to think about testing something.
> >
> > Other than that, not everything is easily testable in your actual CI.
> >
> > > Pavel>_Open_ distribution¹
> > > Pavel> By FOSS source I mean software which
> > > Pavel>  _anybody_ can read, study, copy, modify, distribute
> > >
> > > Can you tell us if org.osgi.enterprise complies with your definition
> > > of "open distribution"?
> >
> > The license refuses you to modify the code, it is *clear vendor lock-in*
> > and:
>
>
> >   """
> >   anybody who cares a bit about open source principles should at least
> >   be aware of its consequences
> >   """
>
> This is a red-herring. The whole purpose of osgi is to provide a
> consistent interface to allow loading of software. If you were to modify
> it, it would fail to be relevant. Blindly applying OSS principles does
> not make sense here.

This is _huge untrue_, there is no need to use license for this.

Is the interface so perfect so it can not be made better?  In future?  I
think this is obvious -- Why you can't prepare compatible and better
interface as concurrent solution?  Why we need to wait for anybody?

Somebody just tries to protect its advantage.

Not modifiable code is vendor-lock-in;  and that is not good project to
hard-depend on.

> From what I can see they are provided simply to allow you to build packages
> just like this.

Are you sure this is not a crossing the law border?

I'm not going to do this because I haven't "signed the papers" on the web
page as discussed in [1] (probably the correct place to download the
sources from).  Downloading of that jar, using it and fixing FTBFS bugs in
that might be illegal.  So I can't do this..

> > You are allowed to download original sources, but you can't modify them.
> > I stopped the observation here as that is simply not acceptable -- this
> > has been discussed in mentioned threads, direct reference is [1]
> > (reference by Vitalii).
> >
>
> See above
>
> Dave Cramer
>
> davec@postgresintl.com
> www.postgresintl.com

[ My last upload was around 2007, so I suggest to approach the Debian Java
Maintainers <pkg-java-maintainers@lists.alioth.debian.org>, however this is a
public list, only CCing one maintainer ]

On 07.03.2016 15:26, Pavel Raiskup wrote:
> Hi all,
>
> --
> I'm taking the liberty of CCing all pgjdbc packagers I'm aware of (please
> opt-out if you don't care and sorry for rush).  I just want to see whether
> we in Fedora are thinking a constructive way.  This is not GNU/Linux
> oriented effort, but it is rather about any open source
> packagers/distributors, feel free to add anybody who might be interested
> in the loop.
> --
>
> _Open_ distribution¹ of pgjdbc is becoming a bit painful for the last
> several releases, mostly because there are some non-free/windows-only
> related _hard_-dependencies (currently osgi.enterprise, waffle-jna) which
> disallow us to build pgjdbc on free distro.
>
> The preferred way would be to solve this upstream (making the dependencies
> optional), but it is not a mandate of pgjdbc upstream to cooperate on this
> -- even patches from us to support pure open source build are not wanted.
> As upstream is not interested in non-maven builds, it will be most
> probably even worse later.
>
> We've done some small observation around GNU/Linux packages, and it seems
> we all reinvent the very similar patches or hacks over and over again.
>
> Because PostgreSQL connector is important part of operating system, we are
> thinking about a small friendly fork of pgjdbc, called pgjdbc-foss.  This
> should allow us to solve the issue rather sooner than later.
>
> That project idea:
>
>    * we should provide an _easy to use_ (documented how to build from
>      source) version-ed tarball, compatible with pgjdbc
>
>    * this tarball would be FOSS source-only, with FOSS dependencies,
>      (non-free deps could be possible in future, but only as opt-in
>      feature)
>
>    * the build would be 1-step process (no need to build pgjdbc-parent-poms
>      first, and others), with some obvious system dependencies
>
>    * that tarball would allow us to 100% build-from-source _without_ tweaks
>
>    * build from this tarball must not rely on maven repositories --
>      untrusted content at distribution level
>
>    * the testsuite should be fixed to allow us to run it easily under
>      non-root user, on a local/cloud build-box
>
> Would you be interested in having one common code-base for
> open-source-distribution-model of pgjdbc, and optionally (preferably)
> cooperate?  That source should be as close as possible to pgjdbc, just
> limited limited set of patches to allow us to build/test/distribute
> correctly and what is more important we could do the job _consistently_
> with a lot _less_ packagers effort.
>
> Just let me know if this is good/bad idea from your packackar's POV.  Some
> links for discussion with upstream about issues [1,..N].
>
> ¹ By that I mean ability to build from FOSS source, _against_
>    FOSS source dependencies.  By FOSS source I mean software which
>    _anybody_ can read, study, copy, modify, distribute.
>
> [1] http://www.postgresql.org/message-id/1831842355.39585708.1455624950515.JavaMail.zimbra@redhat.com
> [2] http://www.postgresql.org/message-id/2113338928.20942725.1448530160996.JavaMail.zimbra@redhat.com
> [3] http://www.postgresql.org/message-id/5479464.pnS2mdyLUu@nb.usersys.redhat.com
> [4] http://www.postgresql.org/message-id/2217774.p6G2ev8LQ6@nb.usersys.redhat.com
>
> Pavel
>

Pavel>Not modifiable code is vendor-lock-in

org.osgi.enterprise jar is Apache 2.0-licensed.
Apache 2.0 allows modification of a source code. Surprise.

Pavel>I our case, it is IMO no need to test the potentially opt-outed feature,

You claim to "invent common build denominator feature", then you claim
"there's no need to test it".
Are you kidding?

As per Dave's words: "can you explain why packaging can't be tested"?
"no need" != "can't" as far as I can understand.
I think package testing should be rather simple.

Pavel> It is not needed to check in upstream
Pavel>that the opt-out feature works

You seem to ignore the main aim of testing. The tests are there to
catch unintentional changes.
If no tests added, any innocent refactoring might break your packaging script.

Vladimir

On Tuesday 08 of March 2016 14:37:56 Vladimir Sitnikov wrote:
> Pavel>Not modifiable code is vendor-lock-in
>
> org.osgi.enterprise jar is Apache 2.0-licensed.
> Apache 2.0 allows modification of a source code. Surprise.

It is actually not correct, as far as I am aware.  The links you provided
so far are results of official build, which puts the sources there by
default.  Can we convince upstream to release official tarballs without
the "signature needed" request that disallows you to modify?

> Pavel>I our case, it is IMO no need to test the potentially opt-outed
> Pavel>feature,
>
> You claim to "invent common build denominator feature", then you claim
> "there's no need to test it".
> Are you kidding?

Yes, here is your excuse :) I talked about, or?

You test your full-feature-set you support ATM.  That is fine, I do not
plan to stop testing something.

Some people (not you but me!) do want to disable something for themselves,
what exactly do you want to test on it?
But we can probably simulate the situation for you -- we can always do two
builds in upstream CI -- with/without the feature, even though you support
only the first scenario.  Do I understand it right this is wanted?

> As per Dave's words: "can you explain why packaging can't be tested"?
> "no need" != "can't" as far as I can understand.
> I think package testing should be rather simple.

It is tested.  I don't think you want it upstream as you don't support
packages we re-distribute.

> Pavel> It is not needed to check in upstream
> Pavel>that the opt-out feature works
>
> You seem to ignore the main aim of testing. The tests are there to
> catch unintentional changes.

No.  I appreciate testing, be fair please.

You know that there are tests _already_;  that would catch the issues we
could potentially add into existing level of your support ... so we can
fix the patches we propose to not do that.

I just say -- don't test that the '--opt-out-waffle' and '--opt-no-osgi',
whatever format it will have.  Then you know that "nobody" except for
packagers use those -- and the guys know how to fix this if something
wrong happens...

> If no tests added, any innocent refactoring might break your packaging
> script.

And, again -- don't be afraid here, that is why we are here.  You can not
provide packaging scripts for the whole world, it is not upstream
responsibility and no upstream does this.

Pavel

On 03/08/2016 10:33 AM, Pavel Raiskup wrote:
> On Monday 07 of March 2016 18:31:41 Vladimir Sitnikov wrote:
>
>> Pavel> even patches from us to support pure open source build are not wanted
>>
>> I'm afraid this ^^ is misleading.
>>
>> Patches are welcome provided they include tests to cover the change.
>> No tests -> no acceptance. It is in line with typical development
>> model, isn't it?
> This is *your* easy excuse to not incorporate change ;).  100% coverage is
> a sci-fi, as anybody must agree.  What am I going to test on Fedora if I
> patch osgi out?  Am I going to test that it does not work?  If yes, I'm
> fine to write the patch.
>
> You refused attempts to post patches which would make some code optional,
> at which point it is not useful to think about testing something.
>
> Other than that, not everything is easily testable in your actual CI.

You hit my pet peeve here. We are not animals. Intelligent people do TDD,
and clean code. The rest should write unit tests.
TDD and clean code are the rules of software development as a profession.
Simply because they are the only known practice to avoid fscking up the
code.

Of course there are cases which cannot be tested, but you have to provide
a reason at least.

On Monday 07 of March 2016 18:31:41 Vladimir Sitnikov wrote:
> Pavel>what is more important we could do the job _consistently_
> Pavel>with a lot _less_ packagers effort.
>
> I think "testing" is the key answer here to get consistent results.

I am talking about the downstream patching effort -- that should be done
consistently in all distributions.

> Pavel>* the build would be 1-step process
> Pavel> Any thoughts?
>
> This all reminds me the 14 competing standards (see [1])
> Do you mean "1-step" as in "1-cpu-instruction"?
> Is "./build.sh" a "1-step"?

By one step I mean something like './build.sh' in terms of package
building.  That is -- no need to build parent-poms (one-file) package
before (or build stub package first, add this to repository and then build
again the full package).

> Pavel> even patches from us to support pure open source build are not wanted
>
> I'm afraid this ^^ is misleading.
>
> Patches are welcome provided they include tests to cover the change.
> No tests -> no acceptance. It is in line with typical development
> model, isn't it?

This is *your* easy excuse to not incorporate change ;).  100% coverage is
a sci-fi, as anybody must agree.  What am I going to test on Fedora if I
patch osgi out?  Am I going to test that it does not work?  If yes, I'm
fine to write the patch.

You refused attempts to post patches which would make some code optional,
at which point it is not useful to think about testing something.

Other than that, not everything is easily testable in your actual CI.

> Pavel>_Open_ distribution¹
> Pavel> By FOSS source I mean software which
> Pavel>  _anybody_ can read, study, copy, modify, distribute
>
> Can you tell us if org.osgi.enterprise complies with your definition
> of "open distribution"?

The license refuses you to modify the code, it is *clear vendor lock-in*
and:

  """
  anybody who cares a bit about open source principles should at least
  be aware of its consequences
  """

Also, your build system should allow us to build without this support.
That is how everything else works.  Can we submit patches for this?

> Just in case you say "no", see [2] (download jar and note that sources
> are in OSGI-OPT folder) for complete source code that is available
> under Apache 2.0 license -> you can read, copy, modify and distribute
> it with no problem.

But as far as I understand, those jars are not original sources -- those
are products of some build (which automatically adds some license file
there? or what is automatized?  Is it easily re-build-able?).

You are allowed to download original sources, but you can't modify them.
I stopped the observation here as that is simply not acceptable -- this
has been discussed in mentioned threads, direct reference is [1]
(reference by Vitalii).

> Taking that into account, why do you think "org.osgi.enterprise" is
> not "open distribution"?
>
> 1: thttps://xkcd.com/927/
> 2: http://mvnrepository.com/artifact/org.osgi/org.osgi.enterprise/5.0.0

:) no, this is different.  If you produce API that needs to be protected
by license, it is either wrong design, or you have bad intentions.

[1] https://lists.fedoraproject.org/pipermail/legal/2012-July/001930.html

Pavel

On Tuesday 08 of March 2016 13:29:13 Árpád Magosányi wrote:
> On 03/08/2016 10:33 AM, Pavel Raiskup wrote:
> > On Monday 07 of March 2016 18:31:41 Vladimir Sitnikov wrote:
> >
> >> Pavel> even patches from us to support pure open source build are not wanted
> >>
> >> I'm afraid this ^^ is misleading.
> >>
> >> Patches are welcome provided they include tests to cover the change.
> >> No tests -> no acceptance. It is in line with typical development
> >> model, isn't it?
> >
> > This is *your* easy excuse to not incorporate change ;).  100% coverage is
> > a sci-fi, as anybody must agree.  What am I going to test on Fedora if I
> > patch osgi out?  Am I going to test that it does not work?  If yes, I'm
> > fine to write the patch.
> >
> > You refused attempts to post patches which would make some code optional,
> > at which point it is not useful to think about testing something.
> >
> > Other than that, not everything is easily testable in your actual CI.
>
> You hit my pet peeve here. We are not animals. Intelligent people do TDD,
> and clean code. The rest should write unit tests.

Please, what is this about?  Feel free to be constructive, but I just try
to convince someone that we need to opt-out something.  If we had 12
optional features, that is 2^12 possible tests, which is sci-fi.

I'm not telling to not do TDD, I'm not offensive.  I do not tell that
people who do-write-unit-tests are animals or not-intelligent, neither I
divide people like that.  I appreciate any work done the open source way.

Feel free to look at the comments Vladimir posted before, you'll see what
I mean by "no test, no acceptance".  There was no space for discussion at
that time.

> TDD and clean code are the rules of software development as a
> profession.  Simply because they are the only known practice to avoid
> fscking up the code.

Agreed, but it is orthogonal issue here.

> Of course there are cases which cannot be tested, but you have to provide
> a reason at least.

Thanks, we all agree on this.

On Tuesday 08 of March 2016 12:27:32 Vitalii Tymchyshyn wrote:
> As for me, the problem is not about JDBC driver, the problem is with FOSS
> world not able to deal with OSGI recently.

Then the actual status works for you.  It is completely fine, but there
are other who do care and who prefer different ideas, and there should be
a way to lock ourselves somewhere.

> It neither starts nor stops with PgJDBC.

Truth.

> And for me it's not a reason to make some useful feature optional
> complicating things for a user (instead of per-JDK jar you are starting
> to produce at least twice as many jars to deal with).

I need to react because of this ^.  PostgreSQL server has a lot of
features available during build, try './configure --help'.  You can choose
what features are important for you, and what not.

The opt-out (unused by upstream probably) in pgjdbc I would like to see is
far from being that sophisticated.  What is the problem with this?

> And it's not a technical limitation - current jars work perfectly in
> non-OSGI environment and runtime dependency resolution is the way java
> world tries to live.

Note the whole Java world probably.  I, for example, can consider myself
as somebody somebody who needs to write some gui app in Java, than
distribute that as native package for distributions.  More!  I want the
distribution guys do it for myself.  But I can't use jdbc, because it is
not provided natively -- and I can't thus use Java, because I can't
connect to PostgreSQL.

> Guys, you really need to decide with OSG in general. How are you going
> to deal with modern versions of containers like JBoss that has OSGi as a
> core feature? That is the problem.

JBoss is not Fedora.  We can't pull everything into Fedora just because we
want to connect 'A with B' where both 'A' and 'B' are already there and we
are fine with that.  To connect 'A' with 'B' is needed 'C (osgi support)'
and 'D (waffle)', where we are not able to provide 'D' and 'C' is not
needed atm (somebody tried to deal with that in Y2012, and we did not move
forward yet in 2016).  We need solution now, and we can enable things
later.

> You used to use API files from Felix, but they dont produce fresh
> versions anymore, possiby after switching to the official API jars.
> Contact them to check it.

I'll discuss this locally and report back, but it is not related ATM.
Some features are naturally expected to be optional, and that is why I'm
here.

Fedora is where it is in this case and I'm unfortunately unable to change
that (I'm not going to provide and support waffle and osgi just to provide
jdbc).

Pavel

> Best regards, Vitalii Tymchyshyn
>
> Вт, 8 бер. 2016 07:08 користувач Pavel Raiskup <praiskup@redhat.com> пише:
>
> > On Tuesday 08 of March 2016 14:37:56 Vladimir Sitnikov wrote:
> > > Pavel>Not modifiable code is vendor-lock-in
> > >
> > > org.osgi.enterprise jar is Apache 2.0-licensed.
> > > Apache 2.0 allows modification of a source code. Surprise.
> >
> > It is actually not correct, as far as I am aware.  The links you provided
> > so far are results of official build, which puts the sources there by
> > default.  Can we convince upstream to release official tarballs without
> > the "signature needed" request that disallows you to modify?
> >
> > > Pavel>I our case, it is IMO no need to test the potentially opt-outed
> > > Pavel>feature,
> > >
> > > You claim to "invent common build denominator feature", then you claim
> > > "there's no need to test it".
> > > Are you kidding?
> >
> > Yes, here is your excuse :) I talked about, or?
> >
> > You test your full-feature-set you support ATM.  That is fine, I do not
> > plan to stop testing something.
> >
> > Some people (not you but me!) do want to disable something for themselves,
> > what exactly do you want to test on it?
> > But we can probably simulate the situation for you -- we can always do two
> > builds in upstream CI -- with/without the feature, even though you support
> > only the first scenario.  Do I understand it right this is wanted?
> >
> > > As per Dave's words: "can you explain why packaging can't be tested"?
> > > "no need" != "can't" as far as I can understand.
> > > I think package testing should be rather simple.
> >
> > It is tested.  I don't think you want it upstream as you don't support
> > packages we re-distribute.
> >
> > > Pavel> It is not needed to check in upstream
> > > Pavel>that the opt-out feature works
> > >
> > > You seem to ignore the main aim of testing. The tests are there to
> > > catch unintentional changes.
> >
> > No.  I appreciate testing, be fair please.
> >
> > You know that there are tests _already_;  that would catch the issues we
> > could potentially add into existing level of your support ... so we can
> > fix the patches we propose to not do that.
> >
> > I just say -- don't test that the '--opt-out-waffle' and '--opt-no-osgi',
> > whatever format it will have.  Then you know that "nobody" except for
> > packagers use those -- and the guys know how to fix this if something
> > wrong happens...
> >
> > > If no tests added, any innocent refactoring might break your packaging
> > > script.
> >
> > And, again -- don't be afraid here, that is why we are here.  You can not
> > provide packaging scripts for the whole world, it is not upstream
> > responsibility and no upstream does this.
> >
> > Pavel
> >
> >
> >
> > --
> > Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-jdbc
> >

Thanks for the comment.  I still think that we should fix upstream, can
we?  But ..

On Tuesday 08 of March 2016 11:05:33 Matteo Melli wrote:
> Hi all,
>
> As asked by Alvaro this is the link to the patch and instruction to remove
> JNA and Waffle from the driver and to generate a package for the
> problematic osgi interface:
> http://www.postgresql.org/message-id/CAKFrgp8Qgj32NcJiDdfPMnrPU=wY=KKR+BKX86G71N3i=2dUPg@mail.gmail.com

Thanks for the patches.  Yes, Pavel wrote similar ones few months back in
https://fedorapeople.org/cgit/pkajaba/public_git/postgresql-jdbc.git

> From there to generate a spec should be quite straightforward.

It is not completely straight-forward for other distributions.  Those
changes need to be done at one place.

> I suggest to use maven instead of ant to build project since
> modification to the pom can be handled easily by macros from
> javapackages-tools and it make easier dependency resolution thanks to
> xmvn.

Agreed.

> As Árpád mentioned this approach will require to maintain the package when
> things break. Anyway, after touching the process of migrating a package
> from Java world (or better say maven world) to distribution packaging, I
> can say that the greater effort has to be put in dependency handling.

That is why things like drivers should not _unconditionally_ depend on
some non-free, hardly useful dependencies.  Consider yourself starting
some thin-range oriented distribution from scratch -- you'll be glad to
hear that upstream allowed you to opt-out features that are redundant for
you.

> An that always bring the need to maintain the spec file. I think that if
> the patch is not so big it should be ok as an extra cost to add, and, I
> think, this is the case.

Right, but it is no need to have that patch in Gentoo, Red Hat Enterprise
Linux, Fedora, Arch, ..., *BSD, etc.  It is better to write it once and
don't push everybody in this chain reinvent the patch (or C&P is from one
package to another).

Pavel

>
> Matteo
>
> 2016-03-08 1:02 GMT+01:00 Álvaro Hernández Tortosa <aht@8kdata.com>:
>
> >
> >
> > On 08/03/16 00:49, Árpád Magosányi wrote:
> >
> >> Hola Álvaro,
> >>
> >
> >     Hi Árpád, thanks for your Spanish salutation :)
> >
> >>
> >> Just some observations. We've been there and done that with upstream.
> >> Maintaining a package with patches this way can easily get to be a big
> >> burden. At some point it becomes a habit, and the big bad surprise comes
> >> when you want to upgrade after a period of lazyness..
> >>
> >
> >     That's surely true. However, the suggested patch to maintain (please
> > see the code when submitted here) is minimal. Indeed, it's not code per se.
> > It's just removing some sections of the code -and their dependencies- that
> > while useful upstream do not make sense in Fedora (like a Windows
> > subsystem).
> >
> >     Indeed, it makes sense packager's patch mechanism. After all, you may
> > not need stuff that upstream *does* want to have (like, again, Windows
> > stuff). But obviously, upstream cannot be forced to dump that stuff.
> >
> >     I doubt maintaining this kind of patches should become a burden. They
> > are not growing and/or diverging patches, but rather static and
> > straightforward to adapt if upstream modifies related files.
> >
> > I do not want to have a stand on the underlying conflict. Maybe the
> >> healthy github pulse and the quick, accepting treatment of #252 are just
> >> misleading me... 3:)
> >>
> >
> >     I guess you're referring to https://github.com/pgjdbc/pgjdbc/pull/252
> > That was ant, and now we're on maven. Things may or may not be different,
> > but as I have so far seen, upstream is not considering removing these
> > dependencies as of today, and that's why a fork has been suggested.
> >
> > _If you do think that there will be things which should be permanently
> >> patched_ , I would recommend the following to keep the gap (hence
> >> maintenance burden) as small as possible:
> >> - Use a fork on github.
> >>
> >     Could be a nice way of maintaining it.
> >
> >> - Have integration branches for all important upstream branches, where a
> >> CI job continously merges upstream and tests whether it is still working.
> >>
> >     Right.
> >
> >> - Open a pull request in upstream for every commit you have. Give them a
> >> chance to sync up.
> >>
> >
> >     As I mentioned, I don't think this will work. The patches that Fedora
> > requires are considered necessary by upstream, as they make sense there
> > (but not in Fedora). That's why there's all this discussion ;P
> >
> > - Change only what is absolutely necessary, including not just code, but
> >> also development practices and standards.
> >>
> >
> >     I would keep it to the minimal, for sure.
> >
> >
> >> Side note: My understanding of maven is that you can exactly control
> >> your direct dependencies in the pom, and you can have a list including
> >> transitives with the dependency plugin. You can check and abort build if
> >> an unexpected dependency comes up. So no need to have an inferior build
> >> system just because dependencies.
> >>
> >
> >     I'm not sure if I got what you mean here, but if you mean ant is an
> > inferior solution the answer is yes and pgjdbc will not move back to and,
> > maven is the choice and for good reasons.
> >
> >     Thanks for your comments,
> >
> >     Álvaro
> >
> >
> > --
> > Álvaro Hernández Tortosa
> >
> >
> > -----------
> > 8Kdata
> >
> >
> >
> >
> >>
> >> On 03/07/2016 10:21 PM, Álvaro Hernández Tortosa wrote:
> >>
> >>>      Hi Pavel.
> >>>
> >>>      As you are well aware :) we (as in 8Kdata) are packaging ToroDB,
> >>> java software that relies on pgjdbc. Since we saw that this issue was
> >>> not moving forward, we investigated the issue on our own.
> >>>
> >>>      While as of today the current ToroDB spec that we have is
> >>> depending on an older version of JDBC --which is highly undesirable--
> >>> we have also built a simple mean of packaging pgjdbc for Fedora
> >>> without having to modify upstream.
> >>>
> >>>      Since RPM packages allow (and some of them heavily use) patches
> >>> against upstream, this is easy to accomplish. Having a parent pom is
> >>> neither a problem --it is packaged as a subpackage (might not be
> >>> ideal, but hey, it works, and it doesn't harm). All in all, with all
> >>> the obvious advantages and disadvantages, we have working source code.
> >>> Rather than a will to patch upstream or fork pgjdbc, we have a working
> >>> spec that packages pgjdbc, removing (by patching) all the non-foss and
> >>> windows dependencies and code, and packaging the parent pom too.
> >>>
> >>>      I believe iterating over this effort is better than either forking
> >>> or patching upstream.
> >>>
> >>>      I don't have the code, but Matteo (cc'ed), who did this work, will
> >>> share it tomorrow.
> >>>
> >>>      I'd suggest to work on this, which seems to me as a compromise
> >>> solution, and agree all on this solution, rather than plan longer-term
> >>> plans that, while ideal for some, aren't for the rest and, and above
> >>> all, will take significantly more effort. Our main goal is to deliver
> >>> packaged and foss pgjdbc to the users asap, and here we have an
> >>> already working solution.
> >>>
> >>>      Matteo, please share the code whenever possible.
> >>>
> >>>      Cheers,
> >>>
> >>>      Álvaro
> >>>
> >>>
> >

On Tuesday 08 of March 2016 05:32:20 Dave Cramer wrote:
> We can and have accepted patches without test cases as long as an
> argument can be made why they can't be tested.

Good to hear.  I've heard something like "no test => no patch, right?!" so
far.

I our case, it is IMO no need to test the potentially opt-outed feature,
which _is_ what is all this about.  It is not needed to check in upstream
that the opt-out feature works (because everybody in downstream distro
packaging will take a look at this and do the opt-out).  You also don't
need to run CI for Gentoo specific usecases I believe (a lot of options
for many packages).  No need to install and test Fedora, we'll do that.

So, I agree -- having higher coverage is better, and if there is something
to test, we can do it.  But we need to first discuss what can be changed.

> > You refused attempts to post patches which would make some code optional,
> > at which point it is not useful to think about testing something.
> >
> > Other than that, not everything is easily testable in your actual CI.
> >
> > > Pavel>_Open_ distribution¹
> > > Pavel> By FOSS source I mean software which
> > > Pavel>  _anybody_ can read, study, copy, modify, distribute
> > >
> > > Can you tell us if org.osgi.enterprise complies with your definition
> > > of "open distribution"?
> >
> > The license refuses you to modify the code, it is *clear vendor lock-in*
> > and:
>
>
> >   """
> >   anybody who cares a bit about open source principles should at least
> >   be aware of its consequences
> >   """
>
> This is a red-herring. The whole purpose of osgi is to provide a
> consistent interface to allow loading of software. If you were to modify
> it, it would fail to be relevant. Blindly applying OSS principles does
> not make sense here.

This is _huge untrue_, there is no need to use license for this.

Is the interface so perfect so it can not be made better?  In future?  I
think this is obvious -- Why you can't prepare compatible and better
interface as concurrent solution?  Why we need to wait for anybody?

Somebody just tries to protect its advantage.

Not modifiable code is vendor-lock-in;  and that is not good project to
hard-depend on.

> From what I can see they are provided simply to allow you to build packages
> just like this.

Are you sure this is not a crossing the law border?

I'm not going to do this because I haven't "signed the papers" on the web
page as discussed in [1] (probably the correct place to download the
sources from).  Downloading of that jar, using it and fixing FTBFS bugs in
that might be illegal.  So I can't do this..

> > You are allowed to download original sources, but you can't modify them.
> > I stopped the observation here as that is simply not acceptable -- this
> > has been discussed in mentioned threads, direct reference is [1]
> > (reference by Vitalii).
> >
>
> See above
>
> Dave Cramer
>
> davec@postgresintl.com
> www.postgresintl.com

Pavel>Not modifiable code is vendor-lock-in

org.osgi.enterprise jar is Apache 2.0-licensed.
Apache 2.0 allows modification of a source code. Surprise.

Pavel>I our case, it is IMO no need to test the potentially opt-outed feature,

You claim to "invent common build denominator feature", then you claim
"there's no need to test it".
Are you kidding?

As per Dave's words: "can you explain why packaging can't be tested"?
"no need" != "can't" as far as I can understand.
I think package testing should be rather simple.

Pavel> It is not needed to check in upstream
Pavel>that the opt-out feature works

You seem to ignore the main aim of testing. The tests are there to
catch unintentional changes.
If no tests added, any innocent refactoring might break your packaging script.

Vladimir

On Tuesday 08 of March 2016 14:37:56 Vladimir Sitnikov wrote:
> Pavel>Not modifiable code is vendor-lock-in
>
> org.osgi.enterprise jar is Apache 2.0-licensed.
> Apache 2.0 allows modification of a source code. Surprise.

It is actually not correct, as far as I am aware.  The links you provided
so far are results of official build, which puts the sources there by
default.  Can we convince upstream to release official tarballs without
the "signature needed" request that disallows you to modify?

> Pavel>I our case, it is IMO no need to test the potentially opt-outed
> Pavel>feature,
>
> You claim to "invent common build denominator feature", then you claim
> "there's no need to test it".
> Are you kidding?

Yes, here is your excuse :) I talked about, or?

You test your full-feature-set you support ATM.  That is fine, I do not
plan to stop testing something.

Some people (not you but me!) do want to disable something for themselves,
what exactly do you want to test on it?
But we can probably simulate the situation for you -- we can always do two
builds in upstream CI -- with/without the feature, even though you support
only the first scenario.  Do I understand it right this is wanted?

> As per Dave's words: "can you explain why packaging can't be tested"?
> "no need" != "can't" as far as I can understand.
> I think package testing should be rather simple.

It is tested.  I don't think you want it upstream as you don't support
packages we re-distribute.

> Pavel> It is not needed to check in upstream
> Pavel>that the opt-out feature works
>
> You seem to ignore the main aim of testing. The tests are there to
> catch unintentional changes.

No.  I appreciate testing, be fair please.

You know that there are tests _already_;  that would catch the issues we
could potentially add into existing level of your support ... so we can
fix the patches we propose to not do that.

I just say -- don't test that the '--opt-out-waffle' and '--opt-no-osgi',
whatever format it will have.  Then you know that "nobody" except for
packagers use those -- and the guys know how to fix this if something
wrong happens...

> If no tests added, any innocent refactoring might break your packaging
> script.

And, again -- don't be afraid here, that is why we are here.  You can not
provide packaging scripts for the whole world, it is not upstream
responsibility and no upstream does this.

Pavel

On 03/08/2016 10:33 AM, Pavel Raiskup wrote:
> On Monday 07 of March 2016 18:31:41 Vladimir Sitnikov wrote:
>
>> Pavel> even patches from us to support pure open source build are not wanted
>>
>> I'm afraid this ^^ is misleading.
>>
>> Patches are welcome provided they include tests to cover the change.
>> No tests -> no acceptance. It is in line with typical development
>> model, isn't it?
> This is *your* easy excuse to not incorporate change ;).  100% coverage is
> a sci-fi, as anybody must agree.  What am I going to test on Fedora if I
> patch osgi out?  Am I going to test that it does not work?  If yes, I'm
> fine to write the patch.
>
> You refused attempts to post patches which would make some code optional,
> at which point it is not useful to think about testing something.
>
> Other than that, not everything is easily testable in your actual CI.

You hit my pet peeve here. We are not animals. Intelligent people do TDD,
and clean code. The rest should write unit tests.
TDD and clean code are the rules of software development as a profession.
Simply because they are the only known practice to avoid fscking up the
code.

Of course there are cases which cannot be tested, but you have to provide
a reason at least.

On Tuesday 08 of March 2016 13:29:13 Árpád Magosányi wrote:
> On 03/08/2016 10:33 AM, Pavel Raiskup wrote:
> > On Monday 07 of March 2016 18:31:41 Vladimir Sitnikov wrote:
> >
> >> Pavel> even patches from us to support pure open source build are not wanted
> >>
> >> I'm afraid this ^^ is misleading.
> >>
> >> Patches are welcome provided they include tests to cover the change.
> >> No tests -> no acceptance. It is in line with typical development
> >> model, isn't it?
> >
> > This is *your* easy excuse to not incorporate change ;).  100% coverage is
> > a sci-fi, as anybody must agree.  What am I going to test on Fedora if I
> > patch osgi out?  Am I going to test that it does not work?  If yes, I'm
> > fine to write the patch.
> >
> > You refused attempts to post patches which would make some code optional,
> > at which point it is not useful to think about testing something.
> >
> > Other than that, not everything is easily testable in your actual CI.
>
> You hit my pet peeve here. We are not animals. Intelligent people do TDD,
> and clean code. The rest should write unit tests.

Please, what is this about?  Feel free to be constructive, but I just try
to convince someone that we need to opt-out something.  If we had 12
optional features, that is 2^12 possible tests, which is sci-fi.

I'm not telling to not do TDD, I'm not offensive.  I do not tell that
people who do-write-unit-tests are animals or not-intelligent, neither I
divide people like that.  I appreciate any work done the open source way.

Feel free to look at the comments Vladimir posted before, you'll see what
I mean by "no test, no acceptance".  There was no space for discussion at
that time.

> TDD and clean code are the rules of software development as a
> profession.  Simply because they are the only known practice to avoid
> fscking up the code.

Agreed, but it is orthogonal issue here.

> Of course there are cases which cannot be tested, but you have to provide
> a reason at least.

Thanks, we all agree on this.

On Tuesday 08 of March 2016 12:27:32 Vitalii Tymchyshyn wrote:
> As for me, the problem is not about JDBC driver, the problem is with FOSS
> world not able to deal with OSGI recently.

Then the actual status works for you.  It is completely fine, but there
are other who do care and who prefer different ideas, and there should be
a way to lock ourselves somewhere.

> It neither starts nor stops with PgJDBC.

Truth.

> And for me it's not a reason to make some useful feature optional
> complicating things for a user (instead of per-JDK jar you are starting
> to produce at least twice as many jars to deal with).

I need to react because of this ^.  PostgreSQL server has a lot of
features available during build, try './configure --help'.  You can choose
what features are important for you, and what not.

The opt-out (unused by upstream probably) in pgjdbc I would like to see is
far from being that sophisticated.  What is the problem with this?

> And it's not a technical limitation - current jars work perfectly in
> non-OSGI environment and runtime dependency resolution is the way java
> world tries to live.

Note the whole Java world probably.  I, for example, can consider myself
as somebody somebody who needs to write some gui app in Java, than
distribute that as native package for distributions.  More!  I want the
distribution guys do it for myself.  But I can't use jdbc, because it is
not provided natively -- and I can't thus use Java, because I can't
connect to PostgreSQL.

> Guys, you really need to decide with OSG in general. How are you going
> to deal with modern versions of containers like JBoss that has OSGi as a
> core feature? That is the problem.

JBoss is not Fedora.  We can't pull everything into Fedora just because we
want to connect 'A with B' where both 'A' and 'B' are already there and we
are fine with that.  To connect 'A' with 'B' is needed 'C (osgi support)'
and 'D (waffle)', where we are not able to provide 'D' and 'C' is not
needed atm (somebody tried to deal with that in Y2012, and we did not move
forward yet in 2016).  We need solution now, and we can enable things
later.

> You used to use API files from Felix, but they dont produce fresh
> versions anymore, possiby after switching to the official API jars.
> Contact them to check it.

I'll discuss this locally and report back, but it is not related ATM.
Some features are naturally expected to be optional, and that is why I'm
here.

Fedora is where it is in this case and I'm unfortunately unable to change
that (I'm not going to provide and support waffle and osgi just to provide
jdbc).

Pavel

> Best regards, Vitalii Tymchyshyn
>
> Вт, 8 бер. 2016 07:08 користувач Pavel Raiskup <praiskup@redhat.com> пише:
>
> > On Tuesday 08 of March 2016 14:37:56 Vladimir Sitnikov wrote:
> > > Pavel>Not modifiable code is vendor-lock-in
> > >
> > > org.osgi.enterprise jar is Apache 2.0-licensed.
> > > Apache 2.0 allows modification of a source code. Surprise.
> >
> > It is actually not correct, as far as I am aware.  The links you provided
> > so far are results of official build, which puts the sources there by
> > default.  Can we convince upstream to release official tarballs without
> > the "signature needed" request that disallows you to modify?
> >
> > > Pavel>I our case, it is IMO no need to test the potentially opt-outed
> > > Pavel>feature,
> > >
> > > You claim to "invent common build denominator feature", then you claim
> > > "there's no need to test it".
> > > Are you kidding?
> >
> > Yes, here is your excuse :) I talked about, or?
> >
> > You test your full-feature-set you support ATM.  That is fine, I do not
> > plan to stop testing something.
> >
> > Some people (not you but me!) do want to disable something for themselves,
> > what exactly do you want to test on it?
> > But we can probably simulate the situation for you -- we can always do two
> > builds in upstream CI -- with/without the feature, even though you support
> > only the first scenario.  Do I understand it right this is wanted?
> >
> > > As per Dave's words: "can you explain why packaging can't be tested"?
> > > "no need" != "can't" as far as I can understand.
> > > I think package testing should be rather simple.
> >
> > It is tested.  I don't think you want it upstream as you don't support
> > packages we re-distribute.
> >
> > > Pavel> It is not needed to check in upstream
> > > Pavel>that the opt-out feature works
> > >
> > > You seem to ignore the main aim of testing. The tests are there to
> > > catch unintentional changes.
> >
> > No.  I appreciate testing, be fair please.
> >
> > You know that there are tests _already_;  that would catch the issues we
> > could potentially add into existing level of your support ... so we can
> > fix the patches we propose to not do that.
> >
> > I just say -- don't test that the '--opt-out-waffle' and '--opt-no-osgi',
> > whatever format it will have.  Then you know that "nobody" except for
> > packagers use those -- and the guys know how to fix this if something
> > wrong happens...
> >
> > > If no tests added, any innocent refactoring might break your packaging
> > > script.
> >
> > And, again -- don't be afraid here, that is why we are here.  You can not
> > provide packaging scripts for the whole world, it is not upstream
> > responsibility and no upstream does this.
> >
> > Pavel
> >
> >
> >
> > --
> > Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-jdbc
> >

Hi Craig and Dave,

On Thursday 10 of March 2016 12:19:33 Craig Ringer wrote:
> So ... any interest in doing this via Maven profiles?
>
> Dave and I agree that that's the way forward as far as PgJDBC goes. It'd
> let you avoid patching PgJDBC sources in the rpm spec and having to
> maintain that spec patch, while still building without needing things not
> packed in Fedora.
>
> Thoughts, comments?

thanks for all your good comments!  This is definitely the preferred way
to go, Allow me some time to process this, I'll update this thread as soon
as possible.

Pavel

Hi Craig and Dave,

On Thursday 10 of March 2016 12:19:33 Craig Ringer wrote:
> So ... any interest in doing this via Maven profiles?
>
> Dave and I agree that that's the way forward as far as PgJDBC goes. It'd
> let you avoid patching PgJDBC sources in the rpm spec and having to
> maintain that spec patch, while still building without needing things not
> packed in Fedora.
>
> Thoughts, comments?

thanks for all your good comments!  This is definitely the preferred way
to go, Allow me some time to process this, I'll update this thread as soon
as possible.

Pavel

On Thursday 10 of March 2016 22:08:18 Craig Ringer wrote:
> > thanks for all your good comments!  This is definitely the preferred way
> > to go, Allow me some time to process this, I'll update this thread as soon
> > as possible.
>
> Cool.
>
> It might be a good idea to speak to maintainers of some of the Maven-based
> Fedora packages, as they'll probably be able to offer some hints/guidance
> on the details when it comes to creating profiles, conditionally compiling
> files, etc.
>
> I haven't verified that you can use maven profiles to avoid the OSGi stuff,
> but it'll certainly do the job for Waffle-JNA and the couple of isolated
> files in PgJDBC that use it.

I'm getting lost TBH, it all sounds like some magic to me (taking into
account that I want to do the thing the right way, which means not
breaking actual code and conditionally allow build on offline linux).

There are two WIP testing projects/branches I prepared with kind help of
our Java guys:

  https://github.com/praiskup/pgjdbc-parent-poms/commits/pgjdbc-linux-fixes
  https://github.com/praiskup/pgjdbc/commits/pgjdbc-linux-fixes

Note that the patches are not final and are somehow destructive, which is
definitely not on schedule for upstream proposal -- I just need to present
where I feel blocked.

When using those changes -- after the parent-poms is built and installed
into ~/.m2, after some dependencies for osgi are installed from maven (I'm
not yet ready to fully avoid osgi code - starting with waffle as you said)
I'm able to "start" compiling with:

  $ python3 /usr/share/java-utils/mvn_build.py \
      -- -DwaffleEnabled=false -Dmaven.test.skip=true

Which is equivalent to:

  $ xmvn --batch-mode --offline -DwaffleEnabled=false \
    -Dmaven.test.skip=true verify org.fedoraproject.xmvn:xmvn-mojo:install \
    org.apache.maven.plugins:maven-javadoc-plugin:aggregate \
    org.fedoraproject.xmvn:xmvn-mojo:builddep

This fails with:
[ERROR] Failed to execute goal com.igormaznitsa:jcp:6.0.1:preprocess
(preprocessSources) on project postgresql: Can't find a source directory
[/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]
-> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed
to execute goal com.igormaznitsa:jcp:6.0.1:preprocess (preprocessSources)
on project postgresql: Can't find a source directory
[/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]

The error log is attached.  Seems like something wrong is with
preprocessor.  Is anybody able to help a bit?  It is quite exhausting and
I start to think that, if we want to have it upstream, we need to have a
shell script or Makefile (that would be is more flexible) :( and I'm not
sure whether that will be friendly with igormaznitsa:jcp thing.
Thoughts?

Thanks,
Pavel

> Caused by: java.lang.IllegalStateException: Can't find a source directory
[/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]

Does the folder in question exists?
Does the build succeed with regular maven?
Vladimir

On Tuesday 22 of March 2016 15:48:26 Vladimir Sitnikov wrote:
> Does the folder in question exists?
> Does the build succeed with regular maven?

No and yes.

Pavel

On Thursday 10 of March 2016 22:08:18 Craig Ringer wrote:
> > thanks for all your good comments!  This is definitely the preferred way
> > to go, Allow me some time to process this, I'll update this thread as soon
> > as possible.
>
> Cool.
>
> It might be a good idea to speak to maintainers of some of the Maven-based
> Fedora packages, as they'll probably be able to offer some hints/guidance
> on the details when it comes to creating profiles, conditionally compiling
> files, etc.
>
> I haven't verified that you can use maven profiles to avoid the OSGi stuff,
> but it'll certainly do the job for Waffle-JNA and the couple of isolated
> files in PgJDBC that use it.

I'm getting lost TBH, it all sounds like some magic to me (taking into
account that I want to do the thing the right way, which means not
breaking actual code and conditionally allow build on offline linux).

There are two WIP testing projects/branches I prepared with kind help of
our Java guys:

  https://github.com/praiskup/pgjdbc-parent-poms/commits/pgjdbc-linux-fixes
  https://github.com/praiskup/pgjdbc/commits/pgjdbc-linux-fixes

Note that the patches are not final and are somehow destructive, which is
definitely not on schedule for upstream proposal -- I just need to present
where I feel blocked.

When using those changes -- after the parent-poms is built and installed
into ~/.m2, after some dependencies for osgi are installed from maven (I'm
not yet ready to fully avoid osgi code - starting with waffle as you said)
I'm able to "start" compiling with:

  $ python3 /usr/share/java-utils/mvn_build.py \
      -- -DwaffleEnabled=false -Dmaven.test.skip=true

Which is equivalent to:

  $ xmvn --batch-mode --offline -DwaffleEnabled=false \
    -Dmaven.test.skip=true verify org.fedoraproject.xmvn:xmvn-mojo:install \
    org.apache.maven.plugins:maven-javadoc-plugin:aggregate \
    org.fedoraproject.xmvn:xmvn-mojo:builddep

This fails with:
[ERROR] Failed to execute goal com.igormaznitsa:jcp:6.0.1:preprocess
(preprocessSources) on project postgresql: Can't find a source directory
[/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]
-> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed
to execute goal com.igormaznitsa:jcp:6.0.1:preprocess (preprocessSources)
on project postgresql: Can't find a source directory
[/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]

The error log is attached.  Seems like something wrong is with
preprocessor.  Is anybody able to help a bit?  It is quite exhausting and
I start to think that, if we want to have it upstream, we need to have a
shell script or Makefile (that would be is more flexible) :( and I'm not
sure whether that will be friendly with igormaznitsa:jcp thing.
Thoughts?

Thanks,
Pavel

> Caused by: java.lang.IllegalStateException: Can't find a source directory
[/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]

Does the folder in question exists?
Does the build succeed with regular maven?
Vladimir

On Tuesday 22 of March 2016 15:48:26 Vladimir Sitnikov wrote:
> Does the folder in question exists?
> Does the build succeed with regular maven?

No and yes.

Pavel

On Friday 01 of April 2016 10:11:11 Craig Ringer wrote:
> On 22 March 2016 at 19:48, Pavel Raiskup <praiskup@redhat.com> wrote:
>
>
> >   https://github.com/praiskup/pgjdbc/commits/pgjdbc-linux-fixes
>
>
> Thanks for proceeding with this. I hope you've been able to get some help
> from Maven folks within RH.

Yes.  But there is 0% experience with java-comment-preprocessor.

> I've made some comments on the commit. Most notably you're exclusing
> ISSPIClient.java from the build.

Fixed.

> Note that the patches are not final and are somehow destructive, which is
> > definitely not on schedule for upstream proposal -- I just need to present
> > where I feel blocked.
> >
>
> Totally fine, thanks for making the effort.
>
> > -DwaffleEnabled=false
>
> OK, so the profile activation of excludeWaffleFiles is controlled by
> the waffleEnabled property being false, in which case you exclude the SSPI
> files.
>
> I personally wonder if it should be an enabled-by-default profile that
> *includes* SSPI files and dependencies, which you can disable to turn SSPI
> support off. But that shouldn't really matter.

Then you need to have by-default-excluded files, which are then
by-default-included-by-default-profile.  I can change that, but it looks
like just matter of taste and actual status looks easy enough.

> I suspect that what's happening here is that when the excludeWaffleFiles
> profile is activated, other profiles get deactivated - specifically,
> activeByDefault profiles. Per the docs (
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html):

I don't think this is the case -- what default profiles could be disabled
unintentionally, or what makes you think so (my maven foo is so weak, I
can miss something obvious)?  I'm not able to control jcp's profiles
explicitly while building the pgjdbc project anyway..

Based on the log, it rather looks like the preprocessor simply ignores the
excpludes we do by maven-compiler-plugin?

> "This [<activeByDefault/>] profile will automatically be active for all
> builds unless another profile in the same POM is activated using one of the
> previously described methods. All profiles that are active by default are
> automatically deactivated when a profile in the POM is activated on the
> command line or through its activation config."

I'm aware of that -- this basically forced me to use properties for
enabling/disabling profiles (one property can trigger more profiles a
straight forward way).

> You can use "mvn help:active-profiles  --batch-mode --offline
> -DwaffleEnabled=false" to see which profiles are activated and compare that
> to "mvn help:active-profiles --batch-mode --offline" (without the extra
> system property).

This, for some reason, did not work for me, I'll have a look once more.

> I'm afraid my Maven's very rusty these days and I wasn't involved in the
> final PgJDBC ant-to-maven conversion, though I agreed it was a good idea.
> I'll try to help.

Thanks!

> Starting with:
>
> * http://stackoverflow.com/q/5309379/398670
> * http://greyfocus.com/2015/06/activebydefault-maven-other-profiles/
>
> You could use an <activeProfiles/> in ~/.m2/settings.xml, an explicit -P
> argument on the command line, or make the profile activated dependent on
> the absence of a property that we never set so it's always active.
>
> I have to say that I've always thought the way <activeByDefault/> has no
> option to suppress automatic deactivation when another profile is activated
> is a real wart. You should be able to declare a profile that's always
> active unless explicitly deactivated by a negated argument to -P more
> cleanly. As it is you need a hacky workaround.
>
> IIRC in my own projects I used to avoid activeByDefault profiles and I'd
> specify profile-lists in different settings.xml configurations. While
> getting quite annoyed.
>
> http://stackoverflow.com/a/2248552/398670 and
> http://stackoverflow.com/q/2246033/398670 explain in more detail. IMO it's
> a real wart in the build system. You're getting bitten by it because you're
> adding build profiles to a codebase that didn't use them before.

I believe I've read the links before, and the problems you mentioning here
is different, but the status is that this does not help :(.

> When using those changes -- after the parent-poms is built and installed
> > into ~/.m2, after some dependencies for osgi are installed from maven (I'm
> > not yet ready to fully avoid osgi code - starting with waffle as you said)
> >
>
> Sensible. TBH I suspect  RH will need to get OSGi into RH proper soon
> anyway, since it is IIRC needed for major things like Glassfish, JBoss AS
> 7, etc.

Ok, I need to ping once more on this, while still trying to implement the
opt-out trigger.

> > This fails with:
> > [ERROR] Failed to execute goal com.igormaznitsa:jcp:6.0.1:preprocess
> > (preprocessSources) on project postgresql: Can't find a source directory
> >
> > [/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]
> >
>
> Looks like the annotation preprocessor didn't run so there's no input,
> suggesting it's possibly controlled by an <activeByDefault/> profile.
> Hopefully not in a 3rd party POM...

Well, I was able to run the testsuite successfully -- so I suspect that
the preprocessor runs fine.  And see below ...

> The error log is attached.  Seems like something wrong is with
> > preprocessor.
>
>
> > It is quite exhausting and
> > I start to think that, if we want to have it upstream, we need to have a
> > shell script or Makefile (that would be is more flexible)
>
>
> It's not more flexible, it's just more *familiar*. When I moved from Java
> to C dev I used to think "gah, I wish I could just do this with maven, this
> makefile is AWFUL".
>
> Maven's pretty much arbitrarily programmable. The problem is that it's done
> in an undeniably awkward way, where complicated stuff requires the
> implementation of a maven plugin to do what you want. I doubt that's
> necessary in this case though. Thankfully.
>
> I do think maven has some massive warts, but I've spent long enough
> swearing at makefiles, at CMake, at SCons (back in the day), at Ant and at
> autoconf to think that all build systems are pretty much awful. It's just
> you're more used to make's awfulness.

I believe you are right here (that maven is equivalent to shell+makefile),
but I'm not yet that good with maven to tell.

> > sure whether that will be friendly with igormaznitsa:jcp thing.
> >
>
> I wasn't aware that the Maven build conversion of PgJDBC used jcp.  I'm a
> little surprised it was necessary, but we do have to support multiple
> different JDBC spec versions, which creates some complexities.
>
> It's probably not going to be a major barrier. The issue you seem to have
> hit here is that the annotation processor didn't run so when when jcp tried
> to

Well, it look like the preprocessor just ignores our "exclude" statements,
I believe in that because if we simply remove all the java files related
to waffle and osgi, we are able to build the package (== the preprocessor
works).

The question is why jcp ignores the <excludes />.

Pavel

On Friday 01 of April 2016 10:11:11 Craig Ringer wrote:
> On 22 March 2016 at 19:48, Pavel Raiskup <praiskup@redhat.com> wrote:
>
>
> >   https://github.com/praiskup/pgjdbc/commits/pgjdbc-linux-fixes
>
>
> Thanks for proceeding with this. I hope you've been able to get some help
> from Maven folks within RH.

Yes.  But there is 0% experience with java-comment-preprocessor.

> I've made some comments on the commit. Most notably you're exclusing
> ISSPIClient.java from the build.

Fixed.

> Note that the patches are not final and are somehow destructive, which is
> > definitely not on schedule for upstream proposal -- I just need to present
> > where I feel blocked.
> >
>
> Totally fine, thanks for making the effort.
>
> > -DwaffleEnabled=false
>
> OK, so the profile activation of excludeWaffleFiles is controlled by
> the waffleEnabled property being false, in which case you exclude the SSPI
> files.
>
> I personally wonder if it should be an enabled-by-default profile that
> *includes* SSPI files and dependencies, which you can disable to turn SSPI
> support off. But that shouldn't really matter.

Then you need to have by-default-excluded files, which are then
by-default-included-by-default-profile.  I can change that, but it looks
like just matter of taste and actual status looks easy enough.

> I suspect that what's happening here is that when the excludeWaffleFiles
> profile is activated, other profiles get deactivated - specifically,
> activeByDefault profiles. Per the docs (
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html):

I don't think this is the case -- what default profiles could be disabled
unintentionally, or what makes you think so (my maven foo is so weak, I
can miss something obvious)?  I'm not able to control jcp's profiles
explicitly while building the pgjdbc project anyway..

Based on the log, it rather looks like the preprocessor simply ignores the
excpludes we do by maven-compiler-plugin?

> "This [<activeByDefault/>] profile will automatically be active for all
> builds unless another profile in the same POM is activated using one of the
> previously described methods. All profiles that are active by default are
> automatically deactivated when a profile in the POM is activated on the
> command line or through its activation config."

I'm aware of that -- this basically forced me to use properties for
enabling/disabling profiles (one property can trigger more profiles a
straight forward way).

> You can use "mvn help:active-profiles  --batch-mode --offline
> -DwaffleEnabled=false" to see which profiles are activated and compare that
> to "mvn help:active-profiles --batch-mode --offline" (without the extra
> system property).

This, for some reason, did not work for me, I'll have a look once more.

> I'm afraid my Maven's very rusty these days and I wasn't involved in the
> final PgJDBC ant-to-maven conversion, though I agreed it was a good idea.
> I'll try to help.

Thanks!

> Starting with:
>
> * http://stackoverflow.com/q/5309379/398670
> * http://greyfocus.com/2015/06/activebydefault-maven-other-profiles/
>
> You could use an <activeProfiles/> in ~/.m2/settings.xml, an explicit -P
> argument on the command line, or make the profile activated dependent on
> the absence of a property that we never set so it's always active.
>
> I have to say that I've always thought the way <activeByDefault/> has no
> option to suppress automatic deactivation when another profile is activated
> is a real wart. You should be able to declare a profile that's always
> active unless explicitly deactivated by a negated argument to -P more
> cleanly. As it is you need a hacky workaround.
>
> IIRC in my own projects I used to avoid activeByDefault profiles and I'd
> specify profile-lists in different settings.xml configurations. While
> getting quite annoyed.
>
> http://stackoverflow.com/a/2248552/398670 and
> http://stackoverflow.com/q/2246033/398670 explain in more detail. IMO it's
> a real wart in the build system. You're getting bitten by it because you're
> adding build profiles to a codebase that didn't use them before.

I believe I've read the links before, and the problems you mentioning here
is different, but the status is that this does not help :(.

> When using those changes -- after the parent-poms is built and installed
> > into ~/.m2, after some dependencies for osgi are installed from maven (I'm
> > not yet ready to fully avoid osgi code - starting with waffle as you said)
> >
>
> Sensible. TBH I suspect  RH will need to get OSGi into RH proper soon
> anyway, since it is IIRC needed for major things like Glassfish, JBoss AS
> 7, etc.

Ok, I need to ping once more on this, while still trying to implement the
opt-out trigger.

> > This fails with:
> > [ERROR] Failed to execute goal com.igormaznitsa:jcp:6.0.1:preprocess
> > (preprocessSources) on project postgresql: Can't find a source directory
> >
> > [/home/praiskup/rh/projects/pgjdbc/pgjdbc/target/generated-sources/annotations]
> >
>
> Looks like the annotation preprocessor didn't run so there's no input,
> suggesting it's possibly controlled by an <activeByDefault/> profile.
> Hopefully not in a 3rd party POM...

Well, I was able to run the testsuite successfully -- so I suspect that
the preprocessor runs fine.  And see below ...

> The error log is attached.  Seems like something wrong is with
> > preprocessor.
>
>
> > It is quite exhausting and
> > I start to think that, if we want to have it upstream, we need to have a
> > shell script or Makefile (that would be is more flexible)
>
>
> It's not more flexible, it's just more *familiar*. When I moved from Java
> to C dev I used to think "gah, I wish I could just do this with maven, this
> makefile is AWFUL".
>
> Maven's pretty much arbitrarily programmable. The problem is that it's done
> in an undeniably awkward way, where complicated stuff requires the
> implementation of a maven plugin to do what you want. I doubt that's
> necessary in this case though. Thankfully.
>
> I do think maven has some massive warts, but I've spent long enough
> swearing at makefiles, at CMake, at SCons (back in the day), at Ant and at
> autoconf to think that all build systems are pretty much awful. It's just
> you're more used to make's awfulness.

I believe you are right here (that maven is equivalent to shell+makefile),
but I'm not yet that good with maven to tell.

> > sure whether that will be friendly with igormaznitsa:jcp thing.
> >
>
> I wasn't aware that the Maven build conversion of PgJDBC used jcp.  I'm a
> little surprised it was necessary, but we do have to support multiple
> different JDBC spec versions, which creates some complexities.
>
> It's probably not going to be a major barrier. The issue you seem to have
> hit here is that the annotation processor didn't run so when when jcp tried
> to

Well, it look like the preprocessor just ignores our "exclude" statements,
I believe in that because if we simply remove all the java files related
to waffle and osgi, we are able to build the package (== the preprocessor
works).

The question is why jcp ignores the <excludes />.

Pavel

Just to give a quick update ..

On Thursday 07 of April 2016 13:57:18 Pavel Raiskup wrote:
> Well, it look like the preprocessor just ignores our "exclude" statements,
> I believe in that because if we simply remove all the java files related
> to waffle and osgi, we are able to build the package (== the preprocessor
> works).

That does not seem to be truth, sorry for mystification.

It seems like jcp is not the bad guy here, but rather
maven-compiler-plugin of version 3.5 (it all works fine with
maven-compiler-plugin v3.3).

Let's wait for the bug [1] resolution (whatever it will be), after that we
can move to submitting PRs¹ -- so if there is anything I can help with
before that, please ping me.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1325060

Thanks, Pavel

¹ The solution would be in -D_FEATURE_Enabled=false options available
  during maven build, both available for pgjdbc-parent-poms and pgjdbc.

Just to give a quick update ..

On Thursday 07 of April 2016 13:57:18 Pavel Raiskup wrote:
> Well, it look like the preprocessor just ignores our "exclude" statements,
> I believe in that because if we simply remove all the java files related
> to waffle and osgi, we are able to build the package (== the preprocessor
> works).

That does not seem to be truth, sorry for mystification.

It seems like jcp is not the bad guy here, but rather
maven-compiler-plugin of version 3.5 (it all works fine with
maven-compiler-plugin v3.3).

Let's wait for the bug [1] resolution (whatever it will be), after that we
can move to submitting PRs¹ -- so if there is anything I can help with
before that, please ping me.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1325060

Thanks, Pavel

¹ The solution would be in -D_FEATURE_Enabled=false options available
  during maven build, both available for pgjdbc-parent-poms and pgjdbc.

On Friday 08 of April 2016 09:22:47 Pavel Raiskup wrote:
> Let's wait for the bug [1] resolution (whatever it will be), after that we
> can move to submitting PRs¹

Here are the proposed PRs:
https://github.com/pgjdbc/pgjdbc-parent-poms/pull/4
https://github.com/pgjdbc/pgjdbc/pull/546

With those patches backported downstream we were able to build pgjdbc,
without additional downstream patching:
https://copr.fedorainfracloud.org/coprs/g/db-sig/pgjdbc-migrate-to-maven/monitor/

Because we are able to build pgjdbc on Fedora, we could potentially use
Copr as CI tool for those patches.

Thanks for the review,
Pavel

On Friday 08 of April 2016 09:22:47 Pavel Raiskup wrote:
> Let's wait for the bug [1] resolution (whatever it will be), after that we
> can move to submitting PRs¹

Here are the proposed PRs:
https://github.com/pgjdbc/pgjdbc-parent-poms/pull/4
https://github.com/pgjdbc/pgjdbc/pull/546

With those patches backported downstream we were able to build pgjdbc,
without additional downstream patching:
https://copr.fedorainfracloud.org/coprs/g/db-sig/pgjdbc-migrate-to-maven/monitor/

Because we are able to build pgjdbc on Fedora, we could potentially use
Copr as CI tool for those patches.

Thanks for the review,
Pavel