Обсуждение: Wrong link not pointing to the release tarball

Hi all,

I'm sure this has been discussed somewhere (but I can't find where).
Rather sending new ticket -- the download page [1] has wrong link to
latest source tarball.  It points to '*.jar' file.

[1] https://jdbc.postgresql.org/download.html

Pavel

On Friday 22 of January 2016 07:32:50 Dave Cramer wrote:
> This is a standard maven artifact. Jar files can be opened using zip tools.
> If it is not usable in that state we can change it back although it adds an
> additional step

Ah, I thought that was not intentional.

Tarballs are the standard for releases of wast majority of FOSS projects
-- as the 'ustar' format is defined by POSIX (and is _really_ simple,
compared to zip) it is available probably everywhere.  Fore example,
PostgreSQL and Maven themselves are distributed in tarballs.

As I write you from GNU/Linux, I definitely prefer .tar.gz, but OTOH, this
is probably not burning if nobody else cares -- I'm just surprised.
I believe we are able to deal with jar/zip files.

Pavel

On Friday 22 of January 2016 14:20:20 Pavel Raiskup wrote:
> On Friday 22 of January 2016 07:32:50 Dave Cramer wrote:
> > This is a standard maven artifact. Jar files can be opened using zip tools.
> > If it is not usable in that state we can change it back although it adds an
> > additional step
>
> Ah, I thought that was not intentional.
>
> Tarballs are the standard for releases of wast majority of FOSS projects
> -- as the 'ustar' format is defined by POSIX (and is _really_ simple,
> compared to zip) it is available probably everywhere.  Fore example,
> PostgreSQL and Maven themselves are distributed in tarballs.
>
> As I write you from GNU/Linux, I definitely prefer .tar.gz, but OTOH, this
> is probably not burning if nobody else cares -- I'm just surprised.
> I believe we are able to deal with jar/zip files.

Actually, I'm afraid that the content of the zip file is not
self-standing .. but I'm not sure.

For example, there is the full 'org/postgresql/..' file hierarchy, which
does not match what is in git.  And also the pom.xml files are missing.

Is that expected?  What is the expected way to build from the [1] release
zip file?

[1] https://jdbc.postgresql.org/download/postgresql-9.4.1207-sources.jar

Pavel

In fact, there are two types of "source artifacts":
1) source.jar -- this one is for IDE integration and manual
inspection. It is not suitable for a build process.
2) *.tar.gz -- I've no idea why is it useful, however, it existed
before mavenization, thus it still exists.

I've no idea *.tar.gz artifact is suitable for build purposes.

Dave, I think Pavel is right in a sense that (Source, 9.4 Build 1207)
link points to postgresql-9.4.1207-sources.jar while it should point
to postgresql-9.4.1207-dist.tar.gz (like the previous versions).

Dave>The recommended way to build is to pull from github.

That is true. "build from github tarball" is tested better (it is used
for driver releases) than "build from *.tar.gz" (it is write-only).

Vladimir

On Friday 22 of January 2016 08:32:25 Dave Cramer wrote:
> As I said this is a maven artifact.

Thanks for explanation.

> We actually don't expect anyone to build from a tarball.

Than you can count us in :) we were always tarball consumers.  Building
from source is essential.  If you could provide source tarball, that would
be perfect not only for us.

> The recommended way to build is to pull from github. However I'm sensing
> that you have a requirement to build from a tarball?

No :) tarball is not strict requirement, basically any well-known format
works well.  The strict requirement is just to build from source which is
"signed-off-by" upstream.  By that I mean that it is clear that the
package comes from your hands (upstream).  Anybody should be able to get
that tarball without problems to check that I (as a packager) do not
cheat.

Pavel


> Dave Cramer
>
> davec@postgresintl.com
> www.postgresintl.com
>
> On 22 January 2016 at 08:26, Pavel Raiskup <praiskup@redhat.com> wrote:
>
> > On Friday 22 of January 2016 14:20:20 Pavel Raiskup wrote:
> > > On Friday 22 of January 2016 07:32:50 Dave Cramer wrote:
> > > > This is a standard maven artifact. Jar files can be opened using zip
> > tools.
> > > > If it is not usable in that state we can change it back although it
> > adds an
> > > > additional step
> > >
> > > Ah, I thought that was not intentional.
> > >
> > > Tarballs are the standard for releases of wast majority of FOSS projects
> > > -- as the 'ustar' format is defined by POSIX (and is _really_ simple,
> > > compared to zip) it is available probably everywhere.  Fore example,
> > > PostgreSQL and Maven themselves are distributed in tarballs.
> > >
> > > As I write you from GNU/Linux, I definitely prefer .tar.gz, but OTOH,
> > this
> > > is probably not burning if nobody else cares -- I'm just surprised.
> > > I believe we are able to deal with jar/zip files.
> >
> > Actually, I'm afraid that the content of the zip file is not
> > self-standing .. but I'm not sure.
> >
> > For example, there is the full 'org/postgresql/..' file hierarchy, which
> > does not match what is in git.  And also the pom.xml files are missing.
> >
> > Is that expected?  What is the expected way to build from the [1] release
> > zip file?
> >
> > [1] https://jdbc.postgresql.org/download/postgresql-9.4.1207-sources.jar
> >
> > Pavel
> >
> >

On Friday 22 of January 2016 16:47:15 Vladimir Sitnikov wrote:
> In fact, there are two types of "source artifacts":
> 1) source.jar -- this one is for IDE integration and manual
> inspection. It is not suitable for a build process.
> 2) *.tar.gz -- I've no idea why is it useful, however, it existed
> before mavenization, thus it still exists.

Thanks, good to hear!

> I've no idea *.tar.gz artifact is suitable for build purposes.

It definitely is suitable, that helps us to build jar file from source --
from beginning.

> Dave, I think Pavel is right in a sense that (Source, 9.4 Build 1207)
> link points to postgresql-9.4.1207-sources.jar while it should point
> to postgresql-9.4.1207-dist.tar.gz (like the previous versions).

If you could fix the link, that would make our life easier then.  I was
not able to find the link.

> Dave>The recommended way to build is to pull from github.
>
> That is true. "build from github tarball" is tested better (it is used
> for driver releases) than "build from *.tar.gz" (it is write-only).

Well, tarball got by `git archive` would be good enough, too.  But with
that we are not completely sure that the particular commit ID matches
exactly the version you released in pre-built binaries.

Thanks, Pavel

What I have seen while researching Fedora stuff is that some projects have '-project' additional files (same artifact, with different classifier) deployed to maven central in both zip and tgz format. As far as I can see, it's pretty simple to achieve with assembly plugin. It even has predefined 'project' descriptor that can be useful.

Best regards, Vitalii Tymchyshyn

Pavel> But with
Pavel>that we are not completely sure that the particular commit ID matches
Pavel>exactly the version you released in pre-built binaries.

Technically speaking, maven creates a git tag (e.g. REL9.4.1207),
then it checks out git tree of that particular commit ID (from github!)
to a clean folder, then it builds the driver, then the results are published.

As long as we use maven, release tag will point to the commit ID
that is used for the release.

Our current *.tar.gz is closer to maven's  "bin.tar.gz". In other words,
our tar.gz is for the case when all artifacts like source jar, release jar,
dependencies, etc, are required at once.

Vladimir

On Friday 22 of January 2016 17:05:38 Vladimir Sitnikov wrote:
> Pavel> But with
> Pavel>that we are not completely sure that the particular commit ID matches
> Pavel>exactly the version you released in pre-built binaries.
>
> Technically speaking, maven creates a git tag (e.g. REL9.4.1207), then
> it checks out git tree of that particular commit ID (from github!) to a
> clean folder, then it builds the driver, then the results are published.
>
> As long as we use maven, release tag will point to the commit ID
> that is used for the release.

This sounds fine, the only bottleneck I can think of is that people among
distributions used to have "that" tarball (concrete name, structure, etc.)
and to switch from the (very) old version to the REL9.4.1208 they will
need to change a bit more things.

I'm OK to fix our spec file (build "declarative" "script") [1] to follow
new format, and those who have not raised their hands yet will need to do
it post-mortem.  Could we fix the link coming from  download.html to the
tarball -- to clearly say what source linux distribution should use?

> Our current *.tar.gz is closer to maven's  "bin.tar.gz". In other words,
> our tar.gz is for the case when all artifacts like source jar, release jar,
> dependencies, etc, are required at once.

Might be, that bin.tar.gz is not known to me :(.

[1] http://pkgs.fedoraproject.org/cgit/rpms/postgresql-jdbc.git/tree/postgresql-jdbc.spec

Pavel

>[1] http://pkgs.fedoraproject.org/cgit/rpms/postgresql-jdbc.git/tree/postgresql-jdbc.spec

>37:  Version: 9.4.%{upstreamrel}

It should be %{upstreamver}, shouldn't it?

># ASL 2.0 applies only to postgresql-jdbc.pom file, the rest is BSD
>License: BSD and ASL 2.0

That is strange. Which "postgresql-jdbc.pom" is referred to here?

> 57: BuildRequires: ant
> 58: BuildRequires: ant-junit

I do not think we require ant.

>85: find -name "*.jar" -or -name "*.class" | xargs rm -f

We have *.class files for a good reason: they include gettext translations.
If you absolutely require to kill that, you need to add "mvn
-Ptranslate compile" step to perform "gettext" step.
However, that would void "built from the same git source" statement.

>111: ln -s postgresql-jdbc.jar postgresql-jdbc2.jar

That is somewhat risky, since the new jar is not usable for old JRE.
In other words, if a software exists that uses Java 5 and jdbc2.jar,
then newly build jar is not compatible with it.

In order to build jre7, and jre6-compatible jars, separate build steps
should be used.
We have https://github.com/pgjdbc/pgjdbc-jre7 & jre6 for that.

Vladimir

On Friday 22 of January 2016 17:30:59 Vladimir Sitnikov wrote:
> >[1] http://pkgs.fedoraproject.org/cgit/rpms/postgresql-jdbc.git/tree/postgresql-jdbc.spec
>
> >37:  Version: 9.4.%{upstreamrel}
>
> It should be %{upstreamver}, shouldn't it?

There used to be typo -- dot vs. dash, in RPM world, you can not have dash
within "Release" tag.  You have *strict* NAME-VERSION-RELEASE, while the
only component which can contain dash is NAME.  Based on this principle is
built most of the stuff in RPM distributions.  Can here

> ># ASL 2.0 applies only to postgresql-jdbc.pom file, the rest is BSD
> >License: BSD and ASL 2.0
>
> That is strange. Which "postgresql-jdbc.pom" is referred to here?

That one which lies right along the spec file I pointed you at.  That pom
file was created in Y2009 by Tom Lane, because we needed to have "some"
pom file (at least as far as I understand, I'm not package that long ;)).
We do not change things if that is not necessary, so there we are...

> > 57: BuildRequires: ant
> > 58: BuildRequires: ant-junit
>
> I do not think we require ant.

We (Fedora) still do.  There are hacks even in the '*.spec' file to make
the package built .. unfortunately.  We were not able to build with maven,
and other distros seem to be even behind use.

> >85: find -name "*.jar" -or -name "*.class" | xargs rm -f
>
> We have *.class files for a good reason: they include gettext translations.
> If you absolutely require to kill that, you need to add "mvn
> -Ptranslate compile" step to perform "gettext" step.
> However, that would void "built from the same git source" statement.

This needs to be done, really.  That is for our users -- we can prove them
that no precompiled code is used, because that is very often issue of Java
package upstreams.  I would reconsider the need of *any* pre-build files
within _source_ tarball -- because precompiled java files are not sources
anymore.

> >111: ln -s postgresql-jdbc.jar postgresql-jdbc2.jar
>
> That is somewhat risky, since the new jar is not usable for old JRE.
> In other words, if a software exists that uses Java 5 and jdbc2.jar,
> then newly build jar is not compatible with it.
>
> In order to build jre7, and jre6-compatible jars, separate build steps
> should be used.
> We have https://github.com/pgjdbc/pgjdbc-jre7 & jre6 for that.

Argh, OK.  I do not see that deep to see the runtime differences -- but I
bet we should remove the compatibility symlinks, right?  As nobody
complained so far I bet nobody uses them.

Vladmir, thanks for the review.  I do not object, its far from perfect --
it worked with very old release tarball;  and we are off topic.  I've seen
Dave fixed the link, so I bet you'll keep that format in future releases
too, which is *perfect*.

Thanks guys,
Pavel

Pavel>but I bet we should remove the compatibility symlinks, right?

I think so.

Pavel> I've seen Dave fixed the link

What I do not like is the link publish is manual process, thus it is
prone to errors.
We should either automate it, or prefer github's tar.gz

Vladimir

On 1/22/2016 5:32 AM, Dave Cramer wrote:
> As I said this is a maven artifact. We actually don't expect anyone to
> build from a tarball.
>
> The recommended way to build is to pull from github. However I'm
> sensing that you have a requirement to build from a tarball?
>

noone packaging for a released OS (using rpm, deb, etc) is going to want
to pull from git, they are going to want a versioned tarball of the
source that represents the precise version they are building so each
time its built they get exactly the same thing.

--
john r pierce, recycling bits in santa cruz

John>noone packaging for a released OS (using rpm, deb, etc) is going
to want to pull from git

I'm afraid you are missing the point.
Here's the URL I recommend:
https://github.com/pgjdbc/pgjdbc/archive/REL9.4.1207.tar.gz
It is a HTTPS URL of a file that never changes. That exact contents
was used to build pgjdbc 9.4.1207.

Alternatively, you might use a URL with commit ID (see [1]). Commit id
is somewhat harder to identify, but it is protected from tampering
(git tag can be recreated with different contents, but the contents of
a particular git commit id would be always the same).

What's the problem with using those kind of links?

[1]: https://github.com/pgjdbc/pgjdbc/archive/f17e2f3074a95dc767d6d896ffd696e045b48466.tar.gz

Vladimir

On 1/22/2016 11:19 AM, Vladimir Sitnikov wrote:
> John>noone packaging for a released OS (using rpm, deb, etc) is going
> to want to pull from git
>
> I'm afraid you are missing the point.
> Here's the URL I recommend:
> https://github.com/pgjdbc/pgjdbc/archive/REL9.4.1207.tar.gz
> It is a HTTPS URL of a file that never changes. That exact contents
> was used to build pgjdbc 9.4.1207.

ok, thats fine.    sorry, I thought you were referring to pulling the
whole source out of git.



--
john r pierce, recycling bits in santa cruz

On Jan 22, 2016 11:19 AM, "Vladimir Sitnikov" <sitnikov.vladimir@gmail.com> wrote:
>
> John>noone packaging for a released OS (using rpm, deb, etc) is going
> to want to pull from git
>
> I'm afraid you are missing the point.
> Here's the URL I recommend:
> https://github.com/pgjdbc/pgjdbc/archive/REL9.4.1207.tar.gz
> It is a HTTPS URL of a file that never changes. That exact contents
> was used to build pgjdbc 9.4.1207.
>
> Alternatively, you might use a URL with commit ID (see [1]). Commit id
> is somewhat harder to identify, but it is protected from tampering
> (git tag can be recreated with different contents, but the contents of
> a particular git commit id would be always the same).
>
> What's the problem with using those kind of links?
>
> [1]: https://github.com/pgjdbc/pgjdbc/archive/f17e2f3074a95dc767d6d896ffd696e045b48466.tar.gz
>

Oh this is way easier. . Is this acceptable
> Vladimir
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc

On Jan 22, 2016 11:26 AM, "John R Pierce" <pierce@hogranch.com> wrote:
>
> On 1/22/2016 11:19 AM, Vladimir Sitnikov wrote:
>>
>> John>noone packaging for a released OS (using rpm, deb, etc) is going
>> to want to pull from git
>>
>> I'm afraid you are missing the point.
>> Here's the URL I recommend:
>> https://github.com/pgjdbc/pgjdbc/archive/REL9.4.1207.tar.gz
>> It is a HTTPS URL of a file that never changes. That exact contents
>> was used to build pgjdbc 9.4.1207.
>
>
> ok, thats fine.    sorry, I thought you were referring to pulling the whole source out of git.
>
>

That is effectively what is contemplated here
>
>
> --
> john r pierce, recycling bits in santa cruz
>
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc

John>ok, thats fine.
John>sorry, I thought you were referring to pulling the whole source out of git.

The missing part is the checksum & gpg.
In other words, you have no idea what should be the checksum of the
"tarball" you are about to download.
And you are not sure if the checksum itself came from a trusted source.
Something like sha1sum.txt.asc should do the trick I suppose.

Note: current https://jdbc.postgresql.org/download.html does not list
checksums & signatures.

I think I can configure addition of "sha1sum.txt.asc" files like in
[1] to pgjdbc's releases page (see [2])


Alternative source can be Maven Central (see [3]).
It is a "standardized" repository with checksums and gpg signatures.

However, if we pick Central as the source of the tarballs, then we'd
better create yet another flavor of a tarball that would not include
jar dependencies, etc, etc.
In other words, "just a build-ready tarball" with no extra stuff.
The drawback of that approach is that tarball would be a build
artifact, and the upstream would never use it to produce "authentic"
build artifacts.


Any thoughts?


[1]: https://github.com/syncthing/syncthing/releases
[2]: https://github.com/pgjdbc/pgjdbc/releases
[3]: https://oss.sonatype.org/content/repositories/releases/org/postgresql/postgresql/9.4.1207/


Vladimir

On Friday 22 of January 2016 22:44:00 Vladimir Sitnikov wrote:
> John>ok, thats fine.
> John>sorry, I thought you were referring to pulling the whole source out of git.
>
> The missing part is the checksum & gpg.
> In other words, you have no idea what should be the checksum of the
> "tarball" you are about to download.

Right, this is really missing part -- especially the gpg signature.
Working with gpg should be rather manual job anyway :/.  It really
outweighs the benefits of automatization.

Note that this thread grows from simple request:  Please fix the http
link.  Now I would raise humble request:  Please don't change the release
tarball process.  Optionally -- having gpg sign would be real improvement.

Pavel

> And you are not sure if the checksum itself came from a trusted source.
> Something like sha1sum.txt.asc should do the trick I suppose.
>
> Note: current https://jdbc.postgresql.org/download.html does not list
> checksums & signatures.
>
> I think I can configure addition of "sha1sum.txt.asc" files like in
> [1] to pgjdbc's releases page (see [2])
>
>
> Alternative source can be Maven Central (see [3]).
> It is a "standardized" repository with checksums and gpg signatures.
>
> However, if we pick Central as the source of the tarballs, then we'd
> better create yet another flavor of a tarball that would not include
> jar dependencies, etc, etc.
> In other words, "just a build-ready tarball" with no extra stuff.
> The drawback of that approach is that tarball would be a build
> artifact, and the upstream would never use it to produce "authentic"
> build artifacts.
>
>
> Any thoughts?
>
>
> [1]: https://github.com/syncthing/syncthing/releases
> [2]: https://github.com/pgjdbc/pgjdbc/releases
> [3]: https://oss.sonatype.org/content/repositories/releases/org/postgresql/postgresql/9.4.1207/
>
>
> Vladimir
>
>
>