Обсуждение: Patch to readd documentation

The attached patch adds the old JDBC documentation, which was lost
during the move.  Along the way I've converted it to XML so it is more
easily possible to use Java-based XSLT processors in it, which probably
goes over better around here.

To build it, use:

ant -Ddocbook.stylesheet='/some/where/xhtml/chunk.xsl' doc

You can download the stylesheets here:

http://sourceforge.net/project/showfiles.php?group_id=21935&package_id=16608

(Eventually, we should support fetching them directly over http and/or
support XML catalogs.)

You might need to use JDK 1.4 or install additional classes, according
to the Ant documentation.

If the build process crashes with an error message like this:

.../gentext.xsl:230:57: Error! Variable accessed before it is bound!

you should try to use Saxon to do the transformation.  I have done the
following:

export CLASSPATH=/usr/share/saxon-6.5.2/saxon.jar:$CLASSPATH

The HTML rendition will end up in build/doc.

The content of the documentation could probably be overhauled as well,
but I just focused on integrating the existing material before it's
forgotten.

On Fri, 23 Jan 2004, Peter Eisentraut wrote:

> The attached patch adds the old JDBC documentation, which was lost
> during the move.  Along the way I've converted it to XML so it is more
> easily possible to use Java-based XSLT processors in it, which probably
> goes over better around here.
>
> (Eventually, we should support fetching them directly over http and/or
> support XML catalogs.)

I looked around to see how other projects handle this type of thing and it
seemed like a number of them packaged the xsl stylesheets as part
of the package.  This provides an easier means of specifying where they
exist and enforces consistency because eveyone is using the same version
of the stylesheet.  By also distributing saxon we can make this a zero
step configuration.  The price we pay for this is a larger distribution,
but in this case I would simply recommend creating another cvs module
solely for documentation because most people aren't interested in building
the documentation, only reading it (if that).

>
> You might need to use JDK 1.4 or install additional classes, according
> to the Ant documentation.
>
> If the build process crashes with an error message like this:
>
> .../gentext.xsl:230:57: Error! Variable accessed before it is bound!

I got "org.apache.xml.dtm.DTMException: No more DTM IDs are available",
but the root cause is the same.  The xalan transformer shipped with the
1.4 jdk has a number of bugs.  I was able to run this sucessfully with the
1.5alpha JDK.

Kris Jurka

Kris Jurka wrote:
> I looked around to see how other projects handle this type of thing
> and it seemed like a number of them packaged the xsl stylesheets as
> part of the package.

We would be on the wrong side of that inclusion: the JDBC driver is 1.2
MB, the XSL stylesheets are 12 MB.

I am firmly opposed to any sort of inclusion of other packages in the
name of convenience.  People who are not able or willing to install
software should use the prebuilt packages.  And it's not like
installing Saxon or XSL stylesheets is a great challenge.  They should
be included in recent versions of free operating systems.

On Sat, 24 Jan 2004, Peter Eisentraut wrote:

> Kris Jurka wrote:
> > I looked around to see how other projects handle this type of thing
> > and it seemed like a number of them packaged the xsl stylesheets as
> > part of the package.
>
> We would be on the wrong side of that inclusion: the JDBC driver is 1.2
> MB, the XSL stylesheets are 12 MB.

I'm not suggesting trying to manage the stylesheets like any other piece
of source code.  We would put the compressed archive in the repository
that would be expanded on demand.  Further if size is an issue we could
trim the stylesheets to only those that we needed to build the
documentation.  The size is also why I suggested creating a new cvs
module, so it wouldn't clutter the main source.

> I am firmly opposed to any sort of inclusion of other packages in the
> name of convenience.  People who are not able or willing to install
> software should use the prebuilt packages.  And it's not like
> installing Saxon or XSL stylesheets is a great challenge.  They should
> be included in recent versions of free operating systems.
>

Installing the software is only half of the problem.  There is no
configure to go out and locate the required files.  Your build
instructions require modifying both the classpath and the ant command
line which I personally found a pain.  Finally as a java project we are
not dealing with just "recent versions of free operating systems."  You
had suggested adding a way to pull the stylesheets on demand over http.
Why not make this part of the documentation distribution?

Kris Jurka

Kris Jurka wrote:
> Installing the software is only half of the problem.  There is no
> configure to go out and locate the required files.

But there are mechanisms that accomplish the same thing, namely XML
catalogs in this case.

> Finally as a java project we
> are not dealing with just "recent versions of free operating
> systems."

Most developers are probably using one of those.  That's the only thing
that matters here.

> You had suggested adding a way to pull the stylesheets on
> demand over http. Why not make this part of the documentation
> distribution?

Because I haven't figured out how to do it yet.  I know how it works
with xsltproc, but I can't get Saxon to do it.

Btw., creating a separate distribution for documentation is just a
perfect way to make sure the documentation never gets read or
maintained.  If that is your plan then you might as well remove it.

On Sat, 24 Jan 2004, Peter Eisentraut wrote:

> Btw., creating a separate distribution for documentation is just a
> perfect way to make sure the documentation never gets read or
> maintained.  If that is your plan then you might as well remove it.

The way to make sure the documentation is read is to put it on a website,
not have an xml document in subdirectory of the main package.  The way to
make sure the documentation is maintained is to make the build process as
easy as possible.

Kris Jurka

Kris Jurka wrote:
> The way to make sure the documentation is maintained is to
> make the build process as easy as possible.

Maybe if dealing with Saxon is deemed to complicated we should use
xsltproc.  That would make the build process a zero-effert step besides
of course having to install xsltproc.

Kris Jurka wrote:
>
> On Sat, 24 Jan 2004, Peter Eisentraut wrote:
>
>
>>Btw., creating a separate distribution for documentation is just a
>>perfect way to make sure the documentation never gets read or
>>maintained.  If that is your plan then you might as well remove it.
>
>
> The way to make sure the documentation is read is to put it on a website,
> not have an xml document in subdirectory of the main package.  The way to
> make sure the documentation is maintained is to make the build process as
> easy as possible.

Can we do a build target that just validates the doc source and doesn't
actually generate documentation from it? If I understand things
correctly, that'd make quick documentation changes easier in that you
don't need additional tools installed, but you can still check you
didn't irretrievably break things before checking in..

I find that most of the doc changes I make elsewhere are updates to
existing text to reflect code changes, rather than large rewrites .. so
actually generating and viewing the final documentation isn't so important.

-O

Oliver Jowett wrote:
> Can we do a build target that just validates the doc source and
> doesn't actually generate documentation from it? If I understand
> things correctly, that'd make quick documentation changes easier in
> that you don't need additional tools installed, but you can still
> check you didn't irretrievably break things before checking in..

To my first approximation, validating the documentation is not any
easier than processing it when considering the tools that need to be
installed and configured.

Peter Eisentraut wrote:
> Oliver Jowett wrote:
>
>>Can we do a build target that just validates the doc source and
>>doesn't actually generate documentation from it? If I understand
>>things correctly, that'd make quick documentation changes easier in
>>that you don't need additional tools installed, but you can still
>>check you didn't irretrievably break things before checking in..
>
>
> To my first approximation, validating the documentation is not any
> easier than processing it when considering the tools that need to be
> installed and configured.

I think you should be able to validate the docs at the XML level using
just the XML libraries provided with recent JDKs (I'll check if this
works shortly). We do need the DocBook XML DTDs available to properly
validate the document, but that seems less onerous than needing a recent
XSLT processor plus DocBook stylesheets.

Ant has a XMLValidate task that might be useful here.

-O

Oliver Jowett wrote:
> I think you should be able to validate the docs at the XML level using
> just the XML libraries provided with recent JDKs (I'll check if this
> works shortly). We do need the DocBook XML DTDs available to properly
> validate the document, but that seems less onerous than needing a recent
> XSLT processor plus DocBook stylesheets.
>
> Ant has a XMLValidate task that might be useful here.

The attached patch implements this. It'll use a local copy of the
DocBook XML DTDs if present, otherwise it'll grab them from the system
URI given in pgjdbc.xml (i.e. from www.open-oasis.org). See the comments
in the patch for more details.

This works here using:

$ ant -version
Apache Ant version 1.5.4 compiled on September 22 2003

$ java -version
java version "1.4.2_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)

-O
Index: build.xml
===================================================================
RCS file: /usr/local/cvsroot/pgjdbc/pgjdbc/build.xml,v
retrieving revision 1.42
diff -u -c -r1.42 build.xml
*** build.xml    26 Jan 2004 14:03:48 -0000    1.42
--- build.xml    27 Jan 2004 03:59:57 -0000
***************
*** 344,347 ****
--- 344,426 ----
     <delete file="${builddir}/doc/pgjdbc.html"/>
    </target>

+   <!-- Blah. Can't reference an external XML catalog until Ant 1.6.
+        So instead we copy the contents of doc/docbook-dtd/catalog.xml
+        here, in the form that Ant's xmlcatalog element wants. -->
+   <xmlcatalog id="docbook-catalog">
+     <dtd publicId="-//OASIS//DTD DocBook XML V4.2//EN"
+       location="doc/docbook-dtd/docbookx.dtd"/>
+     <entity publicId="-//OASIS//DTD DocBook CALS Table Model V4.2//EN"
+       location="doc/docbook-dtd/calstblx.dtd"/>
+     <entity publicId="-//OASIS//DTD XML Exchange Table Model 19990315//EN"
+       location="doc/docbook-dtd/soextblx.dtd"/>
+     <entity publicId="-//OASIS//ELEMENTS DocBook Information Pool V4.2//EN"
+       location="doc/docbook-dtd/dbpoolx.mod"/>
+     <entity publicId="-//OASIS//ELEMENTS DocBook Document Hierarchy V4.2//EN"
+       location="doc/docbook-dtd/dbhierx.mod"/>
+     <entity publicId="-//OASIS//ENTITIES DocBook Additional General Entities V4.2//EN"
+       location="doc/docbook-dtd/dbgenent.mod"/>
+     <entity publicId="-//OASIS//ENTITIES DocBook Notations V4.2//EN"
+       location="doc/docbook-dtd/dbnotnx.mod"/>
+     <entity publicId="-//OASIS//ENTITIES DocBook Character Entities V4.2//EN"
+       location="doc/docbook-dtd/dbcentx.mod"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Diacritical Marks//EN//XML"
+       location="doc/docbook-dtd/ent/iso-dia.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Numeric and Special Graphic//EN//XML"
+       location="doc/docbook-dtd/ent/iso-num.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Publishing//EN//XML"
+       location="doc/docbook-dtd/ent/iso-pub.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES General Technical//EN//XML"
+       location="doc/docbook-dtd/ent/iso-tech.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Latin 1//EN//XML"
+       location="doc/docbook-dtd/ent/iso-lat1.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Latin 2//EN//XML"
+       location="doc/docbook-dtd/ent/iso-lat2.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Greek Letters//EN//XML"
+       location="doc/docbook-dtd/ent/iso-grk1.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Monotoniko Greek//EN//XML"
+       location="doc/docbook-dtd/ent/iso-grk2.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Greek Symbols//EN//XML"
+       location="doc/docbook-dtd/ent/iso-grk3.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Alternative Greek Symbols//EN//XML"
+       location="doc/docbook-dtd/ent/iso-grk4.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Math Symbols: Arrow Relations//EN//XML"
+       location="doc/docbook-dtd/ent/iso-amsa.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Math Symbols: Binary Operators//EN//XML"
+       location="doc/docbook-dtd/ent/iso-amsb.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Math Symbols: Delimiters//EN//XML"
+       location="doc/docbook-dtd/ent/iso-amsc.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Math Symbols: Negated Relations//EN//XML"
+       location="doc/docbook-dtd/ent/iso-amsn.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Math Symbols: Ordinary//EN//XML"
+       location="doc/docbook-dtd/ent/iso-amso.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Added Math Symbols: Relations//EN//XML"
+       location="doc/docbook-dtd/ent/iso-amsr.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Box and Line Drawing//EN//XML"
+       location="doc/docbook-dtd/ent/iso-box.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Russian Cyrillic//EN//XML"
+       location="doc/docbook-dtd/ent/iso-cyr1.ent"/>
+     <entity publicId="ISO 8879:1986//ENTITIES Non-Russian Cyrillic//EN//XML"
+       location="doc/docbook-dtd/ent/iso-cyr2.ent"/>
+   </xmlcatalog>
+
+   <!-- Validate but don't process the documentation.
+        This target expects the DocBook XML DTDs, available at
+        http://www.docbook.org/xml/4.2/docbook-xml-4.2.zip, to be
+        present in doc/docbook-dtd. If they're not present, they will
+        be fetched on each validation from www.oasis-open.org.
+
+        Note that if the DTD fetch fails, validation fails with a fairly
+        useless error message. Run ant with -verbose to get a more useful
+        error. You'll need to set the java properties http.proxyHost and
+        http.proxyPort if you need to go via a proxy to fetch the DTDs.
+   -->
+
+   <target name="validate-doc" depends="prepare">
+     <xmlvalidate warn="true">
+       <fileset dir="doc" includes="*.xml"/>
+       <xmlcatalog refid="docbook-catalog"/>
+     </xmlvalidate>
+   </target>
+
  </project>

On Tue, 27 Jan 2004, Oliver Jowett wrote:

> Oliver Jowett wrote:
> > I think you should be able to validate the docs at the XML level using
> > just the XML libraries provided with recent JDKs (I'll check if this
> > works shortly). We do need the DocBook XML DTDs available to properly
> > validate the document, but that seems less onerous than needing a recent
> > XSLT processor plus DocBook stylesheets.
> >
> > Ant has a XMLValidate task that might be useful here.
>
> The attached patch implements this. It'll use a local copy of the
> DocBook XML DTDs if present, otherwise it'll grab them from the system
> URI given in pgjdbc.xml (i.e. from www.open-oasis.org). See the comments
> in the patch for more details.
>

I don't see what the point of the xmlcatalog in its current incarnation
is.  It should either take a config parameter like building the
documentation does as a path to the dtds or it should package them with
it.  As it stands the user would have to download the files and put them
in this special doc directory.  If that is the expected case we should
just package them ourselves as I've recommended elsewhere.

Kris Jurka

Kris Jurka wrote:
>
> On Tue, 27 Jan 2004, Oliver Jowett wrote:
>
>
>>Oliver Jowett wrote:
>>
>>>I think you should be able to validate the docs at the XML level using
>>>just the XML libraries provided with recent JDKs (I'll check if this
>>>works shortly). We do need the DocBook XML DTDs available to properly
>>>validate the document, but that seems less onerous than needing a recent
>>>XSLT processor plus DocBook stylesheets.
>>>
>>>Ant has a XMLValidate task that might be useful here.
>>
>>The attached patch implements this. It'll use a local copy of the
>>DocBook XML DTDs if present, otherwise it'll grab them from the system
>>URI given in pgjdbc.xml (i.e. from www.open-oasis.org). See the comments
>>in the patch for more details.
>>
>
>
> I don't see what the point of the xmlcatalog in its current incarnation
> is.  It should either take a config parameter like building the
> documentation does as a path to the dtds

Yes. My patch was a proof of concept. If we decide to go this way the
local DTD location can certainly be configurable.

 > or it should package them with
> it.  As it stands the user would have to download the files and put them
> in this special doc directory.  If that is the expected case we should
> just package them ourselves as I've recommended elsewhere.

I thought you were recommending packaging the stylesheets which is a
somewhat bigger proposition. The DTDs are around 300-400k uncompressed;
aren't the stylesheets much bigger (I saw 12MB mentioned)?

I don't really see it's worth packaging them in the CVS tree though.
It's trivial to install them if needed (literally, download and unzip
one file into your build tree; see the comments in the patch). Also it's
not actually necessary to have the DTDs stored locally, they'll get
fetched via HTTP at validation time if not present. Having a local copy
is just faster.

-O

I've reconsidered my scheme to package docbook stylesheets, dtds, and
saxon in a separate jdbc documentation package.  After realizing that the
number of people who will ever build the documentation can comfortably be
counted on my digits, a little complexity isn't going to kill us.  I've
added Oliver's validation target to the build file and added the ability
to specify the dtd and stylesheet locations via a local properties file
(build.local.properties). This will allow overriding any default config
parameter for a local setup.  My build.local.properties for a debian
system looks like:

docbook.stylesheet=/usr/share/sgml/docbook/stylesheet/xsl/nwalsh/xhtml/chunk.xsl
docbook.dtd=/usr/share/sgml/docbook/dtd/xml/4.2

Kris Jurka

Kris,

What about people who build the install packages (i.e. rpms, debs, etc).
  What process do they follow?  Do they build from source?  Do they use
prebuilt files provided on the website?  What files would we expect they
include?  When jdbc was part of the base distribution these were easy
questions; all you installed in the rpms was the jar file since the doc
was all handled separately in it's own rpm.  Now we need to figure this
out ourselves.

For consistency I think we need to have some packaging instructions that
provide guidance on what an install of the jdbc driver should contain
(clearly it needs one or more of the jar files, should also include the
doc - in what formats though, html, pdf, etc, and probably also should
include the example source code).

If many (most?) users of the jdbc driver get it through prebuilt
packages, then we need to spend some time thinking about these issues,
so that these users have a reasonable environment to work in.

thanks,
--Barry


Kris Jurka wrote:
>
> I've reconsidered my scheme to package docbook stylesheets, dtds, and
> saxon in a separate jdbc documentation package.  After realizing that the
> number of people who will ever build the documentation can comfortably be
> counted on my digits, a little complexity isn't going to kill us.  I've
> added Oliver's validation target to the build file and added the ability
> to specify the dtd and stylesheet locations via a local properties file
> (build.local.properties). This will allow overriding any default config
> parameter for a local setup.  My build.local.properties for a debian
> system looks like:
>
> docbook.stylesheet=/usr/share/sgml/docbook/stylesheet/xsl/nwalsh/xhtml/chunk.xsl
> docbook.dtd=/usr/share/sgml/docbook/dtd/xml/4.2
>
> Kris Jurka
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faqs/FAQ.html

On Tue, 27 Jan 2004, Barry Lind wrote:

> Kris,
>
> What about people who build the install packages (i.e. rpms, debs, etc).
>   What process do they follow?  Do they build from source?  Do they use
> prebuilt files provided on the website?  What files would we expect they
> include?  When jdbc was part of the base distribution these were easy
> questions; all you installed in the rpms was the jar file since the doc
> was all handled separately in it's own rpm.  Now we need to figure this
> out ourselves.

Well, we will certainly have to produce a source package.  My
understanding is that even for the main tree the packager doesn't
produce the documentation from source, but simply uses a provided .tar.gz
file of the html doc.  This seems like a reasonable thing to do as well
(although I can't say I understand putting a compressed archive inside of
a compressed archive.)  Where these actually go is up to the actual
packager.

>
> For consistency I think we need to have some packaging instructions that
> provide guidance on what an install of the jdbc driver should contain
> (clearly it needs one or more of the jar files, should also include the
> doc - in what formats though, html, pdf, etc, and probably also should
> include the example source code).

We could certainly produce a "packager's package" which contained the
prebuilt jar files and documentation.  Packaging instructions are great,
but what we'll really need to do is advertise our availability to help and
be responsive to maintainers.  At the moment the 7.4 rpms have bad jar
files because when Lamar went to pull the prebuilt ones, they were out of
date.  He just took whatever he could find and packaged that.  I've
asked him to check with the developers in the future to make sure he's got
things straight.  Maybe we need to start a packager outreach program right
before the next release where we round them all up and make sure we're all
on the same page.

Kris Jurka

Kris Jurka wrote:
> Well, we will certainly have to produce a source package.  My
> understanding is that even for the main tree the packager doesn't
> produce the documentation from source, but simply uses a provided
> .tar.gz file of the html doc.  This seems like a reasonable thing to
> do as well (although I can't say I understand putting a compressed
> archive inside of a compressed archive.)  Where these actually go is
> up to the actual packager.

The traditional approach for a source package is to prebuild as much as
you can while remaining portable.  Typically, this includes things like
yacc, and it certainly includes the documentation.  So far, the JDBC
driver hasn't had anything it could prebuild before shipping a source
package, so currently there is no difference between CVS and a source
package.  But it seems like a good time to start.

Btw., the only reason that the server package pulls these tricks with
the documentation is so that you can still build snapshot source
packages when the documentation happens to be broken.  I wouldn't worry
about that.  Before long we'll have the documentation build worked out
and it will run like a clockwork.

> We could certainly produce a "packager's package" which contained the
> prebuilt jar files and documentation.  Packaging instructions are
> great, but what we'll really need to do is advertise our availability
> to help and be responsive to maintainers.

Just put the spec files in your CVS and be your own maintainer.  That
way you keep everything under control.