Re: JDBC CTS 1.2.1

On Wednesday 27 October 2004 14:15, Vadim Nasardinov wrote:
> Before attempting to tackle the 1.3.1 suite, I tried to get 1.2.1 up
> and running.  My results are as follows:
>
>     PASSED:  1778
>     FAILED:   510

I took a closer look at the failures and discovered that the situation
was a lot better than the above figures suggest.  It turns out that
that the testsuite only has 575 tests or so -- not 1700+, as one might
imagine by looking at the above.

(If you use the slightly newer exclusion list available from
http://java.sun.com/products/jdbc/exclude.1.2.1.txt?button=Download
then the number of tests goes down to 548.)

Each test is run in four different "vehicles": appclient, ejb, jsp,
and servlet.  If you sort passed and failed tests by vehicle, the
following picture emerges:

    -------+-----------+-----+-----+--------
           | appclient | ejb | jsp | servlet
    -------+-----------+-----+-----+--------
    PASSED |       551 | 129 | 549 |     549
    -------+-----------+-----+-----+--------
    FAILED |        24 | 442 |  22 |      22
    -------+-----------+-----+-----+--------
     TOTAL |       575 | 571 | 571 |     571
    -------+-----------+-----+-----+--------


Note that the "ejb" vehicle is an outlier.  I haven't taken a close
look at it yet, but I suspect that the vast majority of tests that
fail in the ejb vehicle do so for reasons unrelated to the JDBC
driver.

A more correct breakdown is as follows:

   PASSED:  549
   FAILED:   22
   TOTAL:   571

Quite respectable, if you ask me.  Furthermore, I suspect that most
(if not all) of the remaining 22 failures are due to lack of support
for in/out parameters in stored procedures in the current backend, as
Dave Cramer pointed out in
  http://archives.postgresql.org/pgsql-jdbc/2004-10/msg00290.php


The updated version of the script that counts passed and failed tests
is available here:
  http://people.redhat.com/~vadimn/scratch/pgsql-jdbc/cts/cts-logs.py


Vadim