Re: ps.setCharacterStream() and memory usage

On Friday 29 October 2004 07:56, Sebastiaan van Erk wrote:
> Coming back to another problem I have with the following insert
> statement: (Postgres 7.4, driver build 215, jdbc 3)
>
>    ps.setCharacterStream(1, reader, (int) messageFile.length());
>    ps.executeUpdate();
>
> The reason I do this (using a character stream to load a text file
> to a TEXT field) is that I wish to avoid loading the file into
> memory completely.
[...]
> [...] the file actually gets copied THREE times into memory, causing
> my Java file to already get a java.lang.OutOfMemory error with an
> insert of a 10M text file (Java allocates 20M for this, since it has
> 2 byte chars, and the driver makes (at least) 3 copies (referenced
> at the same time, not allowing one to be GC'ed), making 60M).

I assume you're talking about
org/postgresql/jdbc2/AbstractJdbc2Statement.java which has this piece
of code (slightly reformatted to fit on a 80-column page):


    public void setCharacterStream(int i, Reader x, int length)
        throws SQLException {

        checkClosed();
        if (length < 0)
            throw new PSQLException(GT.tr("Invalid stream length {0}.",
                                          new Integer(length)));

        if (connection.haveMinimumCompatibleVersion("7.2")) {
            char[] l_chars = new char[length];
            int l_charsRead = 0;
            try {
                while(true) {
                    int n = x.read(l_chars,
                                   l_charsRead,
                                   length - l_charsRead);
                    if (n == -1)
                        break;
                    l_charsRead += n;
                    if (l_charsRead == length)
                        break;
                }
            } catch (IOException l_ioe) {
                throw new PSQLException(GT.tr("Provided Reader failed."),
                                        PSQLState.UNEXPECTED_ERROR, l_ioe);
            }
            setString(i, new String(l_chars, 0, l_charsRead));
        } else {

        }
    }


This does appear to be optimized for reading smallish chunks of text.
If the character source behind the Reader is sufficiently large, I
believe we'd be better off building up a StringBuffer rather than a
character array.  Something like this:

    if (connection.haveMinimumCompatibleVersion("7.2")) {
        StringBuffer sb = new StringBuffer(length);
        BufferedReader br = new BufferedReader(x);
        try {
            while(true) {
                String chunk = br.readLine();
                if (chunk == null) { break; }
                else {sb.append(chunk); }
            }
        } catch (IOException l_ioe) {
            throw new PSQLException(GT.tr("Provided Reader failed."),
                                    PSQLState.UNEXPECTED_ERROR, l_ioe);
        }
        setString(i, sb.toString());
    }

The reason this is better is because

  (a) the String(char[],int,int) constructor always defensively copies
      the passed in character array.  So, as you point out, if you
      pass in a 10-million-character array, a new 10-million character
      array will be allocated.

  (b) in contrast, if you construct a String from a StringBuffer, they
      can share the underlying character array:
      http://www.google.com/search?q=Heinz+Kabutz+StringBuffer+Issue+068&btnI