Re: WIP Incremental JSON Parser

On Wed, Mar 20, 2024 at 11:56 PM Andrew Dunstan <andrew@dunslane.net> wrote:
> Thanks, included that and attended to the other issues we discussed. I think this is pretty close now.

Okay, looking over the thread, there are the following open items:
- extend the incremental test in order to exercise the semantic callbacks [1]

- add Assert calls in impossible error cases [2]

- error out if the non-incremental lex doesn't consume the entire token [2]

- double-check that out of memory is an appropriate failure mode for
the frontend [3]

Just as a general style nit:

> +   if (lex->incremental)
> +   {
> +       lex->input = lex->token_terminator = lex->line_start = json;
> +       lex->input_length = len;
> +       lex->inc_state->is_last_chunk = is_last;
> +   }
> +   else
> +       return JSON_INVALID_LEXER_TYPE;

I think flipping this around would probably make it more readable;
something like:

    if (!lex->incremental)
        return JSON_INVALID_LEXER_TYPE;

    lex->input = ...

[1] https://www.postgresql.org/message-id/CAOYmi%2BnHV55Uhz%2Bo-HKq0GNiWn2L5gMcuyRQEz_fqpGY%3DpFxKA%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAD5tBcLi2ffZkktV2qrsKSBykE-N8CiYgrfbv0vZ-F7%3DxLFeqw%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAOYmi%2BnY%3DrF6dJCzaOuA3d-3FbwXCcecOs_S1NutexFA3dRXAw%40mail.gmail.com