Re: Add tuples_skipped to pg_stat_progress_copy

On Wed, Jan 17, 2024 at 2:22 PM torikoshia <torikoshia@oss.nttdata.com> wrote:
>
> Hi,
>
> 132de9968840c introduced SAVE_ERROR_TO option to COPY and enabled to
> skip malformed data, but there is no way to watch the number of skipped
> rows during COPY.
>
> Attached patch adds tuples_skipped to pg_stat_progress_copy, which
> counts the number of skipped tuples because source data is malformed.
> If SAVE_ERROR_TO is not specified, this column remains zero.
>
> The advantage would be that users can quickly notice and stop COPYing
> when there is a larger amount of skipped data than expected, for
> example.
>
> As described in commit log, it is expected to add more choices for
> SAVE_ERROR_TO like 'log' and using such options may enable us to know
> the number of skipped tuples during COPY, but exposed in
> pg_stat_progress_copy would be easier to monitor.
>
>
> What do you think?

+1

The patch is pretty simple. Here is a comment:

+       (if <literal>SAVE_ERROR_TO</literal> is specified, otherwise zero).
+      </para></entry>
+     </row>

To be precise, this counter only advances when a value other than
'ERROR' is specified to SAVE_ERROR_TO option.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com