Обсуждение: pgsql: Report progress of CREATE INDEX operations

Report progress of CREATE INDEX operations

This uses the progress reporting infrastructure added by c16dc1aca5e0,
adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY.

There are two pieces to this: one is index-AM-agnostic, and the other is
AM-specific.  The latter is fairly elaborate for btrees, including
reportage for parallel index builds and the separate phases that btree
index creation uses; other index AMs, which are much simpler in their
building procedures, have simplistic reporting only, but that seems
sufficient, at least for non-concurrent builds.

The index-AM-agnostic part is fairly complete, providing insight into
the CONCURRENTLY wait phases as well as block-based progress during the
index validation table scan.  (The index validation index scan requires
patching each AM, which has not been included here.)

Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada
Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/ab0dfc961b6a821f23d9c40c723d11380ce195a6

Modified Files
--------------
contrib/amcheck/verify_nbtree.c          |   2 +-
contrib/bloom/blinsert.c                 |   2 +-
contrib/bloom/blutils.c                  |   1 +
doc/src/sgml/indexam.sgml                |  13 ++
doc/src/sgml/monitoring.sgml             | 224 ++++++++++++++++++++++++++++++-
src/backend/access/brin/brin.c           |   5 +-
src/backend/access/gin/gininsert.c       |   2 +-
src/backend/access/gin/ginutil.c         |   1 +
src/backend/access/gist/gist.c           |   1 +
src/backend/access/gist/gistbuild.c      |   2 +-
src/backend/access/hash/hash.c           |   7 +-
src/backend/access/hash/hashsort.c       |   6 +
src/backend/access/heap/heapam_handler.c | 107 +++++++++++++++
src/backend/access/nbtree/nbtree.c       |   9 ++
src/backend/access/nbtree/nbtsort.c      |  56 +++++++-
src/backend/access/nbtree/nbtutils.c     |  24 ++++
src/backend/access/spgist/spginsert.c    |   2 +-
src/backend/access/spgist/spgutils.c     |   1 +
src/backend/catalog/index.c              |  57 +++++++-
src/backend/catalog/system_views.sql     |  27 ++++
src/backend/commands/indexcmds.c         |  72 ++++++++--
src/backend/storage/ipc/standby.c        |   2 +-
src/backend/storage/lmgr/lmgr.c          |  46 ++++++-
src/backend/storage/lmgr/lock.c          |   7 +-
src/backend/utils/adt/amutils.c          |  23 ++++
src/backend/utils/adt/pgstatfuncs.c      |   2 +
src/include/access/amapi.h               |   4 +
src/include/access/genam.h               |   1 +
src/include/access/nbtree.h              |  11 ++
src/include/access/tableam.h             |   7 +
src/include/catalog/catversion.h         |   2 +-
src/include/catalog/pg_proc.dat          |  10 +-
src/include/commands/progress.h          |  37 ++++-
src/include/pgstat.h                     |   5 +-
src/include/storage/lmgr.h               |   4 +-
src/include/storage/lock.h               |   2 +-
src/test/regress/expected/rules.out      |  30 ++++-
37 files changed, 768 insertions(+), 46 deletions(-)

On 2019-Apr-02, Alvaro Herrera wrote:

> Report progress of CREATE INDEX operations

Argh, I pushed an older version of the commit.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,

On 2019-04-02 15:41:46 -0300, Alvaro Herrera wrote:
> On 2019-Apr-02, Alvaro Herrera wrote:
> 
> > Report progress of CREATE INDEX operations
> 
> Argh, I pushed an older version of the commit.

Note that longfin complains about something different than the output
files:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=longfin&dt=2019-04-02%2018%3A49%3A03

lmgr.c:882:14: error: variable 'count' is used uninitialized whenever '?:' condition is false
[-Werror,-Wsometimes-uninitialized]
                                                                                   progress ? &count : NULL));
                                                                                   ^~~~~~~~
lmgr.c:883:12: note: uninitialized use occurs here
                total += count;
                         ^~~~~
lmgr.c:882:14: note: remove the '?:' if its condition is always true
                                                                                   progress ? &count : NULL));
                                                                                   ^~~~~~~~~~~      ~~~~~~~
lmgr.c:878:14: note: initialize the variable 'count' to silence this warning
                int                     count;
                                             ^
                                              = 0
1 error generated.
make[4]: *** [lmgr.o] Error 1
make[4]: *** Waiting for unfinished jobs....

Greetings,

Andres Freund

On 2019-Apr-02, Andres Freund wrote:

> Note that longfin complains about something different than the output
> files:
> 
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=longfin&dt=2019-04-02%2018%3A49%3A03
> 
> lmgr.c:882:14: error: variable 'count' is used uninitialized whenever '?:' condition is false
[-Werror,-Wsometimes-uninitialized]
>                                                                                    progress ? &count : NULL));
>                                                                                    ^~~~~~~~
> lmgr.c:883:12: note: uninitialized use occurs here
>                 total += count;
>                          ^~~~~
> lmgr.c:882:14: note: remove the '?:' if its condition is always true
>                                                                                    progress ? &count : NULL));
>                                                                                    ^~~~~~~~~~~      ~~~~~~~
> lmgr.c:878:14: note: initialize the variable 'count' to silence this warning
>                 int                     count;
>                                              ^

Had noticed, thanks.  Just pushed a blind fix for that ...

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2019-Apr-02, Alvaro Herrera wrote:

> Report progress of CREATE INDEX operations

I just noticed that valgrind is also complaining about
IndexVacuumInfo.report_progress being uninitialized in a couple of
callsites.  Will fix early tomorrow.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services