Обсуждение: pgsql: Replace our traditional initial-catalog-data format with abette
Replace our traditional initial-catalog-data format with a better design. Historically, the initial catalog data to be installed during bootstrap has been written in DATA() lines in the catalog header files. This had lots of disadvantages: the format was badly underdocumented, it was very difficult to edit the data in any mechanized way, and due to the lack of any abstraction the data was verbose, hard to read/understand, and easy to get wrong. Hence, move this data into separate ".dat" files and represent it in a way that can easily be read and rewritten by Perl scripts. The new format is essentially "key => value" for each column; while it's a bit repetitive, explicit labeling of each value makes the data far more readable and less error-prone. Provide a way to abbreviate entries by omitting field values that match a specified default value for their column. This allows removal of a large amount of repetitive boilerplate and also lowers the barrier to adding new columns. Also teach genbki.pl how to translate symbolic OID references into numeric OIDs for more cases than just "regproc"-like pg_proc references. It can now do that for regprocedure-like references (thus solving the problem that regproc is ambiguous for overloaded functions), operators, types, opfamilies, opclasses, and access methods. Use this to turn nearly all OID cross-references in the initial data into symbolic form. This represents a very large step forward in readability and error resistance of the initial catalog data. It should also reduce the difficulty of renumbering OID assignments in uncommitted patches. Also, solve the longstanding problem that frontend code that would like to use OID macros and other information from the catalog headers often had difficulty with backend-only code in the headers. To do this, arrange for all generated macros, plus such other declarations as we deem fit, to be placed in "derived" header files that are safe for frontend inclusion. (Once clients migrate to using these pg_*_d.h headers, it will be possible to get rid of the pg_*_fn.h headers, which only exist to quarantine code away from clients. That is left for follow-on patches, however.) The now-automatically-generated macros include the Anum_xxx and Natts_xxx constants that we used to have to update by hand when adding or removing catalog columns. Replace the former manual method of generating OID macros for pg_type entries with an automatic method, ensuring that all built-in types have OID macros. (But note that this patch does not change the way that OID macros for pg_proc entries are built and used. It's not clear that making that match the other catalogs would be worth extra code churn.) Add SGML documentation explaining what the new data format is and how to work with it. Despite being a very large change in the catalog headers, there is no catversion bump here, because postgres.bki and related output files haven't changed at all. John Naylor, based on ideas from various people; review and minor additional coding by me; previous review by Alvaro Herrera Discussion: https://postgr.es/m/CAJVSVGWO48JbbwXkJz_yBFyGYW-M9YWxnPdxJBUosDC9ou_F0Q@mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/372728b0d49552641f0ea83d9d2e08817de038fa Modified Files -------------- doc/src/sgml/bki.sgml | 661 +- doc/src/sgml/libpq.sgml | 2 +- src/Makefile | 5 + src/backend/Makefile | 34 +- src/backend/catalog/.gitignore | 2 + src/backend/catalog/Catalog.pm | 313 +- src/backend/catalog/Makefile | 87 +- src/backend/catalog/README | 111 - src/backend/catalog/genbki.pl | 478 +- src/backend/utils/Gen_fmgrtab.pl | 62 +- src/backend/utils/Makefile | 15 +- src/include/Makefile | 6 +- src/include/catalog/.gitignore | 2 + src/include/catalog/Makefile | 31 + src/include/catalog/duplicate_oids | 6 +- src/include/catalog/genbki.h | 29 +- src/include/catalog/indexing.h | 2 +- src/include/catalog/pg_aggregate.dat | 598 ++ src/include/catalog/pg_aggregate.h | 336 +- src/include/catalog/pg_am.dat | 34 + src/include/catalog/pg_am.h | 62 +- src/include/catalog/pg_amop.dat | 2472 ++++++ src/include/catalog/pg_amop.h | 1148 +-- src/include/catalog/pg_amproc.dat | 1277 +++ src/include/catalog/pg_amproc.h | 554 +- src/include/catalog/pg_attrdef.h | 20 +- src/include/catalog/pg_attribute.h | 59 +- src/include/catalog/pg_auth_members.h | 22 +- src/include/catalog/pg_authid.dat | 67 + src/include/catalog/pg_authid.h | 71 +- src/include/catalog/pg_cast.dat | 535 ++ src/include/catalog/pg_cast.h | 368 +- src/include/catalog/pg_class.dat | 68 + src/include/catalog/pg_class.h | 76 +- src/include/catalog/pg_collation.dat | 31 + src/include/catalog/pg_collation.h | 42 +- src/include/catalog/pg_constraint.h | 56 +- src/include/catalog/pg_conversion.h | 29 +- src/include/catalog/pg_database.dat | 26 + src/include/catalog/pg_database.h | 34 +- src/include/catalog/pg_db_role_setting.h | 29 +- src/include/catalog/pg_default_acl.h | 27 +- src/include/catalog/pg_depend.h | 49 +- src/include/catalog/pg_description.h | 37 +- src/include/catalog/pg_enum.h | 27 +- src/include/catalog/pg_event_trigger.h | 22 +- src/include/catalog/pg_extension.h | 29 +- src/include/catalog/pg_foreign_data_wrapper.h | 23 +- src/include/catalog/pg_foreign_server.h | 23 +- src/include/catalog/pg_foreign_table.h | 19 +- src/include/catalog/pg_index.h | 38 +- src/include/catalog/pg_inherits.h | 24 +- src/include/catalog/pg_init_privs.h | 42 +- src/include/catalog/pg_language.dat | 31 + src/include/catalog/pg_language.h | 39 +- src/include/catalog/pg_largeobject.h | 19 +- src/include/catalog/pg_largeobject_metadata.h | 18 +- src/include/catalog/pg_namespace.dat | 25 + src/include/catalog/pg_namespace.h | 37 +- src/include/catalog/pg_opclass.dat | 357 + src/include/catalog/pg_opclass.h | 222 +- src/include/catalog/pg_operator.dat | 3340 ++++++++ src/include/catalog/pg_operator.h | 1854 +---- src/include/catalog/pg_opfamily.dat | 248 + src/include/catalog/pg_opfamily.h | 170 +- src/include/catalog/pg_partitioned_table.h | 24 +- src/include/catalog/pg_pltemplate.dat | 51 + src/include/catalog/pg_pltemplate.h | 39 +- src/include/catalog/pg_policy.h | 29 +- src/include/catalog/pg_proc.dat | 10210 ++++++++++++++++++++++++ src/include/catalog/pg_proc.h | 5637 +------------ src/include/catalog/pg_publication.h | 24 +- src/include/catalog/pg_publication_rel.h | 19 +- src/include/catalog/pg_range.dat | 31 + src/include/catalog/pg_range.h | 61 +- src/include/catalog/pg_replication_origin.h | 22 +- src/include/catalog/pg_rewrite.h | 23 +- src/include/catalog/pg_seclabel.h | 16 +- src/include/catalog/pg_sequence.h | 20 +- src/include/catalog/pg_shdepend.h | 42 +- src/include/catalog/pg_shdescription.h | 36 +- src/include/catalog/pg_shseclabel.h | 18 +- src/include/catalog/pg_statistic.h | 44 +- src/include/catalog/pg_statistic_ext.h | 26 +- src/include/catalog/pg_subscription.h | 20 +- src/include/catalog/pg_subscription_rel.h | 19 +- src/include/catalog/pg_tablespace.dat | 22 + src/include/catalog/pg_tablespace.h | 26 +- src/include/catalog/pg_transform.h | 25 +- src/include/catalog/pg_trigger.h | 35 +- src/include/catalog/pg_ts_config.dat | 19 + src/include/catalog/pg_ts_config.h | 29 +- src/include/catalog/pg_ts_config_map.dat | 35 + src/include/catalog/pg_ts_config_map.h | 47 +- src/include/catalog/pg_ts_dict.dat | 20 + src/include/catalog/pg_ts_dict.h | 31 +- src/include/catalog/pg_ts_parser.dat | 20 + src/include/catalog/pg_ts_parser.h | 58 +- src/include/catalog/pg_ts_template.dat | 30 + src/include/catalog/pg_ts_template.h | 49 +- src/include/catalog/pg_type.dat | 942 +++ src/include/catalog/pg_type.h | 565 +- src/include/catalog/pg_user_mapping.h | 19 +- src/include/catalog/reformat_dat_file.pl | 311 + src/include/catalog/toasting.h | 2 +- src/include/catalog/unused_oids | 8 +- src/interfaces/ecpg/ecpglib/pg_type.h | 65 +- src/tools/msvc/Solution.pm | 49 +- src/tools/msvc/clean.bat | 2 + src/tools/pgindent/pgindent | 8 +- 110 files changed, 22928 insertions(+), 12480 deletions(-)
Seems, gaur animal doesn't too long shell command. Could "find | xargs"
help here?
cp ./*.h
'/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/
|| exit; \
chmod 644
'/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/*.h
|| exit; \
for dir in access bootstrap catalog commands common datatype executor
fe_utils foreign lib libpq mb nodes optimizer parser postmaster regex
replication rewrite statistics storage tcop snowball snowball/libstemmer
tsearch tsearch/dicts utils port port/atomics port/win32 port/win32_msvc
port/win32_msvc/sys port/win32/arpa port/win32/netinet port/win32/sys
portability; do \
cp ./$dir/*.h
'/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/$dir/
|| exit; \
chmod 644
'/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/$dir/*.h
|| exit; \
done
/bin/sh[5]: /usr/bin/chmod: The parameter list is too long.
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gaur&dt=2018-04-08%2017%3A20%3A57
Tom Lane wrote:
> Replace our traditional initial-catalog-data format with a better design.
>
> Historically, the initial catalog data to be installed during bootstrap
> has been written in DATA() lines in the catalog header files. This had
> lots of disadvantages: the format was badly underdocumented, it was
> very difficult to edit the data in any mechanized way, and due to the
> lack of any abstraction the data was verbose, hard to read/understand,
> and easy to get wrong.
>
> Hence, move this data into separate ".dat" files and represent it in a way
> that can easily be read and rewritten by Perl scripts. The new format is
> essentially "key => value" for each column; while it's a bit repetitive,
> explicit labeling of each value makes the data far more readable and less
> error-prone. Provide a way to abbreviate entries by omitting field values
> that match a specified default value for their column. This allows removal
> of a large amount of repetitive boilerplate and also lowers the barrier to
> adding new columns.
>
> Also teach genbki.pl how to translate symbolic OID references into
> numeric OIDs for more cases than just "regproc"-like pg_proc references.
> It can now do that for regprocedure-like references (thus solving the
> problem that regproc is ambiguous for overloaded functions), operators,
> types, opfamilies, opclasses, and access methods. Use this to turn
> nearly all OID cross-references in the initial data into symbolic form.
> This represents a very large step forward in readability and error
> resistance of the initial catalog data. It should also reduce the
> difficulty of renumbering OID assignments in uncommitted patches.
>
> Also, solve the longstanding problem that frontend code that would like to
> use OID macros and other information from the catalog headers often had
> difficulty with backend-only code in the headers. To do this, arrange for
> all generated macros, plus such other declarations as we deem fit, to be
> placed in "derived" header files that are safe for frontend inclusion.
> (Once clients migrate to using these pg_*_d.h headers, it will be possible
> to get rid of the pg_*_fn.h headers, which only exist to quarantine code
> away from clients. That is left for follow-on patches, however.)
>
> The now-automatically-generated macros include the Anum_xxx and Natts_xxx
> constants that we used to have to update by hand when adding or removing
> catalog columns.
>
> Replace the former manual method of generating OID macros for pg_type
> entries with an automatic method, ensuring that all built-in types have
> OID macros. (But note that this patch does not change the way that
> OID macros for pg_proc entries are built and used. It's not clear that
> making that match the other catalogs would be worth extra code churn.)
>
> Add SGML documentation explaining what the new data format is and how to
> work with it.
>
> Despite being a very large change in the catalog headers, there is no
> catversion bump here, because postgres.bki and related output files
> haven't changed at all.
>
> John Naylor, based on ideas from various people; review and minor
> additional coding by me; previous review by Alvaro Herrera
>
> Discussion: https://postgr.es/m/CAJVSVGWO48JbbwXkJz_yBFyGYW-M9YWxnPdxJBUosDC9ou_F0Q@mail.gmail.com
>
> Branch
> ------
> master
>
> Details
> -------
> https://git.postgresql.org/pg/commitdiff/372728b0d49552641f0ea83d9d2e08817de038fa
>
> Modified Files
> --------------
> doc/src/sgml/bki.sgml | 661 +-
> doc/src/sgml/libpq.sgml | 2 +-
> src/Makefile | 5 +
> src/backend/Makefile | 34 +-
> src/backend/catalog/.gitignore | 2 +
> src/backend/catalog/Catalog.pm | 313 +-
> src/backend/catalog/Makefile | 87 +-
> src/backend/catalog/README | 111 -
> src/backend/catalog/genbki.pl | 478 +-
> src/backend/utils/Gen_fmgrtab.pl | 62 +-
> src/backend/utils/Makefile | 15 +-
> src/include/Makefile | 6 +-
> src/include/catalog/.gitignore | 2 +
> src/include/catalog/Makefile | 31 +
> src/include/catalog/duplicate_oids | 6 +-
> src/include/catalog/genbki.h | 29 +-
> src/include/catalog/indexing.h | 2 +-
> src/include/catalog/pg_aggregate.dat | 598 ++
> src/include/catalog/pg_aggregate.h | 336 +-
> src/include/catalog/pg_am.dat | 34 +
> src/include/catalog/pg_am.h | 62 +-
> src/include/catalog/pg_amop.dat | 2472 ++++++
> src/include/catalog/pg_amop.h | 1148 +--
> src/include/catalog/pg_amproc.dat | 1277 +++
> src/include/catalog/pg_amproc.h | 554 +-
> src/include/catalog/pg_attrdef.h | 20 +-
> src/include/catalog/pg_attribute.h | 59 +-
> src/include/catalog/pg_auth_members.h | 22 +-
> src/include/catalog/pg_authid.dat | 67 +
> src/include/catalog/pg_authid.h | 71 +-
> src/include/catalog/pg_cast.dat | 535 ++
> src/include/catalog/pg_cast.h | 368 +-
> src/include/catalog/pg_class.dat | 68 +
> src/include/catalog/pg_class.h | 76 +-
> src/include/catalog/pg_collation.dat | 31 +
> src/include/catalog/pg_collation.h | 42 +-
> src/include/catalog/pg_constraint.h | 56 +-
> src/include/catalog/pg_conversion.h | 29 +-
> src/include/catalog/pg_database.dat | 26 +
> src/include/catalog/pg_database.h | 34 +-
> src/include/catalog/pg_db_role_setting.h | 29 +-
> src/include/catalog/pg_default_acl.h | 27 +-
> src/include/catalog/pg_depend.h | 49 +-
> src/include/catalog/pg_description.h | 37 +-
> src/include/catalog/pg_enum.h | 27 +-
> src/include/catalog/pg_event_trigger.h | 22 +-
> src/include/catalog/pg_extension.h | 29 +-
> src/include/catalog/pg_foreign_data_wrapper.h | 23 +-
> src/include/catalog/pg_foreign_server.h | 23 +-
> src/include/catalog/pg_foreign_table.h | 19 +-
> src/include/catalog/pg_index.h | 38 +-
> src/include/catalog/pg_inherits.h | 24 +-
> src/include/catalog/pg_init_privs.h | 42 +-
> src/include/catalog/pg_language.dat | 31 +
> src/include/catalog/pg_language.h | 39 +-
> src/include/catalog/pg_largeobject.h | 19 +-
> src/include/catalog/pg_largeobject_metadata.h | 18 +-
> src/include/catalog/pg_namespace.dat | 25 +
> src/include/catalog/pg_namespace.h | 37 +-
> src/include/catalog/pg_opclass.dat | 357 +
> src/include/catalog/pg_opclass.h | 222 +-
> src/include/catalog/pg_operator.dat | 3340 ++++++++
> src/include/catalog/pg_operator.h | 1854 +----
> src/include/catalog/pg_opfamily.dat | 248 +
> src/include/catalog/pg_opfamily.h | 170 +-
> src/include/catalog/pg_partitioned_table.h | 24 +-
> src/include/catalog/pg_pltemplate.dat | 51 +
> src/include/catalog/pg_pltemplate.h | 39 +-
> src/include/catalog/pg_policy.h | 29 +-
> src/include/catalog/pg_proc.dat | 10210 ++++++++++++++++++++++++
> src/include/catalog/pg_proc.h | 5637 +------------
> src/include/catalog/pg_publication.h | 24 +-
> src/include/catalog/pg_publication_rel.h | 19 +-
> src/include/catalog/pg_range.dat | 31 +
> src/include/catalog/pg_range.h | 61 +-
> src/include/catalog/pg_replication_origin.h | 22 +-
> src/include/catalog/pg_rewrite.h | 23 +-
> src/include/catalog/pg_seclabel.h | 16 +-
> src/include/catalog/pg_sequence.h | 20 +-
> src/include/catalog/pg_shdepend.h | 42 +-
> src/include/catalog/pg_shdescription.h | 36 +-
> src/include/catalog/pg_shseclabel.h | 18 +-
> src/include/catalog/pg_statistic.h | 44 +-
> src/include/catalog/pg_statistic_ext.h | 26 +-
> src/include/catalog/pg_subscription.h | 20 +-
> src/include/catalog/pg_subscription_rel.h | 19 +-
> src/include/catalog/pg_tablespace.dat | 22 +
> src/include/catalog/pg_tablespace.h | 26 +-
> src/include/catalog/pg_transform.h | 25 +-
> src/include/catalog/pg_trigger.h | 35 +-
> src/include/catalog/pg_ts_config.dat | 19 +
> src/include/catalog/pg_ts_config.h | 29 +-
> src/include/catalog/pg_ts_config_map.dat | 35 +
> src/include/catalog/pg_ts_config_map.h | 47 +-
> src/include/catalog/pg_ts_dict.dat | 20 +
> src/include/catalog/pg_ts_dict.h | 31 +-
> src/include/catalog/pg_ts_parser.dat | 20 +
> src/include/catalog/pg_ts_parser.h | 58 +-
> src/include/catalog/pg_ts_template.dat | 30 +
> src/include/catalog/pg_ts_template.h | 49 +-
> src/include/catalog/pg_type.dat | 942 +++
> src/include/catalog/pg_type.h | 565 +-
> src/include/catalog/pg_user_mapping.h | 19 +-
> src/include/catalog/reformat_dat_file.pl | 311 +
> src/include/catalog/toasting.h | 2 +-
> src/include/catalog/unused_oids | 8 +-
> src/interfaces/ecpg/ecpglib/pg_type.h | 65 +-
> src/tools/msvc/Solution.pm | 49 +-
> src/tools/msvc/clean.bat | 2 +
> src/tools/pgindent/pgindent | 8 +-
> 110 files changed, 22928 insertions(+), 12480 deletions(-)
>
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
Teodor Sigaev <teodor@sigaev.ru> writes:
> Seems, gaur animal doesn't too long shell command. Could "find | xargs"
> help here?
Hm. I think better to cd into the target directory so we can eliminate
all the repetitions of the path. Will fix ...
regards, tom lane