Обсуждение: Schema variables - new implementation for Postgres 15
declare x int ;
begin
for i in 1..1000000
loop
let ooo = i;
end loop;
end;
$$;
Вложения
From: Pavel Stehule <pavel.stehule@gmail.com>
--------------------------------------------------
do $$
declare x int ;
begin
for i in 1..1000000
loop
let ooo = i;
end loop;
end;
$$;
variant 1 .. 1500 ms
variant 2 with PLpgSQL support .. 140 ms
variant 2 without PLpgSQL support 9000 ms
--------------------------------------------------
That's impressive! But 1 million times of variable assignment took only 140 ms? It's that one assignment took only 140 nanosecond, which is near one DRAM access? Can PL/pgSQL processing be really so fast?
Regards
Takayuki Tsunakawa
From: Pavel Stehule <pavel.stehule@gmail.com>
--------------------------------------------------
do $$
declare x int ;
begin
for i in 1..1000000
loop
let ooo = i;
end loop;
end;
$$;
variant 1 .. 1500 ms
variant 2 with PLpgSQL support .. 140 ms
variant 2 without PLpgSQL support 9000 ms
--------------------------------------------------
That's impressive! But 1 million times of variable assignment took only 140 ms? It's that one assignment took only 140 nanosecond, which is near one DRAM access? Can PL/pgSQL processing be really so fast?
Regards
Takayuki Tsunakawa
Hi,I am returning back to implementation of schema variables. The schema variables can be used as an alternative to package variables (Oracle's PL/SQL or ADA). The schema variables can be used as fast and safe storage of session information for RLS too.The previous implementation had not cleanly implemented execution of the LET statement. It was something between query and utility, and although it was working - it was out of Postgres concept (with different implementation of queries and utilities).I totally rewrote the implementation of the LET statement. I prepared two variants:First variant is based on the introduction of the new command type CMD_LET and new very small executor node SetVariable (this is a very very reduced analogy of ModifyTable node). The code is consistent and what is important - the LET statement can be prepared. The execution is relatively fast from PLpgSQL too. Without any special support the execution has the same speed like non simple queries. The statement reuses an execution plan, but simple execution is not supported.Second variant is implemented like a classic utility command. There is not any surprise. It is shorter, simple, but the LET statement cannot be prepared (this is the limit of all utility statements). Without special support in PLpgSQL the execution is about 10x slower than the execution of the first variant. But there is a new possibility of using the main parser from PLpgSQL (implemented by Tom for new implementation of assign statement in pg 14), and then this support in plpgsql requires only a few lines). When the statement LET is explicitly supported by PLpgSQL, then execution is very fast (the speed is comparable with the speed of the assign statement) - it is about 10x faster than the first variant.I tested codedo $$
declare x int ;
begin
for i in 1..1000000
loop
let ooo = i;
end loop;
end;
$$;variant 1 .. 1500 msvariant 2 with PLpgSQL support .. 140 msvariant 2 without PLpgSQL support 9000 msThe slower speed of the first variant from PLpgSQL can be fixed. But for this moment, the speed is good enough. This is the worst case, because in the first variant LET statement cannot use optimization for simple query evaluation (now).Now I think so implementation is significantly cleaner, and I hope so it will be more acceptable for committers.I am starting a new thread, because this is a new implementation, and because I am sending two alternative implementations of one functionality.Comments, notes, objections?
RegardsPavel
Вложения
Attachments:
- schema-variables-v-execnode-2021-01.patch
- schema-variables-v-utility-2021-01.patch
On Thu, Apr 15, 2021, at 10:42, Pavel Stehule wrote:Attachments:
- schema-variables-v-execnode-2021-01.patch
- schema-variables-v-utility-2021-01.patch
Applications are currently know to be misusing set_config()+current_setting() to pass information in a session or transaction.Such users might be interested in using Schema variables as a better replacement.However, since set_config() is transactional, it can't be used as a drop-in replacement:+ <para>+ The value of a schema variable is local to the current session. Retrieving+ a variable's value returns either a NULL or a default value, unless its value+ is set to something else in the current session with a LET command. The content+ of a variable is not transactional. This is the same as in regular variables+ in PL languages.+ </para>I think the "The content of a variable is not transactional." part is therefore a bad idea.Another pattern is to use TEMP TABLEs to pass around information in a session or transaction.If the LET command would be transactional, it could be used as a drop-in replacement for such use-cases as well.Furthermore, I think a non-transactional LET command would be insidious,since it looks like any other SQL command, all of which are transactional.(The ones that aren't such as REINDEX CONCURRENTLY will properly throw an error if run inside a transaction block.)A non-transactional LET command IMO would be non-SQL-idiomatic and non-intuitive.
/Joel
From: Pavel Stehule <pavel.stehule@gmail.com>
--------------------------------------------------
I am sorry, but in this topic we have different opinions. The variables in PLpgSQL are not transactional too (same is true in Perl, Python, ...). Session variables in Oracle, MS SQL, DB2, MySQL are not transactional too. My primary focus is PLpgSQL - and I would like to use schema variables as global plpgsql variables (from PLpgSQL perspective) - that means in Postgres's perspective session variables. But in Postgres, I have to write features that will work with others PL too - PLPython, PLPerl, ... Statement SET in ANSI/SQL standard (SQL/PSM) doesn't expect transactional behaviour for variables too. Unfortunately SET keyword is used in Postgres for GUC, and isn't possible to reuse without a compatibility break.
The PostgreSQL configuration is transactional, but it is a different feature designed for different purposes. Using GUC like session variables is just a workaround. It can be useful for some cases, sure. But it is not usual behaviour. And for other cases the transactional behaviour is not practical. Schema variables are not replacement of GUC, schema variables are not replacement of temporal tables. There is a prepared patch for global temp tables. I hope so this patch can be committed to Postgres 15. Global temp tables fixes almost all disadvantages of temporary tables in Postgres. So the schema variable is not a one row table. It is a different creature - designed to support the server's side procedural features.
--------------------------------------------------
+1
I understand (and wish) this feature is intended to ease migration from Oracle PL/SQL, which will further increase the popularity of Postgres. So, the transactional behavior is not necessary unless Oracle has such a feature.
Furthermore, Postgres already has some non-transactonal SQL commands. So, I don't think we need to reject non-transactional LET.
* Sequence operation: SELECT nextval/setval
* SET [SESSION]
* SET ROLE
* SET SESSION AUTHORIZATION
--------------------------------------------------
I have prepared a patch that allows non default transactional behaviour (but this behaviour should not be default - I didn't design schema variables as temp tables replacement). This patch increases the length of the current patch about 1/4, and I have enough work with rebasing with the current patch, so I didn't send it to commitfest now. If schema variables will be inside core, this day I'll send the patch that allows transactional behaviour for schema variables - I promise.
--------------------------------------------------
I prefer the simpler, targeted one without transactional behavior initially, because added complexity might prevent this feature from being committed in PG 15.
Regards
Takayuki Tsunakawa
Hionly rebase
RegardsPavel
Вложения
st 12. 5. 2021 v 6:17 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Hionly rebasesecond try - rebase after serial_scheduler remove
RegardsPavelRegardsPavel
Вложения
Hirebase
RegardsPavel
Вложения
The value of a schema variable is local to the current session. Retrieving a variable's value returns either a NULL or a default value, unless its value is set to something else in the current session with a LET command. The content of a variable is not transactional. This is the same as in regular variables in PL languages.
Schema variables are retrieved by the SELECT SQL command. Their value is set with the LET SQL command. While schema variables share properties with tables, their value cannot be updated with an UPDATE command.
declare var1 int ; i int;
begin
for i in 1..1000000
loop
var1 := i;
end loop;
end;
$$;
DO
Time: 71,515 ms
declare i int ;
begin
for i in 1..1000000
loop
let var1 = i;
end loop;
end;
$$;
DO
Time: 94,658 ms
ERROR: null value is not allowed for NOT NULL schema variable "var1"
DETAIL: The schema variable was not initialized yet.
postgres=# LET var1=current_date;
ERROR: schema variable "var1" is declared IMMUTABLE
-- Gilles Darold http://www.darold.net/
Hi,Review resume:This patch implements Schema Variables that are database objects that can hold a single or composite value following the data type used at variable declaration. Schema variables, like relations, exist within a schema and their access is controlled via GRANT and REVOKE commands. The schema variable can be created by the CREATE VARIABLE command, altered using ALTER VARIABLE and removed using DROP VARIABLE.The value of a schema variable is local to the current session. Retrieving a variable's value returns either a NULL or a default value, unless its value is set to something else in the current session with a LET command. The content of a variable is not transactional. This is the same as in regular variables in PL languages.
Schema variables are retrieved by the SELECT SQL command. Their value is set with the LET SQL command. While schema variables share properties with tables, their value cannot be updated with an UPDATE command.
The patch apply with the patch command without problem and compilation reports no warning or errors. Regression tests pass successfully using make check or make installcheckIt also includes all documentation and regression tests.Performances are near the set of plpgsql variable settings which is impressive:do $$
declare var1 int ; i int;
begin
for i in 1..1000000
loop
var1 := i;
end loop;
end;
$$;
DO
Time: 71,515 msCREATE VARIABLE var1 AS integer;do $$
declare i int ;
begin
for i in 1..1000000
loop
let var1 = i;
end loop;
end;
$$;
DO
Time: 94,658 msThere is just one thing that puzzles me. We can use :CREATE VARIABLE var1 AS date NOT NULL;postgres=# SELECT var1;
ERROR: null value is not allowed for NOT NULL schema variable "var1"which I understand and is the right behavior. But if we use:CREATE IMMUTABLE VARIABLE var1 AS date NOT NULL;postgres=# SELECT var1;ERROR: null value is not allowed for NOT NULL schema variable "var1"It should probably be better to not allow NOT NULL when IMMUTABLE is used because the variable can not be used at all. Also probably IMMUTABLE without a DEFAULT value should also be restricted as it makes no sens. If the user wants the variable to be NULL he must use DEFAULT NULL. This is just a though, the above error messages are explicit and the user can understand what wrong declaration he have done.
DETAIL: The schema variable was not initialized yet.
postgres=# LET var1=current_date;
ERROR: schema variable "var1" is declared IMMUTABLE
Except that I think this patch is ready for committers, so if there is no other opinion in favor of restricting the use of IMMUTABLE with NOT NULL and DEFAULT I will change the status to ready for committers.-- Gilles Darold http://www.darold.net/
Hi,Review resume:This patch implements Schema Variables that are database objects that can hold a single or composite value following the data type used at variable declaration. Schema variables, like relations, exist within a schema and their access is controlled via GRANT and REVOKE commands. The schema variable can be created by the CREATE VARIABLE command, altered using ALTER VARIABLE and removed using DROP VARIABLE.The value of a schema variable is local to the current session. Retrieving a variable's value returns either a NULL or a default value, unless its value is set to something else in the current session with a LET command. The content of a variable is not transactional. This is the same as in regular variables in PL languages.
Schema variables are retrieved by the SELECT SQL command. Their value is set with the LET SQL command. While schema variables share properties with tables, their value cannot be updated with an UPDATE command.
The patch apply with the patch command without problem and compilation reports no warning or errors. Regression tests pass successfully using make check or make installcheckIt also includes all documentation and regression tests.Performances are near the set of plpgsql variable settings which is impressive:do $$
declare var1 int ; i int;
begin
for i in 1..1000000
loop
var1 := i;
end loop;
end;
$$;
DO
Time: 71,515 msCREATE VARIABLE var1 AS integer;do $$
declare i int ;
begin
for i in 1..1000000
loop
let var1 = i;
end loop;
end;
$$;
DO
Time: 94,658 msThere is just one thing that puzzles me. We can use :CREATE VARIABLE var1 AS date NOT NULL;postgres=# SELECT var1;
ERROR: null value is not allowed for NOT NULL schema variable "var1"which I understand and is the right behavior. But if we use:CREATE IMMUTABLE VARIABLE var1 AS date NOT NULL;postgres=# SELECT var1;ERROR: null value is not allowed for NOT NULL schema variable "var1"It should probably be better to not allow NOT NULL when IMMUTABLE is used because the variable can not be used at all. Also probably IMMUTABLE without a DEFAULT value should also be restricted as it makes no sens. If the user wants the variable to be NULL he must use DEFAULT NULL. This is just a though, the above error messages are explicit and the user can understand what wrong declaration he have done.
DETAIL: The schema variable was not initialized yet.
postgres=# LET var1=current_date;
ERROR: schema variable "var1" is declared IMMUTABLE
Except that I think this patch is ready for committers, so if there is no other opinion in favor of restricting the use of IMMUTABLE with NOT NULL and DEFAULT I will change the status to ready for committers.-- Gilles Darold http://www.darold.net/
Вложения
Hiso 28. 8. 2021 v 11:57 odesílatel Gilles Darold <gilles@darold.net> napsal:Hi,Review resume:This patch implements Schema Variables that are database objects that can hold a single or composite value following the data type used at variable declaration. Schema variables, like relations, exist within a schema and their access is controlled via GRANT and REVOKE commands. The schema variable can be created by the CREATE VARIABLE command, altered using ALTER VARIABLE and removed using DROP VARIABLE.The value of a schema variable is local to the current session. Retrieving a variable's value returns either a NULL or a default value, unless its value is set to something else in the current session with a LET command. The content of a variable is not transactional. This is the same as in regular variables in PL languages.
Schema variables are retrieved by the SELECT SQL command. Their value is set with the LET SQL command. While schema variables share properties with tables, their value cannot be updated with an UPDATE command.
The patch apply with the patch command without problem and compilation reports no warning or errors. Regression tests pass successfully using make check or make installcheckIt also includes all documentation and regression tests.Performances are near the set of plpgsql variable settings which is impressive:do $$
declare var1 int ; i int;
begin
for i in 1..1000000
loop
var1 := i;
end loop;
end;
$$;
DO
Time: 71,515 msCREATE VARIABLE var1 AS integer;do $$
declare i int ;
begin
for i in 1..1000000
loop
let var1 = i;
end loop;
end;
$$;
DO
Time: 94,658 msThere is just one thing that puzzles me. We can use :CREATE VARIABLE var1 AS date NOT NULL;postgres=# SELECT var1;
ERROR: null value is not allowed for NOT NULL schema variable "var1"which I understand and is the right behavior. But if we use:CREATE IMMUTABLE VARIABLE var1 AS date NOT NULL;postgres=# SELECT var1;ERROR: null value is not allowed for NOT NULL schema variable "var1"It should probably be better to not allow NOT NULL when IMMUTABLE is used because the variable can not be used at all. Also probably IMMUTABLE without a DEFAULT value should also be restricted as it makes no sens. If the user wants the variable to be NULL he must use DEFAULT NULL. This is just a though, the above error messages are explicit and the user can understand what wrong declaration he have done.
DETAIL: The schema variable was not initialized yet.
postgres=# LET var1=current_date;
ERROR: schema variable "var1" is declared IMMUTABLEI wrote a check that disables this case. Please, see the attached patch. I agree, so this case is confusing, and it is better to disable it.
Great, I also think that this is better to not confuse the user.
postgres=# CREATE IMMUTABLE VARIABLE var1 AS date NOT NULL;
ERROR: IMMUTABLE NOT NULL variable requires default expression
Working as expected. I have moved the patch to "Ready for committers". Thanks for this feature.
-- Gilles Darold http://www.darold.net/
Great, I also think that this is better to not confuse the user.
postgres=# CREATE IMMUTABLE VARIABLE var1 AS date NOT NULL;
ERROR: IMMUTABLE NOT NULL variable requires default expressionWorking as expected. I have moved the patch to "Ready for committers". Thanks for this feature.
-- Gilles Darold http://www.darold.net/
> [schema-variables-20210909.patch] Hi Pavel, The patch applies and compiles fine but 'make check' for the assert-enabled fails on 131 out of 210 tests. (while compiling HEAD checks run without errors for both assert-disabled and assert-enabled) Erik Rijkers test tablespace ... ok 303 ms parallel group (20 tests): oid char pg_lsn int2 varchar txid int4 regproc uuid float4 text name money boolean bit float8 int8 enum numeric rangetypes boolean ... ok 112 ms char ... ok 57 ms name ... ok 106 ms varchar ... ok 74 ms text ... ok 106 ms int2 ... ok 73 ms int4 ... ok 92 ms int8 ... ok 130 ms oid ... ok 55 ms float4 ... ok 102 ms float8 ... ok 126 ms bit ... ok 124 ms numeric ... ok 362 ms txid ... ok 87 ms uuid ... ok 100 ms enum ... ok 142 ms money ... ok 109 ms rangetypes ... ok 433 ms pg_lsn ... ok 64 ms regproc ... ok 91 ms parallel group (20 tests): lseg path circle time macaddr create_function_0 timetz line macaddr8 numerology point interval inet date strings polygon box multirangetypes timestamp timestamptz strings ... ok 166 ms numerology ... ok 89 ms point ... ok 96 ms lseg ... ok 35 ms line ... ok 70 ms box ... ok 255 ms path ... ok 50 ms polygon ... ok 237 ms circle ... ok 53 ms date ... ok 127 ms time ... ok 60 ms timetz ... ok 67 ms timestamp ... ok 379 ms timestamptz ... ok 413 ms interval ... ok 97 ms inet ... ok 118 ms macaddr ... ok 60 ms macaddr8 ... ok 80 ms multirangetypes ... ok 307 ms create_function_0 ... ok 63 ms parallel group (12 tests): comments unicode misc_sanity tstypes xid expressions horology geometry mvcc type_sanity regex opr_sanity geometry ... ok 140 ms horology ... ok 120 ms tstypes ... ok 53 ms regex ... ok 335 ms type_sanity ... ok 155 ms opr_sanity ... ok 355 ms misc_sanity ... ok 43 ms comments ... ok 20 ms expressions ... ok 100 ms unicode ... ok 25 ms xid ... ok 56 ms mvcc ... ok 146 ms test create_function_1 ... ok 10 ms test create_type ... ok 30 ms test create_table ... ok 333 ms test create_function_2 ... ok 11 ms parallel group (5 tests): copydml copyselect insert_conflict insert copy copy ... ok 336 ms copyselect ... ok 34 ms copydml ... ok 28 ms insert ... ok 291 ms insert_conflict ... FAILED (test process exited with exit code 2) 239 ms parallel group (3 tests): create_operator create_procedure create_misc create_misc ... ok 131 ms create_operator ... ok 29 ms create_procedure ... ok 52 ms parallel group (5 tests): create_view create_index_spgist index_including create_index index_including_gist create_index ... FAILED (test process exited with exit code 2) 3801 ms create_index_spgist ... ok 523 ms create_view ... FAILED (test process exited with exit code 2) 339 ms index_including ... FAILED (test process exited with exit code 2) 3801 ms index_including_gist ... FAILED (test process exited with exit code 2) 3801 ms parallel group (16 tests): create_aggregate create_cast typed_table drop_if_exists roleattributes create_am hash_func updatable_views errors infinite_recurse create_function_3 triggers constraints select inherit vacuum create_aggregate ... FAILED (test process exited with exit code 2) 164 ms create_function_3 ... FAILED (test process exited with exit code 2) 164 ms create_cast ... FAILED (test process exited with exit code 2) 164 ms constraints ... FAILED (test process exited with exit code 2) 181 ms triggers ... FAILED (test process exited with exit code 2) 181 ms select ... FAILED (test process exited with exit code 2) 181 ms inherit ... FAILED (test process exited with exit code 2) 181 ms typed_table ... FAILED (test process exited with exit code 2) 163 ms vacuum ... FAILED (test process exited with exit code 2) 180 ms drop_if_exists ... FAILED (test process exited with exit code 2) 163 ms updatable_views ... FAILED (test process exited with exit code 2) 163 ms roleattributes ... FAILED (test process exited with exit code 2) 163 ms create_am ... FAILED (test process exited with exit code 2) 163 ms hash_func ... FAILED (test process exited with exit code 2) 162 ms errors ... FAILED (test process exited with exit code 2) 162 ms infinite_recurse ... FAILED (test process exited with exit code 2) 162 ms test sanity_check ... FAILED (test process exited with exit code 2) 26 ms parallel group (20 tests): select_into subselect select_distinct arrays join namespace hash_index select_having portals transactions aggregates random update delete union btree_index select_implicit select_distinct_on prepared_xacts case select_into ... FAILED (test process exited with exit code 2) 20 ms select_distinct ... FAILED (test process exited with exit code 2) 21 ms select_distinct_on ... FAILED (test process exited with exit code 2) 26 ms select_implicit ... FAILED (test process exited with exit code 2) 26 ms select_having ... FAILED (test process exited with exit code 2) 23 ms subselect ... FAILED (test process exited with exit code 2) 20 ms union ... FAILED (test process exited with exit code 2) 25 ms case ... FAILED (test process exited with exit code 2) 27 ms join ... FAILED (test process exited with exit code 2) 22 ms aggregates ... FAILED (test process exited with exit code 2) 24 ms transactions ... FAILED (test process exited with exit code 2) 24 ms random ... failed (ignored) (test process exited with exit code 2) 24 ms portals ... FAILED (test process exited with exit code 2) 23 ms arrays ... FAILED (test process exited with exit code 2) 20 ms btree_index ... FAILED (test process exited with exit code 2) 25 ms hash_index ... FAILED (test process exited with exit code 2) 22 ms update ... FAILED (test process exited with exit code 2) 23 ms delete ... FAILED (test process exited with exit code 2) 24 ms namespace ... FAILED (test process exited with exit code 2) 21 ms prepared_xacts ... FAILED (test process exited with exit code 2) 25 ms parallel group (20 tests): gist brin identity generated password tablesample lock matview replica_identity rowsecurity security_label object_address drop_operator groupingsets join_hash privileges collate init_privs spgist gin brin ... FAILED (test process exited with exit code 2) 15 ms gin ... FAILED (test process exited with exit code 2) 22 ms gist ... FAILED (test process exited with exit code 2) 13 ms spgist ... FAILED (test process exited with exit code 2) 22 ms privileges ... FAILED (test process exited with exit code 2) 19 ms init_privs ... FAILED (test process exited with exit code 2) 21 ms security_label ... FAILED (test process exited with exit code 2) 17 ms collate ... FAILED (test process exited with exit code 2) 20 ms matview ... FAILED (test process exited with exit code 2) 17 ms lock ... FAILED (test process exited with exit code 2) 15 ms replica_identity ... FAILED (test process exited with exit code 2) 17 ms rowsecurity ... FAILED (test process exited with exit code 2) 17 ms object_address ... FAILED (test process exited with exit code 2) 17 ms tablesample ... FAILED (test process exited with exit code 2) 15 ms groupingsets ... FAILED (test process exited with exit code 2) 17 ms drop_operator ... FAILED (test process exited with exit code 2) 17 ms password ... FAILED (test process exited with exit code 2) 14 ms identity ... FAILED (test process exited with exit code 2) 13 ms generated ... FAILED (test process exited with exit code 2) 14 ms join_hash ... FAILED (test process exited with exit code 2) 18 ms parallel group (2 tests): brin_multi brin_bloom brin_bloom ... FAILED (test process exited with exit code 2) 4 ms brin_multi ... FAILED (test process exited with exit code 2) 4 ms parallel group (14 tests): async create_table_like collate.icu.utf8 misc sysviews alter_operator tidscan tidrangescan alter_generic tsrf incremental_sort misc_functions tid dbsize create_table_like ... FAILED (test process exited with exit code 2) 10 ms alter_generic ... FAILED (test process exited with exit code 2) 15 ms alter_operator ... FAILED (test process exited with exit code 2) 13 ms misc ... FAILED (test process exited with exit code 2) 11 ms async ... FAILED (test process exited with exit code 2) 9 ms dbsize ... FAILED (test process exited with exit code 2) 17 ms misc_functions ... FAILED (test process exited with exit code 2) 14 ms sysviews ... FAILED (test process exited with exit code 2) 12 ms tsrf ... FAILED (test process exited with exit code 2) 14 ms tid ... FAILED (test process exited with exit code 2) 16 ms tidscan ... FAILED (test process exited with exit code 2) 13 ms tidrangescan ... FAILED (test process exited with exit code 2) 13 ms collate.icu.utf8 ... FAILED (test process exited with exit code 2) 9 ms incremental_sort ... FAILED (test process exited with exit code 2) 13 ms parallel group (6 tests): amutils psql_crosstab collate.linux.utf8 psql stats_ext rules rules ... FAILED (test process exited with exit code 2) 8 ms psql ... FAILED (test process exited with exit code 2) 7 ms psql_crosstab ... FAILED (test process exited with exit code 2) 7 ms amutils ... FAILED (test process exited with exit code 2) 7 ms stats_ext ... FAILED (test process exited with exit code 2) 8 ms collate.linux.utf8 ... FAILED (test process exited with exit code 2) 7 ms test select_parallel ... FAILED (test process exited with exit code 2) 3 ms test write_parallel ... FAILED (test process exited with exit code 2) 3 ms parallel group (2 tests): publication subscription publication ... FAILED (test process exited with exit code 2) 4 ms subscription ... FAILED (test process exited with exit code 2) 4 ms parallel group (17 tests): select_views foreign_key xmlmap window functional_deps tsearch cluster combocid bitmapops tsdicts equivclass guc indirect_toast advisory_lock foreign_data dependency portals_p2 select_views ... FAILED (test process exited with exit code 2) 11 ms portals_p2 ... FAILED (test process exited with exit code 2) 20 ms foreign_key ... FAILED (test process exited with exit code 2) 11 ms cluster ... FAILED (test process exited with exit code 2) 15 ms dependency ... FAILED (test process exited with exit code 2) 19 ms guc ... FAILED (test process exited with exit code 2) 16 ms bitmapops ... FAILED (test process exited with exit code 2) 16 ms combocid ... FAILED (test process exited with exit code 2) 15 ms tsearch ... FAILED (test process exited with exit code 2) 14 ms tsdicts ... FAILED (test process exited with exit code 2) 16 ms foreign_data ... FAILED (test process exited with exit code 2) 17 ms window ... FAILED (test process exited with exit code 2) 12 ms xmlmap ... FAILED (test process exited with exit code 2) 12 ms functional_deps ... FAILED (test process exited with exit code 2) 13 ms advisory_lock ... FAILED (test process exited with exit code 2) 16 ms indirect_toast ... FAILED (test process exited with exit code 2) 16 ms equivclass ... FAILED (test process exited with exit code 2) 15 ms parallel group (6 tests): json json_encoding jsonb jsonpath_encoding jsonb_jsonpath jsonpath json ... FAILED (test process exited with exit code 2) 4 ms jsonb ... FAILED (test process exited with exit code 2) 6 ms json_encoding ... FAILED (test process exited with exit code 2) 5 ms jsonpath ... FAILED (test process exited with exit code 2) 10 ms jsonpath_encoding ... FAILED (test process exited with exit code 2) 6 ms jsonb_jsonpath ... FAILED (test process exited with exit code 2) 7 ms parallel group (19 tests): plpgsql limit rowtypes sequence largeobject returning domain polymorphism plancache prepare alter_table truncate temp rangefuncs with copy2 conversion schema_variables xml plancache ... FAILED (test process exited with exit code 2) 16 ms limit ... FAILED (test process exited with exit code 2) 10 ms plpgsql ... FAILED (test process exited with exit code 2) 7 ms copy2 ... FAILED (test process exited with exit code 2) 25 ms temp ... FAILED (test process exited with exit code 2) 21 ms domain ... FAILED (test process exited with exit code 2) 13 ms rangefuncs ... FAILED (test process exited with exit code 2) 22 ms prepare ... FAILED (test process exited with exit code 2) 19 ms conversion ... FAILED (test process exited with exit code 2) 24 ms truncate ... FAILED (test process exited with exit code 2) 19 ms alter_table ... FAILED (test process exited with exit code 2) 18 ms sequence ... FAILED (test process exited with exit code 2) 11 ms polymorphism ... FAILED (test process exited with exit code 2) 13 ms rowtypes ... FAILED (test process exited with exit code 2) 10 ms returning ... FAILED (test process exited with exit code 2) 11 ms largeobject ... FAILED (test process exited with exit code 2) 10 ms with ... FAILED (test process exited with exit code 2) 22 ms xml ... FAILED (test process exited with exit code 2) 23 ms schema_variables ... FAILED (test process exited with exit code 2) 23 ms parallel group (11 tests): explain hash_part partition_info reloptions memoize compression partition_aggregate partition_join indexing partition_prune tuplesort partition_join ... FAILED 902 ms partition_prune ... ok 1006 ms reloptions ... ok 106 ms hash_part ... ok 99 ms indexing ... ok 929 ms partition_aggregate ... ok 791 ms partition_info ... ok 104 ms tuplesort ... ok 1099 ms explain ... ok 90 ms compression ... ok 214 ms memoize ... ok 109 ms parallel group (2 tests): event_trigger oidjoins event_trigger ... ok 107 ms oidjoins ... ok 157 ms test fast_default ... ok 138 ms test stats ... ok 617 ms On 9/9/21 6:59 AM, Pavel Stehule wrote: > Hi > > fresh rebase > > Regards > > Pavel >
> [schema-variables-20210909.patch]
Hi Pavel,
The patch applies and compiles fine but 'make check' for the
assert-enabled fails on 131 out of 210 tests.
(while compiling HEAD checks run without errors for both assert-disabled
and assert-enabled)
Erik Rijkers
test tablespace ... ok 303 ms
parallel group (20 tests): oid char pg_lsn int2 varchar txid int4
regproc uuid float4 text name money boolean bit float8 int8 enum numeric
rangetypes
boolean ... ok 112 ms
char ... ok 57 ms
name ... ok 106 ms
varchar ... ok 74 ms
text ... ok 106 ms
int2 ... ok 73 ms
int4 ... ok 92 ms
int8 ... ok 130 ms
oid ... ok 55 ms
float4 ... ok 102 ms
float8 ... ok 126 ms
bit ... ok 124 ms
numeric ... ok 362 ms
txid ... ok 87 ms
uuid ... ok 100 ms
enum ... ok 142 ms
money ... ok 109 ms
rangetypes ... ok 433 ms
pg_lsn ... ok 64 ms
regproc ... ok 91 ms
parallel group (20 tests): lseg path circle time macaddr
create_function_0 timetz line macaddr8 numerology point interval inet
date strings polygon box multirangetypes timestamp timestamptz
strings ... ok 166 ms
numerology ... ok 89 ms
point ... ok 96 ms
lseg ... ok 35 ms
line ... ok 70 ms
box ... ok 255 ms
path ... ok 50 ms
polygon ... ok 237 ms
circle ... ok 53 ms
date ... ok 127 ms
time ... ok 60 ms
timetz ... ok 67 ms
timestamp ... ok 379 ms
timestamptz ... ok 413 ms
interval ... ok 97 ms
inet ... ok 118 ms
macaddr ... ok 60 ms
macaddr8 ... ok 80 ms
multirangetypes ... ok 307 ms
create_function_0 ... ok 63 ms
parallel group (12 tests): comments unicode misc_sanity tstypes xid
expressions horology geometry mvcc type_sanity regex opr_sanity
geometry ... ok 140 ms
horology ... ok 120 ms
tstypes ... ok 53 ms
regex ... ok 335 ms
type_sanity ... ok 155 ms
opr_sanity ... ok 355 ms
misc_sanity ... ok 43 ms
comments ... ok 20 ms
expressions ... ok 100 ms
unicode ... ok 25 ms
xid ... ok 56 ms
mvcc ... ok 146 ms
test create_function_1 ... ok 10 ms
test create_type ... ok 30 ms
test create_table ... ok 333 ms
test create_function_2 ... ok 11 ms
parallel group (5 tests): copydml copyselect insert_conflict insert copy
copy ... ok 336 ms
copyselect ... ok 34 ms
copydml ... ok 28 ms
insert ... ok 291 ms
insert_conflict ... FAILED (test process exited with
exit code 2) 239 ms
parallel group (3 tests): create_operator create_procedure create_misc
create_misc ... ok 131 ms
create_operator ... ok 29 ms
create_procedure ... ok 52 ms
parallel group (5 tests): create_view create_index_spgist
index_including create_index index_including_gist
create_index ... FAILED (test process exited with
exit code 2) 3801 ms
create_index_spgist ... ok 523 ms
create_view ... FAILED (test process exited with
exit code 2) 339 ms
index_including ... FAILED (test process exited with
exit code 2) 3801 ms
index_including_gist ... FAILED (test process exited with
exit code 2) 3801 ms
parallel group (16 tests): create_aggregate create_cast typed_table
drop_if_exists roleattributes create_am hash_func updatable_views errors
infinite_recurse create_function_3 triggers constraints select inherit
vacuum
create_aggregate ... FAILED (test process exited with
exit code 2) 164 ms
create_function_3 ... FAILED (test process exited with
exit code 2) 164 ms
create_cast ... FAILED (test process exited with
exit code 2) 164 ms
constraints ... FAILED (test process exited with
exit code 2) 181 ms
triggers ... FAILED (test process exited with
exit code 2) 181 ms
select ... FAILED (test process exited with
exit code 2) 181 ms
inherit ... FAILED (test process exited with
exit code 2) 181 ms
typed_table ... FAILED (test process exited with
exit code 2) 163 ms
vacuum ... FAILED (test process exited with
exit code 2) 180 ms
drop_if_exists ... FAILED (test process exited with
exit code 2) 163 ms
updatable_views ... FAILED (test process exited with
exit code 2) 163 ms
roleattributes ... FAILED (test process exited with
exit code 2) 163 ms
create_am ... FAILED (test process exited with
exit code 2) 163 ms
hash_func ... FAILED (test process exited with
exit code 2) 162 ms
errors ... FAILED (test process exited with
exit code 2) 162 ms
infinite_recurse ... FAILED (test process exited with
exit code 2) 162 ms
test sanity_check ... FAILED (test process exited with
exit code 2) 26 ms
parallel group (20 tests): select_into subselect select_distinct arrays
join namespace hash_index select_having portals transactions aggregates
random update delete union btree_index select_implicit
select_distinct_on prepared_xacts case
select_into ... FAILED (test process exited with
exit code 2) 20 ms
select_distinct ... FAILED (test process exited with
exit code 2) 21 ms
select_distinct_on ... FAILED (test process exited with
exit code 2) 26 ms
select_implicit ... FAILED (test process exited with
exit code 2) 26 ms
select_having ... FAILED (test process exited with
exit code 2) 23 ms
subselect ... FAILED (test process exited with
exit code 2) 20 ms
union ... FAILED (test process exited with
exit code 2) 25 ms
case ... FAILED (test process exited with
exit code 2) 27 ms
join ... FAILED (test process exited with
exit code 2) 22 ms
aggregates ... FAILED (test process exited with
exit code 2) 24 ms
transactions ... FAILED (test process exited with
exit code 2) 24 ms
random ... failed (ignored) (test process
exited with exit code 2) 24 ms
portals ... FAILED (test process exited with
exit code 2) 23 ms
arrays ... FAILED (test process exited with
exit code 2) 20 ms
btree_index ... FAILED (test process exited with
exit code 2) 25 ms
hash_index ... FAILED (test process exited with
exit code 2) 22 ms
update ... FAILED (test process exited with
exit code 2) 23 ms
delete ... FAILED (test process exited with
exit code 2) 24 ms
namespace ... FAILED (test process exited with
exit code 2) 21 ms
prepared_xacts ... FAILED (test process exited with
exit code 2) 25 ms
parallel group (20 tests): gist brin identity generated password
tablesample lock matview replica_identity rowsecurity security_label
object_address drop_operator groupingsets join_hash privileges collate
init_privs spgist gin
brin ... FAILED (test process exited with
exit code 2) 15 ms
gin ... FAILED (test process exited with
exit code 2) 22 ms
gist ... FAILED (test process exited with
exit code 2) 13 ms
spgist ... FAILED (test process exited with
exit code 2) 22 ms
privileges ... FAILED (test process exited with
exit code 2) 19 ms
init_privs ... FAILED (test process exited with
exit code 2) 21 ms
security_label ... FAILED (test process exited with
exit code 2) 17 ms
collate ... FAILED (test process exited with
exit code 2) 20 ms
matview ... FAILED (test process exited with
exit code 2) 17 ms
lock ... FAILED (test process exited with
exit code 2) 15 ms
replica_identity ... FAILED (test process exited with
exit code 2) 17 ms
rowsecurity ... FAILED (test process exited with
exit code 2) 17 ms
object_address ... FAILED (test process exited with
exit code 2) 17 ms
tablesample ... FAILED (test process exited with
exit code 2) 15 ms
groupingsets ... FAILED (test process exited with
exit code 2) 17 ms
drop_operator ... FAILED (test process exited with
exit code 2) 17 ms
password ... FAILED (test process exited with
exit code 2) 14 ms
identity ... FAILED (test process exited with
exit code 2) 13 ms
generated ... FAILED (test process exited with
exit code 2) 14 ms
join_hash ... FAILED (test process exited with
exit code 2) 18 ms
parallel group (2 tests): brin_multi brin_bloom
brin_bloom ... FAILED (test process exited with
exit code 2) 4 ms
brin_multi ... FAILED (test process exited with
exit code 2) 4 ms
parallel group (14 tests): async create_table_like collate.icu.utf8
misc sysviews alter_operator tidscan tidrangescan alter_generic tsrf
incremental_sort misc_functions tid dbsize
create_table_like ... FAILED (test process exited with
exit code 2) 10 ms
alter_generic ... FAILED (test process exited with
exit code 2) 15 ms
alter_operator ... FAILED (test process exited with
exit code 2) 13 ms
misc ... FAILED (test process exited with
exit code 2) 11 ms
async ... FAILED (test process exited with
exit code 2) 9 ms
dbsize ... FAILED (test process exited with
exit code 2) 17 ms
misc_functions ... FAILED (test process exited with
exit code 2) 14 ms
sysviews ... FAILED (test process exited with
exit code 2) 12 ms
tsrf ... FAILED (test process exited with
exit code 2) 14 ms
tid ... FAILED (test process exited with
exit code 2) 16 ms
tidscan ... FAILED (test process exited with
exit code 2) 13 ms
tidrangescan ... FAILED (test process exited with
exit code 2) 13 ms
collate.icu.utf8 ... FAILED (test process exited with
exit code 2) 9 ms
incremental_sort ... FAILED (test process exited with
exit code 2) 13 ms
parallel group (6 tests): amutils psql_crosstab collate.linux.utf8 psql
stats_ext rules
rules ... FAILED (test process exited with
exit code 2) 8 ms
psql ... FAILED (test process exited with
exit code 2) 7 ms
psql_crosstab ... FAILED (test process exited with
exit code 2) 7 ms
amutils ... FAILED (test process exited with
exit code 2) 7 ms
stats_ext ... FAILED (test process exited with
exit code 2) 8 ms
collate.linux.utf8 ... FAILED (test process exited with
exit code 2) 7 ms
test select_parallel ... FAILED (test process exited with
exit code 2) 3 ms
test write_parallel ... FAILED (test process exited with
exit code 2) 3 ms
parallel group (2 tests): publication subscription
publication ... FAILED (test process exited with
exit code 2) 4 ms
subscription ... FAILED (test process exited with
exit code 2) 4 ms
parallel group (17 tests): select_views foreign_key xmlmap window
functional_deps tsearch cluster combocid bitmapops tsdicts equivclass
guc indirect_toast advisory_lock foreign_data dependency portals_p2
select_views ... FAILED (test process exited with
exit code 2) 11 ms
portals_p2 ... FAILED (test process exited with
exit code 2) 20 ms
foreign_key ... FAILED (test process exited with
exit code 2) 11 ms
cluster ... FAILED (test process exited with
exit code 2) 15 ms
dependency ... FAILED (test process exited with
exit code 2) 19 ms
guc ... FAILED (test process exited with
exit code 2) 16 ms
bitmapops ... FAILED (test process exited with
exit code 2) 16 ms
combocid ... FAILED (test process exited with
exit code 2) 15 ms
tsearch ... FAILED (test process exited with
exit code 2) 14 ms
tsdicts ... FAILED (test process exited with
exit code 2) 16 ms
foreign_data ... FAILED (test process exited with
exit code 2) 17 ms
window ... FAILED (test process exited with
exit code 2) 12 ms
xmlmap ... FAILED (test process exited with
exit code 2) 12 ms
functional_deps ... FAILED (test process exited with
exit code 2) 13 ms
advisory_lock ... FAILED (test process exited with
exit code 2) 16 ms
indirect_toast ... FAILED (test process exited with
exit code 2) 16 ms
equivclass ... FAILED (test process exited with
exit code 2) 15 ms
parallel group (6 tests): json json_encoding jsonb jsonpath_encoding
jsonb_jsonpath jsonpath
json ... FAILED (test process exited with
exit code 2) 4 ms
jsonb ... FAILED (test process exited with
exit code 2) 6 ms
json_encoding ... FAILED (test process exited with
exit code 2) 5 ms
jsonpath ... FAILED (test process exited with
exit code 2) 10 ms
jsonpath_encoding ... FAILED (test process exited with
exit code 2) 6 ms
jsonb_jsonpath ... FAILED (test process exited with
exit code 2) 7 ms
parallel group (19 tests): plpgsql limit rowtypes sequence largeobject
returning domain polymorphism plancache prepare alter_table truncate
temp rangefuncs with copy2 conversion schema_variables xml
plancache ... FAILED (test process exited with
exit code 2) 16 ms
limit ... FAILED (test process exited with
exit code 2) 10 ms
plpgsql ... FAILED (test process exited with
exit code 2) 7 ms
copy2 ... FAILED (test process exited with
exit code 2) 25 ms
temp ... FAILED (test process exited with
exit code 2) 21 ms
domain ... FAILED (test process exited with
exit code 2) 13 ms
rangefuncs ... FAILED (test process exited with
exit code 2) 22 ms
prepare ... FAILED (test process exited with
exit code 2) 19 ms
conversion ... FAILED (test process exited with
exit code 2) 24 ms
truncate ... FAILED (test process exited with
exit code 2) 19 ms
alter_table ... FAILED (test process exited with
exit code 2) 18 ms
sequence ... FAILED (test process exited with
exit code 2) 11 ms
polymorphism ... FAILED (test process exited with
exit code 2) 13 ms
rowtypes ... FAILED (test process exited with
exit code 2) 10 ms
returning ... FAILED (test process exited with
exit code 2) 11 ms
largeobject ... FAILED (test process exited with
exit code 2) 10 ms
with ... FAILED (test process exited with
exit code 2) 22 ms
xml ... FAILED (test process exited with
exit code 2) 23 ms
schema_variables ... FAILED (test process exited with
exit code 2) 23 ms
parallel group (11 tests): explain hash_part partition_info reloptions
memoize compression partition_aggregate partition_join indexing
partition_prune tuplesort
partition_join ... FAILED 902 ms
partition_prune ... ok 1006 ms
reloptions ... ok 106 ms
hash_part ... ok 99 ms
indexing ... ok 929 ms
partition_aggregate ... ok 791 ms
partition_info ... ok 104 ms
tuplesort ... ok 1099 ms
explain ... ok 90 ms
compression ... ok 214 ms
memoize ... ok 109 ms
parallel group (2 tests): event_trigger oidjoins
event_trigger ... ok 107 ms
oidjoins ... ok 157 ms
test fast_default ... ok 138 ms
test stats ... ok 617 ms
On 9/9/21 6:59 AM, Pavel Stehule wrote:
> Hi
>
> fresh rebase
>
> Regards
>
> Pavel
>
Hičt 9. 9. 2021 v 12:21 odesílatel Erik Rijkers <er@xs4all.nl> napsal:> [schema-variables-20210909.patch]
Hi Pavel,
The patch applies and compiles fine but 'make check' for the
assert-enabled fails on 131 out of 210 tests.This morning I tested it. I'll recheck it.Pavel
I had not this problem yesterday.
-- Gilles Darold http://www.darold.net/
Le 09/09/2021 à 11:40, Pavel Stehule a écrit :Hičt 9. 9. 2021 v 12:21 odesílatel Erik Rijkers <er@xs4all.nl> napsal:> [schema-variables-20210909.patch]
Hi Pavel,
The patch applies and compiles fine but 'make check' for the
assert-enabled fails on 131 out of 210 tests.This morning I tested it. I'll recheck it.Pavel
I had not this problem yesterday.
-- Gilles Darold http://www.darold.net/
> [schema-variables-20210909.patch]
Hi Pavel,
The patch applies and compiles fine but 'make check' for the
assert-enabled fails on 131 out of 210 tests.
(while compiling HEAD checks run without errors for both assert-disabled
and assert-enabled)
Вложения
On 9/10/21 10:06 AM, Pavel Stehule wrote: > Hi > > čt 9. 9. 2021 v 12:21 odesílatel Erik Rijkers <er@xs4all.nl> napsal: > >> >> Hi Pavel, >> >> The patch applies and compiles fine but 'make check' for the >> assert-enabled fails on 131 out of 210 tests. >> >> (while compiling HEAD checks run without errors for both assert-disabled >> and assert-enabled) >> >> > > Please, check, attached patch. I fixed a routine for processing a list of > identifiers - now it works with the identifier's node more sensitive. > Previous implementation of strVal was more tolerant. > [schema-variables-20210910.patch] Apply, compile, make, & check(-world), and my small testsuite OK. So all's well again - Ready for committer! Thanks, Erik Rijkers > Regards > > Pavel > > >> >
On 9/10/21 10:06 AM, Pavel Stehule wrote:
> Hi
>
> čt 9. 9. 2021 v 12:21 odesílatel Erik Rijkers <er@xs4all.nl> napsal:
>
>>
>> Hi Pavel,
>>
>> The patch applies and compiles fine but 'make check' for the
>> assert-enabled fails on 131 out of 210 tests.
>>
>> (while compiling HEAD checks run without errors for both assert-disabled
>> and assert-enabled)
>>
>>
>
> Please, check, attached patch. I fixed a routine for processing a list of
> identifiers - now it works with the identifier's node more sensitive.
> Previous implementation of strVal was more tolerant.
> [schema-variables-20210910.patch]
Apply, compile, make, & check(-world), and my small testsuite OK.
So all's well again - Ready for committer!
Thanks,
Erik Rijkers
> Regards
>
> Pavel
>
>
>>
>
On Fri, Sep 10, 2021 at 10:06:04AM +0200, Pavel Stehule wrote: > Hi > > čt 9. 9. 2021 v 12:21 odesílatel Erik Rijkers <er@xs4all.nl> napsal: > > > > [schema-variables-20210909.patch] > > > > Hi Pavel, > > > > The patch applies and compiles fine but 'make check' for the > > assert-enabled fails on 131 out of 210 tests. > > > > (while compiling HEAD checks run without errors for both assert-disabled > > and assert-enabled) > > > > > > Please, check, attached patch. I fixed a routine for processing a list of > identifiers - now it works with the identifier's node more sensitive. > Previous implementation of strVal was more tolerant. > Hi Pavel, Just noted that there is no support for REASSIGN OWNED BY: """ regression=# create variable random_number numeric; CREATE VARIABLE regression=# alter variable random_number owner to jcm; ALTER VARIABLE regression=# reassign owned by jcm to jaime; ERROR: unexpected classid 9222 """ TEMP variables are not schema variables? at least not attached to the schema one expects: """ regression=# create temp variable random_number numeric ; CREATE VARIABLE regression=# \dV List of variables Schema | Name | Type | Is nullable | Is mutable | Default | Owner | Transaction al end action -----------+---------------+---------+-------------+------------+---------+----------+------------ -------------- pg_temp_4 | random_number | numeric | t | t | | jcasanov | (1 row) regression=# select public.random_number; ERROR: missing FROM-clause entry for table "public" LINE 1: select public.random_number; ^ """ There was a comment that TEMP variables should be DECLAREd instead of CREATEd, i guess that is because those have similar behaviour. At least, I would like to see similar messages when using the ON COMMIT DROP option in a TEMP variable: """ regression=# create temp variable random_number numeric on commit drop; CREATE VARIABLE regression=# \dV Did not find any schema variables. regression=# declare q cursor for select 1; ERROR: DECLARE CURSOR can only be used in transaction blocks """ About that, why are you not using syntax ON COMMIT RESET instead on inventing ON TRANSACTION END RESET? seems better because you already use ON COMMIT DROP. I will test more this patch tomorrow. Great work, very complete. -- Jaime Casanova Director de Servicios Profesionales SystemGuards - Consultores de PostgreSQL
Just noted that there is no support for REASSIGN OWNED BY:
"""
regression=# create variable random_number numeric;
CREATE VARIABLE
regression=# alter variable random_number owner to jcm;
ALTER VARIABLE
regression=# reassign owned by jcm to jaime;
ERROR: unexpected classid 9222
"""
TEMP variables are not schema variables? at least not attached to the
schema one expects:
"""
regression=# create temp variable random_number numeric ;
CREATE VARIABLE
regression=# \dV
List of variables
Schema | Name | Type | Is nullable | Is mutable | Default | Owner | Transaction
al end action
-----------+---------------+---------+-------------+------------+---------+----------+------------
--------------
pg_temp_4 | random_number | numeric | t | t | | jcasanov |
(1 row)
regression=# select public.random_number;
ERROR: missing FROM-clause entry for table "public"
LINE 1: select public.random_number;
^
"""
There was a comment that TEMP variables should be DECLAREd instead of
CREATEd, i guess that is because those have similar behaviour. At least,
I would like to see similar messages when using the ON COMMIT DROP
option in a TEMP variable:
"""
regression=# create temp variable random_number numeric on commit drop;
CREATE VARIABLE
regression=# \dV
Did not find any schema variables.
regression=# declare q cursor for select 1;
ERROR: DECLARE CURSOR can only be used in transaction blocks
"""
CREATE VARIABLE
postgres=# \dV
List of variables
┌────────┬───────────────┬─────────┬─────────────┬────────────┬─────────┬───────┬──────────────────────────┐
│ Schema │ Name │ Type │ Is nullable │ Is mutable │ Default │ Owner │ Transactional end action │
╞════════╪═══════════════╪═════════╪═════════════╪════════════╪═════════╪═══════╪══════════════════════════╡
│ public │ random_number │ numeric │ t │ t │ │ tom2 │ │
└────────┴───────────────┴─────────┴─────────────┴────────────┴─────────┴───────┴──────────────────────────┘
(1 row)
About that, why are you not using syntax ON COMMIT RESET instead on
inventing ON TRANSACTION END RESET? seems better because you already use
ON COMMIT DROP.
I will test more this patch tomorrow. Great work, very complete.
--
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL
Вложения
Hi
Just noted that there is no support for REASSIGN OWNED BY:
"""
regression=# create variable random_number numeric;
CREATE VARIABLE
regression=# alter variable random_number owner to jcm;
ALTER VARIABLE
regression=# reassign owned by jcm to jaime;
ERROR: unexpected classid 9222
"""should be fixed by the attached patch, please check.
TEMP variables are not schema variables? at least not attached to the
schema one expects:temp variables are schema variables like any other. But they are created in temp schema - like temp tables.I designed it in consistency with temporary tables.
"""
regression=# create temp variable random_number numeric ;
CREATE VARIABLE
regression=# \dV
List of variables
Schema | Name | Type | Is nullable | Is mutable | Default | Owner | Transaction
al end action
-----------+---------------+---------+-------------+------------+---------+----------+------------
--------------
pg_temp_4 | random_number | numeric | t | t | | jcasanov |
(1 row)
regression=# select public.random_number;
ERROR: missing FROM-clause entry for table "public"
LINE 1: select public.random_number;
^
"""
There was a comment that TEMP variables should be DECLAREd instead of
CREATEd, i guess that is because those have similar behaviour. At least,
I would like to see similar messages when using the ON COMMIT DROP
option in a TEMP variable:I don't remember this comment. When I talked about similarity with the DECLARE statement, I thought about semantic similarity with T-SQL (Microsoft SQL) DECLARE command. Unfortunately, DECLARE command is pretty messy - it exists in SQL, it exists in SQL/PSM and it exists in T-SQL - and every time has similar syntax, but partially different semantics. For me - CREATE TEMP VARIABLE creates session's life limited variable (by default), similarly like DECLARE @localvariable command from T-SQL.
"""
regression=# create temp variable random_number numeric on commit drop;
CREATE VARIABLE
regression=# \dV
Did not find any schema variables.
regression=# declare q cursor for select 1;
ERROR: DECLARE CURSOR can only be used in transaction blocks
"""I have different resultpostgres=# create temp variable random_number numeric on commit drop;
CREATE VARIABLE
postgres=# \dV
List of variables
┌────────┬───────────────┬─────────┬─────────────┬────────────┬─────────┬───────┬──────────────────────────┐
│ Schema │ Name │ Type │ Is nullable │ Is mutable │ Default │ Owner │ Transactional end action │
╞════════╪═══════════════╪═════════╪═════════════╪════════════╪═════════╪═══════╪══════════════════════════╡
│ public │ random_number │ numeric │ t │ t │ │ tom2 │ │
└────────┴───────────────┴─────────┴─────────────┴────────────┴─────────┴───────┴──────────────────────────┘
(1 row)
About that, why are you not using syntax ON COMMIT RESET instead on
inventing ON TRANSACTION END RESET? seems better because you already use
ON COMMIT DROP.I thought about this question for a very long time, and I think the introduction of a new clause is better, and I will try to explain why.One part of this patch are DDL statements - and all DDL statements are consistent with other DDL statements in Postgres. Schema variables DDL commands are transactional and for TEMP variables we can specify a scope - session or transaction, and then clause ON COMMIT DROP is used. You should not need to specify ON ROLLBACK action, because in this case an removing from system catalogue is only one possible action.Second part of this patch is holding some value in schema variables or initialization with default expression. The default behaviour is not transactional, and the value is stored all session's time by default. But I think it can be very useful to enforce initialization in some specific times - now only the end of the transaction is possible to specify. In the future there can be transaction end, transaction start, rollback, commit, top query start, top query end, ... This logic is different from the logic of DDL commands. For DDL commands I need to specify behaviour just for the COMMIT end. But for reset of non-transactional schema variables I need to specify any possible end of transaction - COMMIT, ROLLBACK or COMMIT or ROLLBACK. In this initial version I implemented "ON COMMIT OR ROLLBACK RESET", and although it is clean I think it is more readable is the clause that I invented "ON TRANSACTION END". "ON COMMIT RESET" is not exact. "ON COMMIT OR ROLLBACK RESET" sounds a little bit strange for me, but we use something similar in trigger definition "ON INSERT OR UPDATE OR DELETE ..." My opinion is not too strong if "ON TRANSACTION END RESET" or "ON COMMIT OR ROLLBACK RESET" is better, and I can change it if people will have different preferences, but I am sure so "ON COMMIT RESET" is not correct in implemented case. And from the perspective of a PLpgSQL developer, I would have initialized the variable on any transaction start, so I need to reset it on any end.RegardsPavel
I will test more this patch tomorrow. Great work, very complete.
--
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL
On Sun, Sep 12, 2021 at 05:38:42PM +0200, Pavel Stehule wrote: > Hi > > > """ > > regression=# create temp variable random_number numeric on commit drop; > > CREATE VARIABLE > > regression=# \dV > > Did not find any schema variables. > > regression=# declare q cursor for select 1; > > ERROR: DECLARE CURSOR can only be used in transaction blocks > > """ > > > > I have different result > > postgres=# create temp variable random_number numeric on commit drop; > CREATE VARIABLE > postgres=# \dV > List of variables > ┌────────┬───────────────┬─────────┬─────────────┬────────────┬─────────┬───────┬──────────────────────────┐ > │ Schema │ Name │ Type │ Is nullable │ Is mutable │ Default │ > Owner │ Transactional end action │ > ╞════════╪═══════════════╪═════════╪═════════════╪════════════╪═════════╪═══════╪══════════════════════════╡ > │ public │ random_number │ numeric │ t │ t │ │ > tom2 │ │ > └────────┴───────────────┴─────────┴─────────────┴────────────┴─────────┴───────┴──────────────────────────┘ > (1 row) > > > Hi, Thanks, will test rebased version. BTW, that is not the temp variable. You can note it because of the schema or the lack of a "Transaction end action". That is a normal non-temp variable that has been created before. A TEMP variable with an ON COMMIT DROP created outside an explicit transaction will disappear immediatly like cursor does in the same situation. -- Jaime Casanova Director de Servicios Profesionales SystemGuards - Consultores de PostgreSQL
Hi,
Thanks, will test rebased version.
BTW, that is not the temp variable. You can note it because of the
schema or the lack of a "Transaction end action". That is a normal
non-temp variable that has been created before. A TEMP variable with an
ON COMMIT DROP created outside an explicit transaction will disappear
immediatly like cursor does in the same situation.
regression=# create temp variable random_number numeric ;
CREATE VARIABLE
regression=# \dV
List of variables
Schema | Name | Type | Is nullable | Is mutable | Default | Owner | Transaction
al end action
-----------+---------------+---------+-------------+------------+---------+----------+------------
--------------
pg_temp_4 | random_number | numeric | t | t | | jcasanov |
(1 row)
regression=# select public.random_number;
ERROR: missing FROM-clause entry for table "public"
LINE 1: select public.random_number;
^
"""
--
Jaime Casanova
Director de Servicios Profesionales
SystemGuards - Consultores de PostgreSQL
Hijust rebase of patchRegardsPavel
Вложения
Hi, I took a quick look at the latest patch version. In general the patch looks pretty complete and clean, and for now I have only some basic comments. The attached patch tweaks some of this, along with a couple additional minor changes that I'll not discuss here. 1) Not sure why we need to call this "schema variables". Most objects are placed in a schema, and we don't say "schema tables" for example. And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so it's a bit inconsistent. The docs actually use "Global variables" in one place for some reason. 2) I find this a bit confusing: SELECT non_existent_variable; test=# select s; ERROR: column "non_existent_variable" does not exist LINE 1: select non_existent_variable; I wonder if this means using SELECT to read variables is a bad idea, and we should have a separate command, just like we have LET (instead of just using UPDATE in some way). 3) I've reworded / tweaked a couple places in the docs, but this really needs a native speaker - I don't have a very good "feeling" for this technical language so it's probably still quite cumbersome. 4) Is sequential scan of the hash table in clean_cache_callback() a good idea? I wonder how fast (with how many variables) it'll become noticeable, but it may be good enough for now and we can add something better (tracing which variables need resetting) later. 5) In what situation would we call clean_cache_callback() without a transaction state? If that happens it seems more like a bug, so maybeelog(ERROR) or Assert() would be more appropriate? 6) free_schema_variable does not actually use the force parameter 7) The target_exprkind expression in transformSelectStmt really needs some explanation. Because that's chance you'll look at this in 6 months and understand what it does? target_exprkind = (pstate->p_expr_kind != EXPR_KIND_LET_TARGET || pstate->parentParseState != NULL) ? EXPR_KIND_SELECT_TARGET : EXPR_KIND_LET_TARGET; 8) immutable variables without a default value IMO this case should not be allowed. On 2021/08/29 you wrote: I thought about this case, and I have one scenario, where this behaviour can be useful. When the variable is declared as IMMUTABLE NOT NULL without not null default, then any access to the content of the variable has to fail. I think it can be used for detection, where and when the variable is first used. So this behavior is allowed just because I think, so this feature can be interesting for debugging. If this idea is too strange, I have no problem to disable this case. This seems like a really strange use case. In a production code you'll not do this, because then the variable is useless and the code does not work at all (it'll just fail whenever it attempts to access the var). And if you can modify the code, there are other / better ways to do this (raising an exception, ...). So this seems pretty useless to me, +1 to disabling it. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Вложения
FWIW the patch was marked as RFC for about a year, but there was plenty of discussion / changes since then, so that seemed premature. I've switched it back to WoA. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Hi,
I took a quick look at the latest patch version. In general the patch
looks pretty complete and clean, and for now I have only some basic
comments. The attached patch tweaks some of this, along with a couple
additional minor changes that I'll not discuss here.
1) Not sure why we need to call this "schema variables". Most objects
are placed in a schema, and we don't say "schema tables" for example.
And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so it's a bit
inconsistent.
The docs actually use "Global variables" in one place for some reason.
2) I find this a bit confusing:
SELECT non_existent_variable;
test=# select s;
ERROR: column "non_existent_variable" does not exist
LINE 1: select non_existent_variable;
I wonder if this means using SELECT to read variables is a bad idea, and
we should have a separate command, just like we have LET (instead of
just using UPDATE in some way).
3) I've reworded / tweaked a couple places in the docs, but this really
needs a native speaker - I don't have a very good "feeling" for this
technical language so it's probably still quite cumbersome.
4) Is sequential scan of the hash table in clean_cache_callback() a
good idea? I wonder how fast (with how many variables) it'll become
noticeable, but it may be good enough for now and we can add something
better (tracing which variables need resetting) later.
5) In what situation would we call clean_cache_callback() without a
transaction state? If that happens it seems more like a bug, so
maybeelog(ERROR) or Assert() would be more appropriate?
6) free_schema_variable does not actually use the force parameter
7) The target_exprkind expression in transformSelectStmt really needs
some explanation. Because that's chance you'll look at this in 6 months
and understand what it does?
target_exprkind =
(pstate->p_expr_kind != EXPR_KIND_LET_TARGET ||
pstate->parentParseState != NULL) ?
EXPR_KIND_SELECT_TARGET : EXPR_KIND_LET_TARGET;
8) immutable variables without a default value
IMO this case should not be allowed. On 2021/08/29 you wrote:
I thought about this case, and I have one scenario, where this
behaviour can be useful. When the variable is declared as IMMUTABLE
NOT NULL without not null default, then any access to the content of
the variable has to fail. I think it can be used for detection,
where and when the variable is first used. So this behavior is
allowed just because I think, so this feature can be interesting for
debugging. If this idea is too strange, I have no problem to disable
this case.
This seems like a really strange use case. In a production code you'll
not do this, because then the variable is useless and the code does not
work at all (it'll just fail whenever it attempts to access the var).
And if you can modify the code, there are other / better ways to do this
(raising an exception, ...).
So this seems pretty useless to me, +1 to disabling it.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sat, Nov 06, 2021 at 04:45:19AM +0100, Pavel Stehule wrote: > st 3. 11. 2021 v 14:05 odesílatel Tomas Vondra <tomas.vondra@enterprisedb.com> napsal: > > 1) Not sure why we need to call this "schema variables". Most objects > > are placed in a schema, and we don't say "schema tables" for example. > > And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so it's a bit > > inconsistent. +1 At least the error messages need to be consistent. It doesn't make sense to have both of these: + elog(ERROR, "cache lookup failed for schema variable %u", varid); + elog(ERROR, "cache lookup failed for variable %u", varid); > Yes, there is inconsistency, but I think it is necessary. The name > "variable" is too generic. Theoretically we can use other adjectives like > session variables or global variables and the name will be valid. But it > doesn't describe the fundamentals of design. This is similar to the > package's variables from PL/SQL. These variables are global, session's > variables too. But the usual name is "package variables". So schema > variables are assigned to schemes, and I think a good name can be "schema > variables". But it is not necessary to repeat keyword schema in the CREATE > COMMAND. > > My opinion is not too strong in this case, and I can accept just > "variables" or "session's variables" or "global variables", but I am not > sure if these names describe this feature well, because still they are too > generic. There are too many different implementations of session global > variables (see PL/SQL or T-SQL or DB2). I would prefer "session variable". To me, this feature seems similar to a CTE (which exists for a single statement), or a temporary table (which exists for a single transaction). So "session" conveys a lot more of its meaning than "schema". But don't rename everything just for me... -- Justin
On Sat, Nov 06, 2021 at 04:45:19AM +0100, Pavel Stehule wrote:
> st 3. 11. 2021 v 14:05 odesílatel Tomas Vondra <tomas.vondra@enterprisedb.com> napsal:
> > 1) Not sure why we need to call this "schema variables". Most objects
> > are placed in a schema, and we don't say "schema tables" for example.
> > And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so it's a bit
> > inconsistent.
+1
At least the error messages need to be consistent.
It doesn't make sense to have both of these:
+ elog(ERROR, "cache lookup failed for schema variable %u", varid);
+ elog(ERROR, "cache lookup failed for variable %u", varid);
> Yes, there is inconsistency, but I think it is necessary. The name
> "variable" is too generic. Theoretically we can use other adjectives like
> session variables or global variables and the name will be valid. But it
> doesn't describe the fundamentals of design. This is similar to the
> package's variables from PL/SQL. These variables are global, session's
> variables too. But the usual name is "package variables". So schema
> variables are assigned to schemes, and I think a good name can be "schema
> variables". But it is not necessary to repeat keyword schema in the CREATE
> COMMAND.
>
> My opinion is not too strong in this case, and I can accept just
> "variables" or "session's variables" or "global variables", but I am not
> sure if these names describe this feature well, because still they are too
> generic. There are too many different implementations of session global
> variables (see PL/SQL or T-SQL or DB2).
I would prefer "session variable".
To me, this feature seems similar to a CTE (which exists for a single
statement), or a temporary table (which exists for a single transaction). So
"session" conveys a lot more of its meaning than "schema".
But don't rename everything just for me...
--
Justin
On 11/6/21 16:40, Pavel Stehule wrote: > > > so 6. 11. 2021 v 15:57 odesílatel Justin Pryzby <pryzby@telsasoft.com > <mailto:pryzby@telsasoft.com>> napsal: > > On Sat, Nov 06, 2021 at 04:45:19AM +0100, Pavel Stehule wrote: > > st 3. 11. 2021 v 14:05 odesílatel Tomas Vondra > <tomas.vondra@enterprisedb.com > <mailto:tomas.vondra@enterprisedb.com>> napsal: > > > 1) Not sure why we need to call this "schema variables". Most > objects > > > are placed in a schema, and we don't say "schema tables" for > example. > > > And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so > it's a bit > > > inconsistent. > > +1 > > At least the error messages need to be consistent. > It doesn't make sense to have both of these: > > + elog(ERROR, "cache lookup failed for schema variable > %u", varid); > + elog(ERROR, "cache lookup failed for variable %u", > varid); > > > Yes, there is inconsistency, but I think it is necessary. The name > > "variable" is too generic. Theoretically we can use other > adjectives like > > session variables or global variables and the name will be valid. > But it > > doesn't describe the fundamentals of design. This is similar to the > > package's variables from PL/SQL. These variables are global, > session's > > variables too. But the usual name is "package variables". So schema > > variables are assigned to schemes, and I think a good name can be > "schema > > variables". But it is not necessary to repeat keyword schema in > the CREATE > > COMMAND. > > > > My opinion is not too strong in this case, and I can accept just > > "variables" or "session's variables" or "global variables", but I > am not > > sure if these names describe this feature well, because still > they are too > > generic. There are too many different implementations of session > global > > variables (see PL/SQL or T-SQL or DB2). > > I would prefer "session variable". > > To me, this feature seems similar to a CTE (which exists for a single > statement), or a temporary table (which exists for a single > transaction). So > "session" conveys a lot more of its meaning than "schema". > > > It depends on where you are looking. There are two perspectives - data > and metadata. And if I use data perspective, then it is session related. > If I use metadata perspective, then it can be persistent or temporal > like tables. I think you mean "temporary" not "temporal". This really confused me for a while, because temporal means "involving time" (e.g. a table with from/to timestamp range, etc). > I see strong similarity with Global Temporary Tables - but > I think naming "local temporary tables" and "global temporary tables" > can be not intuitive or messy for a lot of people too. Right, it's a bit like global temporary tables, in the sense that there's a shared definition but local (session) state. > Anyway, if people will try to find this feature on Google, then > probably use keywords "session variables", so maybe my preference of > more technical terminology is obscure and not practical, and the name > "session variables" can be more practical for other people. Hmmm, maybe. > If I use the system used for GTT - then the exact name can be "Global > Session Variable". Can we use this name? Or shortly just Session > Variables because we don't support local session variables now. So a "local variable" would be defined just for a given session, just like a temporary table? Wouldn't that have the same issues with catalog bloat as temporary tables? I'd probably vote for "session variables". We can call it local/global session variables in the future, if we end up implementing that. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 11/6/21 04:45, Pavel Stehule wrote: > Hi > > st 3. 11. 2021 v 14:05 odesílatel Tomas Vondra > <tomas.vondra@enterprisedb.com <mailto:tomas.vondra@enterprisedb.com>> > napsal: > > Hi, > > I took a quick look at the latest patch version. In general the patch > looks pretty complete and clean, and for now I have only some basic > comments. The attached patch tweaks some of this, along with a couple > additional minor changes that I'll not discuss here. > > > 1) Not sure why we need to call this "schema variables". Most objects > are placed in a schema, and we don't say "schema tables" for example. > And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so it's a bit > inconsistent. > > > Yes, there is inconsistency, but I think it is necessary. The name > "variable" is too generic. Theoretically we can use other adjectives > like session variables or global variables and the name will be valid. > But it doesn't describe the fundamentals of design. This is similar to > the package's variables from PL/SQL. These variables are global, > session's variables too. But the usual name is "package variables". So > schema variables are assigned to schemes, and I think a good name can be > "schema variables". But it is not necessary to repeat keyword schema in > the CREATE COMMAND. > > My opinion is not too strong in this case, and I can accept just > "variables" or "session's variables" or "global variables", but I am not > sure if these names describe this feature well, because still they are > too generic. There are too many different implementations of session > global variables (see PL/SQL or T-SQL or DB2). > OK. "Session variable" seems better to me, but I'm not sure how well that matches other databases. I'm not sure how much should we feel constrained by naming in other databases, though. > > The docs actually use "Global variables" in one place for some reason. > > > 2) I find this a bit confusing: > > SELECT non_existent_variable; > test=# select s; > ERROR: column "non_existent_variable" does not exist > LINE 1: select non_existent_variable; > > I wonder if this means using SELECT to read variables is a bad idea, and > we should have a separate command, just like we have LET (instead of > just using UPDATE in some way). > > > I am sure so I want to use variables in SELECTs. One interesting case is > using variables in RLS. > How much more complicated would it be without the SELECT? > I prefer to fix this error message to "column or variable ... does not > exist" > Not sure it's a good idea to make the error message more ambiguous. Most people won't use variables at all, and the message will be less clear for them. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 11/6/21 04:45, Pavel Stehule wrote:
> Hi
>
> st 3. 11. 2021 v 14:05 odesílatel Tomas Vondra
> <tomas.vondra@enterprisedb.com <mailto:tomas.vondra@enterprisedb.com>>
> napsal:
>
> Hi,
>
> I took a quick look at the latest patch version. In general the patch
> looks pretty complete and clean, and for now I have only some basic
> comments. The attached patch tweaks some of this, along with a couple
> additional minor changes that I'll not discuss here.
>
>
> 1) Not sure why we need to call this "schema variables". Most objects
> are placed in a schema, and we don't say "schema tables" for example.
> And it's CREATE VARIABLE and not CREATE SCHEMA VARIABLE, so it's a bit
> inconsistent.
>
>
> Yes, there is inconsistency, but I think it is necessary. The name
> "variable" is too generic. Theoretically we can use other adjectives
> like session variables or global variables and the name will be valid.
> But it doesn't describe the fundamentals of design. This is similar to
> the package's variables from PL/SQL. These variables are global,
> session's variables too. But the usual name is "package variables". So
> schema variables are assigned to schemes, and I think a good name can be
> "schema variables". But it is not necessary to repeat keyword schema in
> the CREATE COMMAND.
>
> My opinion is not too strong in this case, and I can accept just
> "variables" or "session's variables" or "global variables", but I am not
> sure if these names describe this feature well, because still they are
> too generic. There are too many different implementations of session
> global variables (see PL/SQL or T-SQL or DB2).
>
OK. "Session variable" seems better to me, but I'm not sure how well
that matches other databases. I'm not sure how much should we feel
constrained by naming in other databases, though.
>
> The docs actually use "Global variables" in one place for some reason.
>
>
> 2) I find this a bit confusing:
>
> SELECT non_existent_variable;
> test=# select s;
> ERROR: column "non_existent_variable" does not exist
> LINE 1: select non_existent_variable;
>
> I wonder if this means using SELECT to read variables is a bad idea, and
> we should have a separate command, just like we have LET (instead of
> just using UPDATE in some way).
>
>
> I am sure so I want to use variables in SELECTs. One interesting case is
> using variables in RLS.
>
How much more complicated would it be without the SELECT?
> I prefer to fix this error message to "column or variable ... does not
> exist"
>
Not sure it's a good idea to make the error message more ambiguous. Most
people won't use variables at all, and the message will be less clear
for them.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Wed, Nov 03, 2021 at 02:05:02PM +0100, Tomas Vondra wrote:
> 3) I've reworded / tweaked a couple places in the docs, but this really
> needs a native speaker - I don't have a very good "feeling" for this
> technical language so it's probably still quite cumbersome.
On Daniel's suggestion, I have reviewed the docs, and then proofread the rest
of the patch. My amendments are in 0003.
--
Justin
my last patch has 276KB and your patch has 293KB?
On Mon, Nov 15, 2021 at 09:00:13PM +0100, Pavel Stehule wrote:
> Thank you for review and fixes, I try to complete some version for next
> work, and looks so your patch 0001 is broken
>
> gedit reports to me broken unicode \A0\A0\A0\A0\A0
>
> my last patch has 276KB and your patch has 293KB?
On Mon, Nov 15, 2021 at 09:06:08PM +0100, Pavel Stehule wrote:
> >
> > my last patch has 276KB and your patch has 293KB?
>
> Please, can you resend your version of patch 0001?
https://www.postgresql.org/message-id/20211106013904.GG17618@telsasoft.com
0001 is exactly your patch applied to HEAD, and 0002 are Tomas' changes
relative to your patch.
0003 is my contribution on top. My intent is that you wouldn't apply 0001, but
rather apply my 0003 on top of your existing branch, and then review 0002/0003,
and then squish the changes into your patch.
I see the 0xa0 stuff in your original patch before my changes, but I'm not sure
what went wrong.
Let me know if you have any issue applying my changes on top of your existing,
local branch ?
--
Justin
8) immutable variables without a default value
IMO this case should not be allowed. On 2021/08/29 you wrote:
I thought about this case, and I have one scenario, where this
behaviour can be useful. When the variable is declared as IMMUTABLE
NOT NULL without not null default, then any access to the content of
the variable has to fail. I think it can be used for detection,
where and when the variable is first used. So this behavior is
allowed just because I think, so this feature can be interesting for
debugging. If this idea is too strange, I have no problem to disable
this case.
ERROR: IMMUTABLE NOT NULL variable requires default expression
On Mon, Nov 15, 2021 at 09:00:13PM +0100, Pavel Stehule wrote:
> Thank you for review and fixes, I try to complete some version for next
> work, and looks so your patch 0001 is broken
>
> gedit reports to me broken unicode \A0\A0\A0\A0\A0
>
> my last patch has 276KB and your patch has 293KB?
On Mon, Nov 15, 2021 at 09:06:08PM +0100, Pavel Stehule wrote:
> >
> > my last patch has 276KB and your patch has 293KB?
>
> Please, can you resend your version of patch 0001?
https://www.postgresql.org/message-id/20211106013904.GG17618@telsasoft.com
0001 is exactly your patch applied to HEAD, and 0002 are Tomas' changes
relative to your patch.
0003 is my contribution on top. My intent is that you wouldn't apply 0001, but
rather apply my 0003 on top of your existing branch, and then review 0002/0003,
and then squish the changes into your patch.
I see the 0xa0 stuff in your original patch before my changes, but I'm not sure
what went wrong.
Let me know if you have any issue applying my changes on top of your existing,
local branch ?
--
Justin
Вложения
Op 19-12-2021 om 07:23 schreef Pavel Stehule: > > I am sending new versions of patches. > > I hope I solved all Tomas's objections. > > 1. The schema variables were renamed to session variables > 2. I fixed issues related to creating, dropping variables under > subtransactions + regress tests > 3. I fixed issues in pg_dump + regress tests > > [0001-schema-variables-20211219.patch] > [0002-schema-variables-20211219.patch] Hi Pavel, I get an error during test 'session_variables'. (on the upside, my own little testsuite runs without error) thanks, Erik Rijkers
Вложения
Op 19-12-2021 om 07:23 schreef Pavel Stehule:
>
> I am sending new versions of patches.
>
> I hope I solved all Tomas's objections.
>
> 1. The schema variables were renamed to session variables
> 2. I fixed issues related to creating, dropping variables under
> subtransactions + regress tests
> 3. I fixed issues in pg_dump + regress tests
>
> [0001-schema-variables-20211219.patch]
> [0002-schema-variables-20211219.patch]
Hi Pavel,
I get an error during test 'session_variables'.
(on the upside, my own little testsuite runs without error)
thanks,
Erik Rijkers
ne 19. 12. 2021 v 8:09 odesílatel Erik Rijkers <er@xs4all.nl> napsal:Op 19-12-2021 om 07:23 schreef Pavel Stehule:
>
> I am sending new versions of patches.
>
> I hope I solved all Tomas's objections.
>
> 1. The schema variables were renamed to session variables
> 2. I fixed issues related to creating, dropping variables under
> subtransactions + regress tests
> 3. I fixed issues in pg_dump + regress tests
>
> [0001-schema-variables-20211219.patch]
> [0002-schema-variables-20211219.patch]
Hi Pavel,
I get an error during test 'session_variables'.
(on the upside, my own little testsuite runs without error)
thanks,please, can you send me regress diff?
RegardsPavel
Erik Rijkers
Op 19-12-2021 om 08:13 schreef Pavel Stehule: > > > ne 19. 12. 2021 v 8:09 odesílatel Erik Rijkers <er@xs4all.nl > > > > > [0001-schema-variables-20211219.patch] > > [0002-schema-variables-20211219.patch] > > Hi Pavel, > > I get an error during test 'session_variables'. > > (on the upside, my own little testsuite runs without error) > > thanks, > > > please, can you send me regress diff? > I did attach it but if you did not receive it, see also cfbot, especially https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.156992 Erik
Op 19-12-2021 om 08:13 schreef Pavel Stehule:
>
>
> ne 19. 12. 2021 v 8:09 odesílatel Erik Rijkers <er@xs4all.nl
> >
>
> > [0001-schema-variables-20211219.patch]
> > [0002-schema-variables-20211219.patch]
>
> Hi Pavel,
>
> I get an error during test 'session_variables'.
>
> (on the upside, my own little testsuite runs without error)
>
> thanks,
>
>
> please, can you send me regress diff?
>
I did attach it but if you did not receive it, see also cfbot, especially
https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.156992
Erik
Вложения
> [0001-schema-variables-20211219-2.patch] > [0002-schema-variables-20211219-2.patch] Hi Pavel, You said earlier > 1. The schema variables were renamed to session variable But I still see: $ grep -Eic 'schema variable' postgres.html 15 (postgres.html from 'make postgres.html') So that rename doesn't seem finished. Erik
> [0001-schema-variables-20211219-2.patch]
> [0002-schema-variables-20211219-2.patch]
Hi Pavel,
You said earlier
> 1. The schema variables were renamed to session variable
But I still see:
$ grep -Eic 'schema variable' postgres.html
15
(postgres.html from 'make postgres.html')
So that rename doesn't seem finished.
Erik
Вложения
Вложения
Od: Pavel Stehule <pavel.stehule@gmail.com>
Date: po 27. 12. 2021 v 5:30
Subject: Re: Schema variables - new implementation for Postgres 15
To: Justin Pryzby <pryzby@telsasoft.com>
> > Maybe "session variable" should be added to the glossary.
>
> done
+ A persistent database object that holds an value in session memory.
+ This memory is not shared across sessions, and after session end, this
+ memory (the value) is released. The access (read or write) to session variables
+ is controlled by access rigths similary to other database object access rigts.
an value => a value
rigths => rights
rigts => rights
Вложения
Вложения
On Wed, 3 Nov 2021 at 13:05, Tomas Vondra <tomas.vondra@enterprisedb.com> wrote: > > 2) I find this a bit confusing: > > SELECT non_existent_variable; > test=# select s; > ERROR: column "non_existent_variable" does not exist > LINE 1: select non_existent_variable; > > I wonder if this means using SELECT to read variables is a bad idea, and > we should have a separate command, just like we have LET (instead of > just using UPDATE in some way). > Hmm. This way of reading variables worries me for a different reason -- I think it makes it all too easy to break existing applications by inadvertently (or deliberately) defining variables that conflict with column names referred to in existing queries. For example, if I define a variable called "relkind", then psql's \sv meta-command is broken because the query it performs can't distinguish between the column and the variable. Similarly, there's ambiguity between alias.colname and schema.variablename. So, for example, if I do the following: CREATE SCHEMA n; CREATE VARIABLE n.nspname AS int; then lots of things are broken, including pg_dump and a number of psql meta-commands. I don't think it's acceptable to make it so easy for a user to break the system in this way. Those are examples that a malicious user might use, but even without such examples, I think it would be far too easy to inadvertently break a large application by defining a variable that conflicted with a column name you didn't know about. Regards, Dean
On Wed, 3 Nov 2021 at 13:05, Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>
> 2) I find this a bit confusing:
>
> SELECT non_existent_variable;
> test=# select s;
> ERROR: column "non_existent_variable" does not exist
> LINE 1: select non_existent_variable;
>
> I wonder if this means using SELECT to read variables is a bad idea, and
> we should have a separate command, just like we have LET (instead of
> just using UPDATE in some way).
>
Hmm. This way of reading variables worries me for a different reason
-- I think it makes it all too easy to break existing applications by
inadvertently (or deliberately) defining variables that conflict with
column names referred to in existing queries.
For example, if I define a variable called "relkind", then psql's \sv
meta-command is broken because the query it performs can't distinguish
between the column and the variable.
Similarly, there's ambiguity between alias.colname and
schema.variablename. So, for example, if I do the following:
CREATE SCHEMA n;
CREATE VARIABLE n.nspname AS int;
then lots of things are broken, including pg_dump and a number of psql
meta-commands. I don't think it's acceptable to make it so easy for a
user to break the system in this way.
Those are examples that a malicious user might use, but even without
such examples, I think it would be far too easy to inadvertently break
a large application by defining a variable that conflicted with a
column name you didn't know about.
Regards,
Dean
On Thu, Jan 13, 2022, at 18:24, Dean Rasheed wrote:> Those are examples that a malicious user might use, but even without> such examples, I think it would be far too easy to inadvertently break> a large application by defining a variable that conflicted with a> column name you didn't know about.I think there is also a readability problem with the non-locality of this feature.I think it would be better to have an explicit namespace for these global variables, so that when reading code, they would stand-out.As a bonus, that would also solve the risk of breaking code, as you pointed out.Most code should never need any global variables at all, so in the rare occasions when they are needed, I think it's perfectly fine if some more verbose fully-qualified syntax was needed to use them, rather than to pollute the namespace and risk breaking code.
I want to bring up an idea presented earlier in a different thread:How about exploiting reserved SQL keywords followed by a dot, as special labels?This could solve the problem with this patch, as well as the other root label patch to access function parameters.It's an unorthodox idea, but due to legacy, I think we need to be creative, if we want a safe solution with no risk of breaking any code, which I think should be a requirement.Taking inspiration from Javascript, how about using the SQL reserved keyword "window"?In Javascript, "window.variableName" means that the variable variableName declared at the global scope.
Furthermore:"from" could be used to access function/procedure IN parameters."to" could be used to access function OUT parameters."from" or "to" could be used to access function INOUT parameters.Examples:SELECT u.user_idINTO to.user_idFROM users uWHERE u.username = from.username;-- After authentication, the authenticated user_id could be stored as a global variable:window.user_id := to.user_id;-- The authenticated user_id could then be used in queries that should filter on user_id:SELECT o.order_idFROM orders oWHERE o.user_id = window.user_id;This would require endorsement from the SQL committee of course, otherwise we would face problems if they suddenly would introduce syntax where a reserved keyword could be followed by a dot.I think from a readability perspective, it works, since the different meanings can be distinguished by writing one in UPPERCASE and the other in lowercase./Joel
On Thu, Jan 13, 2022, at 20:12, Pavel Stehule wrote:>I cannot imagine how the "window" keyword can work in SQL context. In Javascript "window" is an object - it is not a keyword, and it makes sense in usual Javascript context inside HTML browsers.I was thinking since Javascript is by far the most known programming language, the "window" word would be familiar and easy to remember, but I agree, it's not perfect.
Hm, "values" would be nice, it's reserved in SQL:2016 [1] and in DB2/Mimer/MySQL/Oracle/SQL Server/Teradata [2], but unfortunately not in PostgreSQL [1], so perhaps not doable.Syntax:values.[schema name].[variable name]
On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com> wrote: > > I like the idea of prioritizing tables over variables with warnings when collision is detected. It cannot break anything.And it allows to using short identifiers when there is not collision. Yeah, that seems OK, as long as it's clearly documented. I don't think a warning is necessary. (FWIW, testing with dbfiddle, that appears to match Db2's behaviour). Regards, Dean
On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
> I like the idea of prioritizing tables over variables with warnings when collision is detected. It cannot break anything. And it allows to using short identifiers when there is not collision.
Yeah, that seems OK, as long as it's clearly documented. I don't think
a warning is necessary.
(FWIW, testing with dbfiddle, that appears to match Db2's behaviour).
Regards,
Dean
Hi, On Thu, Jan 13, 2022 at 07:32:26PM +0100, Pavel Stehule wrote: > čt 13. 1. 2022 v 19:23 odesílatel Dean Rasheed <dean.a.rasheed@gmail.com> > napsal: > > > On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com> > > wrote: > > > > > > I like the idea of prioritizing tables over variables with warnings when > > collision is detected. It cannot break anything. And it allows to using > > short identifiers when there is not collision. > > > > Yeah, that seems OK, as long as it's clearly documented. I don't think > > a warning is necessary. What should be the behavior for a cached plan that uses a variable when a conflicting relation is later created? I think that it should be the same as a search_path change and the plan should be discarded. > The warning can be disabled by default, but I think it should be there. > This is a signal, so some in the database schema should be renamed. Maybe - > session_variables_ambiguity_warning. I agree that having a way to know that a variable has been bypassed can be useful. > > (FWIW, testing with dbfiddle, that appears to match Db2's behaviour). > > > > Thank you for check Do you know what's oracle's behavior on that? I've been looking at the various dependency handling, and I noticed that collation are ignored, while they're accepted syntax-wise: =# create collation mycollation (locale = 'fr-FR', provider = 'icu'); CREATE COLLATION =# create variable myvariable text collate mycollation; CREATE VARIABLE =# select classid::regclass, objid, objsubid, refclassid::regclass, refobjid, refobjsubid from pg_depend where classid::regclass::text= 'pg_variable' or refclassid::regclass::text = 'pg_variable'; classid | objid | objsubid | refclassid | refobjid | refobjsubid -------------+-------+----------+--------------+----------+------------- pg_variable | 16407 | 0 | pg_namespace | 2200 | 0 (1 row) =# let myvariable = 'AA'; LET =# select 'AA' collate "en-x-icu" < myvariable; ?column? ---------- f (1 row) =# select 'AA' collate "en-x-icu" < myvariable collate mycollation; ERROR: 42P21: collation mismatch between explicit collations "en-x-icu" and "mycollation" LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat... So it's missing both dependency recording for variable's collation and also teaching various code that variables can have a collation. It's also missing some invalidation detection. For instance: =# create variable myval text; CREATE VARIABLE =# let myval = 'pg_class'; LET =# prepare s(text) as select relname from pg_class where relname = $1 or relname = myval; PREPARE =# set plan_cache_mode = force_generic_plan ; SET =# execute s (''); relname ---------- pg_class (1 row) =# drop variable myval ; DROP VARIABLE =# create variable myval int; CREATE VARIABLE =# execute s (''); ERROR: XX000: cache lookup failed for session variable 16408 The plan should have been discarded and the new plan should fail for type problem. Strangely, subsequent calls don't error out: =# execute s(''); relname --------- (0 rows) But doing an explain shows that there's a problem: =# explain execute s(''); ERROR: XX000: cache lookup failed for variable 16408
Hi,
On Thu, Jan 13, 2022 at 07:32:26PM +0100, Pavel Stehule wrote:
> čt 13. 1. 2022 v 19:23 odesílatel Dean Rasheed <dean.a.rasheed@gmail.com>
> napsal:
>
> > On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com>
> > wrote:
> > >
> > > I like the idea of prioritizing tables over variables with warnings when
> > collision is detected. It cannot break anything. And it allows to using
> > short identifiers when there is not collision.
> >
> > Yeah, that seems OK, as long as it's clearly documented. I don't think
> > a warning is necessary.
What should be the behavior for a cached plan that uses a variable when a
conflicting relation is later created? I think that it should be the same as a
search_path change and the plan should be discarded.
> The warning can be disabled by default, but I think it should be there.
> This is a signal, so some in the database schema should be renamed. Maybe -
> session_variables_ambiguity_warning.
I agree that having a way to know that a variable has been bypassed can be
useful.
> > (FWIW, testing with dbfiddle, that appears to match Db2's behaviour).
> >
>
> Thank you for check
Do you know what's oracle's behavior on that?
I've been looking at the various dependency handling, and I noticed that
collation are ignored, while they're accepted syntax-wise:
=# create collation mycollation (locale = 'fr-FR', provider = 'icu');
CREATE COLLATION
=# create variable myvariable text collate mycollation;
CREATE VARIABLE
=# select classid::regclass, objid, objsubid, refclassid::regclass, refobjid, refobjsubid from pg_depend where classid::regclass::text = 'pg_variable' or refclassid::regclass::text = 'pg_variable';
classid | objid | objsubid | refclassid | refobjid | refobjsubid
-------------+-------+----------+--------------+----------+-------------
pg_variable | 16407 | 0 | pg_namespace | 2200 | 0
(1 row)
=# let myvariable = 'AA';
LET
=# select 'AA' collate "en-x-icu" < myvariable;
?column?
----------
f
(1 row)
=# select 'AA' collate "en-x-icu" < myvariable collate mycollation;
ERROR: 42P21: collation mismatch between explicit collations "en-x-icu" and "mycollation"
LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat...
So it's missing both dependency recording for variable's collation and also
teaching various code that variables can have a collation.
It's also missing some invalidation detection. For instance:
=# create variable myval text;
CREATE VARIABLE
=# let myval = 'pg_class';
LET
=# prepare s(text) as select relname from pg_class where relname = $1 or relname = myval;
PREPARE
=# set plan_cache_mode = force_generic_plan ;
SET
=# execute s ('');
relname
----------
pg_class
(1 row)
=# drop variable myval ;
DROP VARIABLE
=# create variable myval int;
CREATE VARIABLE
=# execute s ('');
ERROR: XX000: cache lookup failed for session variable 16408
The plan should have been discarded and the new plan should fail for type
problem.
Strangely, subsequent calls don't error out:
=# execute s('');
relname
---------
(0 rows)
But doing an explain shows that there's a problem:
=# explain execute s('');
ERROR: XX000: cache lookup failed for variable 16408
For example, if I define a variable called "relkind", then psql's \sv
meta-command is broken because the query it performs can't distinguish
between the column and the variable.
Hi, On Fri, Jan 14, 2022 at 07:49:09AM -0300, Marcos Pegoraro wrote: > > > > For example, if I define a variable called "relkind", then psql's \sv > > meta-command is broken because the query it performs can't distinguish > > between the column and the variable. > > > If variables use : as prefix you´ll never have these conflicts. > > select relkind from pg_class where relkind = :relkind This is already used by psql client side variables, so this is not an option.
For example, if I define a variable called "relkind", then psql's \sv
meta-command is broken because the query it performs can't distinguish
between the column and the variable.If variables use : as prefix you´ll never have these conflicts.select relkind from pg_class where relkind = :relkind
Hi,
On Thu, Jan 13, 2022 at 07:32:26PM +0100, Pavel Stehule wrote:
> čt 13. 1. 2022 v 19:23 odesílatel Dean Rasheed <dean.a.rasheed@gmail.com>
> napsal:
>
> > On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com>
> > wrote:
> > >
> > > I like the idea of prioritizing tables over variables with warnings when
> > collision is detected. It cannot break anything. And it allows to using
> > short identifiers when there is not collision.
> >
> > Yeah, that seems OK, as long as it's clearly documented. I don't think
> > a warning is necessary.
What should be the behavior for a cached plan that uses a variable when a
conflicting relation is later created? I think that it should be the same as a
search_path change and the plan should be discarded.
> The warning can be disabled by default, but I think it should be there.
> This is a signal, so some in the database schema should be renamed. Maybe -
> session_variables_ambiguity_warning.
I agree that having a way to know that a variable has been bypassed can be
useful.
> > (FWIW, testing with dbfiddle, that appears to match Db2's behaviour).
> >
>
> Thank you for check
Do you know what's oracle's behavior on that?
I've been looking at the various dependency handling, and I noticed that
collation are ignored, while they're accepted syntax-wise:
=# "
CREATE COLLATION
=# create variable myvariable text collate mycollation;
CREATE VARIABLE
=# select classid::regclass, objid, objsubid, refclassid::regclass, refobjid, refobjsubid from pg_depend where classid::regclass::text = 'pg_variable' or refclassid::regclass::text = 'pg_variable';
classid | objid | objsubid | refclassid | refobjid | refobjsubid
-------------+-------+----------+--------------+----------+-------------
pg_variable | 16407 | 0 | pg_namespace | 2200 | 0
(1 row)
=# let myvariable = 'AA';
LET
=# select 'AA' collate "en-x-icu" < myvariable;
?column?
----------
f
(1 row)
=# select 'AA' collate "en-x-icu" < myvariable collate mycollation;
ERROR: 42P21: collation mismatch between explicit collations "en-x-icu" and "mycollation"
LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat...
│ ?column? │
╞══════════╡
│ f │
└──────────┘
(1 row)
postgres=# select 'AA' collate "en-x-icu" < mycol collate mycollation from foo;
ERROR: collation mismatch between explicit collations "en-x-icu" and "mycollation"
LINE 1: select 'AA' collate "en-x-icu" < mycol collate mycollation f...
^
So it's missing both dependency recording for variable's collation and also
teaching various code that variables can have a collation.
It's also missing some invalidation detection. For instance:
=# create variable myval text;
CREATE VARIABLE
=# let myval = 'pg_class';
LET
=# prepare s(text) as select relname from pg_class where relname = $1 or relname = myval;
PREPARE
=# set plan_cache_mode = force_generic_plan ;
SET
=# execute s ('');
relname
----------
pg_class
(1 row)
=# drop variable myval ;
DROP VARIABLE
=# create variable myval int;
CREATE VARIABLE
=# execute s ('');
ERROR: XX000: cache lookup failed for session variable 16408
The plan should have been discarded and the new plan should fail for type
problem.
Strangely, subsequent calls don't error out:
=# execute s('');
relname
---------
(0 rows)
But doing an explain shows that there's a problem:
=# explain execute s('');
ERROR: XX000: cache lookup failed for variable 16408
Вложения
=# "
CREATE COLLATION
=# create variable myvariable text collate mycollation;
CREATE VARIABLE
=# select classid::regclass, objid, objsubid, refclassid::regclass, refobjid, refobjsubid from pg_depend where classid::regclass::text = 'pg_variable' or refclassid::regclass::text = 'pg_variable';
classid | objid | objsubid | refclassid | refobjid | refobjsubid
-------------+-------+----------+--------------+----------+-------------
pg_variable | 16407 | 0 | pg_namespace | 2200 | 0
(1 row)fixed
=# let myvariable = 'AA';
LET
=# select 'AA' collate "en-x-icu" < myvariable;
?column?
----------
f
(1 row)
=# select 'AA' collate "en-x-icu" < myvariable collate mycollation;
ERROR: 42P21: collation mismatch between explicit collations "en-x-icu" and "mycollation"
LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat...What do you expect? I don't understand collating well, but it looks correct. Minimally the tables have the same behavior.create collation mycollation (locale = 'fr-FR', provider = 'icu');create table foo(mycol text collate mycollation);select 'AA' collate "en-x-icu" < mycol from foo;┌──────────┐
│ ?column? │
╞══════════╡
│ f │
└──────────┘
(1 row)
postgres=# select 'AA' collate "en-x-icu" < mycol collate mycollation from foo;
ERROR: collation mismatch between explicit collations "en-x-icu" and "mycollation"
LINE 1: select 'AA' collate "en-x-icu" < mycol collate mycollation f...
^
provider = 'icu',
locale = '@colStrength=secondary', -- or 'und-u-ks-level2'
deterministic = false
);
CREATE COLLATION
postgres=# create variable testv as text col
postgres=# create variable testv as text collate nd2;
CREATE VARIABLE
postgres=# let testv = 'Ahoj';
LET
postgres=# select testv = 'AHOJ';
┌──────────┐
│ ?column? │
╞══════════╡
│ t │
└──────────┘
(1 row)
┌──────────┐
│ ?column? │
╞══════════╡
│ f │
└──────────┘
(1 row)
Hi, On Tue, Jan 18, 2022 at 10:01:01PM +0100, Pavel Stehule wrote: > pá 14. 1. 2022 v 3:44 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > > =# let myvariable = 'AA'; > > LET > > > > =# select 'AA' collate "en-x-icu" < myvariable; > > ?column? > > ---------- > > f > > (1 row) > > > > =# select 'AA' collate "en-x-icu" < myvariable collate mycollation; > > ERROR: 42P21: collation mismatch between explicit collations "en-x-icu" > > and "mycollation" > > LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat... > > > > What do you expect? I don't understand collating well, but it looks > correct. Minimally the tables have the same behavior. Indeed, I actually didn't know that such object's collation were implicit and could be overloaded without a problem as long as there's no conflict between all the explicit collations. So I agree that the current behavior is ok, including a correct handling for wanted conflicts: =# create variable var1 text collate "fr-x-icu"; CREATE VARIABLE =# create variable var2 text collate "en-x-icu"; CREATE VARIABLE =# let var1 = 'hoho'; LET =# let var2 = 'hoho'; LET =# select var1 < var2; ERROR: 42P22: could not determine which collation to use for string comparison HINT: Use the COLLATE clause to set the collation explicitly. > Please, can you check the attached patches? All the issue I mentioned are fixed, thanks! I see a few problems with the other new features added though. The new session_variables_ambiguity_warning GUC is called even in contexts where it shouldn't apply. For instance: =# set session_variables_ambiguity_warning = 1; SET =# create variable v text; CREATE VARIABLE =# DO $$ DECLARE v text; BEGIN v := 'test'; RAISE NOTICE 'v: %', v; END; $$ LANGUAGE plpgsql; WARNING: 42702: session variable "v" is shadowed by column LINE 1: v := 'test' ^ DETAIL: The identifier can be column reference or session variable reference. HINT: The column reference is preferred against session variable reference. QUERY: v := 'test' But this "v := 'test'" shouldn't be a substitute for a LET, and it indeed doesn't work: =# DO $$ BEGIN v := 'test'; RAISE NOTICE 'v: %', v; END; $$ LANGUAGE plpgsql; ERROR: 42601: "v" is not a known variable LINE 3: v := 'test'; But the RAISE NOTICE does see the session variable (which should be the correct behavior I think), so the warning should have been raised for this instruction (and in that case the message is incorrect, as it's not shadowing a column). Also, the pg_dump handling emits a COLLATION option for session variables even for default collation, while it should only emit it if the collation is not the type's default collation. As a reference, for attributes the SQL used is: "CASE WHEN a.attcollation <> t.typcollation " "THEN a.attcollation ELSE 0 END AS attcollation,\n" Also, should \dV or \dV+ show the collation? And a few comments on the new chunks in this version of the patch (I didn't look in detail at the whole patch yet): + <para> + The session variables can be overshadowed by columns in an query. When query + holds identifier or qualified identifier that can be used as session variable + identifier and as column identifier too, then it is used as column identifier + every time. This situation can be logged by enabling configuration + parameter <xref linkend="guc-session-variables-ambiguity-warning"/>. + </para> Is "overshadowed" correct? The rest of the patch only says "shadow(ed)". While at it, here's some proposition to improve the phrasing: + The session variables can be shadowed by column references in a query. When a + query contains identifiers or qualified identifiers that could be used as both + a session variable identifiers and as column identifier, then the column + identifier is preferred every time. Warnings can be emitted when this situation + happens by enabling configuration parameter <xref + linkend="guc-session-variables-ambiguity-warning"/>. Similarly, the next documentation could be rephrased to: + When on, a warning is raised when any identifier in a query could be used as both + a column identifier or a session variable identifier. + The default is <literal>off</literal>. Few other nitpicking: + * If we really detect collision of column and variable identifier, + * then we prefer column, because we don't want to allow to break + * an existing valid queries by new variable. s/an existing/existing +-- it is ambigonuous, but columns are preferred ambiguous? @@ -369,6 +367,19 @@ VariableCreate(const char *varName, /* dependency on extension */ recordDependencyOnCurrentExtension(&myself, false); + /* + * Normal dependency from a domain to its collation. We know the default + * collation is pinned, so don't bother recording it. + */ + if (OidIsValid(varCollation) && + varCollation != DEFAULT_COLLATION_OID) The comment mentions domains rather than session variables. And for the initial patch, while looking around I found this comment on fix_alternative_subplan(): @@ -1866,7 +1969,9 @@ fix_alternative_subplan(PlannerInfo *root, AlternativeSubPlan *asplan, * replacing Aggref nodes that should be replaced by initplan output Params, * choosing the best implementation for AlternativeSubPlans, * looking up operator opcode info for OpExpr and related nodes, - * and adding OIDs from regclass Const nodes into root->glob->relationOids. + * and adding OIDs from regclass Const nodes into root->glob->relationOids, + * and replacing PARAM_VARIABLE paramid, that is the oid of the session variable + * to offset the array by query used session variables. ??? I don't really understand the comment, and the "???" looks a bit suspicious. I'm assuming it's a reference to this new behavior in fix_param_node(): * fix_param_node * Do set_plan_references processing on a Param + * Collect session variables list and replace variable oid by + * index to collected list. * * If it's a PARAM_MULTIEXPR, replace it with the appropriate Param from * root->multiexpr_params; otherwise no change is needed. * Just for paranoia's sake, we make a copy of the node in either case. + * + * If it's a PARAM_VARIABLE, then we should to calculate paramid. Some improvement on the comments would be welcome there, probably including some mention to the "glob->sessionVariables" collected list?
Hi,
On Tue, Jan 18, 2022 at 10:01:01PM +0100, Pavel Stehule wrote:
> pá 14. 1. 2022 v 3:44 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> >
> > =# let myvariable = 'AA';
> > LET
> >
> > =# select 'AA' collate "en-x-icu" < myvariable;
> > ?column?
> > ----------
> > f
> > (1 row)
> >
> > =# select 'AA' collate "en-x-icu" < myvariable collate mycollation;
> > ERROR: 42P21: collation mismatch between explicit collations "en-x-icu"
> > and "mycollation"
> > LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat...
> >
>
> What do you expect? I don't understand collating well, but it looks
> correct. Minimally the tables have the same behavior.
Indeed, I actually didn't know that such object's collation were implicit and
could be overloaded without a problem as long as there's no conflict between
all the explicit collations. So I agree that the current behavior is ok,
including a correct handling for wanted conflicts:
=# create variable var1 text collate "fr-x-icu";
CREATE VARIABLE
=# create variable var2 text collate "en-x-icu";
CREATE VARIABLE
=# let var1 = 'hoho';
LET
=# let var2 = 'hoho';
LET
=# select var1 < var2;
ERROR: 42P22: could not determine which collation to use for string comparison
HINT: Use the COLLATE clause to set the collation explicitly.
> Please, can you check the attached patches?
All the issue I mentioned are fixed, thanks!
I see a few problems with the other new features added though. The new
session_variables_ambiguity_warning GUC is called even in contexts where it
shouldn't apply. For instance:
=# set session_variables_ambiguity_warning = 1;
SET
=# create variable v text;
CREATE VARIABLE
=# DO $$
DECLARE v text;
BEGIN
v := 'test';
RAISE NOTICE 'v: %', v;
END;
$$ LANGUAGE plpgsql;
WARNING: 42702: session variable "v" is shadowed by column
LINE 1: v := 'test'
^
DETAIL: The identifier can be column reference or session variable reference.
HINT: The column reference is preferred against session variable reference.
QUERY: v := 'test'
But this "v := 'test'" shouldn't be a substitute for a LET, and it indeed
doesn't work:
=# DO $$
BEGIN
v := 'test';
RAISE NOTICE 'v: %', v;
END;
$$ LANGUAGE plpgsql;
ERROR: 42601: "v" is not a known variable
LINE 3: v := 'test';
But the RAISE NOTICE does see the session variable (which should be the correct
behavior I think), so the warning should have been raised for this instruction
(and in that case the message is incorrect, as it's not shadowing a column).
HINT: The column reference and query parameter is preferred against session variable reference.
Hi, On Wed, Jan 19, 2022 at 09:09:41PM +0100, Pavel Stehule wrote: > st 19. 1. 2022 v 9:01 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > RAISE NOTICE should use local variables, and in this case is a question if we > should raise a warning. There are two possible analogies - we can see session > variables like global variables, and then the warning should not be raised, > or we can see relation between session variables and plpgsql variables > similar like session variables and some with higher priority, and then > warning should be raised. If we want to ensure that the new session variable > doesn't break code, then session variables should have lower priority than > plpgsql variables too. And because the plpgsql protection against collision > cannot be used, then I prefer raising the warning. Ah that's indeed a good point. I agree, they're from a different part of the system so they should be treated as different things, and thus raising a warning. It's consistent with the chosen conservative approach anyway. > PLpgSQL assignment should not be in collision with session variables ever Agreed. > > > > > =# DO $$ > > BEGIN > > v := 'test'; > > RAISE NOTICE 'v: %', v; > > END; > > $$ LANGUAGE plpgsql; > > ERROR: 42601: "v" is not a known variable > > LINE 3: v := 'test'; > > > > But the RAISE NOTICE does see the session variable (which should be the > > correct > > behavior I think), so the warning should have been raised for this > > instruction > > (and in that case the message is incorrect, as it's not shadowing a > > column). > > > > In this case I can detect node type, and I can identify external param > node, but I cannot to detect if this code was executed from PLpgSQL or from > some other > > So I can to modify warning text to some Yes, that's what I had in mind too. > DETAIL: The identifier can be column reference or query parameter or > session variable reference. > HINT: The column reference and query parameter is preferred against > session variable reference. > > I cannot to use term "plpgsql variable" becase I cannot to ensure validity > of this message > > Maybe is better to don't talk about source of this issue, and just talk > about result - so the warning text should be just > > MESSAGE: "session variable \"xxxx\" is shadowed > DETAIL: "session variables can be shadowed by columns, routine's variables > and routine's arguments with same name" > > Is it better? I clearly prefer the 2nd version.
Hi,
On Tue, Jan 18, 2022 at 10:01:01PM +0100, Pavel Stehule wrote:
> pá 14. 1. 2022 v 3:44 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> >
> > =# let myvariable = 'AA';
> > LET
> >
> > =# select 'AA' collate "en-x-icu" < myvariable;
> > ?column?
> > ----------
> > f
> > (1 row)
> >
> > =# select 'AA' collate "en-x-icu" < myvariable collate mycollation;
> > ERROR: 42P21: collation mismatch between explicit collations "en-x-icu"
> > and "mycollation"
> > LINE 1: select 'AA' collate "en-x-icu" < myvariable collate mycollat...
> >
>
> What do you expect? I don't understand collating well, but it looks
> correct. Minimally the tables have the same behavior.
Indeed, I actually didn't know that such object's collation were implicit and
could be overloaded without a problem as long as there's no conflict between
all the explicit collations. So I agree that the current behavior is ok,
including a correct handling for wanted conflicts:
=# create variable var1 text collate "fr-x-icu";
CREATE VARIABLE
=# create variable var2 text collate "en-x-icu";
CREATE VARIABLE
=# let var1 = 'hoho';
LET
=# let var2 = 'hoho';
LET
=# select var1 < var2;
ERROR: 42P22: could not determine which collation to use for string comparison
HINT: Use the COLLATE clause to set the collation explicitly.
> Please, can you check the attached patches?
All the issue I mentioned are fixed, thanks!
I see a few problems with the other new features added though. The new
session_variables_ambiguity_warning GUC is called even in contexts where it
shouldn't apply. For instance:
=# set session_variables_ambiguity_warning = 1;
SET
=# create variable v text;
CREATE VARIABLE
=# DO $$
DECLARE v text;
BEGIN
v := 'test';
RAISE NOTICE 'v: %', v;
END;
$$ LANGUAGE plpgsql;
WARNING: 42702: session variable "v" is shadowed by column
LINE 1: v := 'test'
^
DETAIL: The identifier can be column reference or session variable reference.
HINT: The column reference is preferred against session variable reference.
QUERY: v := 'test'
But this "v := 'test'" shouldn't be a substitute for a LET, and it indeed
doesn't work:
=# DO $$
BEGIN
v := 'test';
RAISE NOTICE 'v: %', v;
END;
$$ LANGUAGE plpgsql;
ERROR: 42601: "v" is not a known variable
LINE 3: v := 'test';
But the RAISE NOTICE does see the session variable (which should be the correct
behavior I think), so the warning should have been raised for this instruction
(and in that case the message is incorrect, as it's not shadowing a column).
Also, the pg_dump handling emits a COLLATION option for session variables even
for default collation, while it should only emit it if the collation is not the
type's default collation. As a reference, for attributes the SQL used is:
"CASE WHEN a.attcollation <> t.typcollation "
"THEN a.attcollation ELSE 0 END AS attcollation,\n"
<--> * Find all the user attributes and their types.
<--> *
<--> * Since we only want to dump COLLATE clauses for attributes whose
<--> * collation is different from their type's default, we use a CASE here to
<--> * suppress uninteresting attcollations cheaply.
<--> */
Also, should \dV or \dV+ show the collation?
And a few comments on the new chunks in this version of the patch (I didn't
look in detail at the whole patch yet):
+ <para>
+ The session variables can be overshadowed by columns in an query. When query
+ holds identifier or qualified identifier that can be used as session variable
+ identifier and as column identifier too, then it is used as column identifier
+ every time. This situation can be logged by enabling configuration
+ parameter <xref linkend="guc-session-variables-ambiguity-warning"/>.
+ </para>
Is "overshadowed" correct? The rest of the patch only says "shadow(ed)".
While at it, here's some proposition to improve the phrasing:
+ The session variables can be shadowed by column references in a query. When a
+ query contains identifiers or qualified identifiers that could be used as both
+ a session variable identifiers and as column identifier, then the column
+ identifier is preferred every time. Warnings can be emitted when this situation
+ happens by enabling configuration parameter <xref
+ linkend="guc-session-variables-ambiguity-warning"/>.
Similarly, the next documentation could be rephrased to:
+ When on, a warning is raised when any identifier in a query could be used as both
+ a column identifier or a session variable identifier.
+ The default is <literal>off</literal>.
Few other nitpicking:
+ * If we really detect collision of column and variable identifier,
+ * then we prefer column, because we don't want to allow to break
+ * an existing valid queries by new variable.
s/an existing/existing
+-- it is ambigonuous, but columns are preferred
ambiguous?
@@ -369,6 +367,19 @@ VariableCreate(const char *varName,
/* dependency on extension */
recordDependencyOnCurrentExtension(&myself, false);
+ /*
+ * Normal dependency from a domain to its collation. We know the default
+ * collation is pinned, so don't bother recording it.
+ */
+ if (OidIsValid(varCollation) &&
+ varCollation != DEFAULT_COLLATION_OID)
The comment mentions domains rather than session variables.
And for the initial patch, while looking around I found this comment on
fix_alternative_subplan():
@@ -1866,7 +1969,9 @@ fix_alternative_subplan(PlannerInfo *root, AlternativeSubPlan *asplan,
* replacing Aggref nodes that should be replaced by initplan output Params,
* choosing the best implementation for AlternativeSubPlans,
* looking up operator opcode info for OpExpr and related nodes,
- * and adding OIDs from regclass Const nodes into root->glob->relationOids.
+ * and adding OIDs from regclass Const nodes into root->glob->relationOids,
+ * and replacing PARAM_VARIABLE paramid, that is the oid of the session variable
+ * to offset the array by query used session variables. ???
I don't really understand the comment, and the "???" looks a bit suspicious.
I'm assuming it's a reference to this new behavior in fix_param_node():
* fix_param_node
* Do set_plan_references processing on a Param
+ * Collect session variables list and replace variable oid by
+ * index to collected list.
*
* If it's a PARAM_MULTIEXPR, replace it with the appropriate Param from
* root->multiexpr_params; otherwise no change is needed.
* Just for paranoia's sake, we make a copy of the node in either case.
+ *
+ * If it's a PARAM_VARIABLE, then we should to calculate paramid.
Some improvement on the comments would be welcome there, probably including
some mention to the "glob->sessionVariables" collected list?
Вложения
Hi, On Fri, Jan 21, 2022 at 09:23:34PM +0100, Pavel Stehule wrote: > > st 19. 1. 2022 v 9:01 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > Also, the pg_dump handling emits a COLLATION option for session variables > > even > > for default collation, while it should only emit it if the collation is > > not the > > type's default collation. As a reference, for attributes the SQL used is: > > > > "CASE WHEN a.attcollation > > <> t.typcollation " > > "THEN a.attcollation ELSE > > 0 END AS attcollation,\n" > > > > Isn't it a different issue? I don't see filtering DEFAULT_COLLATION_OID in > pg_dump code. But this case protects against a redundant COLLATE clause, > and for consistency, this check should be done for variables too. Yes, sorry my message was a bit ambiguous as for all native collatable types the "default" collation is the type's default collation, I thought that the code extract would make it clear enough. In any case your fix is exactly what I had in mind so it's perfect, thanks! > I am sending updated patches Thanks a lot! I will try to review them over the weekend.
Hi, On Fri, Jan 21, 2022 at 09:23:34PM +0100, Pavel Stehule wrote: > > I am sending updated patches I've been looking a bit deeper at the feature and I noticed that there's no locking involved around the session variable usage, and I don't think that's ok. AFAICS any variable used in a session will be cached in the local hash table and will never try to access some catalog or cache, so I don't have any naive scenario that would immediately crash, but this has some other implications that seems debatable. For instance, right now nothing prevents a variable from being dropped while another session is using it. Obviously we can't lock a session variable forever just because a session assigned a value once ages ago, especially outside of the current transaction. But if a session set a variable in the local transaction, I don't think that it's ok to have a subsequent query failing because someone else concurrently dropped the variable. I only backlogged this current thread but I didn't see that being discussed.
Hi,
On Fri, Jan 21, 2022 at 09:23:34PM +0100, Pavel Stehule wrote:
>
> I am sending updated patches
I've been looking a bit deeper at the feature and I noticed that there's no
locking involved around the session variable usage, and I don't think that's
ok. AFAICS any variable used in a session will be cached in the local hash
table and will never try to access some catalog or cache, so I don't have any
naive scenario that would immediately crash, but this has some other
implications that seems debatable.
For instance, right now nothing prevents a variable from being dropped while
another session is using it.
Obviously we can't lock a session variable forever just because a session
assigned a value once ages ago, especially outside of the current transaction.
But if a session set a variable in the local transaction, I don't think that
it's ok to have a subsequent query failing because someone else concurrently
dropped the variable.
I only backlogged this current thread but I didn't see that being discussed.
Hi, On Sun, Jan 23, 2022 at 09:25:56AM +0100, Pavel Stehule wrote: > ne 23. 1. 2022 v 9:10 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > Isn't there enough stability of the system cache? sinval is sent at the > moment when changes in the system catalog are visible. So inside query > execution I don't see that the variable was dropped in another session. Yes, inside a single query it should probably be ok, but I'm talking about multiple query execution in the same transaction.
Hi,
On Sun, Jan 23, 2022 at 09:25:56AM +0100, Pavel Stehule wrote:
> ne 23. 1. 2022 v 9:10 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> Isn't there enough stability of the system cache? sinval is sent at the
> moment when changes in the system catalog are visible. So inside query
> execution I don't see that the variable was dropped in another session.
Yes, inside a single query it should probably be ok, but I'm talking about
multiple query execution in the same transaction.
I tested it now. a sinval message is waiting on the transaction end. So when a variable is used, then it is working fine until the transaction ends.But when the session makes some DDL, then send sinval to self, and at this moment, the variable can be dropped before the transaction ends.
Hi,Le dim. 23 janv. 2022 à 22:34, Pavel Stehule <pavel.stehule@gmail.com> a écrit :I tested it now. a sinval message is waiting on the transaction end. So when a variable is used, then it is working fine until the transaction ends.But when the session makes some DDL, then send sinval to self, and at this moment, the variable can be dropped before the transaction ends.a backend can accept sinval in very common scenarios, like acquiring a heavyweight lock. That includes accessing a relation thats not in the catcache, so that's really critical to have a protection here.
Вложения
Hi, On Mon, Jan 24, 2022 at 12:33:11PM +0100, Pavel Stehule wrote: > > here is updated patch with locking support Thanks for updating the patch! While the locking is globally working as intended, I found a few problems with it. First, I don't think that acquiring the lock in get_session_variable_type_typmod_collid() and prepare_variable_for_reading() is the correct approach. In transformColumnRef() and transformLetStmt() you first call IdentifyVariable() to check if the given name is a variable without locking it and later try to lock the variable if you get a valid Oid. This is bug prone as any other backend could drop the variable between the two calls and you would end up with a cache lookup failure. I think the lock should be acquired during IdentifyVariable. It should probably be optional as one codepath only needs the information to raise a warning when a variable is shadowed, so a concurrent drop isn't a problem there. For prepare_variable_for_reading(), the callers are CopySessionVariable() and GetSessionVariable(). IIUC those should take care of executor-time locks, but shouldn't there be some changes for planning, like in AcquirePlannerLocks()? Some other comments on this part of the patch: @@ -717,6 +730,9 @@ RemoveSessionVariable(Oid varid) Relation rel; HeapTuple tup; + /* Wait, when dropped variable is not used */ + LockDatabaseObject(VariableRelationId, varid, 0, AccessExclusiveLock); Why do you explicitly try to acquire an AEL on the variable here? RemoveObjects / get_object_address should guarantee that this was already done. You could add an assert LockHeldByMe() here, but no other code path do it so it would probably waste cycles in assert builds for nothing as it's a fundamental guarantee. @@ -747,6 +763,9 @@ RemoveSessionVariable(Oid varid) * only when current transaction will be commited. */ register_session_variable_xact_action(varid, ON_COMMIT_RESET); + + /* Release lock */ + UnlockDatabaseObject(VariableRelationId, varid, 0, AccessExclusiveLock); } Why releasing the lock here? It will be done at the end of the transaction, and you certainly don't want other backends to start using this variable in between. Also, since you acquired the lock a second time it only decreases the lock count in the locallock so the lock isn't released anyway. + * Returns type, typmod and collid of session variable. + * + * As a side effect this function acquires AccessShareLock on the + * related session variable. */ void -get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, Oid *collid) +get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, Oid *collid, + bool lock_held) lock_held is a bit misleading. If you keep some similar parameter for this or another function, maybe name it lock_it or something like that instead? Also, the comment isn't accurate and should say that an ASL is acquired iff the variable is true. + /* + * Acquire a lock on session variable, which we won't release until commit. + * This ensure that one backend cannot to drop session variable used by + * second backend. + */ (and similar comments) I don't think it's necessary to explain why we acquire locks, we should just say that the lock will be kept for the whole transaction (and not until a commit) And while looking at nearby code, it's probably worthwhile to add an Assert in create_sessionvars_hashtable() to validate that sessionvars htab is NULL.
Hi,
On Mon, Jan 24, 2022 at 12:33:11PM +0100, Pavel Stehule wrote:
>
> here is updated patch with locking support
Thanks for updating the patch!
While the locking is globally working as intended, I found a few problems with
it.
First, I don't think that acquiring the lock in
get_session_variable_type_typmod_collid() and prepare_variable_for_reading() is
the correct approach. In transformColumnRef() and transformLetStmt() you first
call IdentifyVariable() to check if the given name is a variable without
locking it and later try to lock the variable if you get a valid Oid. This is
bug prone as any other backend could drop the variable between the two calls
and you would end up with a cache lookup failure. I think the lock should be
acquired during IdentifyVariable. It should probably be optional as one
codepath only needs the information to raise a warning when a variable is
shadowed, so a concurrent drop isn't a problem there.
Hi, On Tue, Jan 25, 2022 at 09:35:09AM +0100, Pavel Stehule wrote: > út 25. 1. 2022 v 6:18 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > I think the lock should be > > acquired during IdentifyVariable. It should probably be optional as one > > codepath only needs the information to raise a warning when a variable is > > shadowed, so a concurrent drop isn't a problem there. > > > > There is a problem, because before the IdentifyVariable call I don't know > if the variable will be shadowed or not. > > If I lock a variable inside IdentifyVariable, then I need to remember if I > did lock there, or if the variable was locked already, and If the variable > is shadowed and if lock is fresh, then I can unlock the variable. But in transformColumnRef() you already know if you found a matching column or not when calling IdentifyVariable(), so you know if an existing variable will shadow it right? Couldn't you call something like lockit = node == NULL; varid = IdentifyVariable(cref->fields, &attrname, ¬_unique, lockit); The only other caller is transformLetStmt(), which should always lock the variable anyway.
Hi,
On Tue, Jan 25, 2022 at 09:35:09AM +0100, Pavel Stehule wrote:
> út 25. 1. 2022 v 6:18 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> > I think the lock should be
> > acquired during IdentifyVariable. It should probably be optional as one
> > codepath only needs the information to raise a warning when a variable is
> > shadowed, so a concurrent drop isn't a problem there.
> >
>
> There is a problem, because before the IdentifyVariable call I don't know
> if the variable will be shadowed or not.
>
> If I lock a variable inside IdentifyVariable, then I need to remember if I
> did lock there, or if the variable was locked already, and If the variable
> is shadowed and if lock is fresh, then I can unlock the variable.
But in transformColumnRef() you already know if you found a matching column or
not when calling IdentifyVariable(), so you know if an existing variable will
shadow it right?
Couldn't you call something like
lockit = node == NULL;
varid = IdentifyVariable(cref->fields, &attrname, ¬_unique, lockit);
The only other caller is transformLetStmt(), which should always lock the
variable anyway.
Hi,
On Mon, Jan 24, 2022 at 12:33:11PM +0100, Pavel Stehule wrote:
>
> here is updated patch with locking support
Thanks for updating the patch!
While the locking is globally working as intended, I found a few problems with
it.
First, I don't think that acquiring the lock in
get_session_variable_type_typmod_collid() and prepare_variable_for_reading() is
the correct approach. In transformColumnRef() and transformLetStmt() you first
call IdentifyVariable() to check if the given name is a variable without
locking it and later try to lock the variable if you get a valid Oid. This is
bug prone as any other backend could drop the variable between the two calls
and you would end up with a cache lookup failure. I think the lock should be
acquired during IdentifyVariable. It should probably be optional as one
codepath only needs the information to raise a warning when a variable is
shadowed, so a concurrent drop isn't a problem there.
For prepare_variable_for_reading(), the callers are CopySessionVariable() and
GetSessionVariable(). IIUC those should take care of executor-time locks, but
shouldn't there be some changes for planning, like in AcquirePlannerLocks()?
Some other comments on this part of the patch:
@@ -717,6 +730,9 @@ RemoveSessionVariable(Oid varid)
Relation rel;
HeapTuple tup;
+ /* Wait, when dropped variable is not used */
+ LockDatabaseObject(VariableRelationId, varid, 0, AccessExclusiveLock);
Why do you explicitly try to acquire an AEL on the variable here?
RemoveObjects / get_object_address should guarantee that this was already done.
You could add an assert LockHeldByMe() here, but no other code path do it so it
would probably waste cycles in assert builds for nothing as it's a fundamental
guarantee.
@@ -747,6 +763,9 @@ RemoveSessionVariable(Oid varid)
* only when current transaction will be commited.
*/
register_session_variable_xact_action(varid, ON_COMMIT_RESET);
+
+ /* Release lock */
+ UnlockDatabaseObject(VariableRelationId, varid, 0, AccessExclusiveLock);
}
Why releasing the lock here? It will be done at the end of the transaction,
and you certainly don't want other backends to start using this variable in
between. Also, since you acquired the lock a second time it only decreases the
lock count in the locallock so the lock isn't released anyway.
+ * Returns type, typmod and collid of session variable.
+ *
+ * As a side effect this function acquires AccessShareLock on the
+ * related session variable.
*/
void
-get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, Oid *collid)
+get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, Oid *collid,
+ bool lock_held)
lock_held is a bit misleading. If you keep some similar parameter for this or
another function, maybe name it lock_it or something like that instead?
Also, the comment isn't accurate and should say that an ASL is acquired iff the
variable is true.
+ /*
+ * Acquire a lock on session variable, which we won't release until commit.
+ * This ensure that one backend cannot to drop session variable used by
+ * second backend.
+ */
(and similar comments)
I don't think it's necessary to explain why we acquire locks, we should just
say that the lock will be kept for the whole transaction (and not until a
commit)
And while looking at nearby code, it's probably worthwhile to add an Assert in
create_sessionvars_hashtable() to validate that sessionvars htab is NULL.
Вложения
Hi, On Tue, Jan 25, 2022 at 10:53:00PM +0100, Pavel Stehule wrote: > > út 25. 1. 2022 v 6:18 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > First, I don't think that acquiring the lock in > > get_session_variable_type_typmod_collid() and > > prepare_variable_for_reading() is > > the correct approach. In transformColumnRef() and transformLetStmt() you > > first > > call IdentifyVariable() to check if the given name is a variable without > > locking it and later try to lock the variable if you get a valid Oid. > > This is > > bug prone as any other backend could drop the variable between the two > > calls > > and you would end up with a cache lookup failure. I think the lock should > > be > > acquired during IdentifyVariable. It should probably be optional as one > > codepath only needs the information to raise a warning when a variable is > > shadowed, so a concurrent drop isn't a problem there. > > > > I moved lock to IdentifyVariable routine +IdentifyVariable(List *names, char **attrname, bool lockit, bool *not_unique) +{ [...] + return varoid_without_attr; + } + else + { + *attrname = c; + return varoid_with_attr; [...] + + if (OidIsValid(varid) && lockit) + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + return varid; There are still some code paths that may not lock the target variable when required. Also, the function comment doesn't say much about attrname handling, it should be clarifed. I think it should initially be set to NULL, to make sure that it's always a valid pointer after the function returns. > attached updated patch Various comments on the patch: No test for GRANT/REVOKE ... ALL VARIABLES IN SCHEMA, maybe it would be good to have one? Documentation: catalogs.sgml: You're still using the old-style 4 columns table, it should be a single column like the rest of the file. + <para> + The <command>CREATE VARIABLE</command> command creates a session variable. + Session variables, like relations, exist within a schema and their access is + controlled via <command>GRANT</command> and <command>REVOKE</command> + commands. Changing a session variable is non-transactional. + </para> The "changing a session variable is non-transactional" is ambiguous. I think that only the value part isn't transactional, the variable metadata themselves (ALTER VARIABLE and other DDL) are transactional right? This should be explicitly described here (although it's made less ambiguous in the next paragraph). + <para> + Session variables are retrieved by the <command>SELECT</command> SQL + command. Their value is set with the <command>LET</command> SQL command. + While session variables share properties with tables, their value cannot be + updated with an <command>UPDATE</command> command. + </para> should this part mention that session variables can be shadowed? For now the only mention to that is in advanced.sgml. + The <literal>DEFAULT</literal> clause can be used to assign a default + value to a session variable. The expression is lazily evaluated during the session first use of the variable. This should be documented as any usage of volatile expression will be impacted. + The <literal>ON TRANSACTION END RESET</literal> + clause causes the session variable to be reset to its default value when + the transaction is committed or rolled back. As far as I can see this clauses doesn't play well with IMMUTABLE VARIABLE, as you can reassign a value once the transaction ends. Same for DISCARD [ ALL | VARIABLES ], or LET var = NULL (or DEFAULT if no default value). Is that intended? + <literal>LET</literal> extends the syntax defined in the SQL + standard. The <literal>SET</literal> command from the SQL standard + is used for different purposes in <productname>PostgreSQL</productname>. I don't fully understand that. Are (session) variables defined in the SQL standard? If yes, all the other documentation pages should clarify that as they currently say that this is a postgres extension. If not, this part should made it clear what is defined in the standard. In revoke.sgml: + REVOKE [ GRANT OPTION FOR ] + { { READ | WRITE } [, ...] | ALL [ PRIVILEGES ] } + ON VARIABLE <replaceable>variable_name</replaceable> [, ...] + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] there's no extra documentation for that, and therefore no clarification on variable_name. VariableIsVisible(): + * If it is in the path, it might still not be visible; it could be + * hidden by another relation of the same name earlier in the path. So + * we must do a slow check for conflicting relations. should it be "another variable of the same name"? Tab completion: CREATE IMMUTABLE VARIABLE is not handled pg_variable.c: Do we really need both session_variable_get_name() and get_session_variable_name()? +/* + * Fetch all fields of session variable from the syscache. + */ +void +initVariable(Variable *var, Oid varid, bool missing_ok, bool fast_only) As least fast_only should be documented in the function comment, especially regarding var->varname, since: + var->oid = varid; + var->name = pstrdup(NameStr(varform->varname)); [...] + if (!fast_only) + { + Datum aclDatum; + bool isnull; + + /* name */ + var->name = pstrdup(NameStr(varform->varname));A [...] + else + { + var->name = NULL; is the output value guaranteed or not? In any case it shouldn't be set twice. Also, I don't see any caller for missing_ok == true, should we remove it? +/* + * Create entry in pg_variable table + */ +ObjectAddress +VariableCreate(const char *varName, [...] + /* dependency on any roles mentioned in ACL */ + if (varacl != NULL) + { + int nnewmembers; + Oid *newmembers; + + nnewmembers = aclmembers(varacl, &newmembers); + updateAclDependencies(VariableRelationId, varid, 0, + varOwner, + 0, NULL, + nnewmembers, newmembers); Shouldn't you use recordDependencyOnNewAcl() instead? Also, sn't it missing a recordDependencyOnOwner()? sessionvariable.c: + * Although session variables are not transactional, we don't + * want (and we cannot) to run cleaning immediately (when we + * got sinval message). The value of session variables can + * be still used or the operation that emits cleaning can be + * reverted. Unfortunatelly, this check can be done only in + * when transaction is committed (the check against system + * catalog requires transaction state). This was the original idea, but since there's now locking to make all DDL safe, the metadata should be considered fully transactional and no session should still be able to use a concurrently dropped variable. Also, the invalidation messages are not sent until the transaction is committed. So is that approach still needed (at least for things outside ON COMMIT DROP / ON TRANSACTION END RESET)? I'm also attaching a 3rd patch with some proposition for documentation rewording (including consistent use of *session* variable), a few comments rewording, copyright year bump and minor things like that. Note that I still didn't really review pg_variable.c or sessionvariable.c since there might be significant changes there for either the sinval / immutable part I mentioned.
Вложения
sessionvariable.c:
+ * Although session variables are not transactional, we don't
+ * want (and we cannot) to run cleaning immediately (when we
+ * got sinval message). The value of session variables can
+ * be still used or the operation that emits cleaning can be
+ * reverted. Unfortunatelly, this check can be done only in
+ * when transaction is committed (the check against system
+ * catalog requires transaction state).
This was the original idea, but since there's now locking to make all DDL safe,
the metadata should be considered fully transactional and no session should
still be able to use a concurrently dropped variable. Also, the invalidation
messages are not sent until the transaction is committed. So is that approach
still needed (at least for things outside ON COMMIT DROP / ON TRANSACTION END
RESET
Hi, On Wed, Jan 26, 2022 at 02:43:54PM +0100, Pavel Stehule wrote: > > I think this is still necessary. The lock protects the variable against > drop from the second session, but not for reverted deletion from the > current session. > > This implementation is due Tomas's request for > > CREATE VARIABLE xx AS int; > LET xx = 100; > BEGIN; > DROP VARIABLE xx; > ROLLBACK; > SELECT xx; --> 100 > > and the variable still holds the last value before DROP I thought about this case, but assumed that the own session wouldn't process the inval until commit. Agreed then, although the comment should clarify the transactional behavior and why it's still necessary. > Personally, this is a corner case (for me, and I think so for users it is > not too interesting, and important), and this behavior is not necessary - > originally I implemented just the RESET variable in this case. On the other > hand, this is a nice feature, and there is an analogy with TRUNCATE > behavior. > > More, I promised, as a second step, implementation of optional > transactional behavior of session variables. And related code is necessary > for it. So I prefer to use related code without change. That's another good reason, so fine by me!
Hi,
On Tue, Jan 25, 2022 at 10:53:00PM +0100, Pavel Stehule wrote:
>
> út 25. 1. 2022 v 6:18 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
> >
> > First, I don't think that acquiring the lock in
> > get_session_variable_type_typmod_collid() and
> > prepare_variable_for_reading() is
> > the correct approach. In transformColumnRef() and transformLetStmt() you
> > first
> > call IdentifyVariable() to check if the given name is a variable without
> > locking it and later try to lock the variable if you get a valid Oid.
> > This is
> > bug prone as any other backend could drop the variable between the two
> > calls
> > and you would end up with a cache lookup failure. I think the lock should
> > be
> > acquired during IdentifyVariable. It should probably be optional as one
> > codepath only needs the information to raise a warning when a variable is
> > shadowed, so a concurrent drop isn't a problem there.
> >
>
> I moved lock to IdentifyVariable routine
+IdentifyVariable(List *names, char **attrname, bool lockit, bool *not_unique)
+{
[...]
+ return varoid_without_attr;
+ }
+ else
+ {
+ *attrname = c;
+ return varoid_with_attr;
[...]
+
+ if (OidIsValid(varid) && lockit)
+ LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock);
+
+ return varid;
There are still some code paths that may not lock the target variable when
required.
Also, the function comment doesn't say much about attrname handling, it should
be clarifed. I think it should initially be set to NULL, to make sure that
it's always a valid pointer after the function returns.
> attached updated patch
Various comments on the patch:
No test for GRANT/REVOKE ... ALL VARIABLES IN SCHEMA, maybe it would be good to
have one?
Documentation:
catalogs.sgml:
You're still using the old-style 4 columns table, it should be a single column
like the rest of the file.
+ <para>
+ The <command>CREATE VARIABLE</command> command creates a session variable.
+ Session variables, like relations, exist within a schema and their access is
+ controlled via <command>GRANT</command> and <command>REVOKE</command>
+ commands. Changing a session variable is non-transactional.
+ </para>
The "changing a session variable is non-transactional" is ambiguous. I think
that only the value part isn't transactional, the variable metadata themselves
(ALTER VARIABLE and other DDL) are transactional right? This should be
explicitly described here (although it's made less ambiguous in the next
paragraph).
+ <para>
+ Session variables are retrieved by the <command>SELECT</command> SQL
+ command. Their value is set with the <command>LET</command> SQL command.
+ While session variables share properties with tables, their value cannot be
+ updated with an <command>UPDATE</command> command.
+ </para>
should this part mention that session variables can be shadowed? For now the
only mention to that is in advanced.sgml.
+ The <literal>DEFAULT</literal> clause can be used to assign a default
+ value to a session variable.
The expression is lazily evaluated during the session first use of the
variable. This should be documented as any usage of volatile expression will
be impacted.
+ The <literal>ON TRANSACTION END RESET</literal>
+ clause causes the session variable to be reset to its default value when
+ the transaction is committed or rolled back.
As far as I can see this clauses doesn't play well with IMMUTABLE VARIABLE, as
you can reassign a value once the transaction ends. Same for DISCARD [ ALL |
VARIABLES ], or LET var = NULL (or DEFAULT if no default value). Is that
intended?
+ <literal>LET</literal> extends the syntax defined in the SQL
+ standard. The <literal>SET</literal> command from the SQL standard
+ is used for different purposes in <productname>PostgreSQL</productname>.
I don't fully understand that. Are (session) variables defined in the SQL
standard? If yes, all the other documentation pages should clarify that as
they currently say that this is a postgres extension. If not, this part should
made it clear what is defined in the standard.
The <command>LET</command> is a <productname>PostgreSQL</productname>
extension.
</para>
In revoke.sgml:
+ REVOKE [ GRANT OPTION FOR ]
+ { { READ | WRITE } [, ...] | ALL [ PRIVILEGES ] }
+ ON VARIABLE <replaceable>variable_name</replaceable> [, ...]
+ FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...]
+ [ CASCADE | RESTRICT ]
there's no extra documentation for that, and therefore no clarification on
variable_name.
VariableIsVisible():
+ * If it is in the path, it might still not be visible; it could be
+ * hidden by another relation of the same name earlier in the path. So
+ * we must do a slow check for conflicting relations.
should it be "another variable of the same name"?
Tab completion: CREATE IMMUTABLE VARIABLE is not handled
pg_variable.c:
Do we really need both session_variable_get_name() and
get_session_variable_name()?
+/*
+ * Fetch all fields of session variable from the syscache.
+ */
+void
+initVariable(Variable *var, Oid varid, bool missing_ok, bool fast_only)
As least fast_only should be documented in the function comment, especially
regarding var->varname, since:
+ var->oid = varid;
+ var->name = pstrdup(NameStr(varform->varname));
[...]
+ if (!fast_only)
+ {
+ Datum aclDatum;
+ bool isnull;
+
+ /* name */
+ var->name = pstrdup(NameStr(varform->varname));A
[...]
+ else
+ {
+ var->name = NULL;
is the output value guaranteed or not? In any case it shouldn't be set twice.
Also, I don't see any caller for missing_ok == true, should we remove it?
+/*
+ * Create entry in pg_variable table
+ */
+ObjectAddress
+VariableCreate(const char *varName,
[...]
+ /* dependency on any roles mentioned in ACL */
+ if (varacl != NULL)
+ {
+ int nnewmembers;
+ Oid *newmembers;
+
+ nnewmembers = aclmembers(varacl, &newmembers);
+ updateAclDependencies(VariableRelationId, varid, 0,
+ varOwner,
+ 0, NULL,
+ nnewmembers, newmembers);
Shouldn't you use recordDependencyOnNewAcl() instead? Also, sn't it missing a
recordDependencyOnOwner()?
sessionvariable.c:
+ * Although session variables are not transactional, we don't
+ * want (and we cannot) to run cleaning immediately (when we
+ * got sinval message). The value of session variables can
+ * be still used or the operation that emits cleaning can be
+ * reverted. Unfortunatelly, this check can be done only in
+ * when transaction is committed (the check against system
+ * catalog requires transaction state).
This was the original idea, but since there's now locking to make all DDL safe,
the metadata should be considered fully transactional and no session should
still be able to use a concurrently dropped variable. Also, the invalidation
messages are not sent until the transaction is committed. So is that approach
still needed (at least for things outside ON COMMIT DROP / ON TRANSACTION END
RESET)?
I'm also attaching a 3rd patch with some proposition for documentation
rewording (including consistent use of *session* variable), a few comments
rewording, copyright year bump and minor things like that.
Note that I still didn't really review pg_variable.c or sessionvariable.c since
there might be significant changes there for either the sinval / immutable part
I mentioned.
Вложения
Hi, On Fri, Jan 28, 2022 at 07:51:08AM +0100, Pavel Stehule wrote: > st 26. 1. 2022 v 8:23 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > + The <literal>ON TRANSACTION END RESET</literal> > > + clause causes the session variable to be reset to its default value > > when > > + the transaction is committed or rolled back. > > > > As far as I can see this clauses doesn't play well with IMMUTABLE > > VARIABLE, as > > you can reassign a value once the transaction ends. Same for DISCARD [ > > ALL | > > VARIABLES ], or LET var = NULL (or DEFAULT if no default value). Is that > > intended? > > > > I think so it is expected. The life scope of assigned (immutable) value is > limited to transaction (when ON TRANSACTION END RESET). > DISCARD is used for reset of session, and after it, you can write the value > first time. > > I enhanced doc in IMMUTABLE clause I think it's still somewhat unclear: - done, no other change will be allowed in the session lifetime. + done, no other change will be allowed in the session variable content's + lifetime. The lifetime of content of session variable can be + controlled by <literal>ON TRANSACTION END RESET</literal> clause. + </para> The "session variable content lifetime" is quite peculiar, as the ON TRANSACTION END RESET is adding transactional behavior to something that's not supposed to be transactional, so more documentation about it seems appropriate. Also DISCARD can be used any time so that's a totally different aspect of the immutable variable content lifetime that's not described here. NULL handling also seems inconsistent. An explicit default NULL value makes it truly immutable, but manually assigning NULL is a different codepath that has a different user behavior: # create immutable variable var_immu int default null; CREATE VARIABLE # let var_immu = 1; ERROR: 22005: session variable "var_immu" is declared IMMUTABLE # create immutable variable var_immu2 int ; CREATE VARIABLE # let var_immu2 = null; LET # let var_immu2 = null; LET # let var_immu2 = 1; LET For var_immu2 I think that the last 2 queries should have errored out. > > In revoke.sgml: > > + REVOKE [ GRANT OPTION FOR ] > > + { { READ | WRITE } [, ...] | ALL [ PRIVILEGES ] } > > + ON VARIABLE <replaceable>variable_name</replaceable> [, ...] > > + FROM { [ GROUP ] <replaceable > > class="parameter">role_name</replaceable> | PUBLIC } [, ...] > > + [ CASCADE | RESTRICT ] > > > > there's no extra documentation for that, and therefore no clarification on > > variable_name. > > > > This is same like function_name, domain_name, ... Ah right. > > pg_variable.c: > > Do we really need both session_variable_get_name() and > > get_session_variable_name()? > > > > They are different - first returns possibly qualified name, second returns > only name. Currently it is used just for error messages in > transformAssignmentIndirection, and I think so it is good for consistency > with other usage of this routine (transformAssignmentIndirection). I agree that consistency with other usage is a good thing, but both functions have very similar and confusing names. Usually when you need the qualified name the calling code just takes care of doing so. Wouldn't it be better to add say get_session_variable_namespace() and construct the target string in the calling code? Also, I didn't dig a lot but I didn't see other usage with optionally qualified name there? I'm not sure how it would make sense anyway since LET semantics are different and the current call for session variable emit incorrect messages: # create table tt(id integer); CREATE TABLE # create variable vv tt; CREATE VARIABLE # let vv.meh = 1; ERROR: 42703: cannot assign to field "meh" of column "meh" because there is no such column in data type tt LINE 1: let vv.meh = 1;
Hi,
On Fri, Jan 28, 2022 at 07:51:08AM +0100, Pavel Stehule wrote:
> st 26. 1. 2022 v 8:23 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> > + The <literal>ON TRANSACTION END RESET</literal>
> > + clause causes the session variable to be reset to its default value
> > when
> > + the transaction is committed or rolled back.
> >
> > As far as I can see this clauses doesn't play well with IMMUTABLE
> > VARIABLE, as
> > you can reassign a value once the transaction ends. Same for DISCARD [
> > ALL |
> > VARIABLES ], or LET var = NULL (or DEFAULT if no default value). Is that
> > intended?
> >
>
> I think so it is expected. The life scope of assigned (immutable) value is
> limited to transaction (when ON TRANSACTION END RESET).
> DISCARD is used for reset of session, and after it, you can write the value
> first time.
>
> I enhanced doc in IMMUTABLE clause
I think it's still somewhat unclear:
- done, no other change will be allowed in the session lifetime.
+ done, no other change will be allowed in the session variable content's
+ lifetime. The lifetime of content of session variable can be
+ controlled by <literal>ON TRANSACTION END RESET</literal> clause.
+ </para>
The "session variable content lifetime" is quite peculiar, as the ON
TRANSACTION END RESET is adding transactional behavior to something that's not
supposed to be transactional, so more documentation about it seems appropriate.
Also DISCARD can be used any time so that's a totally different aspect of the
immutable variable content lifetime that's not described here.
NULL handling also seems inconsistent. An explicit default NULL value makes it
truly immutable, but manually assigning NULL is a different codepath that has a
different user behavior:
# create immutable variable var_immu int default null;
CREATE VARIABLE
# let var_immu = 1;
ERROR: 22005: session variable "var_immu" is declared IMMUTABLE
# create immutable variable var_immu2 int ;
CREATE VARIABLE
# let var_immu2 = null;
LET
# let var_immu2 = null;
LET
# let var_immu2 = 1;
LET
For var_immu2 I think that the last 2 queries should have errored out.
> > In revoke.sgml:
> > + REVOKE [ GRANT OPTION FOR ]
> > + { { READ | WRITE } [, ...] | ALL [ PRIVILEGES ] }
> > + ON VARIABLE <replaceable>variable_name</replaceable> [, ...]
> > + FROM { [ GROUP ] <replaceable
> > class="parameter">role_name</replaceable> | PUBLIC } [, ...]
> > + [ CASCADE | RESTRICT ]
> >
> > there's no extra documentation for that, and therefore no clarification on
> > variable_name.
> >
>
> This is same like function_name, domain_name, ...
Ah right.
> > pg_variable.c:
> > Do we really need both session_variable_get_name() and
> > get_session_variable_name()?
> >
>
> They are different - first returns possibly qualified name, second returns
> only name. Currently it is used just for error messages in
> transformAssignmentIndirection, and I think so it is good for consistency
> with other usage of this routine (transformAssignmentIndirection).
I agree that consistency with other usage is a good thing, but both functions
have very similar and confusing names. Usually when you need the qualified
name the calling code just takes care of doing so. Wouldn't it be better to
add say get_session_variable_namespace() and construct the target string in the
calling code?
Also, I didn't dig a lot but I didn't see other usage with optionally qualified
name there? I'm not sure how it would make sense anyway since LET semantics
are different and the current call for session variable emit incorrect
messages:
# create table tt(id integer);
CREATE TABLE
# create variable vv tt;
CREATE VARIABLE
# let vv.meh = 1;
ERROR: 42703: cannot assign to field "meh" of column "meh" because there is no such column in data type tt
LINE 1: let vv.meh = 1;
CREATE TABLE
CREATE VARIABLE
postgres=# let vv.meh = 1;
ERROR: cannot assign to field "meh" of column or variable "vv" because there is no such column in data type tt
LINE 1: let vv.meh = 1;
^
Вложения
Вложения
Hi, On Sun, Jan 30, 2022 at 08:09:18PM +0100, Pavel Stehule wrote: > > rebase after 02b8048ba5dc36238f3e7c3c58c5946220298d71 Here are a few comments, mostly about pg_variable.c and sessionvariable.c. I stopped before reading the whole patch as I have some concern about the sinval machanism, which ould change a bit the rest of the patch. I'm also attaching a patch (with .txt extension to avoid problem with the cfbot) with some comment update propositions. In sessionvariable.c, why VariableEOXAction and VariableEOXActionCodes? Can't the parser emit directly the char value, like e.g. relpersistence? extraneous returns for 2 functions: +void +get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, + Oid *collid) +{ [...] + return; +} +void +initVariable(Variable *var, Oid varid, bool fast_only) +{ [...] + return; +} VariableCreate(): Maybe add a bunch of AssertArg() for all the mandatory parametrers? Also, the check for variable already existing should be right after the AssertArg(), and using SearchSysCacheExistsX(). Maybe also adding an Assert(OidIsValid(xxxoid)) just after the CatalogTupleInsert(), similarly to some other creation functions? event-triggers.sgml needs updating for the firing matrix, as session variable are compatible with even triggers. +typedef enum SVariableXActAction +{ + ON_COMMIT_DROP, /* used for ON COMMIT DROP */ + ON_COMMIT_RESET, /* used for DROP VARIABLE */ + RESET, /* used for ON TRANSACTION END RESET */ + RECHECK /* recheck if session variable is living */ +} SVariableXActAction; The names seem a bit generic, maybe add a prefix like SVAR_xxx? ON_COMMIT_RESET is also confusing as it looks like an SQL clause. Maybe PERFORM_DROP or something? +static List *xact_drop_actions = NIL; +static List *xact_reset_actions = NIL; Maybe add a comment saying both are lists of SVariableXActAction? +typedef SVariableData * SVariable; looks like a missing bump to typedefs.list. +char * +get_session_variable_name(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + char *varname; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varname = NameStr(varform->varname); + + ReleaseSysCache(tup); + + return varname; +} This kind of function should return a palloc'd copy of the name. +void +ResetSessionVariables(void) [...] + list_free_deep(xact_drop_actions); + xact_drop_actions = NIL; + + list_free_deep(xact_reset_actions); + xact_drop_actions = NIL; +} The 2nd chunk should be xact_reset_actions = NIL +static void register_session_variable_xact_action(Oid varid, SVariableXActAction action); +static void delete_session_variable_xact_action(Oid varid, SVariableXActAction action); The naming is a bit confusing, maybe unregister_session_cable_xact_action() for consistency? +void +register_session_variable_xact_action(Oid varid, + SVariableXActAction action) the function is missing the static keyword. In AtPreEOXact_SessionVariable_on_xact_actions(), those 2 instructions are executed twice (once in the middle and once at the end): list_free_deep(xact_drop_actions); xact_drop_actions = NIL; + * If this entry was created during the current transaction, + * creating_subid is the ID of the creating subxact; if created in a prior + * transaction, creating_subid is zero. I don't see any place in the code where creating_subid can be zero? It looks like it's only there for future transactional implementation, but for now this attribute seems unnecessary? /* at transaction end recheck sinvalidated variables */ RegisterXactCallback(sync_sessionvars_xact_callback, NULL); I don't think it's ok to use xact callback for in-core code. The function explicitly says: > * These functions are intended for use by dynamically loaded modules. > * For built-in modules we generally just hardwire the appropriate calls > * (mainly because it's easier to control the order that way, where needed). Also, this function and AtPreEOXact_SessionVariable_on_xact_actions() are skipping all or part of the processing if there is no active transaction. Is that really ok? I'm particularly sceptical about AtPreEOXact_SessionVariable_on_xact_actions and the RECHECK actions, as the xact_reset_actions list is reset whether the recheck was done or not, so it seems to me that it could be leaking some entries in the hash table. If the database has a lot of object, it seems possible (while unlikely) that a subsequent CREATE VARIABLE can get the same oid leading to incorrect results? If that's somehow ok, wouldn't it be better to rearrange the code to call those functions less often, and only when they can do their work, or at least split the recheck in some different function / list? +static void +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) [...] + if (hashvalue != 0) + { [...] + } + else + sync_sessionvars_all = true; The rechecks being somewhat expensive, I think it could be a win to remove all pending rechecks when setting the sync_sessionvars_all.
Вложения
Hi,
On Sun, Jan 30, 2022 at 08:09:18PM +0100, Pavel Stehule wrote:
>
> rebase after 02b8048ba5dc36238f3e7c3c58c5946220298d71
Here are a few comments, mostly about pg_variable.c and sessionvariable.c. I
stopped before reading the whole patch as I have some concern about the sinval
machanism, which ould change a bit the rest of the patch. I'm also attaching a
patch (with .txt extension to avoid problem with the cfbot) with some comment
update propositions.
In sessionvariable.c, why VariableEOXAction and VariableEOXActionCodes? Can't
the parser emit directly the char value, like e.g. relpersistence?
extraneous returns for 2 functions:
+void
+get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod,
+ Oid *collid)
+{
[...]
+ return;
+}
+void
+initVariable(Variable *var, Oid varid, bool fast_only)
+{
[...]
+ return;
+}
VariableCreate():
Maybe add a bunch of AssertArg() for all the mandatory parametrers?
Also, the check for variable already existing should be right after the
AssertArg(), and using SearchSysCacheExistsX().
Maybe also adding an Assert(OidIsValid(xxxoid)) just after the
CatalogTupleInsert(), similarly to some other creation functions?
event-triggers.sgml needs updating for the firing matrix, as session variable
are compatible with even triggers.
+typedef enum SVariableXActAction
+{
+ ON_COMMIT_DROP, /* used for ON COMMIT DROP */
+ ON_COMMIT_RESET, /* used for DROP VARIABLE */
+ RESET, /* used for ON TRANSACTION END RESET */
+ RECHECK /* recheck if session variable is living */
+} SVariableXActAction;
The names seem a bit generic, maybe add a prefix like SVAR_xxx?
ON_COMMIT_RESET is also confusing as it looks like an SQL clause. Maybe
PERFORM_DROP or something?
<--> * and if this transaction or subtransaction will be committed,
<--> * we want to enforce variable cleaning. (we don't need to wait for
<--> * sinval message). The cleaning action for one session variable
<--> * can be repeated in the action list, and it doesn't do any problem
<--> * (so we don't need to ensure uniqueness). We need separate action
<--> * than RESET, because RESET is executed on any transaction end,
<--> * but we want to execute cleaning only when thecurrent transaction
<--> * will be committed.
<--> */
<-->register_session_variable_xact_action(varid, SVAR_ON_COMMIT_RESET);
+static List *xact_drop_actions = NIL;
+static List *xact_reset_actions = NIL;
Maybe add a comment saying both are lists of SVariableXActAction?
+typedef SVariableData * SVariable;
looks like a missing bump to typedefs.list.
+char *
+get_session_variable_name(Oid varid)
+{
+ HeapTuple tup;
+ Form_pg_variable varform;
+ char *varname;
+
+ tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for session variable %u", varid);
+
+ varform = (Form_pg_variable) GETSTRUCT(tup);
+
+ varname = NameStr(varform->varname);
+
+ ReleaseSysCache(tup);
+
+ return varname;
+}
This kind of function should return a palloc'd copy of the name.
+void
+ResetSessionVariables(void)
[...]
+ list_free_deep(xact_drop_actions);
+ xact_drop_actions = NIL;
+
+ list_free_deep(xact_reset_actions);
+ xact_drop_actions = NIL;
+}
The 2nd chunk should be xact_reset_actions = NIL
+static void register_session_variable_xact_action(Oid varid, SVariableXActAction action);
+static void delete_session_variable_xact_action(Oid varid, SVariableXActAction action);
The naming is a bit confusing, maybe unregister_session_cable_xact_action() for
consistency?
+void
+register_session_variable_xact_action(Oid varid,
+ SVariableXActAction action)
the function is missing the static keyword.
In AtPreEOXact_SessionVariable_on_xact_actions(), those 2 instructions are
executed twice (once in the middle and once at the end):
list_free_deep(xact_drop_actions);
xact_drop_actions = NIL;
+ * If this entry was created during the current transaction,
+ * creating_subid is the ID of the creating subxact; if created in a prior
+ * transaction, creating_subid is zero.
I don't see any place in the code where creating_subid can be zero? It looks
like it's only there for future transactional implementation, but for now this
attribute seems unnecessary?
select xx;
savepoint s1;
drop variable xx;
rollback to s1;
/* at transaction end recheck sinvalidated variables */
RegisterXactCallback(sync_sessionvars_xact_callback, NULL);
I don't think it's ok to use xact callback for in-core code. The function
explicitly says:
> * These functions are intended for use by dynamically loaded modules.
> * For built-in modules we generally just hardwire the appropriate calls
> * (mainly because it's easier to control the order that way, where needed).
Also, this function and AtPreEOXact_SessionVariable_on_xact_actions() are
skipping all or part of the processing if there is no active transaction. Is
that really ok?
I'm particularly sceptical about AtPreEOXact_SessionVariable_on_xact_actions
and the RECHECK actions, as the xact_reset_actions list is reset whether the
recheck was done or not, so it seems to me that it could be leaking some
entries in the hash table. If the database has a lot of object, it seems
possible (while unlikely) that a subsequent CREATE VARIABLE can get the same
oid leading to incorrect results?
If that's somehow ok, wouldn't it be better to rearrange the code to call those
functions less often, and only when they can do their work, or at least split
the recheck in some different function / list?
+static void
+pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue)
[...]
+ if (hashvalue != 0)
+ {
[...]
+ }
+ else
+ sync_sessionvars_all = true;
The rechecks being somewhat expensive, I think it could be a win to remove all
pending rechecks when setting the sync_sessionvars_all.
Вложения
+ * If this entry was created during the current transaction,
+ * creating_subid is the ID of the creating subxact; if created in a prior
+ * transaction, creating_subid is zero.
I don't see any place in the code where creating_subid can be zero? It looks
like it's only there for future transactional implementation, but for now this
attribute seems unnecessary?The comment is not 100% valid. I removed the sentence about zero value of creating_subid.
Вложения
Hi, On Wed, Mar 02, 2022 at 06:03:06AM +0100, Pavel Stehule wrote: > > I lost commit with this change. I am sending updated patch. Thanks a lot Pavel! I did a more thorough review of the patch. I'm attaching a diff (in .txt extension) for comment improvement suggestions. I may have misunderstood things so feel free to discard some of it. I will mention the comment I didn't understand in this mail. First, I spotted some problem in the invalidation logic. + * Assign sinval mark to session variable. This mark probably + * signalized, so the session variable was dropped. But this + * should be rechecked later against system catalog. + */ +static void +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) You mention that hashvalue can only be zero for commands that can't affect session variables (like VACUUM or ANALYZE), but that's not true. It can also happen in case of sinval queue overflow (see InvalidateSystemCaches()). So in that case we should trigger a full recheck, with some heuristics on how to detect that a cached variable is still valid. Unfortunately the oid can wraparound so some other check is needed to make it safe. Also, even if we get a non-zero hashvalue in the inval callback, we can't assume that there weren't any collision in the hash. So the additional check should be used there too. We had a long off-line discussion about this with Pavel yesterday on what heuristic to use there. Unlike other caches where discarding an entry when it shouldn't have been is not really problematic, the cache here contains the real variable value so we can't discard it unless the variable was really dropped. It should be possible to make it work, so I will let Pavel comment on which approach he wants to use and what the drawbacks are. I guess that this will be the most critical part of this patch to decide whether the approach is acceptable or not. The rest is only minor stylistic comments. Using -DRAW_EXPRESSION_COVERAGE_TEST I see that T_LetStmt is missing in raw_expression_tree_walker. ALTER and DROP both suggest "IMMUTABLE VARIABLE" as valid completion, while it should only be usable in the CREATE [ IMMUTABLE ] VARIABLE form. +initVariable(Variable *var, Oid varid, bool fast_only) +{ + var->collation = varform->varcollation; + var->eoxaction = varform->vareoxaction; + var->is_not_null = varform->varisnotnull; + var->is_immutable = varform->varisimmutable; nit: eoxaction is defined after is_not_null and is_immutable, it would be better to keep the initialization order consistent (same in VariableCreate). + values[Anum_pg_variable_varcollation - 1] = ObjectIdGetDatum((char) varCollation); + values[Anum_pg_variable_vareoxaction - 1] = CharGetDatum(eoxaction); seems like the char cast is on the wrong variable? + * [...] We have to hold two separate action lists: + * one for dropping the session variable from system catalog, and + * another one for resetting its value. Both are necessary, since + * dropping a session variable also needs to enforce a reset of + * the value. I don't fully understand that comment. Maybe you meant that the opposite isn't true, ie. highlight that a reset should *not* drop the variable thus two lists? +typedef enum SVariableXActAction +{ + SVAR_ON_COMMIT_DROP, /* used for ON COMMIT DROP */ + SVAR_ON_COMMIT_RESET, /* used for DROP VARIABLE */ + SVAR_RESET, /* used for ON TRANSACTION END RESET */ + SVAR_RECHECK /* verify if session variable still exists */ +} SVariableXActAction; + +typedef struct SVariableXActActionItem +{ + Oid varid; /* varid of session variable */ + SVariableXActAction action; /* reset or drop */ the stored action isn't simply "reset or drop", even though the resulting action will be a reset or a drop (or a no-op) right? Since it's storing a enum define just before, I'd just drop the comment on action, and maybe specify that SVAR_RECHECK will do appropriate cleanup if the session variable doesn't exist. + * Release the variable defined by varid from sessionvars + * hashtab. + */ +static void +free_session_variable(SVariable svar) The function name is a bit confusing given the previous function. Maybe this one should be called forget_session_variable() instead, or something like that? I think the function comment should also mention that caller is responsible for making sure that the sessionvars htab exists before calling it, for extra clarity, or just add an assert for that. +static void +free_session_variable_varid(Oid varid) Similary, maybe renaming this function forget_session_variable_by_id()? +static void +create_sessionvars_hashtable(void) +{ + HASHCTL ctl; + + /* set callbacks */ + if (first_time) + { + /* Read sinval messages */ + CacheRegisterSyscacheCallback(VARIABLEOID, + pg_variable_cache_callback, + (Datum) 0); + + first_time = false; + } + + /* needs its own long lived memory context */ + if (SVariableMemoryContext == NULL) + { + SVariableMemoryContext = + AllocSetContextCreate(TopMemoryContext, + "session variables", + ALLOCSET_START_SMALL_SIZES); + } As far as I can see the SVariableMemoryContext can be reset but never set to NULL, so I think the initialization can be done in the first_time case, and otherwise asserted that it's not NULL. + if (!isnull && svar->typid != typid) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("type \"%s\" of assigned value is different than type \"%s\" of session variable \"%s.% Why testing isnull? I don't think it's ok to allow NULL::text in an int variable for instance. This isn't valid in other context (like inserting in a table) + * result of default expression always). Don't do this check, when variable + * is initialized. + */ + if (!init_mode && I think the last part of the comment is a bit misleading. Maybe "when variable is being initialized" (and similary same for the function comment). + * We try not to break the previous value, if something is wrong. + * + * As side efect this function acquires AccessShareLock on + * related session variable until commit. + */ +void +SetSessionVariable(Oid varid, Datum value, bool isNull, Oid typid) I don't understand what you mean by "We try not to break the previous value, if something is wrong". + /* Initialize svar when not initialized or when stored value is null */ + if (!found) + { + Variable var; + + /* don't need defexpr and acl here */ + initVariable(&var, varid, true); + init_session_variable(svar, &var); + } + + set_session_variable(svar, value, isNull, typid, false); Shouldn't the comment be on the set_session_variable() vall rather than on the !found block? + * Returns the value of the session variable specified by varid. Check correct + * result type. Optionally the result can be copied. + */ +Datum +GetSessionVariable(Oid varid, bool *isNull, Oid expected_typid, bool copy) All callers use copy == true, couldn't we get rid of it and say it returns a copy of the value if any? + * Create new ON_COMMIT_DROP xact action. We have to drop + * ON COMMIT DROP variable, although this variable should not + * be used. So we need to register this action in CREATE VARIABLE + * time. I don't understand this comment. +AtPreEOXact_SessionVariable_on_xact_actions(bool isCommit) +{ + ListCell *l; + + foreach(l, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + /* Iterate only over non dropped entries */ + if (xact_ai->deleting_subid == InvalidSubTransactionId) + { + Assert(xact_ai->action == SVAR_ON_COMMIT_DROP); The assert sould probably be in the block above. + * We want to reset session variable (release it from + * local memory) when RESET is required or when session + * variable was removed explicitly (DROP VARIABLE) or + * implicitly (ON COMMIT DROP). Explicit releasing should + * be done only if the transaction is commited. + */ + if ((xact_ai->action == SVAR_RESET) || + (xact_ai->action == SVAR_ON_COMMIT_RESET && + xact_ai->deleting_subid == InvalidSubTransactionId && + isCommit)) + free_session_variable_varid(xact_ai->varid); This chunk is a bit hard to follow. Also, for SVAR_RESET wouldn't it be better to only make the svar invalid and keep it in the htab? If so, this could be split in two different branches which would be easier to follow. + if (!isCommit && + xact_ai->creating_subid == mySubid && + xact_ai->action != SVAR_RESET && + xact_ai->action != SVAR_RECHECK) + { + /* cur_item must be removed */ + xact_reset_actions = foreach_delete_current(xact_reset_actions, cur_item); + pfree(xact_ai); I think that be definition only the SVAR_ON_COMMIT_DROP (cleaning entry for a dropped session variable) will ever need to be removed there, so we should check for that instead of not being something else? + /* + * Prepare session variables, if not prepared in queryDesc + */ + if (queryDesc->num_session_variables > 0) I don't understand that comment. +static void +svariableStartupReceiver(DestReceiver *self, int operation, TupleDesc typeinfo) +{ + svariableState *myState = (svariableState *) self; + int natts = typeinfo->natts; + int outcols = 0; + int i; + + for (i = 0; i < natts; i++) + { + Form_pg_attribute attr = TupleDescAttr(typeinfo, i); + + if (attr->attisdropped) + continue; + + if (++outcols > 1) + elog(ERROR, "svariable DestReceiver can take only one attribute"); + + myState->typid = attr->atttypid; + myState->typmod = attr->atttypmod; + myState->typlen = attr->attlen; + myState->slot_offset = i; + } + + myState->rows = 0; +} Maybe add an initial Assert to make sure that caller did call SetVariableDestReceiverParams(), and final check that one attribute was found? @@ -1794,15 +1840,39 @@ fix_expr_common(PlannerInfo *root, Node *node) g->cols = cols; } } + else if (IsA(node, Param)) + { + Param *p = (Param *) node; + + if (p->paramkind == PARAM_VARIABLE) + { + PlanInvalItem *inval_item = makeNode(PlanInvalItem); + + /* paramid is still session variable id */ + inval_item->cacheId = VARIABLEOID; + inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(p->paramvarid)); + + /* Append this variable to global, register dependency */ + root->glob->invalItems = lappend(root->glob->invalItems, + inval_item); + } + } I didn't see any test covering invalidation of cached plan using session variables. Could you add some? While at it, maybe use different values on the sesssion_variable.sql tests rather than 100 in many places, so it's easier to identifier which case broke in case of problem. +static Node * +makeParamSessionVariable(ParseState *pstate, + Oid varid, Oid typid, int32 typmod, Oid collid, + char *attrname, int location) +{ [...] + /* + * There are two ways to access session variables - direct, used by simple + * plpgsql expressions, where it is not necessary to emulate stability. + * And Buffered access, which is used everywhere else. We should ensure + * stable values, and because session variables are global, then we should + * work with copied values instead of directly accessing variables. For + * direct access, the varid is best. For buffered access, we need + * to assign an index to the buffer - later, when we know what variables are + * used. Now, we just remember, so we use session variables. I don't understand the last part, starting with "For buffered access, we need...". Also, the beginning of the comment seems like something more general and may be moved somewhere, maybe at the beginning of sessionvariable.c? + * stmt->query is SelectStmt node. An tranformation of + * this node doesn't support SetToDefault node. Instead injecting + * of transformSelectStmt or parse state, we can directly + * transform target list here if holds SetToDefault node. + */ + if (stmt->set_default) I don't understand this comment. Especially since the next transformTargetList() will emit SetToDefault node that will be handled later in that function and then in RewriteQuery. + /* + * rewrite SetToDefaults needs varid in Query structure + */ + query->resultVariable = varid; I also don't understand that comment. Is is always set just in case there's a SetToDefault, or something else? + /* translate paramvarid to session variable name */ + if (param->paramkind == PARAM_VARIABLE) + { + appendStringInfo(context->buf, "%s", + generate_session_variable_name(param->paramvarid)); + return; + } A bit more work seems to be needed for deparsing session variables: # create variable myvar text; CREATE VARIABLE # create view myview as select myvar; CREATE VIEW # \d+ myview View "public.myview" Column | Type | Collation | Nullable | Default | Storage | Description --------+------+-----------+----------+---------+----------+------------- myvar | text | | | | extended | View definition: SELECT myvar AS myvar; There shouldn't be an explicit alias I think.
Вложения
On Thu, Mar 03, 2022 at 03:06:52PM +0800, Julien Rouhaud wrote: > Hi, > > On Wed, Mar 02, 2022 at 06:03:06AM +0100, Pavel Stehule wrote: > > > > I lost commit with this change. I am sending updated patch. Also, another thing is the size of the patch. It's probably the minimum to have a consistent working implementation, but maybe we can still split it to make review easier? For instance, maybe having: - the pg_variable part on its own, without a way to use them, maybe with syscache helpers - the main session variable implementation and test coverage - plpgsql support and test coverage - pg_dump support and test coverage It wouldn't make the main patch that small but could still help quite a bit. Any better suggestion?
On Thu, Mar 03, 2022 at 03:06:52PM +0800, Julien Rouhaud wrote: > Hi, > > On Wed, Mar 02, 2022 at 06:03:06AM +0100, Pavel Stehule wrote: > > > > I lost commit with this change. I am sending updated patch. > > Thanks a lot Pavel! > > I did a more thorough review of the patch. I'm attaching a diff (in .txt > extension) for comment improvement suggestions. I may have misunderstood But the attachment actually was a *.patch, so cfbot tried and failed to apply it.
Вложения
On Sat, Mar 19, 2022 at 04:46:13PM -0500, Justin Pryzby wrote: > On Thu, Mar 03, 2022 at 03:06:52PM +0800, Julien Rouhaud wrote: > > Hi, > > > > On Wed, Mar 02, 2022 at 06:03:06AM +0100, Pavel Stehule wrote: > > > > > > I lost commit with this change. I am sending updated patch. > > > > Thanks a lot Pavel! > > > > I did a more thorough review of the patch. I'm attaching a diff (in .txt > > extension) for comment improvement suggestions. I may have misunderstood > > But the attachment actually was a *.patch, so cfbot tried and failed to apply > it. Argh, I indeed failed to rename the patch. Thanks!
A bit more work seems to be needed for deparsing session variables:
# create variable myvar text;
CREATE VARIABLE
# create view myview as select myvar;
CREATE VIEW
# \d+ myview
View "public.myview"
Column | Type | Collation | Nullable | Default | Storage | Description
--------+------+-----------+----------+---------+----------+-------------
myvar | text | | | | extended |
View definition:
SELECT myvar AS myvar;
There shouldn't be an explicit alias I think.
Hi, On Wed, Mar 23, 2022 at 09:58:59PM +0100, Pavel Stehule wrote: > > A bit more work seems to be needed for deparsing session variables: > > > > # create variable myvar text; > > CREATE VARIABLE > > > > # create view myview as select myvar; > > CREATE VIEW > > > > # \d+ myview > > View "public.myview" > > Column | Type | Collation | Nullable | Default | Storage | Description > > --------+------+-----------+----------+---------+----------+------------- > > myvar | text | | | | extended | > > View definition: > > SELECT myvar AS myvar; > > > > There shouldn't be an explicit alias I think. > > > > I check this issue, and I afraid so it is not fixable. The target list > entry related to session variable has not some magic value like ?column? > that can be used for check if tle->resname is implicit or explicit > > And in this time I cannot to use FigureColname because it doesn't work with > transformed nodes. More - the Param node can be nested in SubscriptingRef > or FieldSelect. It doesn't work perfectly now. See following example: > > create type xt as (a int, b int); > create view b as select (10, ((random()*100)::int)::xt).b; > \d+ b > SELECT (ROW(10, (random() * 100::double precision)::integer)::xt).b AS b; Fair enough. Since there is other code that already behaves the same I agree that it's better to not add special cases in ruleutils.c and have an explicit alias in the deparsed view, which isn't incorrect.
On Thu, Mar 03, 2022 at 03:06:52PM +0800, Julien Rouhaud wrote:
> Hi,
>
> On Wed, Mar 02, 2022 at 06:03:06AM +0100, Pavel Stehule wrote:
> >
> > I lost commit with this change. I am sending updated patch.
Also, another thing is the size of the patch. It's probably the minimum to
have a consistent working implementation, but maybe we can still split it to
make review easier?
For instance, maybe having:
- the pg_variable part on its own, without a way to use them, maybe with
syscache helpers
- the main session variable implementation and test coverage
- plpgsql support and test coverage
- pg_dump support and test coverage
It wouldn't make the main patch that small but could still help quite a bit.
Any better suggestion?
Вложения
- v20220410-0011-documentation.patch
- v20220410-0010-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220410-0007-Enhancing-psql-for-session-variables.patch
- v20220410-0009-Regress-tests-for-session-variables.patch
- v20220410-0008-typedefs.patch
- v20220410-0006-Support-of-LET-command-in-PLpgSQL.patch
- v20220410-0005-LET-command.patch
- v20220410-0004-DISCARD-VARIABLES-command.patch
- v20220410-0003-session-variables.patch
- v20220410-0002-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220410-0001-Catalogue-support-for-session-variables.patch
On Sun, Apr 10, 2022 at 08:30:39PM +0200, Pavel Stehule wrote: > I am sending fresh rebased patch + separation to more patches. This split > is initial, and can be changed later The 0001 patch requires this, but it's not included until 0003. src/include/commands/session_variable.h Each patch should compile and pass tests with the preceding patches, without the following patches. I think the regression tests should be included with their corresponding patch. Maybe it's ok to separate out the changes for pg_dump, docs, and psql - but they'd have to be merged together eventually. I realize some of this runs counter to Julien's suggestion to split patches. The version should be changed: + if (fout->remoteVersion < 150000) I enabled these, which causes the regression tests fail: +#define COPY_PARSE_PLAN_TREES +#define WRITE_READ_PARSE_PLAN_TREES +#define RAW_EXPRESSION_COVERAGE_TEST /home/pryzbyj/src/postgres/src/test/regress/results/session_variables.out 2022-04-10 15:37:32.926306124 -0500 @@ -16,7 +16,7 @@ SET ROLE TO var_test_role; -- should fail LET var1 = 10; -ERROR: permission denied for session variable var1 +ERROR: unrecognized node type: 368 ...
Hi, On Sun, Apr 10, 2022 at 03:43:33PM -0500, Justin Pryzby wrote: > On Sun, Apr 10, 2022 at 08:30:39PM +0200, Pavel Stehule wrote: > > I am sending fresh rebased patch + separation to more patches. This split > > is initial, and can be changed later > > The 0001 patch requires this, but it's not included until 0003. > src/include/commands/session_variable.h > > Each patch should compile and pass tests with the preceding patches, without > the following patches. I think the regression tests should be included with > their corresponding patch. Maybe it's ok to separate out the changes for > pg_dump, docs, and psql - but they'd have to be merged together eventually. > I realize some of this runs counter to Julien's suggestion to split patches. Note that most of my suggestions were only to make the patch easier to review, which was mostly trying to limit a bit the core of the new code. Unfortunately, given the feature we can't really split the patch in many and smaller parts and expect them to be completely self contained, so I'm not against splitting smaller chunks like psql support and whatnot. But I'm not convinced that it will make it easier to review.
On Sun, Apr 10, 2022 at 08:30:39PM +0200, Pavel Stehule wrote:
> I am sending fresh rebased patch + separation to more patches. This split
> is initial, and can be changed later
The 0001 patch requires this, but it's not included until 0003.
src/include/commands/session_variable.h
Each patch should compile and pass tests with the preceding patches, without
the following patches. I think the regression tests should be included with
their corresponding patch. Maybe it's ok to separate out the changes for
pg_dump, docs, and psql - but they'd have to be merged together eventually.
I realize some of this runs counter to Julien's suggestion to split patches.
The version should be changed:
+ if (fout->remoteVersion < 150000)
I enabled these, which causes the regression tests fail:
+#define COPY_PARSE_PLAN_TREES
+#define WRITE_READ_PARSE_PLAN_TREES
+#define RAW_EXPRESSION_COVERAGE_TEST
/home/pryzbyj/src/postgres/src/test/regress/results/session_variables.out 2022-04-10 15:37:32.926306124 -0500
@@ -16,7 +16,7 @@
SET ROLE TO var_test_role;
-- should fail
LET var1 = 10;
-ERROR: permission denied for session variable var1
+ERROR: unrecognized node type: 368
...
Вложения
- v20220412-0001-Catalogue-support-for-session-variables.patch
- v20220412-0005-DISCARD-VARIABLES-command.patch
- v20220412-0004-Support-of-LET-command-in-PLpgSQL.patch
- v20220412-0003-LET-command.patch
- v20220412-0002-session-variables.patch
- v20220412-0007-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220412-0006-Enhancing-psql-for-session-variables.patch
- v20220412-0008-typedefs.patch
- v20220412-0009-Regress-tests-for-session-variables.patch
- v20220412-0010-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220412-0011-documentation.patch
Hi,
On Wed, Mar 02, 2022 at 06:03:06AM +0100, Pavel Stehule wrote:
>
> I lost commit with this change. I am sending updated patch.
Thanks a lot Pavel!
I did a more thorough review of the patch. I'm attaching a diff (in .txt
extension) for comment improvement suggestions. I may have misunderstood
things so feel free to discard some of it. I will mention the comment I didn't
understand in this mail.
First, I spotted some problem in the invalidation logic.
+ * Assign sinval mark to session variable. This mark probably
+ * signalized, so the session variable was dropped. But this
+ * should be rechecked later against system catalog.
+ */
+static void
+pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue)
You mention that hashvalue can only be zero for commands that can't
affect session variables (like VACUUM or ANALYZE), but that's not true. It can
also happen in case of sinval queue overflow (see InvalidateSystemCaches()).
So in that case we should trigger a full recheck, with some heuristics on how
to detect that a cached variable is still valid. Unfortunately the oid can
wraparound so some other check is needed to make it safe.
Also, even if we get a non-zero hashvalue in the inval callback, we can't
assume that there weren't any collision in the hash. So the additional check
should be used there too.
We had a long off-line discussion about this with Pavel yesterday on what
heuristic to use there. Unlike other caches where discarding an entry when it
shouldn't have been is not really problematic, the cache here contains the real
variable value so we can't discard it unless the variable was really dropped.
It should be possible to make it work, so I will let Pavel comment on which
approach he wants to use and what the drawbacks are. I guess that this will be
the most critical part of this patch to decide whether the approach is
acceptable or not.
The rest is only minor stylistic comments.
Using -DRAW_EXPRESSION_COVERAGE_TEST I see that T_LetStmt is missing in
raw_expression_tree_walker.
ALTER and DROP both suggest "IMMUTABLE VARIABLE" as valid completion, while
it should only be usable in the CREATE [ IMMUTABLE ] VARIABLE form.
+initVariable(Variable *var, Oid varid, bool fast_only)
+{
+ var->collation = varform->varcollation;
+ var->eoxaction = varform->vareoxaction;
+ var->is_not_null = varform->varisnotnull;
+ var->is_immutable = varform->varisimmutable;
nit: eoxaction is defined after is_not_null and is_immutable, it would be
better to keep the initialization order consistent (same in VariableCreate).
+ values[Anum_pg_variable_varcollation - 1] = ObjectIdGetDatum((char) varCollation);
+ values[Anum_pg_variable_vareoxaction - 1] = CharGetDatum(eoxaction);
seems like the char cast is on the wrong variable?
+ * [...] We have to hold two separate action lists:
+ * one for dropping the session variable from system catalog, and
+ * another one for resetting its value. Both are necessary, since
+ * dropping a session variable also needs to enforce a reset of
+ * the value.
I don't fully understand that comment. Maybe you meant that the opposite isn't
true, ie. highlight that a reset should *not* drop the variable thus two lists?
+typedef enum SVariableXActAction
+{
+ SVAR_ON_COMMIT_DROP, /* used for ON COMMIT DROP */
+ SVAR_ON_COMMIT_RESET, /* used for DROP VARIABLE */
+ SVAR_RESET, /* used for ON TRANSACTION END RESET */
+ SVAR_RECHECK /* verify if session variable still exists */
+} SVariableXActAction;
+
+typedef struct SVariableXActActionItem
+{
+ Oid varid; /* varid of session variable */
+ SVariableXActAction action; /* reset or drop */
the stored action isn't simply "reset or drop", even though the resulting
action will be a reset or a drop (or a no-op) right? Since it's storing a enum
define just before, I'd just drop the comment on action, and maybe specify that
SVAR_RECHECK will do appropriate cleanup if the session variable doesn't exist.
+ * Release the variable defined by varid from sessionvars
+ * hashtab.
+ */
+static void
+free_session_variable(SVariable svar)
The function name is a bit confusing given the previous function. Maybe this
one should be called forget_session_variable() instead, or something like that?
I think the function comment should also mention that caller is responsible for
making sure that the sessionvars htab exists before calling it, for extra
clarity, or just add an assert for that.
+static void
+free_session_variable_varid(Oid varid)
Similary, maybe renaming this function forget_session_variable_by_id()?
+static void
+create_sessionvars_hashtable(void)
+{
+ HASHCTL ctl;
+
+ /* set callbacks */
+ if (first_time)
+ {
+ /* Read sinval messages */
+ CacheRegisterSyscacheCallback(VARIABLEOID,
+ pg_variable_cache_callback,
+ (Datum) 0);
+
+ first_time = false;
+ }
+
+ /* needs its own long lived memory context */
+ if (SVariableMemoryContext == NULL)
+ {
+ SVariableMemoryContext =
+ AllocSetContextCreate(TopMemoryContext,
+ "session variables",
+ ALLOCSET_START_SMALL_SIZES);
+ }
As far as I can see the SVariableMemoryContext can be reset but never set to
NULL, so I think the initialization can be done in the first_time case, and
otherwise asserted that it's not NULL.
+ if (!isnull && svar->typid != typid)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("type \"%s\" of assigned value is different than type \"%s\" of session variable \"%s.%
Why testing isnull? I don't think it's ok to allow NULL::text in an int
variable for instance. This isn't valid in other context (like inserting in a
table)
+ * result of default expression always). Don't do this check, when variable
+ * is initialized.
+ */
+ if (!init_mode &&
I think the last part of the comment is a bit misleading. Maybe "when variable
is being initialized" (and similary same for the function comment).
+ * We try not to break the previous value, if something is wrong.
+ *
+ * As side efect this function acquires AccessShareLock on
+ * related session variable until commit.
+ */
+void
+SetSessionVariable(Oid varid, Datum value, bool isNull, Oid typid)
I don't understand what you mean by "We try not to break the previous value, if
something is wrong".
+ /* Initialize svar when not initialized or when stored value is null */
+ if (!found)
+ {
+ Variable var;
+
+ /* don't need defexpr and acl here */
+ initVariable(&var, varid, true);
+ init_session_variable(svar, &var);
+ }
+
+ set_session_variable(svar, value, isNull, typid, false);
Shouldn't the comment be on the set_session_variable() vall rather than on the
!found block?
+ * Returns the value of the session variable specified by varid. Check correct
+ * result type. Optionally the result can be copied.
+ */
+Datum
+GetSessionVariable(Oid varid, bool *isNull, Oid expected_typid, bool copy)
All callers use copy == true, couldn't we get rid of it and say it returns a
copy of the value if any?
+ * Create new ON_COMMIT_DROP xact action. We have to drop
+ * ON COMMIT DROP variable, although this variable should not
+ * be used. So we need to register this action in CREATE VARIABLE
+ * time.
I don't understand this comment.
+AtPreEOXact_SessionVariable_on_xact_actions(bool isCommit)
+{
+ ListCell *l;
+
+ foreach(l, xact_drop_actions)
+ {
+ SVariableXActActionItem *xact_ai =
+ (SVariableXActActionItem *) lfirst(l);
+
+ /* Iterate only over non dropped entries */
+ if (xact_ai->deleting_subid == InvalidSubTransactionId)
+ {
+ Assert(xact_ai->action == SVAR_ON_COMMIT_DROP);
The assert sould probably be in the block above.
+ * We want to reset session variable (release it from
+ * local memory) when RESET is required or when session
+ * variable was removed explicitly (DROP VARIABLE) or
+ * implicitly (ON COMMIT DROP). Explicit releasing should
+ * be done only if the transaction is commited.
+ */
+ if ((xact_ai->action == SVAR_RESET) ||
+ (xact_ai->action == SVAR_ON_COMMIT_RESET &&
+ xact_ai->deleting_subid == InvalidSubTransactionId &&
+ isCommit))
+ free_session_variable_varid(xact_ai->varid);
This chunk is a bit hard to follow. Also, for SVAR_RESET wouldn't it be better
to only make the svar invalid and keep it in the htab? If so, this could be
split in two different branches which would be easier to follow.
+ if (!isCommit &&
+ xact_ai->creating_subid == mySubid &&
+ xact_ai->action != SVAR_RESET &&
+ xact_ai->action != SVAR_RECHECK)
+ {
+ /* cur_item must be removed */
+ xact_reset_actions = foreach_delete_current(xact_reset_actions, cur_item);
+ pfree(xact_ai);
I think that be definition only the SVAR_ON_COMMIT_DROP (cleaning entry for a
dropped session variable) will ever need to be removed there, so we should
check for that instead of not being something else?
+ /*
+ * Prepare session variables, if not prepared in queryDesc
+ */
+ if (queryDesc->num_session_variables > 0)
I don't understand that comment.
+static void
+svariableStartupReceiver(DestReceiver *self, int operation, TupleDesc typeinfo)
+{
+ svariableState *myState = (svariableState *) self;
+ int natts = typeinfo->natts;
+ int outcols = 0;
+ int i;
+
+ for (i = 0; i < natts; i++)
+ {
+ Form_pg_attribute attr = TupleDescAttr(typeinfo, i);
+
+ if (attr->attisdropped)
+ continue;
+
+ if (++outcols > 1)
+ elog(ERROR, "svariable DestReceiver can take only one attribute");
+
+ myState->typid = attr->atttypid;
+ myState->typmod = attr->atttypmod;
+ myState->typlen = attr->attlen;
+ myState->slot_offset = i;
+ }
+
+ myState->rows = 0;
+}
Maybe add an initial Assert to make sure that caller did call
SetVariableDestReceiverParams(), and final check that one attribute was found?
@@ -1794,15 +1840,39 @@ fix_expr_common(PlannerInfo *root, Node *node)
g->cols = cols;
}
}
+ else if (IsA(node, Param))
+ {
+ Param *p = (Param *) node;
+
+ if (p->paramkind == PARAM_VARIABLE)
+ {
+ PlanInvalItem *inval_item = makeNode(PlanInvalItem);
+
+ /* paramid is still session variable id */
+ inval_item->cacheId = VARIABLEOID;
+ inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID,
+ ObjectIdGetDatum(p->paramvarid));
+
+ /* Append this variable to global, register dependency */
+ root->glob->invalItems = lappend(root->glob->invalItems,
+ inval_item);
+ }
+ }
I didn't see any test covering invalidation of cached plan using session
variables. Could you add some? While at it, maybe use different values on the
sesssion_variable.sql tests rather than 100 in many places, so it's easier to
identifier which case broke in case of problem.
+static Node *
+makeParamSessionVariable(ParseState *pstate,
+ Oid varid, Oid typid, int32 typmod, Oid collid,
+ char *attrname, int location)
+{
[...]
+ /*
+ * There are two ways to access session variables - direct, used by simple
+ * plpgsql expressions, where it is not necessary to emulate stability.
+ * And Buffered access, which is used everywhere else. We should ensure
+ * stable values, and because session variables are global, then we should
+ * work with copied values instead of directly accessing variables. For
+ * direct access, the varid is best. For buffered access, we need
+ * to assign an index to the buffer - later, when we know what variables are
+ * used. Now, we just remember, so we use session variables.
I don't understand the last part, starting with "For buffered access, we
need...". Also, the beginning of the comment seems like something more general
and may be moved somewhere, maybe at the beginning of sessionvariable.c?
+ * stmt->query is SelectStmt node. An tranformation of
+ * this node doesn't support SetToDefault node. Instead injecting
+ * of transformSelectStmt or parse state, we can directly
+ * transform target list here if holds SetToDefault node.
+ */
+ if (stmt->set_default)
I don't understand this comment. Especially since the next
transformTargetList() will emit SetToDefault node that will be handled later in
that function and then in RewriteQuery.
+ /*
+ * rewrite SetToDefaults needs varid in Query structure
+ */
+ query->resultVariable = varid;
I also don't understand that comment. Is is always set just in case there's a
SetToDefault, or something else?
+ /* translate paramvarid to session variable name */
+ if (param->paramkind == PARAM_VARIABLE)
+ {
+ appendStringInfo(context->buf, "%s",
+ generate_session_variable_name(param->paramvarid));
+ return;
+ }
A bit more work seems to be needed for deparsing session variables:
# create variable myvar text;
CREATE VARIABLE
# create view myview as select myvar;
CREATE VIEW
# \d+ myview
View "public.myview"
Column | Type | Collation | Nullable | Default | Storage | Description
--------+------+-----------+----------+---------+----------+-------------
myvar | text | | | | extended |
View definition:
SELECT myvar AS myvar;
There shouldn't be an explicit alias I think.
Вложения
- v20220621-0011-documentation.patch
- v20220621-0010-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220621-0009-Regress-tests-for-session-variables.patch
- v20220621-0008-typedefs.patch
- v20220621-0007-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220621-0006-Enhancing-psql-for-session-variables.patch
- v20220621-0004-Support-of-LET-command-in-PLpgSQL.patch
- v20220621-0005-DISCARD-VARIABLES-command.patch
- v20220621-0003-LET-command.patch
- v20220621-0002-session-variables.patch
- v20220621-0001-Catalogue-support-for-session-variables.patch
Вложения
- v20220621-0009-Regress-tests-for-session-variables.patch
- v20220621-0008-typedefs.patch
- v20220621-0010-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220621-0007-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220621-0005-DISCARD-VARIABLES-command.patch
- v20220621-0004-Support-of-LET-command-in-PLpgSQL.patch
- v20220621-0011-documentation.patch
- v20220621-0003-LET-command.patch
- v20220621-0006-Enhancing-psql-for-session-variables.patch
- v20220621-0002-session-variables.patch
- v20220621-0001-Catalogue-support-for-session-variables.patch
Вложения
- v20220705-0012-documentation.patch
- v20220705-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220705-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220705-0009-typedefs.patch
- v20220705-0010-Regress-tests-for-session-variables.patch
- v20220705-0007-Enhancing-psql-for-session-variables.patch
- v20220705-0006-DISCARD-VARIABLES-command.patch
- v20220705-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220705-0004-LET-command.patch
- v20220705-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220705-0002-session-variables.patch
- v20220705-0001-Catalogue-support-for-session-variables.patch
Hi Pavel, On Tue, Jul 05, 2022 at 08:42:09AM +0200, Pavel Stehule wrote: > Hi > > fresh rebase + type check. Before returning any value, the related type is > checked if it is valid still Great news, thanks a lot for keeping working on it! I'm still in PTO since last Friday, but I'm planning to start reviewing this patch as soon as I come back. It might take a while as my knowledge of this patch are a bit blurry but hopefully it shouldn't take too long.
Hi Pavel,
On Tue, Jul 05, 2022 at 08:42:09AM +0200, Pavel Stehule wrote:
> Hi
>
> fresh rebase + type check. Before returning any value, the related type is
> checked if it is valid still
Great news, thanks a lot for keeping working on it! I'm still in PTO since
last Friday, but I'm planning to start reviewing this patch as soon as I come
back. It might take a while as my knowledge of this patch are a bit blurry but
hopefully it shouldn't take too long.
Hifresh rebase + type check. Before returning any value, the related type is checked if it is valid still
RegardsPavel
Вложения
- 0009-typedefs.patch
- 0012-documentation.patch
- 0013-This-commit-should-to-help-to-find-and-reason-why-re.patch
- 0010-Regress-tests-for-session-variables.patch
- 0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- 0007-Enhancing-psql-for-session-variables.patch
- 0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- 0006-DISCARD-VARIABLES-command.patch
- 0005-Support-of-LET-command-in-PLpgSQL.patch
- 0004-LET-command.patch
- 0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- 0002-session-variables.patch
- 0001-Catalogue-support-for-session-variables.patch
Hiút 5. 7. 2022 v 8:42 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Hifresh rebase + type check. Before returning any value, the related type is checked if it is valid stillThis set of patches should to help me with investigation of regress test fail reported by cfbot
RegardsPavelRegardsPavel
Вложения
- 0012-documentation.patch
- 0010-Regress-tests-for-session-variables.patch
- 0009-typedefs.patch
- 0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- 0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- 0007-Enhancing-psql-for-session-variables.patch
- 0013-This-commit-should-to-help-to-find-and-reason-why-re.patch
- 0006-DISCARD-VARIABLES-command.patch
- 0005-Support-of-LET-command-in-PLpgSQL.patch
- 0004-LET-command.patch
- 0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- 0002-session-variables.patch
- 0001-Catalogue-support-for-session-variables.patch
On Wed, Jul 06, 2022 at 10:30:31PM +0200, Pavel Stehule wrote: > This set of patches should to help me with investigation of regress test > fail reported by cfbot Do you know you can do the same as what cfbot does under your own github account ? Please see: src/tools/ci/README -- Justin
On Wed, Jul 06, 2022 at 10:30:31PM +0200, Pavel Stehule wrote:
> This set of patches should to help me with investigation of regress test
> fail reported by cfbot
Do you know you can do the same as what cfbot does under your own github
account ? Please see: src/tools/ci/README
--
Justin
Вложения
- v20220707-0012-documentation.patch
- v20220707-0010-Regress-tests-for-session-variables.patch
- v20220707-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220707-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220707-0009-typedefs.patch
- v20220707-0006-DISCARD-VARIABLES-command.patch
- v20220707-0007-Enhancing-psql-for-session-variables.patch
- v20220707-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220707-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220707-0004-LET-command.patch
- v20220707-0002-session-variables.patch
- v20220707-0001-Catalogue-support-for-session-variables.patch
Вложения
- v20220709-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220709-0009-typedefs.patch
- v20220709-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220709-0010-Regress-tests-for-session-variables.patch
- v20220709-0012-documentation.patch
- v20220709-0007-Enhancing-psql-for-session-variables.patch
- v20220709-0006-DISCARD-VARIABLES-command.patch
- v20220709-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220709-0004-LET-command.patch
- v20220709-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220709-0002-session-variables.patch
- v20220709-0001-Catalogue-support-for-session-variables.patch
Вложения
- v20220721-0013-debug-func.patch
- v20220721-0009-typedefs.patch
- v20220721-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220721-0010-Regress-tests-for-session-variables.patch
- v20220721-0012-documentation.patch
- v20220721-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220721-0007-Enhancing-psql-for-session-variables.patch
- v20220721-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220721-0006-DISCARD-VARIABLES-command.patch
- v20220721-0004-LET-command.patch
- v20220721-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220721-0002-session-variables.patch
- v20220721-0001-Catalogue-support-for-session-variables.patch
On 7/21/22 08:16, Pavel Stehule wrote: > Hi > > new update of session variable;s implementation > > - fresh rebase > - new possibility to trace execution with DEBUG1 notification > - new SRF function pg_debug_show_used_session_variables that returns > content of sessionvars hashtab > - redesign of work with list of variables for reset, recheck, on commit > drop, on commit reset Hi Pavel, I don't know exactly what failed but the docs (html/pdf) don't build: cd ~/pg_stuff/pg_sandbox/pgsql.schema_variables/doc/src/sgml $ make html /usr/bin/xmllint --path . --noout --valid postgres.sgml postgres.sgml:374: element link: validity error : IDREF attribute linkend references an unknown ID "catalog-pg-variable" make: *** [Makefile:135: html-stamp] Error 4 Erik Rijkers > > Regards > > Pavel >
Hi, On Thu, Jul 21, 2022 at 09:09:47AM +0200, Erik Rijkers wrote: > On 7/21/22 08:16, Pavel Stehule wrote: > > Hi > > > > new update of session variable;s implementation > > > > - fresh rebase > > - new possibility to trace execution with DEBUG1 notification > > - new SRF function pg_debug_show_used_session_variables that returns > > content of sessionvars hashtab > > - redesign of work with list of variables for reset, recheck, on commit > > drop, on commit reset Thanks for working on those! I will keep reviewing the patchset. > I don't know exactly what failed but the docs (html/pdf) don't build: > > cd ~/pg_stuff/pg_sandbox/pgsql.schema_variables/doc/src/sgml > > $ make html > /usr/bin/xmllint --path . --noout --valid postgres.sgml > postgres.sgml:374: element link: validity error : IDREF attribute linkend > references an unknown ID "catalog-pg-variable" > make: *** [Makefile:135: html-stamp] Error 4 Apparently most of the changes in catalogs.sgml didn't survive the last rebase. I do see the needed section in v20220709-0012-documentation.patch: > + <sect1 id="catalog-pg-variable"> > + <title><structname>pg_variable</structname></title> > [...]
Hi,
On Thu, Jul 21, 2022 at 09:09:47AM +0200, Erik Rijkers wrote:
> On 7/21/22 08:16, Pavel Stehule wrote:
> > Hi
> >
> > new update of session variable;s implementation
> >
> > - fresh rebase
> > - new possibility to trace execution with DEBUG1 notification
> > - new SRF function pg_debug_show_used_session_variables that returns
> > content of sessionvars hashtab
> > - redesign of work with list of variables for reset, recheck, on commit
> > drop, on commit reset
Thanks for working on those! I will keep reviewing the patchset.
> I don't know exactly what failed but the docs (html/pdf) don't build:
>
> cd ~/pg_stuff/pg_sandbox/pgsql.schema_variables/doc/src/sgml
>
> $ make html
> /usr/bin/xmllint --path . --noout --valid postgres.sgml
> postgres.sgml:374: element link: validity error : IDREF attribute linkend
> references an unknown ID "catalog-pg-variable"
> make: *** [Makefile:135: html-stamp] Error 4
Apparently most of the changes in catalogs.sgml didn't survive the last rebase.
I do see the needed section in v20220709-0012-documentation.patch:
> + <sect1 id="catalog-pg-variable">
> + <title><structname>pg_variable</structname></title>
> [...]
Вложения
- v20220722-0013-documentation.patch
- v20220722-0012-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220722-0009-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220722-0010-typedefs.patch
- v20220722-0011-Regress-tests-for-session-variables.patch
- v20220722-0008-Enhancing-psql-for-session-variables.patch
- v20220722-0007-DISCARD-VARIABLES-command.patch
- v20220722-0006-Support-of-LET-command-in-PLpgSQL.patch
- v20220722-0004-debug-func.patch
- v20220722-0005-LET-command.patch
- v20220722-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220722-0002-session-variables.patch
- v20220722-0001-Catalogue-support-for-session-variables.patch
Hi, On Fri, Jul 22, 2022 at 10:58:25AM +0200, Pavel Stehule wrote: > > Apparently most of the changes in catalogs.sgml didn't survive the last > > rebase. > > I do see the needed section in v20220709-0012-documentation.patch: > > > > > + <sect1 id="catalog-pg-variable"> > > > + <title><structname>pg_variable</structname></title> > > > [...] > > > > should be fixed now Thanks! I confirm that the documentation compiles now. As mentioned off-list, I still think that the main comment in sessionvariable.c needs to be adapted to the new approach. At the very least it still refers to the previous 2 lists, but as far as I can see there are now 4 lists: + /* Both lists hold fields of SVariableXActActionItem type */ + static List *xact_on_commit_drop_actions = NIL; + static List *xact_on_commit_reset_actions = NIL; + + /* + * the ON COMMIT DROP and ON TRANSACTION END RESET variables + * are purged from memory every time. + */ + static List *xact_reset_varids = NIL; + + /* + * Holds list variable's id that that should be + * checked against system catalog if still live. + */ + static List *xact_recheck_varids = NIL; Apart from that, I'm not sure how much of the previous behavior changed. It would be easier to review the new patchset having some up to date general description of the approach. If that's overall the same, just implemented slightly differently I will just go ahead and dig into the patchset (although the comments will still have to be changed eventually). Also, one of the things that changes since the last version is: @@ -1980,15 +1975,13 @@ AtEOSubXact_SessionVariable_on_xact_actions(bool isCommit, SubTransactionId mySu */ foreach(cur_item, xact_on_commit_reset_actions) { SVariableXActActionItem *xact_ai = (SVariableXActActionItem *) lfirst(cur_item); - if (!isCommit && - xact_ai->creating_subid == mySubid && - xact_ai->action == SVAR_ON_COMMIT_DROP) + if (!isCommit && xact_ai->creating_subid == mySubid) We previously discussed this off-line, but for some quick context the test was buggy as it wasn't possible to have an SVAR_ON_COMMIT_DROP action in the xact_on_commit_reset_actions list. However I don't see any change in the regression tests since the last version and the tests are all green in both versions. It means that was fixed but there's no test covering it. The local memory management is probably the hardest part of this patchset, so I'm a bit worried if there's nothing that can catch a bug leading to leaked values or entries in some processing list. Do you think it's possible to add some test that would have caught the previous bug?
On 7/22/22 10:58, Pavel Stehule wrote: > > čt 21. 7. 2022 v 9:34 odesílatel Julien Rouhaud <rjuju123@gmail.com > <mailto:rjuju123@gmail.com>> napsal: > > [v20220722] patches Hi Pavel, Thanks, docs now build. Attached a few small text-changes. Also, the pg_restore-doc still has the old name 'schema_variable' instead of session_variable: -A schema_variable --variable=schema_variable Surely those should be changed as well. Erik Rijkers
Вложения
Hi Erik, On Sun, Jul 24, 2022 at 03:39:32PM +0200, Erik Rijkers wrote: > Attached a few small text-changes. When you send patches like this, could you rename them to something other than *.patch and *.diff ? Otherwise, cfbot tries to apply *only* your patches to master, which fails due to missing the original patches that your changes are on top of, and makes it look like the author's patch needs to be rebased. http://cfbot.cputube.org/pavel-stehule.html - Apply patches: FAILED Alternately, (especially if your patch fixes a bug), you can resend the author's patches, rebased, as [1.patch, ..., N.patch] plus your changes as N+1.patch. Then, cfbot tests your patches, and the author can easily review and then integrate your changes. (This is especially nice if the patches currently need to be rebased, and you can make the cfbot pass at the same time as sending fixes). Cheers, -- Justin
Hi,
On Fri, Jul 22, 2022 at 10:58:25AM +0200, Pavel Stehule wrote:
> > Apparently most of the changes in catalogs.sgml didn't survive the last
> > rebase.
> > I do see the needed section in v20220709-0012-documentation.patch:
> >
> > > + <sect1 id="catalog-pg-variable">
> > > + <title><structname>pg_variable</structname></title>
> > > [...]
> >
>
> should be fixed now
Thanks! I confirm that the documentation compiles now.
As mentioned off-list, I still think that the main comment in sessionvariable.c
needs to be adapted to the new approach. At the very least it still refers to
the previous 2 lists, but as far as I can see there are now 4 lists:
+ /* Both lists hold fields of SVariableXActActionItem type */
+ static List *xact_on_commit_drop_actions = NIL;
+ static List *xact_on_commit_reset_actions = NIL;
+
+ /*
+ * the ON COMMIT DROP and ON TRANSACTION END RESET variables
+ * are purged from memory every time.
+ */
+ static List *xact_reset_varids = NIL;
+
+ /*
+ * Holds list variable's id that that should be
+ * checked against system catalog if still live.
+ */
+ static List *xact_recheck_varids = NIL;
Apart from that, I'm not sure how much of the previous behavior changed.
It would be easier to review the new patchset having some up to date general
description of the approach. If that's overall the same, just implemented
slightly differently I will just go ahead and dig into the patchset (although
the comments will still have to be changed eventually).
Also, one of the things that changes since the last version is:
@@ -1980,15 +1975,13 @@ AtEOSubXact_SessionVariable_on_xact_actions(bool isCommit, SubTransactionId mySu
*/
foreach(cur_item, xact_on_commit_reset_actions)
{
SVariableXActActionItem *xact_ai =
(SVariableXActActionItem *) lfirst(cur_item);
- if (!isCommit &&
- xact_ai->creating_subid == mySubid &&
- xact_ai->action == SVAR_ON_COMMIT_DROP)
+ if (!isCommit && xact_ai->creating_subid == mySubid)
We previously discussed this off-line, but for some quick context the test was
buggy as it wasn't possible to have an SVAR_ON_COMMIT_DROP action in the
xact_on_commit_reset_actions list. However I don't see any change in the
regression tests since the last version and the tests are all green in both
versions.
It means that was fixed but there's no test covering it. The local memory
management is probably the hardest part of this patchset, so I'm a bit worried
if there's nothing that can catch a bug leading to leaked values or entries in
some processing list. Do you think it's possible to add some test that would
have caught the previous bug?
Вложения
- v20220727-0012-documentation.patch
- v20220727-0009-typedefs.patch
- v20220727-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220727-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220727-0010-Regress-tests-for-session-variables.patch
- v20220727-0006-DISCARD-VARIABLES-command.patch
- v20220727-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220727-0007-Enhancing-psql-for-session-variables.patch
- v20220727-0004-LET-command.patch
- v20220727-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220727-0002-session-variables.patch
- v20220727-0001-Catalogue-support-for-session-variables.patch
Hi, On Wed, Jul 27, 2022 at 09:59:18PM +0200, Pavel Stehule wrote: > > ne 24. 7. 2022 v 13:12 odesílatel Julien Rouhaud <rjuju123@gmail.com> > napsal: > > > Hi, > > > > On Fri, Jul 22, 2022 at 10:58:25AM +0200, Pavel Stehule wrote: > > > > Apparently most of the changes in catalogs.sgml didn't survive the last > > > > rebase. > > > > I do see the needed section in v20220709-0012-documentation.patch: > > > > > > > > > + <sect1 id="catalog-pg-variable"> > > > > > + <title><structname>pg_variable</structname></title> > > > > > [...] > > > > > > > > > > should be fixed now > > > > Thanks! I confirm that the documentation compiles now. > > > > As mentioned off-list, I still think that the main comment in > > sessionvariable.c > > needs to be adapted to the new approach. At the very least it still > > refers to > > the previous 2 lists, but as far as I can see there are now 4 lists: > > > > + /* Both lists hold fields of SVariableXActActionItem type */ > > + static List *xact_on_commit_drop_actions = NIL; > > + static List *xact_on_commit_reset_actions = NIL; > > + > > + /* > > + * the ON COMMIT DROP and ON TRANSACTION END RESET variables > > + * are purged from memory every time. > > + */ > > + static List *xact_reset_varids = NIL; > > + > > + /* > > + * Holds list variable's id that that should be > > + * checked against system catalog if still live. > > + */ > > + static List *xact_recheck_varids = NIL; > > > > Apart from that, I'm not sure how much of the previous behavior changed. > > > > It would be easier to review the new patchset having some up to date > > general > > description of the approach. If that's overall the same, just implemented > > slightly differently I will just go ahead and dig into the patchset > > (although > > the comments will still have to be changed eventually). > > > > Also, one of the things that changes since the last version is: > > > > @@ -1980,15 +1975,13 @@ AtEOSubXact_SessionVariable_on_xact_actions(bool > > isCommit, SubTransactionId mySu > > */ > > foreach(cur_item, xact_on_commit_reset_actions) > > { > > SVariableXActActionItem *xact_ai = > > (SVariableXActActionItem *) > > lfirst(cur_item); > > > > - if (!isCommit && > > - xact_ai->creating_subid == mySubid && > > - xact_ai->action == SVAR_ON_COMMIT_DROP) > > + if (!isCommit && xact_ai->creating_subid == mySubid) > > > > We previously discussed this off-line, but for some quick context the test > > was > > buggy as it wasn't possible to have an SVAR_ON_COMMIT_DROP action in the > > xact_on_commit_reset_actions list. However I don't see any change in the > > regression tests since the last version and the tests are all green in both > > versions. > > > > It means that was fixed but there's no test covering it. The local memory > > management is probably the hardest part of this patchset, so I'm a bit > > worried > > if there's nothing that can catch a bug leading to leaked values or > > entries in > > some processing list. Do you think it's possible to add some test that > > would > > have caught the previous bug? > > > > I am sending an updated patch. I had to modify sinval message handling. > Previous implementation was not robust and correct (there was some > possibility, so value stored in session's variable was lost after aborted > drop variable. There are new regress tests requested by Julien and some > others describing the mentioned issue. I rewrote the implementation's > description part in sessionvariable.c. Thanks a lot, that's very helpful! I looked at the new description and I'm not sure that I understand the need for the "format change" code that tries to detect whether the underlying types was modified. It seems quite fragile, wouldn't it be better to have the same behavior as for relation (detect and prevent such changes in the first place), since both cases share the same requirements about underlying data types? For instance, it should be totally acceptable to drop an attribute from a custom data type if a session variable is using it, same as if a table is using it but as is it would be rejected for session variables. While at it, the new comments contain a lot of non breakable spaces rather than normal spaces. I also just realized that there's a sessionvariable.c while the header is named session_variable.h.
Hi,
On Wed, Jul 27, 2022 at 09:59:18PM +0200, Pavel Stehule wrote:
>
> ne 24. 7. 2022 v 13:12 odesílatel Julien Rouhaud <rjuju123@gmail.com>
> napsal:
>
> > Hi,
> >
> > On Fri, Jul 22, 2022 at 10:58:25AM +0200, Pavel Stehule wrote:
> > > > Apparently most of the changes in catalogs.sgml didn't survive the last
> > > > rebase.
> > > > I do see the needed section in v20220709-0012-documentation.patch:
> > > >
> > > > > + <sect1 id="catalog-pg-variable">
> > > > > + <title><structname>pg_variable</structname></title>
> > > > > [...]
> > > >
> > >
> > > should be fixed now
> >
> > Thanks! I confirm that the documentation compiles now.
> >
> > As mentioned off-list, I still think that the main comment in
> > sessionvariable.c
> > needs to be adapted to the new approach. At the very least it still
> > refers to
> > the previous 2 lists, but as far as I can see there are now 4 lists:
> >
> > + /* Both lists hold fields of SVariableXActActionItem type */
> > + static List *xact_on_commit_drop_actions = NIL;
> > + static List *xact_on_commit_reset_actions = NIL;
> > +
> > + /*
> > + * the ON COMMIT DROP and ON TRANSACTION END RESET variables
> > + * are purged from memory every time.
> > + */
> > + static List *xact_reset_varids = NIL;
> > +
> > + /*
> > + * Holds list variable's id that that should be
> > + * checked against system catalog if still live.
> > + */
> > + static List *xact_recheck_varids = NIL;
> >
> > Apart from that, I'm not sure how much of the previous behavior changed.
> >
> > It would be easier to review the new patchset having some up to date
> > general
> > description of the approach. If that's overall the same, just implemented
> > slightly differently I will just go ahead and dig into the patchset
> > (although
> > the comments will still have to be changed eventually).
> >
> > Also, one of the things that changes since the last version is:
> >
> > @@ -1980,15 +1975,13 @@ AtEOSubXact_SessionVariable_on_xact_actions(bool
> > isCommit, SubTransactionId mySu
> > */
> > foreach(cur_item, xact_on_commit_reset_actions)
> > {
> > SVariableXActActionItem *xact_ai =
> > (SVariableXActActionItem *)
> > lfirst(cur_item);
> >
> > - if (!isCommit &&
> > - xact_ai->creating_subid == mySubid &&
> > - xact_ai->action == SVAR_ON_COMMIT_DROP)
> > + if (!isCommit && xact_ai->creating_subid == mySubid)
> >
> > We previously discussed this off-line, but for some quick context the test
> > was
> > buggy as it wasn't possible to have an SVAR_ON_COMMIT_DROP action in the
> > xact_on_commit_reset_actions list. However I don't see any change in the
> > regression tests since the last version and the tests are all green in both
> > versions.
> >
> > It means that was fixed but there's no test covering it. The local memory
> > management is probably the hardest part of this patchset, so I'm a bit
> > worried
> > if there's nothing that can catch a bug leading to leaked values or
> > entries in
> > some processing list. Do you think it's possible to add some test that
> > would
> > have caught the previous bug?
> >
>
> I am sending an updated patch. I had to modify sinval message handling.
> Previous implementation was not robust and correct (there was some
> possibility, so value stored in session's variable was lost after aborted
> drop variable. There are new regress tests requested by Julien and some
> others describing the mentioned issue. I rewrote the implementation's
> description part in sessionvariable.c.
Thanks a lot, that's very helpful!
I looked at the new description and I'm not sure that I understand the need for
the "format change" code that tries to detect whether the underlying types was
modified. It seems quite fragile, wouldn't it be better to have the same
behavior as for relation (detect and prevent such changes in the first place),
since both cases share the same requirements about underlying data types? For
instance, it should be totally acceptable to drop an attribute from a custom
data type if a session variable is using it, same as if a table is using it but
as is it would be rejected for session variables.
While at it, the new comments contain a lot of non breakable spaces rather than
normal spaces. I also just realized that there's a sessionvariable.c while the
header is named session_variable.h.
Вложения
- v20220801-0002-session-variables.patch
- v20220801-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220801-0012-documentation.patch
- v20220801-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220801-0009-typedefs.patch
- v20220801-0010-Regress-tests-for-session-variables.patch
- v20220801-0006-DISCARD-VARIABLES-command.patch
- v20220801-0007-Enhancing-psql-for-session-variables.patch
- v20220801-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220801-0004-LET-command.patch
- v20220801-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220801-0001-Catalogue-support-for-session-variables.patch
Вложения
- v20220819-0013-documentation.patch
- v20220819-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220819-0009-typedefs.patch
- v20220819-0007-Enhancing-psql-for-session-variables.patch
- v20220819-0010-Regress-tests-for-session-variables.patch
- v20220819-0006-DISCARD-VARIABLES-command.patch
- v20220819-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220819-0004-LET-command.patch
- v20220819-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220819-0002-session-variables.patch
- v20220819-0001-Catalogue-support-for-session-variables.patch
- v20220819-0012-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220819-0011-fix.patch
HiI am sending fresh update- enhanced work with composite types - now the used composite type can be enhanced, reduced and stored value is converted to expected format- enhancing find_composite_type_dependencies to support session variables, so the type of any field of used composite type cannot be changed
RegardsPavel
Вложения
- v20220819-2-0013-documentation.patch
- v20220819-2-0011-fix.patch
- v20220819-2-0012-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220819-2-0010-Regress-tests-for-session-variables.patch
- v20220819-2-0009-typedefs.patch
- v20220819-2-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220819-2-0006-DISCARD-VARIABLES-command.patch
- v20220819-2-0007-Enhancing-psql-for-session-variables.patch
- v20220819-2-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220819-2-0004-LET-command.patch
- v20220819-2-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220819-2-0002-session-variables.patch
- v20220819-2-0001-Catalogue-support-for-session-variables.patch
> diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c > index f6b740df0a..b3bee39457 100644 > --- a/src/backend/parser/parse_relation.c > +++ b/src/backend/parser/parse_relation.c > @@ -3667,8 +3667,8 @@ errorMissingColumn(ParseState *pstate, > ereport(ERROR, > (errcode(ERRCODE_UNDEFINED_COLUMN), > relname ? > - errmsg("column %s.%s does not exist", relname, colname) : > - errmsg("column \"%s\" does not exist", colname), > + errmsg("column or variable %s.%s does not exist", relname, colname) : > + errmsg("column or variable \"%s\" does not exist", colname), > state->rfirst ? closestfirst ? > errhint("Perhaps you meant to reference the column \"%s.%s\".", > state->rfirst->eref->aliasname, closestfirst) : This is in your patch 12. I wonder -- if relname is not null, then surely this is a column and not a variable, right? So only the second errmsg() here should be changed, and the first one should remain as in the original. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Op 19-08-2022 om 17:29 schreef Pavel Stehule: > pá 19. 8. 2022 v 15:57 odesílatel Pavel Stehule <pavel.stehule@gmail.com> > napsal: > >> Hi >> >> I am sending fresh update >> >> - enhanced work with composite types - now the used composite type can be >> enhanced, reduced and stored value is converted to expected format >> - enhancing find_composite_type_dependencies to support session variables, >> so the type of any field of used composite type cannot be changed >> > > update - fix cpp check v20220819-2-0001-Catalogue-support-for-session-variables.patch v20220819-2-0002-session-variables.patch v20220819-2-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch v20220819-2-0004-LET-command.patch v20220819-2-0005-Support-of-LET-command-in-PLpgSQL.patch v20220819-2-0006-DISCARD-VARIABLES-command.patch v20220819-2-0007-Enhancing-psql-for-session-variables.patch v20220819-2-0008-Possibility-to-dump-session-variables-by-pg_dump.patch v20220819-2-0009-typedefs.patch v20220819-2-0010-Regress-tests-for-session-variables.patch v20220819-2-0011-fix.patch v20220819-2-0012-This-patch-changes-error-message-column-doesn-t-exis.patch v20220819-2-0013-documentation.patch make check fails as a result of the errors in the attached session_variables.out. Erik > >> Regards >> >> Pavel >> >
Вложения
Op 20-08-2022 om 15:32 schreef Erik Rijkers: > Op 19-08-2022 om 17:29 schreef Pavel Stehule: > > make check fails as a result of the errors in the attached > session_variables.out. > Sorry, that should have been this diffs file, of course (attached). Erik
Вложения
Op 20-08-2022 om 15:32 schreef Erik Rijkers:
> Op 19-08-2022 om 17:29 schreef Pavel Stehule:
>
> make check fails as a result of the errors in the attached
> session_variables.out.
>
Sorry, that should have been this diffs file, of course (attached).
Erik
Op 20-08-2022 om 15:41 schreef Pavel Stehule: > so 20. 8. 2022 v 15:36 odesílatel Erik Rijkers <er@xs4all.nl> napsal: > >> Op 20-08-2022 om 15:32 schreef Erik Rijkers: >>> Op 19-08-2022 om 17:29 schreef Pavel Stehule: >>> >>> make check fails as a result of the errors in the attached >>> session_variables.out. >>> >> >> >> Sorry, that should have been this diffs file, of course (attached). >> > > It looks like some problem with not well initialized memory, but I have no > idea how it is possible. What are your configure options? > I compiled both assert-enable and 'normal', and I only just noticed that the assert-enable one did pass tests normally. Below is the config that produced the failing tests: ./configure --prefix=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables --bindir=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables/bin.fast --libdir=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables/lib.fast --with-pgport=6986 --quiet --enable-depend --with-openssl --with-perl --with-libxml --with-libxslt --with-zlib --enable-tap-tests --with-extra-version=_0820_schema_variables_1509 --with-lz4 --with-icu (debian 9, gcc 12.2.0) > >> >> Erik >
On Sat, Aug 20, 2022 at 03:55:07PM +0200, Erik Rijkers wrote: > > Op 20-08-2022 om 15:41 schreef Pavel Stehule: > > so 20. 8. 2022 v 15:36 odesílatel Erik Rijkers <er@xs4all.nl> napsal: > > > > > Op 20-08-2022 om 15:32 schreef Erik Rijkers: > > > > Op 19-08-2022 om 17:29 schreef Pavel Stehule: > > > > > > > > make check fails as a result of the errors in the attached > > > > session_variables.out. > > > > > > > > > > > > > Sorry, that should have been this diffs file, of course (attached). > > > > > > > It looks like some problem with not well initialized memory, but I have no > > idea how it is possible. What are your configure options? > > > > I compiled both assert-enable and 'normal', and I only just noticed that the > assert-enable one did pass tests normally. > > > Below is the config that produced the failing tests: > > ./configure > --prefix=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables --bindir=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables/bin.fast --libdir=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables/lib.fast > --with-pgport=6986 --quiet --enable-depend --with-openssl --with-perl > --with-libxml --with-libxslt --with-zlib --enable-tap-tests > --with-extra-version=_0820_schema_variables_1509 --with-lz4 --with-icu I also tried locally (didn't look at the patch yet), with debug/assert enabled, and had similar error: diff -dU10 /Users/rjuju/git/postgresql/src/test/regress/expected/session_variables.out /Users/rjuju/git/pg/pgmaster_debug/src/test/regress/results/session_variables.out --- /Users/rjuju/git/postgresql/src/test/regress/expected/session_variables.out 2022-08-20 22:25:17.000000000 +0800 +++ /Users/rjuju/git/pg/pgmaster_debug/src/test/regress/results/session_variables.out 2022-08-20 22:30:50.000000000 +0800 @@ -983,23 +983,23 @@ -- should to fail SELECT public.svar; svar --------- (10,20) (1 row) ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; -- should to fail too (different type, different generation number); SELECT public.svar; - svar ----------- - (10,20,) + svar +-------------------- + (10,20,2139062142) (1 row) LET public.svar = ROW(10,20,30); -- should be ok again for new value SELECT public.svar; svar ------------ (10,20,30) (1 row) @@ -1104,31 +1104,31 @@ (1 row) DROP VARIABLE public.svar; DROP TYPE public.svar_test_type; CREATE TYPE public.svar_test_type AS (a int, b int); CREATE VARIABLE public.svar AS public.svar_test_type; CREATE VARIABLE public.svar2 AS public.svar_test_type; LET public.svar = (10, 20); ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; SELECT public.svar; - svar ----------- - (10,20,) + svar +------------ + (10,20,16) (1 row) LET public.svar2 = (10, 20, 30); ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; SELECT public.svar; - svar -------- - (10,) + svar +--------- + (10,16) (1 row) SELECT public.svar2; svar2 --------- (10,30) (1 row)
On Sat, Aug 20, 2022 at 03:55:07PM +0200, Erik Rijkers wrote:
>
> Op 20-08-2022 om 15:41 schreef Pavel Stehule:
> > so 20. 8. 2022 v 15:36 odesílatel Erik Rijkers <er@xs4all.nl> napsal:
> >
> > > Op 20-08-2022 om 15:32 schreef Erik Rijkers:
> > > > Op 19-08-2022 om 17:29 schreef Pavel Stehule:
> > > >
> > > > make check fails as a result of the errors in the attached
> > > > session_variables.out.
> > > >
> > >
> > >
> > > Sorry, that should have been this diffs file, of course (attached).
> > >
> >
> > It looks like some problem with not well initialized memory, but I have no
> > idea how it is possible. What are your configure options?
> >
>
> I compiled both assert-enable and 'normal', and I only just noticed that the
> assert-enable one did pass tests normally.
>
>
> Below is the config that produced the failing tests:
>
> ./configure
> --prefix=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables --bindir=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables/bin.fast --libdir=/home/aardvark/pg_stuff/pg_installations/pgsql.schema_variables/lib.fast
> --with-pgport=6986 --quiet --enable-depend --with-openssl --with-perl
> --with-libxml --with-libxslt --with-zlib --enable-tap-tests
> --with-extra-version=_0820_schema_variables_1509 --with-lz4 --with-icu
I also tried locally (didn't look at the patch yet), with debug/assert enabled,
and had similar error:
diff -dU10 /Users/rjuju/git/postgresql/src/test/regress/expected/session_variables.out /Users/rjuju/git/pg/pgmaster_debug/src/test/regress/results/session_variables.out
--- /Users/rjuju/git/postgresql/src/test/regress/expected/session_variables.out 2022-08-20 22:25:17.000000000 +0800
+++ /Users/rjuju/git/pg/pgmaster_debug/src/test/regress/results/session_variables.out 2022-08-20 22:30:50.000000000 +0800
@@ -983,23 +983,23 @@
-- should to fail
SELECT public.svar;
svar
---------
(10,20)
(1 row)
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
-- should to fail too (different type, different generation number);
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+--------------------
+ (10,20,2139062142)
(1 row)
LET public.svar = ROW(10,20,30);
-- should be ok again for new value
SELECT public.svar;
svar
------------
(10,20,30)
(1 row)
@@ -1104,31 +1104,31 @@
(1 row)
DROP VARIABLE public.svar;
DROP TYPE public.svar_test_type;
CREATE TYPE public.svar_test_type AS (a int, b int);
CREATE VARIABLE public.svar AS public.svar_test_type;
CREATE VARIABLE public.svar2 AS public.svar_test_type;
LET public.svar = (10, 20);
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,16)
(1 row)
LET public.svar2 = (10, 20, 30);
ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
SELECT public.svar;
- svar
--------
- (10,)
+ svar
+---------
+ (10,16)
(1 row)
SELECT public.svar2;
svar2
---------
(10,30)
(1 row)
Вложения
- v20220820-1-0012-documentation.patch
- v20220820-1-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220820-1-0010-Regress-tests-for-session-variables.patch
- v20220820-1-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220820-1-0009-typedefs.patch
- v20220820-1-0007-Enhancing-psql-for-session-variables.patch
- v20220820-1-0006-DISCARD-VARIABLES-command.patch
- v20220820-1-0004-LET-command.patch
- v20220820-1-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220820-1-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220820-1-0002-session-variables.patch
- v20220820-1-0001-Catalogue-support-for-session-variables.patch
> diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
> index f6b740df0a..b3bee39457 100644
> --- a/src/backend/parser/parse_relation.c
> +++ b/src/backend/parser/parse_relation.c
> @@ -3667,8 +3667,8 @@ errorMissingColumn(ParseState *pstate,
> ereport(ERROR,
> (errcode(ERRCODE_UNDEFINED_COLUMN),
> relname ?
> - errmsg("column %s.%s does not exist", relname, colname) :
> - errmsg("column \"%s\" does not exist", colname),
> + errmsg("column or variable %s.%s does not exist", relname, colname) :
> + errmsg("column or variable \"%s\" does not exist", colname),
> state->rfirst ? closestfirst ?
> errhint("Perhaps you meant to reference the column \"%s.%s\".",
> state->rfirst->eref->aliasname, closestfirst) :
This is in your patch 12. I wonder -- if relname is not null, then
surely this is a column and not a variable, right? So only the second
errmsg() here should be changed, and the first one should remain as in
the original.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Op 20-08-2022 om 20:09 schreef Pavel Stehule: > Hi > >> LET public.svar2 = (10, 20, 30); >> ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; >> SELECT public.svar; >> - svar >> -------- >> - (10,) >> + svar >> +--------- >> + (10,16) >> (1 row) >> >> SELECT public.svar2; >> svar2 >> --------- >> (10,30) >> (1 row) >> > > I hope so I found this error. It should be fixed > > [patches v20220820-1-0001 -> 0012] I'm afraid it still gives the same errors during 'make check', and again only errors when compiling without --enable-cassert Thanks, Erik > Regards > > Pavel >
Вложения
On Sat, Aug 20, 2022 at 08:44:49PM +0200, Erik Rijkers wrote: > Op 20-08-2022 om 20:09 schreef Pavel Stehule: > > Hi > > > > > LET public.svar2 = (10, 20, 30); > > > ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; > > > SELECT public.svar; > > > - svar > > > -------- > > > - (10,) > > > + svar > > > +--------- > > > + (10,16) > > > (1 row) > > > > > > SELECT public.svar2; > > > svar2 > > > --------- > > > (10,30) > > > (1 row) > > > > > > > I hope so I found this error. It should be fixed > > > [patches v20220820-1-0001 -> 0012] > > > I'm afraid it still gives the same errors during 'make check', and again > only errors when compiling without --enable-cassert It still fails for me for both --enable-cassert and --disable-cassert, with a different number of errors though. The cfbot is green, but it's unclear to me which version was applied on the last run. AFAICS there's no log available for the branch creation if it succeeds. --enable-cassert: LET public.svar = (10, 20); ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; SELECT public.svar; - svar ----------- - (10,20,) + svar +------------ + (10,20,16) (1 row) LET public.svar2 = (10, 20, 30); ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; SELECT public.svar; - svar -------- - (10,) + svar +--------- + (10,16) (1 row) --disable-cassert: ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; -- should to fail too (different type, different generation number); SELECT public.svar; - svar ----------- - (10,20,) + svar +------------ + (10,20,32) (1 row) LET public.svar = ROW(10,20,30); -- should be ok again for new value SELECT public.svar; svar ------------ (10,20,30) (1 row) @@ -1104,31 +1104,31 @@ (1 row) DROP VARIABLE public.svar; DROP TYPE public.svar_test_type; CREATE TYPE public.svar_test_type AS (a int, b int); CREATE VARIABLE public.svar AS public.svar_test_type; CREATE VARIABLE public.svar2 AS public.svar_test_type; LET public.svar = (10, 20); ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; SELECT public.svar; - svar ----------- - (10,20,) + svar +------------ + (10,20,16) (1 row) LET public.svar2 = (10, 20, 30); ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; SELECT public.svar; - svar -------- - (10,) + svar +--------- + (10,16) (1 row)
On Sat, Aug 20, 2022 at 08:44:49PM +0200, Erik Rijkers wrote:
> Op 20-08-2022 om 20:09 schreef Pavel Stehule:
> > Hi
> >
> > > LET public.svar2 = (10, 20, 30);
> > > ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
> > > SELECT public.svar;
> > > - svar
> > > --------
> > > - (10,)
> > > + svar
> > > +---------
> > > + (10,16)
> > > (1 row)
> > >
> > > SELECT public.svar2;
> > > svar2
> > > ---------
> > > (10,30)
> > > (1 row)
> > >
> >
> > I hope so I found this error. It should be fixed
> > > [patches v20220820-1-0001 -> 0012]
>
>
> I'm afraid it still gives the same errors during 'make check', and again
> only errors when compiling without --enable-cassert
It still fails for me for both --enable-cassert and --disable-cassert, with a
different number of errors though.
The cfbot is green, but it's unclear to me which version was applied on the
last run. AFAICS there's no log available for the branch creation if it
succeeds.
--enable-cassert:
LET public.svar = (10, 20);
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,16)
(1 row)
LET public.svar2 = (10, 20, 30);
ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
SELECT public.svar;
- svar
--------
- (10,)
+ svar
+---------
+ (10,16)
(1 row)
--disable-cassert:
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
-- should to fail too (different type, different generation number);
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,32)
(1 row)
LET public.svar = ROW(10,20,30);
-- should be ok again for new value
SELECT public.svar;
svar
------------
(10,20,30)
(1 row)
@@ -1104,31 +1104,31 @@
(1 row)
DROP VARIABLE public.svar;
DROP TYPE public.svar_test_type;
CREATE TYPE public.svar_test_type AS (a int, b int);
CREATE VARIABLE public.svar AS public.svar_test_type;
CREATE VARIABLE public.svar2 AS public.svar_test_type;
LET public.svar = (10, 20);
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,16)
(1 row)
LET public.svar2 = (10, 20, 30);
ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
SELECT public.svar;
- svar
--------
- (10,)
+ svar
+---------
+ (10,16)
(1 row)
On Sat, Aug 20, 2022 at 08:44:49PM +0200, Erik Rijkers wrote:
> Op 20-08-2022 om 20:09 schreef Pavel Stehule:
> > Hi
> >
> > > LET public.svar2 = (10, 20, 30);
> > > ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
> > > SELECT public.svar;
> > > - svar
> > > --------
> > > - (10,)
> > > + svar
> > > +---------
> > > + (10,16)
> > > (1 row)
> > >
> > > SELECT public.svar2;
> > > svar2
> > > ---------
> > > (10,30)
> > > (1 row)
> > >
> >
> > I hope so I found this error. It should be fixed
> > > [patches v20220820-1-0001 -> 0012]
>
>
> I'm afraid it still gives the same errors during 'make check', and again
> only errors when compiling without --enable-cassert
It still fails for me for both --enable-cassert and --disable-cassert, with a
different number of errors though.
The cfbot is green, but it's unclear to me which version was applied on the
last run. AFAICS there's no log available for the branch creation if it
succeeds.
--enable-cassert:
LET public.svar = (10, 20);
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,16)
(1 row)
LET public.svar2 = (10, 20, 30);
ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
SELECT public.svar;
- svar
--------
- (10,)
+ svar
+---------
+ (10,16)
(1 row)
--disable-cassert:
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
-- should to fail too (different type, different generation number);
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,32)
(1 row)
LET public.svar = ROW(10,20,30);
-- should be ok again for new value
SELECT public.svar;
svar
------------
(10,20,30)
(1 row)
@@ -1104,31 +1104,31 @@
(1 row)
DROP VARIABLE public.svar;
DROP TYPE public.svar_test_type;
CREATE TYPE public.svar_test_type AS (a int, b int);
CREATE VARIABLE public.svar AS public.svar_test_type;
CREATE VARIABLE public.svar2 AS public.svar_test_type;
LET public.svar = (10, 20);
ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
SELECT public.svar;
- svar
-----------
- (10,20,)
+ svar
+------------
+ (10,20,16)
(1 row)
LET public.svar2 = (10, 20, 30);
ALTER TYPE public.svar_test_type DROP ATTRIBUTE b;
SELECT public.svar;
- svar
--------
- (10,)
+ svar
+---------
+ (10,16)
(1 row)
Вложения
- v20220821-1-0012-documentation.patch
- v20220821-1-0010-Regress-tests-for-session-variables.patch
- v20220821-1-0011-This-patch-changes-error-message-column-doesn-t-exis.patch
- v20220821-1-0009-typedefs.patch
- v20220821-1-0008-Possibility-to-dump-session-variables-by-pg_dump.patch
- v20220821-1-0007-Enhancing-psql-for-session-variables.patch
- v20220821-1-0006-DISCARD-VARIABLES-command.patch
- v20220821-1-0005-Support-of-LET-command-in-PLpgSQL.patch
- v20220821-1-0004-LET-command.patch
- v20220821-1-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220821-1-0002-session-variables.patch
- v20220821-1-0001-Catalogue-support-for-session-variables.patch
Op 21-08-2022 om 09:54 schreef Pavel Stehule: > ne 21. 8. 2022 v 6:36 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > >> On Sat, Aug 20, 2022 at 08:44:49PM +0200, Erik Rijkers wrote: >>> Op 20-08-2022 om 20:09 schreef Pavel Stehule: > > should be fixed now> Yep, all tests OK now. Thanks! Erik
Hi Pavel, On Sun, Aug 21, 2022 at 09:54:03AM +0200, Pavel Stehule wrote: > > should be fixed now I started reviewing the patchset, beginning with 0001 (at least the parts that don't substantially change later) and have a few comments. - you define new AclMode READ and WRITE. Those bits are precious and I don't think it's ok to consume 2 bits for session variables, especially since those are the last two bits available since the recent GUC access control patch (ACL_SET and ACL_ALTER_SYSTEM). Maybe we could existing INSERT and UPDATE privileges instead, like it's done for sequences? - make check and make-check-world don't pass with this test only. Given that the split is mostly done to ease review and probably not intended to be committed this way, we probably shouldn't spend efforts to clean up the split apart from making sure that each patch compiles cleanly on its own. But in this case it brought my attention to misc_sanity.sql test. Looking at patch 0010, I see: diff --git a/src/test/regress/expected/misc_sanity.out b/src/test/regress/expected/misc_sanity.out index a57fd142a9..ce9bad7211 100644 --- a/src/test/regress/expected/misc_sanity.out +++ b/src/test/regress/expected/misc_sanity.out @@ -60,7 +60,9 @@ ORDER BY 1, 2; pg_index | indpred | pg_node_tree pg_largeobject | data | bytea pg_largeobject_metadata | lomacl | aclitem[] -(11 rows) + pg_variable | varacl | aclitem[] + pg_variable | vardefexpr | pg_node_tree +(13 rows) This is the test for relations with varlena columns without TOAST table. I don't think that's correct to add those exceptions, and there should be a TOAST table declared for pg_variable too, as noted in the comment above that query. - nitpicking: s/catalogue/catalog/ Some other comments on other patches while testing things around: - For sessionvariable.c (in 0002), I see that there are still all the comments and code about checking type validity based on a generation number and other heuristics. I still fail to understand why this is needed at all as the stored datum should remain compatible as long as we prevent the few incompatible DDL that are also prevented when there's a relation dependency. As an example, I try to quickly disable all that code with the following: diff --git a/src/backend/commands/sessionvariable.c b/src/backend/commands/sessionvariable.c index 9b4f9482a4..7c8808dc46 100644 --- a/src/backend/commands/sessionvariable.c +++ b/src/backend/commands/sessionvariable.c @@ -794,6 +794,8 @@ svartype_verify_composite_fast(SVariableType svt) static int64 get_svariable_valid_type_gennum(SVariableType svt) { + return 1; + HeapTuple tuple; bool fast_check = true; @@ -905,6 +907,8 @@ get_svariabletype(Oid typid) static bool session_variable_use_valid_type(SVariable svar) { + return true; + Assert(svar); Assert(svar->svartype); And session_variable.sql regression test still works just fine. Am I missing something? While at it, the initial comment should probably say "free local memory" rather than "purge memory". - doc are missing for GRANT/REVOKE ... ON ALL VARIABLES - plpgsql.sgml: + <sect3> + <title><command>Session variables and constants</command></title> I don't think it's ok to use "constant" as an alias for immutable session variable as immutable session variable can actually be changed. Similarly, in catalogs.sgml: + <structfield>varisimmutable</structfield> <type>boolean</type> + </para> + <para> + True if the variable is immutable (cannot be modified). The default value is false. + </para></entry> + </row> I think there should be a note and a link to the corresponding part in create_variable.sgml to explain what exactly is an immutable variable, as the implemented behavior (for nullable immutable variable) is somewhat unexpected. - other nitpicking: pg_variable and struct Variable seems a bit inconsistent. For instance one uses vartype and vartypmod and the other typid and typmod, while both use varname and varnamespace. I think we should avoid discrepancy here. Also, there's a sessionvariable.c and a session_variable.h. Let's use session_variable.[ch], as it seems more readable? -typedef patch: missing SVariableTypeData, some commits need a pgindent, e.g: +typedef SVariableTypeData * SVariableType; +typedef SVariableData * SVariable; +static SessionVariableValue * RestoreSessionVariables(char **start_address, + int *num_session_variables); +static Query *transformLetStmt(ParseState *pstate, + LetStmt * stmt); (and multiple others)
> +-- test on query with workers > +CREATE TABLE svar_test(a int); > +INSERT INTO svar_test SELECT * FROM generate_series(1,1000000); When I looked at this, I noticed this huge table. I don't think you should create such a large table just for this. To exercise parallel workers with a smaller table, decrease min_parallel_table_scan_size and others as done in other regression tests. -- Justin
> +-- test on query with workers
> +CREATE TABLE svar_test(a int);
> +INSERT INTO svar_test SELECT * FROM generate_series(1,1000000);
When I looked at this, I noticed this huge table.
I don't think you should create such a large table just for this.
To exercise parallel workers with a smaller table, decrease
min_parallel_table_scan_size and others as done in other regression tests.
--
Justin
Hi Pavel,
On Sun, Aug 21, 2022 at 09:54:03AM +0200, Pavel Stehule wrote:
>
> should be fixed now
I started reviewing the patchset, beginning with 0001 (at least the parts that
don't substantially change later) and have a few comments.
- you define new AclMode READ and WRITE. Those bits are precious and I don't
think it's ok to consume 2 bits for session variables, especially since those
are the last two bits available since the recent GUC access control patch
(ACL_SET and ACL_ALTER_SYSTEM). Maybe we could existing INSERT and UPDATE
privileges instead, like it's done for sequences?
On Mon, Aug 22, 2022 at 09:13:39PM +0200, Pavel Stehule wrote: > po 22. 8. 2022 v 9:33 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > > - you define new AclMode READ and WRITE. Those bits are precious and I > > don't > > think it's ok to consume 2 bits for session variables, especially since > > those > > are the last two bits available since the recent GUC access control patch > > (ACL_SET and ACL_ALTER_SYSTEM). Maybe we could existing INSERT and > > UPDATE > > privileges instead, like it's done for sequences? > > > > > I have not a strong opinion about it. AclMode is uint32 - so I think there > are still 15bites reserved. I think so UPDATE and SELECT rights can work, > but maybe it is better to use separate rights WRITE, READ to be stronger > signalized so the variable is not the relation. On other hand large objects > use ACL_UPDATE, ACL_SELECT too, and it works. So I am neutral in this > question. Has somebody here some opinion on this point? If not I'll modify > the patch like Julien proposes. Actually no, because AclMode is also used to store the grant option part. The comment before AclMode warns about it: * The present representation of AclItem limits us to 16 distinct rights, * even though AclMode is defined as uint32. See utils/acl.h.
Hi, On Tue, Jan 18, 2022 at 10:01:01PM +0100, Pavel Stehule wrote: > > pá 14. 1. 2022 v 3:44 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > On Thu, Jan 13, 2022 at 07:32:26PM +0100, Pavel Stehule wrote: > > > čt 13. 1. 2022 v 19:23 odesílatel Dean Rasheed <dean.a.rasheed@gmail.com > > > > > > > On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com> > > > > wrote: > > > > > > > > > > I like the idea of prioritizing tables over variables with warnings > > when > > > > collision is detected. It cannot break anything. And it allows to using > > > > short identifiers when there is not collision. > > > > > > > > Yeah, that seems OK, as long as it's clearly documented. I don't think > > > > a warning is necessary. > > > > What should be the behavior for a cached plan that uses a variable when a > > conflicting relation is later created? I think that it should be the same > > as a > > search_path change and the plan should be discarded. > > > > > The warning can be disabled by default, but I think it should be there. > > > This is a signal, so some in the database schema should be renamed. > > Maybe - > > > session_variables_ambiguity_warning. > > > > I agree that having a way to know that a variable has been bypassed can be > > useful. > > > > done I've been thinking a bit more about the shadowing, and one scenario we didn't discuss is something like this naive example: CREATE TABLE tt(a text, b text); CREATE TYPE abc AS (a text, b text, c text); CREATE VARIABLE tt AS abc; INSERT INTO tt SELECT 'a', 'b'; LET tt = ('x', 'y', 'z'); SELECT tt.a, tt.b, tt.c FROM tt; Which, with the default configuration, currently returns a | b | c ---+---+--- a | b | z (1 row) I feel a bit uncomfortable that the system allows mixing variable attributes and relation columns for the same relation name. This is even worse here as part of the variable attributes are shadowed. It feels like a good way to write valid queries that clearly won't do what you think they do, a bit like the correlated sub-query trap, so maybe we should have a way to prevent it. What do you think?
Hi,
On Tue, Jan 18, 2022 at 10:01:01PM +0100, Pavel Stehule wrote:
>
> pá 14. 1. 2022 v 3:44 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> > On Thu, Jan 13, 2022 at 07:32:26PM +0100, Pavel Stehule wrote:
> > > čt 13. 1. 2022 v 19:23 odesílatel Dean Rasheed <dean.a.rasheed@gmail.com
> > >
> > > > On Thu, 13 Jan 2022 at 17:42, Pavel Stehule <pavel.stehule@gmail.com>
> > > > wrote:
> > > > >
> > > > > I like the idea of prioritizing tables over variables with warnings
> > when
> > > > collision is detected. It cannot break anything. And it allows to using
> > > > short identifiers when there is not collision.
> > > >
> > > > Yeah, that seems OK, as long as it's clearly documented. I don't think
> > > > a warning is necessary.
> >
> > What should be the behavior for a cached plan that uses a variable when a
> > conflicting relation is later created? I think that it should be the same
> > as a
> > search_path change and the plan should be discarded.
> >
> > > The warning can be disabled by default, but I think it should be there.
> > > This is a signal, so some in the database schema should be renamed.
> > Maybe -
> > > session_variables_ambiguity_warning.
> >
> > I agree that having a way to know that a variable has been bypassed can be
> > useful.
> >
>
> done
I've been thinking a bit more about the shadowing, and one scenario we didn't
discuss is something like this naive example:
CREATE TABLE tt(a text, b text);
CREATE TYPE abc AS (a text, b text, c text);
CREATE VARIABLE tt AS abc;
INSERT INTO tt SELECT 'a', 'b';
LET tt = ('x', 'y', 'z');
SELECT tt.a, tt.b, tt.c FROM tt;
Which, with the default configuration, currently returns
a | b | c
---+---+---
a | b | z
(1 row)
I feel a bit uncomfortable that the system allows mixing variable attributes
and relation columns for the same relation name. This is even worse here as
part of the variable attributes are shadowed.
It feels like a good way to write valid queries that clearly won't do what you
think they do, a bit like the correlated sub-query trap, so maybe we should
have a way to prevent it.
What do you think?
ERROR: column tt.c does not exist
LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
^
(2022-08-23 11:06:03) postgres=# set session_variables_ambiguity_warning to on;
SET
(2022-08-23 11:06:19) postgres=# SELECT tt.a, tt.b, tt.c FROM tt;
WARNING: session variable "tt.a" is shadowed
LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
WARNING: session variable "tt.b" is shadowed
LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
WARNING: session variable "public.tt" is shadowed
LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
^
DETAIL: Session variables can be shadowed by tables or table's aliases with the same name.
ERROR: column tt.c does not exist
LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
^
Вложения
- v20220823-1-0012-documentation.patch
- v20220823-1-0011-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220823-1-0009-typedefs.patch
- v20220823-1-0008-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220823-1-0010-regress-tests-for-session-variables.patch
- v20220823-1-0006-DISCARD-VARIABLES-command.patch
- v20220823-1-0007-enhancing-psql-for-session-variables.patch
- v20220823-1-0005-support-of-LET-command-in-PLpgSQL.patch
- v20220823-1-0004-LET-command.patch
- v20220823-1-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220823-1-0002-session-variables.patch
- v20220823-1-0001-catalog-support-for-session-variables.patch
On Mon, Aug 22, 2022 at 09:13:39PM +0200, Pavel Stehule wrote:
> po 22. 8. 2022 v 9:33 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> >
> > - you define new AclMode READ and WRITE. Those bits are precious and I
> > don't
> > think it's ok to consume 2 bits for session variables, especially since
> > those
> > are the last two bits available since the recent GUC access control patch
> > (ACL_SET and ACL_ALTER_SYSTEM). Maybe we could existing INSERT and
> > UPDATE
> > privileges instead, like it's done for sequences?
> >
> >
> I have not a strong opinion about it. AclMode is uint32 - so I think there
> are still 15bites reserved. I think so UPDATE and SELECT rights can work,
> but maybe it is better to use separate rights WRITE, READ to be stronger
> signalized so the variable is not the relation. On other hand large objects
> use ACL_UPDATE, ACL_SELECT too, and it works. So I am neutral in this
> question. Has somebody here some opinion on this point? If not I'll modify
> the patch like Julien proposes.
Actually no, because AclMode is also used to store the grant option part. The
comment before AclMode warns about it:
* The present representation of AclItem limits us to 16 distinct rights,
* even though AclMode is defined as uint32. See utils/acl.h.
On Tue, Aug 23, 2022 at 11:27:45AM +0200, Pavel Stehule wrote: > út 23. 8. 2022 v 7:56 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > > I've been thinking a bit more about the shadowing, and one scenario we > > didn't > > discuss is something like this naive example: > > > > CREATE TABLE tt(a text, b text); > > > > CREATE TYPE abc AS (a text, b text, c text); > > CREATE VARIABLE tt AS abc; > > > > INSERT INTO tt SELECT 'a', 'b'; > > LET tt = ('x', 'y', 'z'); > > > > SELECT tt.a, tt.b, tt.c FROM tt; > > > > Which, with the default configuration, currently returns > > > > a | b | c > > ---+---+--- > > a | b | z > > (1 row) > > > > I feel a bit uncomfortable that the system allows mixing variable > > attributes > > and relation columns for the same relation name. This is even worse here > > as > > part of the variable attributes are shadowed. > > > > It feels like a good way to write valid queries that clearly won't do what > > you > > think they do, a bit like the correlated sub-query trap, so maybe we should > > have a way to prevent it. > > > > What do you think? > > > > I thought about it before. I think valid RTE (but with the missing column) > can shadow the variable too. > > With this change your query fails: > > (2022-08-23 11:05:55) postgres=# SELECT tt.a, tt.b, tt.c FROM tt; > ERROR: column tt.c does not exist > LINE 1: SELECT tt.a, tt.b, tt.c FROM tt; > ^ > (2022-08-23 11:06:03) postgres=# set session_variables_ambiguity_warning to > on; > SET > (2022-08-23 11:06:19) postgres=# SELECT tt.a, tt.b, tt.c FROM tt; > WARNING: session variable "tt.a" is shadowed > LINE 1: SELECT tt.a, tt.b, tt.c FROM tt; > ^ > DETAIL: Session variables can be shadowed by columns, routine's variables > and routine's arguments with the same name. > WARNING: session variable "tt.b" is shadowed > LINE 1: SELECT tt.a, tt.b, tt.c FROM tt; > ^ > DETAIL: Session variables can be shadowed by columns, routine's variables > and routine's arguments with the same name. > WARNING: session variable "public.tt" is shadowed > LINE 1: SELECT tt.a, tt.b, tt.c FROM tt; > ^ > DETAIL: Session variables can be shadowed by tables or table's aliases > with the same name. > ERROR: column tt.c does not exist > LINE 1: SELECT tt.a, tt.b, tt.c FROM tt; Great, thanks a lot! Could you add some regression tests for that scenario in the next version, since this is handled by some new code? It will also probably be useful to remind any possible committer about that choice.
Hi Pavel,
On Sun, Aug 21, 2022 at 09:54:03AM +0200, Pavel Stehule wrote:
>
> should be fixed now
I started reviewing the patchset, beginning with 0001 (at least the parts that
don't substantially change later) and have a few comments.
- you define new AclMode READ and WRITE. Those bits are precious and I don't
think it's ok to consume 2 bits for session variables, especially since those
are the last two bits available since the recent GUC access control patch
(ACL_SET and ACL_ALTER_SYSTEM). Maybe we could existing INSERT and UPDATE
privileges instead, like it's done for sequences?
- make check and make-check-world don't pass with this test only. Given that
the split is mostly done to ease review and probably not intended to be
committed this way, we probably shouldn't spend efforts to clean up the split
apart from making sure that each patch compiles cleanly on its own. But in
this case it brought my attention to misc_sanity.sql test. Looking at patch
0010, I see:
diff --git a/src/test/regress/expected/misc_sanity.out b/src/test/regress/expected/misc_sanity.out
index a57fd142a9..ce9bad7211 100644
--- a/src/test/regress/expected/misc_sanity.out
+++ b/src/test/regress/expected/misc_sanity.out
@@ -60,7 +60,9 @@ ORDER BY 1, 2;
pg_index | indpred | pg_node_tree
pg_largeobject | data | bytea
pg_largeobject_metadata | lomacl | aclitem[]
-(11 rows)
+ pg_variable | varacl | aclitem[]
+ pg_variable | vardefexpr | pg_node_tree
+(13 rows)
This is the test for relations with varlena columns without TOAST table. I
don't think that's correct to add those exceptions, and there should be a TOAST
table declared for pg_variable too, as noted in the comment above that query.
- nitpicking: s/catalogue/catalog/
Some other comments on other patches while testing things around:
- For sessionvariable.c (in 0002), I see that there are still all the comments
and code about checking type validity based on a generation number and other
heuristics. I still fail to understand why this is needed at all as the
stored datum should remain compatible as long as we prevent the few
incompatible DDL that are also prevented when there's a relation dependency.
As an example, I try to quickly disable all that code with the following:
diff --git a/src/backend/commands/sessionvariable.c b/src/backend/commands/sessionvariable.c
index 9b4f9482a4..7c8808dc46 100644
--- a/src/backend/commands/sessionvariable.c
+++ b/src/backend/commands/sessionvariable.c
@@ -794,6 +794,8 @@ svartype_verify_composite_fast(SVariableType svt)
static int64
get_svariable_valid_type_gennum(SVariableType svt)
{
+ return 1;
+
HeapTuple tuple;
bool fast_check = true;
@@ -905,6 +907,8 @@ get_svariabletype(Oid typid)
static bool
session_variable_use_valid_type(SVariable svar)
{
+ return true;
+
Assert(svar);
Assert(svar->svartype);
And session_variable.sql regression test still works just fine. Am I missing
something?
While at it, the initial comment should probably say "free local memory" rather
than "purge memory".
- doc are missing for GRANT/REVOKE ... ON ALL VARIABLES
- plpgsql.sgml:
+ <sect3>
+ <title><command>Session variables and constants</command></title>
I don't think it's ok to use "constant" as an alias for immutable session
variable as immutable session variable can actually be changed.
Similarly, in catalogs.sgml:
+ <structfield>varisimmutable</structfield> <type>boolean</type>
+ </para>
+ <para>
+ True if the variable is immutable (cannot be modified). The default value is false.
+ </para></entry>
+ </row>
I think there should be a note and a link to the corresponding part in
create_variable.sgml to explain what exactly is an immutable variable, as the
implemented behavior (for nullable immutable variable) is somewhat unexpected.
- other nitpicking: pg_variable and struct Variable seems a bit inconsistent.
For instance one uses vartype and vartypmod and the other typid and typmod,
while both use varname and varnamespace. I think we should avoid discrepancy
here.
Also, there's a sessionvariable.c and a session_variable.h. Let's use
session_variable.[ch], as it seems more readable?
-typedef patch: missing SVariableTypeData, some commits need a pgindent, e.g:
+typedef SVariableTypeData * SVariableType;
+typedef SVariableData * SVariable;
+static SessionVariableValue * RestoreSessionVariables(char **start_address,
+ int *num_session_variables);
+static Query *transformLetStmt(ParseState *pstate,
+ LetStmt * stmt);
(and multiple others)
Вложения
- v20220824-1-0012-documentation.patch
- v20220824-1-0011-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220824-1-0009-typedefs.patch
- v20220824-1-0008-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220824-1-0010-regress-tests-for-session-variables.patch
- v20220824-1-0006-DISCARD-VARIABLES-command.patch
- v20220824-1-0005-support-of-LET-command-in-PLpgSQL.patch
- v20220824-1-0007-enhancing-psql-for-session-variables.patch
- v20220824-1-0004-LET-command.patch
- v20220824-1-0003-typecheck-check-of-consistency-of-format-of-stored-v.patch
- v20220824-1-0002-session-variables.patch
- v20220824-1-0001-catalog-support-for-session-variables.patch
On Tue, Aug 23, 2022 at 11:27:45AM +0200, Pavel Stehule wrote:
> út 23. 8. 2022 v 7:56 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
>
> >
> > I've been thinking a bit more about the shadowing, and one scenario we
> > didn't
> > discuss is something like this naive example:
> >
> > CREATE TABLE tt(a text, b text);
> >
> > CREATE TYPE abc AS (a text, b text, c text);
> > CREATE VARIABLE tt AS abc;
> >
> > INSERT INTO tt SELECT 'a', 'b';
> > LET tt = ('x', 'y', 'z');
> >
> > SELECT tt.a, tt.b, tt.c FROM tt;
> >
> > Which, with the default configuration, currently returns
> >
> > a | b | c
> > ---+---+---
> > a | b | z
> > (1 row)
> >
> > I feel a bit uncomfortable that the system allows mixing variable
> > attributes
> > and relation columns for the same relation name. This is even worse here
> > as
> > part of the variable attributes are shadowed.
> >
> > It feels like a good way to write valid queries that clearly won't do what
> > you
> > think they do, a bit like the correlated sub-query trap, so maybe we should
> > have a way to prevent it.
> >
> > What do you think?
> >
>
> I thought about it before. I think valid RTE (but with the missing column)
> can shadow the variable too.
>
> With this change your query fails:
>
> (2022-08-23 11:05:55) postgres=# SELECT tt.a, tt.b, tt.c FROM tt;
> ERROR: column tt.c does not exist
> LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
> ^
> (2022-08-23 11:06:03) postgres=# set session_variables_ambiguity_warning to
> on;
> SET
> (2022-08-23 11:06:19) postgres=# SELECT tt.a, tt.b, tt.c FROM tt;
> WARNING: session variable "tt.a" is shadowed
> LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
> ^
> DETAIL: Session variables can be shadowed by columns, routine's variables
> and routine's arguments with the same name.
> WARNING: session variable "tt.b" is shadowed
> LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
> ^
> DETAIL: Session variables can be shadowed by columns, routine's variables
> and routine's arguments with the same name.
> WARNING: session variable "public.tt" is shadowed
> LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
> ^
> DETAIL: Session variables can be shadowed by tables or table's aliases
> with the same name.
> ERROR: column tt.c does not exist
> LINE 1: SELECT tt.a, tt.b, tt.c FROM tt;
Great, thanks a lot!
Could you add some regression tests for that scenario in the next version,
since this is handled by some new code? It will also probably be useful to
remind any possible committer about that choice.
Op 24-08-2022 om 08:37 schreef Pavel Stehule: >> > > I fixed these. > > [v20220824-1-*.patch] Hi Pavel, I noticed just now that variable assignment (i.e., LET) unexpectedly (for me anyway) cast the type of the input value. Surely that's wrong? The documentation says clearly enough: 'The result must be of the same data type as the session variable.' Example: create variable x integer; let x=1.5; select x, pg_typeof(x); x | pg_typeof ---+----------- 2 | integer (1 row) Is this correct? If such casts (there are several) are intended then the text of the documentation should be changed. Thanks, Erik
Op 24-08-2022 om 08:37 schreef Pavel Stehule:
>>
>
> I fixed these.
>
> [v20220824-1-*.patch]
Hi Pavel,
I noticed just now that variable assignment (i.e., LET) unexpectedly
(for me anyway) cast the type of the input value. Surely that's wrong?
The documentation says clearly enough:
'The result must be of the same data type as the session variable.'
Example:
create variable x integer;
let x=1.5;
select x, pg_typeof(x);
x | pg_typeof
---+-----------
2 | integer
(1 row)
Is this correct?
If such casts (there are several) are intended then the text of the
documentation should be changed.
postgres$# declare i int;
postgres$# begin
postgres$# i := 1.5;
postgres$# raise notice '%', i;
postgres$# end;
postgres$# $$;
NOTICE: 2
DO
CREATE TABLE
(2022-08-25 19:38:13) postgres=# insert into foo1 values(1.5);
INSERT 0 1
(2022-08-25 19:38:21) postgres=# select * from foo1;
┌───┐
│ a │
╞═══╡
│ 2 │
└───┘
(1 row)
Thanks,
Erik
- For sessionvariable.c (in 0002), I see that there are still all the comments
and code about checking type validity based on a generation number and other
heuristics. I still fail to understand why this is needed at all as the
stored datum should remain compatible as long as we prevent the few
incompatible DDL that are also prevented when there's a relation dependency.
As an example, I try to quickly disable all that code with the following:
Вложения
- v20220825-1-0011-documentation.patch
- v20220825-1-0008-typedefs.patch
- v20220825-1-0010-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220825-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220825-1-0009-regress-tests-for-session-variables.patch
- v20220825-1-0006-enhancing-psql-for-session-variables.patch
- v20220825-1-0005-DISCARD-VARIABLES-command.patch
- v20220825-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220825-1-0003-LET-command.patch
- v20220825-1-0002-session-variables.patch
- v20220825-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20220827-1-0010-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220827-1-0011-documentation.patch
- v20220827-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220827-1-0008-typedefs.patch
- v20220827-1-0009-regress-tests-for-session-variables.patch
- v20220827-1-0005-DISCARD-VARIABLES-command.patch
- v20220827-1-0006-enhancing-psql-for-session-variables.patch
- v20220827-1-0003-LET-command.patch
- v20220827-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220827-1-0002-session-variables.patch
- v20220827-1-0001-catalog-support-for-session-variables.patch
Hi, On Sat, Aug 27, 2022 at 01:17:45PM +0200, Pavel Stehule wrote: > > after some thinking I think that instead of sequence I can use LSN. The > combination oid, LSN should be unique forever Yeah I was about suggesting doing that instead of a sequence, so +1 for that approach! I've been spending a bit of time trying to improve the test coverage on the protection for concurrently deleted and recreated variables, and thought that a new isolation test should be enough. I'm attaching a diff (in .txt extension) that could be applied to 009-regress-tests-for-session-variables.patch, but while working on that I discovered a few problems. First, the pg_debug_show_used_session_variables() function reports what's currently locally known, but there's no guarantee that AcceptInvalidationMessages() will be called prior to its execution. For instance if you're in a transaction and already hold a lock on the function and execute it again. It therefore means that it can display that a locally cached variable isn't dropped and still holds a value, while it's not the case. While it may be surprising, I think that's still the wanted behavior as you want to know what is the cache state. FTR this is tested in the last permutation in the attached patch (although the expected output contains the up-to-date information, so you can see the failure). But if invalidation are processed when calling the function, the behavior seems surprising as far as I can see the cleanup seems to be done in 2 steps: mark t he hash entry as removed and then remove the hash entry. For instance: (conn 1) CREATE VARIABLE myvar AS text; (conn 1) LET myvar = 'something'; (conn 2) DROP VARIABLE myvar; (conn 1) SELECT schema, name, removed FROM pg_debug_show_used_session_variables(); schema | name | removed --------+-------+--------- public | myvar | t (1 row) (conn 1) SELECT schema, name, removed FROM pg_debug_show_used_session_variables(); schema | name | removed --------+------+--------- (0 rows) Why are two steps necessary here, and is that really wanted? Finally, I noticed that it's quite easy to get cache lookup failures when using transactions. AFAICS it's because the current code first checks in the local cache (which often isn't immediately invalidated when in a transaction), returns an oid (of an already dropped variable), then the code acquires a lock on that non-existent variable, which internally accepts invalidation after the lock is acquired. The rest of the code can then fail with some "cache lookup error" in the various functions as the invalidation has now been processed. This is also tested in the attached isolation test. I think that using a retry approach based on SharedInvalidMessageCounter change detection, like RangeVarGetRelidExtended(), in IdentifyVariable() should be enough to fix that class of problem, but maybe some other general functions would need similar protection too. While looking at the testing, I also noticed that the main regression tests comments are now outdated since the new (and more permissive) approach for dropped variable detection. For instance: + ALTER TYPE public.svar_test_type DROP ATTRIBUTE c; + -- should to fail + SELECT public.svar; + svar + --------- + (10,20) + (1 row) + + ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + -- should to fail too (different type, different generation number); + SELECT public.svar; + svar + ---------- + (10,20,) + (1 row) I'm also unsure if this one is showing a broken behavior or not: + CREATE VARIABLE public.avar AS int; + -- should to fail + SELECT avar FROM xxtab; + avar + ------ + 10 + (1 row) + + -- should be ok + SELECT public.avar FROM xxtab; + avar + ------ + + (1 row) For reference, with the code as-is I get the following diff when testing the attached isolation test: --- /Users/rjuju/git/postgresql/src/test/isolation/expected/session-variable.out 2022-08-29 15:41:11.000000000 +0800 +++ /Users/rjuju/git/pg/pgmaster_debug/src/test/isolation/output_iso/results/session-variable.out 2022-08-29 15:42:17.000000000+0800 @@ -16,21 +16,21 @@ step let: LET myvar = 'test'; step val: SELECT myvar; myvar ----- test (1 row) step s1: BEGIN; step drop: DROP VARIABLE myvar; step val: SELECT myvar; -ERROR: column or variable "myvar" does not exist +ERROR: cache lookup failed for session variable 16386 step sr1: ROLLBACK; starting permutation: let val dbg drop create dbg val step let: LET myvar = 'test'; step val: SELECT myvar; myvar ----- test (1 row) @@ -68,20 +68,16 @@ schema|name |removed ------+-----+------- public|myvar|f (1 row) step drop: DROP VARIABLE myvar; step create: CREATE VARIABLE myvar AS text step dbg: SELECT schema, name, removed FROM pg_debug_show_used_session_variables(); schema|name |removed ------+-----+------- -public|myvar|t +public|myvar|f (1 row) step val: SELECT myvar; -myvar ------ - -(1 row) - +ERROR: cache lookup failed for session variable 16389 step sr1: ROLLBACK;
Вложения
Op 24-08-2022 om 08:37 schreef Pavel Stehule:
>>
>
> I fixed these.
>
> [v20220824-1-*.patch]
Hi Pavel,
I noticed just now that variable assignment (i.e., LET) unexpectedly
(for me anyway) cast the type of the input value. Surely that's wrong?
The documentation says clearly enough:
'The result must be of the same data type as the session variable.'
Example:
create variable x integer;
let x=1.5;
select x, pg_typeof(x);
x | pg_typeof
---+-----------
2 | integer
(1 row)
Is this correct?
If such casts (there are several) are intended then the text of the
documentation should be changed.
<term><literal>sql_expression</literal></term>
<listitem>
<para>
- An SQL expression, in parentheses. The result must be of the same data type as the session
- variable.
+ An SQL expression (can be subquery in parenthesis). The result must
+ be of castable to the same data type as the session variable (in
+ implicit or assignment context).
</para>
</listitem>
</varlistentry>
Thanks,
Erik
Hi,
On Sat, Aug 27, 2022 at 01:17:45PM +0200, Pavel Stehule wrote:
>
> after some thinking I think that instead of sequence I can use LSN. The
> combination oid, LSN should be unique forever
Yeah I was about suggesting doing that instead of a sequence, so +1 for that
approach!
I've been spending a bit of time trying to improve the test coverage on the
protection for concurrently deleted and recreated variables, and thought that a
new isolation test should be enough. I'm attaching a diff (in .txt extension)
that could be applied to 009-regress-tests-for-session-variables.patch, but
while working on that I discovered a few problems.
First, the pg_debug_show_used_session_variables() function reports what's
currently locally known, but there's no guarantee that
AcceptInvalidationMessages() will be called prior to its execution. For
instance if you're in a transaction and already hold a lock on the function and
execute it again.
It therefore means that it can display that a locally cached variable isn't
dropped and still holds a value, while it's not the case. While it may be
surprising, I think that's still the wanted behavior as you want to know what
is the cache state. FTR this is tested in the last permutation in the attached
patch (although the expected output contains the up-to-date information, so you
can see the failure).
But if invalidation are processed when calling the function, the behavior seems
surprising as far as I can see the cleanup seems to be done in 2 steps: mark t
he hash entry as removed and then remove the hash entry. For instance:
(conn 1) CREATE VARIABLE myvar AS text;
(conn 1) LET myvar = 'something';
(conn 2) DROP VARIABLE myvar;
(conn 1) SELECT schema, name, removed FROM pg_debug_show_used_session_variables();
schema | name | removed
--------+-------+---------
public | myvar | t
(1 row)
(conn 1) SELECT schema, name, removed FROM pg_debug_show_used_session_variables();
schema | name | removed
--------+------+---------
(0 rows)
Why are two steps necessary here, and is that really wanted?
SET
(2022-08-30 19:38:55) postgres=# CREATE VARIABLE myvar AS text;
DEBUG: record for session variable "myvar" (oid:16390) was created in pg_variable
CREATE VARIABLE
(2022-08-30 19:39:03) postgres=# LET myvar = 'something';
DEBUG: session variable "public.myvar" (oid:16390) has new entry in memory (emitted by WRITE)
DEBUG: session variable "public.myvar" (oid:16390) has new value
LET
(2022-08-30 19:39:11) postgres=# SELECT schema, name, removed FROM pg_debug_show_used_session_variables();
DEBUG: pg_variable_cache_callback 84 2941368844
DEBUG: session variable "public.myvar" (oid:16390) should be rechecked (forced by sinval)
DEBUG: pg_debug_show_used_session_variables start
DEBUG: effective call of sync_sessionvars_all()
DEBUG: pg_debug_show_used_session_variables end
DEBUG: session variable "public.myvar" (oid:16390) is removing from memory
┌────────┬───────┬─────────┐
│ schema │ name │ removed │
╞════════╪═══════╪═════════╡
│ public │ myvar │ t │
└────────┴───────┴─────────┘
(1 row)
(2022-08-30 19:39:32) postgres=# SELECT schema, name, removed FROM pg_debug_show_used_session_variables();
DEBUG: pg_debug_show_used_session_variables start
DEBUG: pg_debug_show_used_session_variables end
┌────────┬──────┬─────────┐
│ schema │ name │ removed │
╞════════╪══════╪═════════╡
└────────┴──────┴─────────┘
(0 rows)
LET
(2022-08-31 06:25:59) postgres=# SELECT schema, name, removed FROM pg_debug_show_used_session_variables();
┌────────┬──────┬─────────┐
│ schema │ name │ removed │
╞════════╪══════╪═════════╡
│ public │ x │ f │
└────────┴──────┴─────────┘
(1 row)
┌────────┬──────┬─────────┐
│ schema │ name │ removed │
╞════════╪══════╪═════════╡
└────────┴──────┴─────────┘
(0 rows)
Finally, I noticed that it's quite easy to get cache lookup failures when using
transactions. AFAICS it's because the current code first checks in the local
cache (which often isn't immediately invalidated when in a transaction),
returns an oid (of an already dropped variable), then the code acquires a lock
on that non-existent variable, which internally accepts invalidation after the
lock is acquired. The rest of the code can then fail with some "cache lookup
error" in the various functions as the invalidation has now been processed.
This is also tested in the attached isolation test.
I think that using a retry approach based on SharedInvalidMessageCounter change
detection, like RangeVarGetRelidExtended(), in IdentifyVariable() should be
enough to fix that class of problem, but maybe some other general functions
would need similar protection too.
While looking at the testing, I also noticed that the main regression tests
comments are now outdated since the new (and more permissive) approach for
dropped variable detection. For instance:
+ ALTER TYPE public.svar_test_type DROP ATTRIBUTE c;
+ -- should to fail
+ SELECT public.svar;
+ svar
+ ---------
+ (10,20)
+ (1 row)
+
+ ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int;
+ -- should to fail too (different type, different generation number);
+ SELECT public.svar;
+ svar
+ ----------
+ (10,20,)
+ (1 row)
+ CREATE VARIABLE public.avar AS int;
+ -- should to fail
+ SELECT avar FROM xxtab;
+ avar
+ ------
+ 10
+ (1 row)
+
+ -- should be ok
+ SELECT public.avar FROM xxtab;
+ avar
+ ------
+
+ (1 row)
For reference, with the code as-is I get the following diff when testing the
attached isolation test:
--- /Users/rjuju/git/postgresql/src/test/isolation/expected/session-variable.out 2022-08-29 15:41:11.000000000 +0800
+++ /Users/rjuju/git/pg/pgmaster_debug/src/test/isolation/output_iso/results/session-variable.out 2022-08-29 15:42:17.000000000 +0800
@@ -16,21 +16,21 @@
step let: LET myvar = 'test';
step val: SELECT myvar;
myvar
-----
test
(1 row)
step s1: BEGIN;
step drop: DROP VARIABLE myvar;
step val: SELECT myvar;
-ERROR: column or variable "myvar" does not exist
+ERROR: cache lookup failed for session variable 16386
step sr1: ROLLBACK;
starting permutation: let val dbg drop create dbg val
step let: LET myvar = 'test';
step val: SELECT myvar;
myvar
-----
test
(1 row)
@@ -68,20 +68,16 @@
schema|name |removed
------+-----+-------
public|myvar|f
(1 row)
step drop: DROP VARIABLE myvar;
step create: CREATE VARIABLE myvar AS text
step dbg: SELECT schema, name, removed FROM pg_debug_show_used_session_variables();
schema|name |removed
------+-----+-------
-public|myvar|t
+public|myvar|f
(1 row)
step val: SELECT myvar;
-myvar
------
-
-(1 row)
-
+ERROR: cache lookup failed for session variable 16389
step sr1: ROLLBACK;
Вложения
- v20220901-1-0011-documentation.patch
- v20220901-1-0010-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220901-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220901-1-0008-typedefs.patch
- v20220901-1-0009-regress-tests-for-session-variables.patch
- v20220901-1-0005-DISCARD-VARIABLES-command.patch
- v20220901-1-0006-enhancing-psql-for-session-variables.patch
- v20220901-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220901-1-0003-LET-command.patch
- v20220901-1-0002-session-variables.patch
- v20220901-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20220902-1-0011-documentation.patch
- v20220902-1-0010-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220902-1-0008-typedefs.patch
- v20220902-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220902-1-0009-regress-tests-for-session-variables.patch
- v20220902-1-0006-enhancing-psql-for-session-variables.patch
- v20220902-1-0005-DISCARD-VARIABLES-command.patch
- v20220902-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220902-1-0003-LET-command.patch
- v20220902-1-0002-session-variables.patch
- v20220902-1-0001-catalog-support-for-session-variables.patch
Hi, On Fri, Sep 02, 2022 at 07:42:00AM +0200, Pavel Stehule wrote: > > rebased today After some off-list discussion with Pavel, I'm sending some proposal patches (in .txt extension) to apply to the last patchset. To sum up, when a session issues a DROP VARIABLE, the session will receive an sinval notification for its own drop and we don't want to process it immediately as we need to hold the value in case the transaction is rollbacked. The current patch avoided that by forcing a single processing of sinval per transaction, and forcing it before dropping the variable. It works but it seems to me that postponing all but the first VARIABLEOID sinval to the next transaction is a big hammer, and the sooner we can free some memory the better. For an alternative approach the attached patch store the lxid in the SVariable itself when dropping a currently set variable, so we can process all sinval and simply defer to the next transaction the memory cleanup of the variable(s) we know we just dropped. What do you think of that approach? As I was working on some changes I also made a pass on session_variable.c. I tried to improve a bit some comments, and also got rid of the "first_time" variable. The name wasn't really great, and AFAICS it can be replaced by testing whether the memory context has been created yet or not. But once that done I noticed the get_rowtype_value() function. I don't think this function is necessary as the core code already knows how to deal with stored datum when the underlying composite type was modified. I tried to bypass that function and always simply return the stored value and all the tests run fine. Is there really any cases when this extra code is needed? FTR I tried to do a bunch of additional testing using relation as base type for variable, as you can do more with those than plain composite types, but it still always works just fine. However, while doing so I noticed that find_composite_type_dependencies() failed to properly handle dependencies on relation (plain tables, matviews and partitioned tables). I'm also adding 2 additional patches to fix this corner case and add an additional regression test for the plain table case.
Вложения
On Sat, Sep 03, 2022 at 11:00:51PM +0800, Julien Rouhaud wrote: > Hi, > > On Fri, Sep 02, 2022 at 07:42:00AM +0200, Pavel Stehule wrote: > > > > rebased today > > After some off-list discussion with Pavel, I'm sending some proposal patches > (in .txt extension) to apply to the last patchset. > > To sum up, when a session issues a DROP VARIABLE, the session will receive an > sinval notification for its own drop and we don't want to process it > immediately as we need to hold the value in case the transaction is rollbacked. > The current patch avoided that by forcing a single processing of sinval per > transaction, and forcing it before dropping the variable. It works but it > seems to me that postponing all but the first VARIABLEOID sinval to the next > transaction is a big hammer, and the sooner we can free some memory the better. > > For an alternative approach the attached patch store the lxid in the SVariable > itself when dropping a currently set variable, so we can process all sinval and > simply defer to the next transaction the memory cleanup of the variable(s) we > know we just dropped. What do you think of that approach? > > As I was working on some changes I also made a pass on session_variable.c. I > tried to improve a bit some comments, and also got rid of the "first_time" > variable. The name wasn't really great, and AFAICS it can be replaced by > testing whether the memory context has been created yet or not. > > But once that done I noticed the get_rowtype_value() function. I don't think > this function is necessary as the core code already knows how to deal with > stored datum when the underlying composite type was modified. I tried to > bypass that function and always simply return the stored value and all the > tests run fine. Is there really any cases when this extra code is needed? > > FTR I tried to do a bunch of additional testing using relation as base type for > variable, as you can do more with those than plain composite types, but it > still always works just fine. > > However, while doing so I noticed that find_composite_type_dependencies() > failed to properly handle dependencies on relation (plain tables, matviews and > partitioned tables). I'm also adding 2 additional patches to fix this corner > case and add an additional regression test for the plain table case. I forgot to mention this chunk: + /* + * Although the value of domain type should be valid (it is + * checked when it is assigned to session variable), we have to + * check related constraints anytime. It can be more expensive + * than in PL/pgSQL. PL/pgSQL forces domain checks when value + * is assigned to the variable or when value is returned from + * function. Fortunately, domain types manage cache of constraints by + * self. + */ + if (svar->is_domain) + { + MemoryContext oldcxt = CurrentMemoryContext; + + /* + * Store domain_check extra in CurTransactionContext. When we are + * in other transaction, the domain_check_extra cache is not valid. + */ + if (svar->domain_check_extra_lxid != MyProc->lxid) + svar->domain_check_extra = NULL; + + domain_check(svar->value, svar->isnull, + svar->typid, &svar->domain_check_extra, + CurTransactionContext); + + svar->domain_check_extra_lxid = MyProc->lxid; + + MemoryContextSwitchTo(oldcxt); + } I agree that storing the domain_check_extra in the transaction context sounds sensible, but the memory context handling is not quite right. Looking at domain_check, it doesn't change the current memory context, so as-is all the code related to oldcxt is unnecessary. Some other callers like expandedrecord.c do switch to a short lived context to limit the lifetime of the possible leak by the expression evaluation, but I don't think that's an option here.
On Sat, Sep 03, 2022 at 11:00:51PM +0800, Julien Rouhaud wrote:
> Hi,
>
> On Fri, Sep 02, 2022 at 07:42:00AM +0200, Pavel Stehule wrote:
> >
> > rebased today
>
> After some off-list discussion with Pavel, I'm sending some proposal patches
> (in .txt extension) to apply to the last patchset.
>
> To sum up, when a session issues a DROP VARIABLE, the session will receive an
> sinval notification for its own drop and we don't want to process it
> immediately as we need to hold the value in case the transaction is rollbacked.
> The current patch avoided that by forcing a single processing of sinval per
> transaction, and forcing it before dropping the variable. It works but it
> seems to me that postponing all but the first VARIABLEOID sinval to the next
> transaction is a big hammer, and the sooner we can free some memory the better.
>
> For an alternative approach the attached patch store the lxid in the SVariable
> itself when dropping a currently set variable, so we can process all sinval and
> simply defer to the next transaction the memory cleanup of the variable(s) we
> know we just dropped. What do you think of that approach?
>
> As I was working on some changes I also made a pass on session_variable.c. I
> tried to improve a bit some comments, and also got rid of the "first_time"
> variable. The name wasn't really great, and AFAICS it can be replaced by
> testing whether the memory context has been created yet or not.
>
> But once that done I noticed the get_rowtype_value() function. I don't think
> this function is necessary as the core code already knows how to deal with
> stored datum when the underlying composite type was modified. I tried to
> bypass that function and always simply return the stored value and all the
> tests run fine. Is there really any cases when this extra code is needed?
>
> FTR I tried to do a bunch of additional testing using relation as base type for
> variable, as you can do more with those than plain composite types, but it
> still always works just fine.
>
> However, while doing so I noticed that find_composite_type_dependencies()
> failed to properly handle dependencies on relation (plain tables, matviews and
> partitioned tables). I'm also adding 2 additional patches to fix this corner
> case and add an additional regression test for the plain table case.
I forgot to mention this chunk:
+ /*
+ * Although the value of domain type should be valid (it is
+ * checked when it is assigned to session variable), we have to
+ * check related constraints anytime. It can be more expensive
+ * than in PL/pgSQL. PL/pgSQL forces domain checks when value
+ * is assigned to the variable or when value is returned from
+ * function. Fortunately, domain types manage cache of constraints by
+ * self.
+ */
+ if (svar->is_domain)
+ {
+ MemoryContext oldcxt = CurrentMemoryContext;
+
+ /*
+ * Store domain_check extra in CurTransactionContext. When we are
+ * in other transaction, the domain_check_extra cache is not valid.
+ */
+ if (svar->domain_check_extra_lxid != MyProc->lxid)
+ svar->domain_check_extra = NULL;
+
+ domain_check(svar->value, svar->isnull,
+ svar->typid, &svar->domain_check_extra,
+ CurTransactionContext);
+
+ svar->domain_check_extra_lxid = MyProc->lxid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
I agree that storing the domain_check_extra in the transaction context sounds
sensible, but the memory context handling is not quite right.
Looking at domain_check, it doesn't change the current memory context, so as-is
all the code related to oldcxt is unnecessary.
Some other callers like expandedrecord.c do switch to a short lived context to
limit the lifetime of the possible leak by the expression evaluation, but I
don't think that's an option here.
Вложения
- v20220904-1-0009-regress-tests-for-session-variables.patch
- v20220904-1-0008-typedefs.patch
- v20220904-1-0011-documentation.patch
- v20220904-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220904-1-0010-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220904-1-0006-enhancing-psql-for-session-variables.patch
- v20220904-1-0005-DISCARD-VARIABLES-command.patch
- v20220904-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220904-1-0003-LET-command.patch
- v20220904-1-0002-session-variables.patch
- v20220904-1-0001-catalog-support-for-session-variables.patch
On Tue, Sep 06, 2022 at 08:43:59AM +0200, Pavel Stehule wrote: > Hi > > After talking with Julian I removed "debug" fields name and nsname from > SVariable structure. When it is possible it is better to read these fields > from catalog without risk of obsoletely or necessity to refresh these > fields. In other cases we display only oid of variable instead name and > nsname (It is used just for debug purposes). Thanks! I'm just adding back the forgotten Cc list.
Вложения
- v20220906-1-0001-catalog-support-for-session-variables.patch
- v20220906-1-0002-session-variables.patch
- v20220906-1-0003-LET-command.patch
- v20220906-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220906-1-0005-DISCARD-VARIABLES-command.patch
- v20220906-1-0006-enhancing-psql-for-session-variables.patch
- v20220906-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220906-1-0008-typedefs.patch
- v20220906-1-0009-regress-tests-for-session-variables.patch
- v20220906-1-0010-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220906-1-0011-documentation.patch
Hi, On Tue, Sep 06, 2022 at 06:23:12PM +0800, Julien Rouhaud wrote: > On Tue, Sep 06, 2022 at 08:43:59AM +0200, Pavel Stehule wrote: > > Hi > > > > After talking with Julian I removed "debug" fields name and nsname from > > SVariable structure. When it is possible it is better to read these fields > > from catalog without risk of obsoletely or necessity to refresh these > > fields. In other cases we display only oid of variable instead name and > > nsname (It is used just for debug purposes). > > Thanks! I'm just adding back the forgotten Cc list. About the last change: +static void +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) +{ [...] + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) should be rechecked (forced by sinval)", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid), + svar->varid); There's no guarantee that the variable still exists in cache (for variables dropped in the current transaction), or even that the callback is called while in a transaction state, so we should only display the oid here. FTR just to be sure I ran all the new regression tests (with this fix) with CCA and log_min_messages = DEBUG1 and it didn't hit any problem, so it doesn't seem that there's any other issue hidden somewhere. Other than that I don't see any remaining problems in session_variable.c. I still have a few nitpicking comments though: +static SVariable +prepare_variable_for_reading(Oid varid) +{ [...] + /* Store result before releasing Executor memory */ + set_session_variable(svar, value, isnull, true); + + MemoryContextSwitchTo(oldcxt); + + FreeExecutorState(estate); The comment and code is a bit misleading, as it's not immediately obvious that set_session_variable() doesn't rely on the current memory contex for allocations. Simply moving the MemoryContextSwitchTo() before the set_session_variable() would be better. +typedef struct SVariableData +{ [...] + bool is_domain; + Oid basetypeid; + void *domain_check_extra; + LocalTransactionId domain_check_extra_lxid; AFAICS basetypeid isn't needed anymore. + /* Both lists hold fields of SVariableXActActionItem type */ + static List *xact_on_commit_drop_actions = NIL; + static List *xact_on_commit_reset_actions = NIL; Is it possible to merge both in a single list? I don't think that there's much to gain trying to separate those. They shouldn't contain a lot of entries, and they're usually scanned at the same time anyway. This is especially important as one of the tricky parts of this patchset is maintaining those lists across subtransactions, and since both have the same heuristics all the related code is duplicated. I see that in AtPreEOXact_SessionVariable_on_xact_actions() both lists are handled interleaved with the xact_recheck_varids, but I don't see any reason why we couldn't process both action lists first and then process the rechecks. I did a quick test and don't see any failure in the regression tests. +void +RemoveSessionVariable(Oid varid) +{ I looks like a layering violation to have (part of) the code for CREATE VARIABLE in pg_variable.[ch] and the code for DROP VARIABLE in session_variable.[ch]. I think it was done mostly because it was the initial sync_sessionvars_all() that was responsible to avoid cleaning up memory for variables dropped in the current transaction, but that's not a requirement anymore. So I don't see anything preventing us from moving RemoveSessionVariable() in pg_variable, and export some function in session_variable to do the additional work for properly maintaining the hash table if needed (with that knowledge held in session_variable, not in pg_variable). You should only need to pass the oid of the variable and the eoxaction. Simlarly, why not move DefineSessionVariable() in pg_variable and expose some API in session_variable to register the needed SVAR_ON_COMMIT_DROP action? Also, while not a problem I don't think that the CommandCounterIncrement() is necessary in DefineSessionVariable(). CREATE VARIABLE is a single operation and you can't have anything else running in the same ProcessUtility() call. And since cd3e27464cc you have the guarantee that a CommandCounterIncrement() will happen at the end of the utility command processing. While at it, maybe it would be good to add some extra tests in src/test/modules/test_extensions. I'm thinking a version 1.0 that creates a variable and initialize the value (and and extra step after creating the extension to make sure that the value is really set), and an upgrade to 2.0 that creates a temp variable on commit drop, that has to fail due to the dependecy on the extension.
Hi,
On Tue, Sep 06, 2022 at 06:23:12PM +0800, Julien Rouhaud wrote:
> On Tue, Sep 06, 2022 at 08:43:59AM +0200, Pavel Stehule wrote:
> > Hi
> >
> > After talking with Julian I removed "debug" fields name and nsname from
> > SVariable structure. When it is possible it is better to read these fields
> > from catalog without risk of obsoletely or necessity to refresh these
> > fields. In other cases we display only oid of variable instead name and
> > nsname (It is used just for debug purposes).
>
> Thanks! I'm just adding back the forgotten Cc list.
About the last change:
+static void
+pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue)
+{
[...]
+ elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) should be rechecked (forced by sinval)",
+ get_namespace_name(get_session_variable_namespace(svar->varid)),
+ get_session_variable_name(svar->varid),
+ svar->varid);
There's no guarantee that the variable still exists in cache (for variables
dropped in the current transaction), or even that the callback is called while
in a transaction state, so we should only display the oid here.
FTR just to be sure I ran all the new regression tests (with this fix) with CCA
and log_min_messages = DEBUG1 and it didn't hit any problem, so it doesn't seem
that there's any other issue hidden somewhere.
Other than that I don't see any remaining problems in session_variable.c. I
still have a few nitpicking comments though:
+static SVariable
+prepare_variable_for_reading(Oid varid)
+{
[...]
+ /* Store result before releasing Executor memory */
+ set_session_variable(svar, value, isnull, true);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ FreeExecutorState(estate);
The comment and code is a bit misleading, as it's not immediately obvious that
set_session_variable() doesn't rely on the current memory contex for
allocations. Simply moving the MemoryContextSwitchTo() before the
set_session_variable() would be better.
+typedef struct SVariableData
+{
[...]
+ bool is_domain;
+ Oid basetypeid;
+ void *domain_check_extra;
+ LocalTransactionId domain_check_extra_lxid;
AFAICS basetypeid isn't needed anymore.
+ /* Both lists hold fields of SVariableXActActionItem type */
+ static List *xact_on_commit_drop_actions = NIL;
+ static List *xact_on_commit_reset_actions = NIL;
Is it possible to merge both in a single list? I don't think that there's much
to gain trying to separate those. They shouldn't contain a lot of entries, and
they're usually scanned at the same time anyway.
This is especially important as one of the tricky parts of this patchset is
maintaining those lists across subtransactions, and since both have the same
heuristics all the related code is duplicated.
I see that in AtPreEOXact_SessionVariable_on_xact_actions() both lists are
handled interleaved with the xact_recheck_varids, but I don't see any reason
why we couldn't process both action lists first and then process the rechecks.
I did a quick test and don't see any failure in the regression tests.
<--><--><--><-->register_session_variable_xact_action(varid,
<--><--><--><--><--><--><--><--><--><--><--><--><--> SVAR_ON_COMMIT_RESET);
<--><-->}
+void
+RemoveSessionVariable(Oid varid)
+{
I looks like a layering violation to have (part of) the code for CREATE
VARIABLE in pg_variable.[ch] and the code for DROP VARIABLE in
session_variable.[ch].
I think it was done mostly because it was the initial sync_sessionvars_all()
that was responsible to avoid cleaning up memory for variables dropped in the
current transaction, but that's not a requirement anymore. So I don't see
anything preventing us from moving RemoveSessionVariable() in pg_variable, and
export some function in session_variable to do the additional work for properly
maintaining the hash table if needed (with that knowledge held in
session_variable, not in pg_variable). You should only need to pass the oid of
the variable and the eoxaction.
Simlarly, why not move DefineSessionVariable() in pg_variable and expose some
API in session_variable to register the needed SVAR_ON_COMMIT_DROP action?
Also, while not a problem I don't think that the CommandCounterIncrement() is
necessary in DefineSessionVariable(). CREATE VARIABLE is a single operation
and you can't have anything else running in the same ProcessUtility() call.
And since cd3e27464cc you have the guarantee that a CommandCounterIncrement()
will happen at the end of the utility command processing.
While at it, maybe it would be good to add some extra tests in
src/test/modules/test_extensions. I'm thinking a version 1.0 that creates a
variable and initialize the value (and and extra step after creating the
extension to make sure that the value is really set), and an upgrade to 2.0
that creates a temp variable on commit drop, that has to fail due to the
dependecy on the extension.
Вложения
- v20220910-1-0010-documentation.patch
- v20220910-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220910-1-0008-regress-tests-for-session-variables.patch
- v20220910-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220910-1-0006-enhancing-psql-for-session-variables.patch
- v20220910-1-0005-DISCARD-VARIABLES-command.patch
- v20220910-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220910-1-0003-LET-command.patch
- v20220910-1-0002-session-variables.patch
- v20220910-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20220911-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20220911-1-0010-documentation.patch
- v20220911-1-0008-regress-tests-for-session-variables.patch
- v20220911-1-0006-enhancing-psql-for-session-variables.patch
- v20220911-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20220911-1-0005-DISCARD-VARIABLES-command.patch
- v20220911-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20220911-1-0003-LET-command.patch
- v20220911-1-0002-session-variables.patch
- v20220911-1-0001-catalog-support-for-session-variables.patch
Hi, On Sun, Sep 11, 2022 at 09:29:49PM +0200, Pavel Stehule wrote: >> >> Originally it was not possible, because there was no xact_reset_varids list, and without this list the processing >> ON_COMMIT_DROP started DROP VARIABLE command, and there was a request for ON_COMMIT_RESET action. >> Now, it is possible, because in RemoveSessionVariable is conditional execution: >> >> <--><--><-->if (!svar->eox_reset) >> <--><--><--><-->register_session_variable_xact_action(varid, >> <--><--><--><--><--><--><--><--><--><--><--><--><--> SVAR_ON_COMMIT_RESET); >> <--><-->} >> >> So when we process ON_COMMIT_DROP actions, we know that the reset will not be processed by ON_COMMIT_RESET action, >> and then these lists can be merged. >> >> so I merged these two lists to one Thanks! This really helps with code readability, and after looking at it I found some issues (see below). > > changes: > > - some minor cleaning > - refactoring of RemoveSessionVariable - move part of code to pg_variable.c Thanks. I think we could still do more to split what code belongs to pg_variable.c and session_variable.c. In my opinion, the various DDL code should only invoke functions in pg_variable.c, which themselves can call function in session_variable.c if needed, and session_variable shouldn't know about CreateSessionVarStmt (which should probably be rename CreateVariableStmt?) or VariableRelationId. After an off-list bikeshedding session with Pavel, we came up with SessionVariableCreatePostprocess() and SessionVariableDropPostprocess() for the functions in session_variable.c called by pg_variable.c when handling CREATE VARIABLE and DROP VARIABLE commands. I'm attaching a new patchset with this change and some more (see below). I'm not sending .txt files as this is rebased on top on the recent GUC refactoring patch. It won't change the cfbot outcome though, as I also add new regression tests that are for now failing (see below). I tried to keep the changes in extra "FIXUP" patches when possible, but the API changes in the first patch cause conflicts in the next one, so the big session variable patch has to contain the needed changes. In this patchset, I also changed the following: - global pass on the comments in session_variable - removed now useless sessionvars_types - added missing prototypes for static functions (for consistency), and moved all the static functions before the static function - minor other nitpicking / stylistic changes Here are the problems I found: - IdentifyVariable() /* * Lock relation. This will also accept any pending invalidation * messages. If we got back InvalidOid, indicating not found, then * there's nothing to lock, but we accept invalidation messages * anyway, to flush any negative catcache entries that may be * lingering. */ + if (!OidIsValid(varid)) + AcceptInvalidationMessages(); + else if (OidIsValid(varid)) + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + if (inval_count == SharedInvalidMessageCounter) + break; + + retry = true; + old_varid = varid; + } AFAICS it's correct, but just to be extra cautious I'd explicitly set varid to InvalidOid before looping, so you restart in the same condition as the first iteration (since varid is initialize when declared). Also, the comments should be modified, it's "Lock variable", not "Lock relation", same for the comment in the previous chunk ("we've locked the relation that used to have this name..."). +Datum +pg_debug_show_used_session_variables(PG_FUNCTION_ARGS) +{ +[...] + else + { + /* + * When session variable was removed from catalog, but still + * it in memory. The memory was not purged yet. + */ + nulls[1] = true; + nulls[2] = true; + nulls[4] = true; + values[5] = BoolGetDatum(true); + nulls[6] = true; + nulls[7] = true; + nulls[8] = true; + } I'm wondering if we could try to improve things a bit here. Maybe display the variable oid instead of its name as we still have that information, the type (using FORMAT_TYPE_ALLOW_INVALID as there's no guarantee that the type would still exist without the dependency) and whether the variable is valid (at least per its stored value). We can keep NULL for the privileges, as there's no API avoid erroring if the role has been dropped. +{ oid => '8488', descr => 'debug list of used session variables', + proname => 'pg_debug_show_used_session_variables', prorows => '1000', proretset => 't', + provolatile => 's', prorettype => 'record', proargtypes => '', + proallargtypes => '{oid,text,text,oid,text,bool,bool,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o}', + proargnames => '{varid,schema,name,typid,typname,removed,has_value,can_read,can_write}', Since we change READ / WRITE acl for SELECT / UPDATE, we should rename the column can_select and can_update. +static void +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) +{ + [...] + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) + { + if (hashvalue == 0 || svar->hashvalue == hashvalue) + { + [...] + xact_recheck_varids = list_append_unique_oid(xact_recheck_varids, + svar->varid); This has a pretty terrible complexity. It can degenerate badly, and there isn't any CHECK_FOR_INTERRUPTS so you could easily lock a backend for quite some time. I think we should just keep appending oids, and do a list_sort(list, list_oid_cmp) and list_deduplicate_oid(list) before processing the list, in sync_sessionvars_all() and AtPreEOXact_SessionVariable_on_xact_actions(). Maybe while at it we could reuse sync_sessionvars_all in AtPreEOXact_SessionVariable_on_xact_actions (with a way to ask for the lxid check or not), rather than duplicating the whole logic twice? +/* + * Fast drop of the complete content of all session variables hash table. + * This is code for DISCARD VARIABLES command. This command + * cannot be run inside transaction, so we don't need to handle + * end of transaction actions. + */ +void +ResetSessionVariables(void) +{ + /* Destroy hash table and reset related memory context */ + if (sessionvars) + { + hash_destroy(sessionvars); + sessionvars = NULL; + + hash_destroy(sessionvars_types); + sessionvars_types = NULL; + } + + /* Release memory allocated by session variables */ + if (SVariableMemoryContext != NULL) + MemoryContextReset(SVariableMemoryContext); + + /* + * There are not any session variables left, so simply trim xact + * action list, and other lists. + */ + list_free_deep(xact_on_commit_actions); + xact_on_commit_actions = NIL; + + /* We should clean xact_reset_varids */ + list_free(xact_reset_varids); + xact_reset_varids = NIL; + + /* we should clean xact_recheck_varids */ + list_free(xact_recheck_varids); + xact_recheck_varids = NIL; +} The initial comment is wrong. This function is used for both DISCARD VARIABLES and DISCARD ALL, but only DISCARD ALL isn't allowed in a transaction (I fixed the comment in the attached patchset). We should allow DISCARD VARIABLES in a transaction, therefore it needs some more thinking on which list can be freed, and in which context it should hold its data. AFAICS the only problematic case is ON COMMIT DROP, but an extra check wouldn't hurt. For instance: rjuju=# BEGIN; BEGIN rjuju=# CREATE TEMP VARIABLE v AS int ON COMMIT DROP; CREATE VARIABLE rjuju=# DISCARD VARIABLES ; DISCARD VARIABLES rjuju=# COMMIT; COMMIT rjuju=# \dV List of variables Schema | Name | Type | Collation | Nullable | Mutable | Default | Owner | Transactional end action -----------+------+---------+-----------+----------+---------+---------+-------+-------------------------- pg_temp_3 | v | integer | | t | t | | rjuju | ON COMMIT DROP (1 row) Note that I still think that keeping a single List for both SVariableXActAction helps for readability, even if it means cherry-picking which items should be removed on DISCARD VARIABLES (which shouldn't be a very frequent operation anyway). Also, xact_recheck_varids is allocated in SVariableMemoryContext, so DISCARD VARIABLE will crash if there's any pending recheck action. There's only one regression test for DISCARD VARIABLE, which clearly wasn't enough. There should be one for the ON COMMIT DROP (which can be added in normal regression test), one one with all action list populated (that need to be in isolation tester). Both are added in the patchset in a suggestion patch, and for now the first test fails and the second crashes. - set_session_variable() is documented to either succeed or not change the currently set value. While it's globally true, I see 2 things that could be problematic: - free_session_variable_value() could technically fail. However, I don't see how it could be happening unless there's a memory corruption, so this would result in either an abort, or a backend in a very bad state. Anyway, since pfree() can clearly ereport(ERROR) we should probably do something about it. That being said, I don't really see the point of trying to preserve a value that looks like random pointer, which will probably cause a segfault the next time it's used. Maybe add a PG_TRY block around the call and mark it as invalid (and set freeval to false) if that happens? - the final elog(DEBUG1) can also fail. It also seems highly unlikely, so maybe accept that this exception is ok? For now I'm adding such a comment in a suggestion patch. - prepare_variable_for_reading() and SetSessionVariable(): + /* Ensure so all entries in sessionvars hash table are valid */ + sync_sessionvars_all(); + + /* Protect used session variable against drop until transaction end */ + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); It's possible that a session variable is dropped after calling sync_sessionvars_all(), and we would receive the sinval when acquiring the lock on VariableRelationId but not process it until the next sync_sessionvars_all call. I think we should acquire the lock first and then call sync_sessionvars_all. I did that in the suggestion patch.
Вложения
- v20220916-0001-catalog-support-for-session-variables.patch
- v20220916-0002-FIXUP-0001-catalog-support-for-session-var.patch
- v20220916-0003-session-variables.patch
- v20220916-0004-FIXUP-0003-session-variables.patch
- v20220916-0005-LET-command.patch
- v20220916-0006-support-of-LET-command-in-PLpgSQL.patch
- v20220916-0007-DISCARD-VARIABLES-command.patch
- v20220916-0008-enhancing-psql-for-session-variables.patch
- v20220916-0009-possibility-to-dump-session-variables-by-p.patch
- v20220916-0010-regress-tests-for-session-variables.patch
- v20220916-0011-FIXUP-regress-tests-for-session-variables.patch
- v20220916-0012-this-patch-changes-error-message-column-do.patch
- v20220916-0013-documentation.patch
On Fri, Sep 16, 2022 at 11:59:04AM +0800, Julien Rouhaud wrote: > Hi, > > On Sun, Sep 11, 2022 at 09:29:49PM +0200, Pavel Stehule wrote: > >> > >> Originally it was not possible, because there was no xact_reset_varids list, and without this list the processing > >> ON_COMMIT_DROP started DROP VARIABLE command, and there was a request for ON_COMMIT_RESET action. > >> Now, it is possible, because in RemoveSessionVariable is conditional execution: > >> > >> <--><--><-->if (!svar->eox_reset) > >> <--><--><--><-->register_session_variable_xact_action(varid, > >> <--><--><--><--><--><--><--><--><--><--><--><--><--> SVAR_ON_COMMIT_RESET); > >> <--><-->} > >> > >> So when we process ON_COMMIT_DROP actions, we know that the reset will not be processed by ON_COMMIT_RESET action, > >> and then these lists can be merged. > >> > >> so I merged these two lists to one > > Thanks! This really helps with code readability, and after looking at it I > found some issues (see below). > > > > changes: > > > > - some minor cleaning > > - refactoring of RemoveSessionVariable - move part of code to pg_variable.c > > Thanks. I think we could still do more to split what code belongs to > pg_variable.c and session_variable.c. In my opinion, the various DDL code > should only invoke functions in pg_variable.c, which themselves can call > function in session_variable.c if needed, and session_variable shouldn't know > about CreateSessionVarStmt (which should probably be rename > CreateVariableStmt?) or VariableRelationId. After an off-list bikeshedding > session with Pavel, we came up with SessionVariableCreatePostprocess() and > SessionVariableDropPostprocess() for the functions in session_variable.c called > by pg_variable.c when handling CREATE VARIABLE and DROP VARIABLE commands. > > I'm attaching a new patchset with this change and some more (see below). I'm > not sending .txt files as this is rebased on top on the recent GUC refactoring > patch. It won't change the cfbot outcome though, as I also add new regression > tests that are for now failing (see below). I tried to keep the changes in > extra "FIXUP" patches when possible, but the API changes in the first patch > cause conflicts in the next one, so the big session variable patch has to > contain the needed changes. > > In this patchset, I also changed the following: > > - global pass on the comments in session_variable > - removed now useless sessionvars_types > - added missing prototypes for static functions (for consistency), and moved > all the static functions before the static function > - minor other nitpicking / stylistic changes > > Here are the problems I found: > > - IdentifyVariable() > > /* > * Lock relation. This will also accept any pending invalidation > * messages. If we got back InvalidOid, indicating not found, then > * there's nothing to lock, but we accept invalidation messages > * anyway, to flush any negative catcache entries that may be > * lingering. > */ > + if (!OidIsValid(varid)) > + AcceptInvalidationMessages(); > + else if (OidIsValid(varid)) > + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); > + > + if (inval_count == SharedInvalidMessageCounter) > + break; > + > + retry = true; > + old_varid = varid; > + } > > AFAICS it's correct, but just to be extra cautious I'd explicitly set varid to > InvalidOid before looping, so you restart in the same condition as the first > iteration (since varid is initialize when declared). Also, the comments should > be modified, it's "Lock variable", not "Lock relation", same for the comment in > the previous chunk ("we've locked the relation that used to have this > name..."). > > +Datum > +pg_debug_show_used_session_variables(PG_FUNCTION_ARGS) > +{ > +[...] > + else > + { > + /* > + * When session variable was removed from catalog, but still > + * it in memory. The memory was not purged yet. > + */ > + nulls[1] = true; > + nulls[2] = true; > + nulls[4] = true; > + values[5] = BoolGetDatum(true); > + nulls[6] = true; > + nulls[7] = true; > + nulls[8] = true; > + } > > I'm wondering if we could try to improve things a bit here. Maybe display the > variable oid instead of its name as we still have that information, the type > (using FORMAT_TYPE_ALLOW_INVALID as there's no guarantee that the type would > still exist without the dependency) and whether the variable is valid (at least > per its stored value). We can keep NULL for the privileges, as there's no API > avoid erroring if the role has been dropped. > > +{ oid => '8488', descr => 'debug list of used session variables', > + proname => 'pg_debug_show_used_session_variables', prorows => '1000', proretset => 't', > + provolatile => 's', prorettype => 'record', proargtypes => '', > + proallargtypes => '{oid,text,text,oid,text,bool,bool,bool,bool}', > + proargmodes => '{o,o,o,o,o,o,o,o,o}', > + proargnames => '{varid,schema,name,typid,typname,removed,has_value,can_read,can_write}', > > Since we change READ / WRITE acl for SELECT / UPDATE, we should rename the > column can_select and can_update. > > +static void > +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) > +{ > + [...] > + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) > + { > + if (hashvalue == 0 || svar->hashvalue == hashvalue) > + { > + [...] > + xact_recheck_varids = list_append_unique_oid(xact_recheck_varids, > + svar->varid); > > This has a pretty terrible complexity. It can degenerate badly, and there > isn't any CHECK_FOR_INTERRUPTS so you could easily lock a backend for quite > some time. > > I think we should just keep appending oids, and do a list_sort(list, > list_oid_cmp) and list_deduplicate_oid(list) before processing the list, in > sync_sessionvars_all() and AtPreEOXact_SessionVariable_on_xact_actions(). > > Maybe while at it we could reuse sync_sessionvars_all in > AtPreEOXact_SessionVariable_on_xact_actions (with a way to ask > for the lxid check or not), rather than duplicating the whole logic twice? > > +/* > + * Fast drop of the complete content of all session variables hash table. > + * This is code for DISCARD VARIABLES command. This command > + * cannot be run inside transaction, so we don't need to handle > + * end of transaction actions. > + */ > +void > +ResetSessionVariables(void) > +{ > + /* Destroy hash table and reset related memory context */ > + if (sessionvars) > + { > + hash_destroy(sessionvars); > + sessionvars = NULL; > + > + hash_destroy(sessionvars_types); > + sessionvars_types = NULL; > + } > + > + /* Release memory allocated by session variables */ > + if (SVariableMemoryContext != NULL) > + MemoryContextReset(SVariableMemoryContext); > + > + /* > + * There are not any session variables left, so simply trim xact > + * action list, and other lists. > + */ > + list_free_deep(xact_on_commit_actions); > + xact_on_commit_actions = NIL; > + > + /* We should clean xact_reset_varids */ > + list_free(xact_reset_varids); > + xact_reset_varids = NIL; > + > + /* we should clean xact_recheck_varids */ > + list_free(xact_recheck_varids); > + xact_recheck_varids = NIL; > +} > > The initial comment is wrong. This function is used for both DISCARD VARIABLES > and DISCARD ALL, but only DISCARD ALL isn't allowed in a transaction (I fixed > the comment in the attached patchset). > We should allow DISCARD VARIABLES in a transaction, therefore it needs some > more thinking on which list can be freed, and in which context it should hold > its data. AFAICS the only problematic case is ON COMMIT DROP, but an extra > check wouldn't hurt. For instance: > > rjuju=# BEGIN; > BEGIN > > rjuju=# CREATE TEMP VARIABLE v AS int ON COMMIT DROP; > CREATE VARIABLE > > rjuju=# DISCARD VARIABLES ; > DISCARD VARIABLES > > rjuju=# COMMIT; > COMMIT > > rjuju=# \dV > List of variables > Schema | Name | Type | Collation | Nullable | Mutable | Default | Owner | Transactional end action > -----------+------+---------+-----------+----------+---------+---------+-------+-------------------------- > pg_temp_3 | v | integer | | t | t | | rjuju | ON COMMIT DROP > (1 row) > > Note that I still think that keeping a single List for both SVariableXActAction > helps for readability, even if it means cherry-picking which items should be > removed on DISCARD VARIABLES (which shouldn't be a very frequent operation > anyway). > > Also, xact_recheck_varids is allocated in SVariableMemoryContext, so DISCARD > VARIABLE will crash if there's any pending recheck action. > > There's only one regression test for DISCARD VARIABLE, which clearly wasn't > enough. There should be one for the ON COMMIT DROP (which can be added in > normal regression test), one one with all action list populated (that need to > be in isolation tester). Both are added in the patchset in a suggestion patch, > and for now the first test fails and the second crashes. > > > - set_session_variable() is documented to either succeed or not change the > currently set value. While it's globally true, I see 2 things that could be > problematic: > > - free_session_variable_value() could technically fail. However, I don't see > how it could be happening unless there's a memory corruption, so this would > result in either an abort, or a backend in a very bad state. Anyway, since > pfree() can clearly ereport(ERROR) we should probably do something about > it. That being said, I don't really see the point of trying to preserve a > value that looks like random pointer, which will probably cause a segfault > the next time it's used. Maybe add a PG_TRY block around the call and mark > it as invalid (and set freeval to false) if that happens? > > - the final elog(DEBUG1) can also fail. It also seems highly unlikely, so > maybe accept that this exception is ok? For now I'm adding such a comment > in a suggestion patch. > > - prepare_variable_for_reading() and SetSessionVariable(): > > + /* Ensure so all entries in sessionvars hash table are valid */ > + sync_sessionvars_all(); > + > + /* Protect used session variable against drop until transaction end */ > + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); > > It's possible that a session variable is dropped after calling > sync_sessionvars_all(), and we would receive the sinval when acquiring the lock > on VariableRelationId but not process it until the next sync_sessionvars_all > call. I think we should acquire the lock first and then call > sync_sessionvars_all. I did that in the suggestion patch. Rebased patcshet against recent conflicts, thanks to Pavel for the reminder. While sending a new patch, I realized that I forgot mentionning this in execMain.c: @@ -200,6 +201,61 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) Assert(queryDesc->sourceText != NULL); estate->es_sourceText = queryDesc->sourceText; + /* + * The executor doesn't work with session variables directly. Values of + * related session variables are copied to dedicated array, and this array + * is passed to executor. + */ + if (queryDesc->num_session_variables > 0) + { + /* + * When paralel access to query parameters (including related session + * variables) is required, then related session variables are restored + * (deserilized) in queryDesc already. So just push pointer of this + * array to executor's estate. + */ + estate->es_session_variables = queryDesc->session_variables; + estate->es_num_session_variables = queryDesc->num_session_variables; + } + else if (queryDesc->plannedstmt->sessionVariables) + { + ListCell *lc; + int nSessionVariables; + int i = 0; + + /* + * In this case, the query uses session variables, but we have to + * prepare the array with passed values (of used session variables) + * first. + */ + nSessionVariables = list_length(queryDesc->plannedstmt->sessionVariables); + + /* Create the array used for passing values of used session variables */ + estate->es_session_variables = (SessionVariableValue *) + palloc(nSessionVariables * sizeof(SessionVariableValue)); + + /* Fill the array */ + [...] + + estate->es_num_session_variables = nSessionVariables; + } I haven't looked at that part yet, but the comments are a bit obscure. IIUC the first branch is for parallel workers only, if the main backend provided the array, and the 2nd chunk is for the main backend. If so, it could be made clearer, and maybe add an assert about IsParallelWorker() (or !IsParallelWorker()) as needed?
Вложения
- v20220922-0001-catalog-support-for-session-variables.patch
- v20220922-0002-FIXUP-0001-catalog-support-for-session-var.patch
- v20220922-0003-session-variables.patch
- v20220922-0004-FIXUP-0003-session-variables.patch
- v20220922-0005-LET-command.patch
- v20220922-0006-support-of-LET-command-in-PLpgSQL.patch
- v20220922-0007-DISCARD-VARIABLES-command.patch
- v20220922-0008-enhancing-psql-for-session-variables.patch
- v20220922-0009-possibility-to-dump-session-variables-by-p.patch
- v20220922-0010-regress-tests-for-session-variables.patch
- v20220922-0011-FIXUP-regress-tests-for-session-variables.patch
- v20220922-0012-this-patch-changes-error-message-column-do.patch
- v20220922-0013-documentation.patch
Hi, The patch has rotten again, sending an updated version. Also, after talking with Pavel, he can't work on this patch before a few days so I'm adding some extra fixup patches for the things I reported in the last few emails, so that the cfbot can hopefully turn green. On Thu, Sep 22, 2022 at 2:41 PM Julien Rouhaud <rjuju123@gmail.com> wrote: > > On Fri, Sep 16, 2022 at 11:59:04AM +0800, Julien Rouhaud wrote: > > Hi, > > > > On Sun, Sep 11, 2022 at 09:29:49PM +0200, Pavel Stehule wrote: > > >> > > >> Originally it was not possible, because there was no xact_reset_varids list, and without this list the processing > > >> ON_COMMIT_DROP started DROP VARIABLE command, and there was a request for ON_COMMIT_RESET action. > > >> Now, it is possible, because in RemoveSessionVariable is conditional execution: > > >> > > >> <--><--><-->if (!svar->eox_reset) > > >> <--><--><--><-->register_session_variable_xact_action(varid, > > >> <--><--><--><--><--><--><--><--><--><--><--><--><--> SVAR_ON_COMMIT_RESET); > > >> <--><-->} > > >> > > >> So when we process ON_COMMIT_DROP actions, we know that the reset will not be processed by ON_COMMIT_RESET action, > > >> and then these lists can be merged. > > >> > > >> so I merged these two lists to one > > > > Thanks! This really helps with code readability, and after looking at it I > > found some issues (see below). > > > > > > changes: > > > > > > - some minor cleaning > > > - refactoring of RemoveSessionVariable - move part of code to pg_variable.c > > > > Thanks. I think we could still do more to split what code belongs to > > pg_variable.c and session_variable.c. In my opinion, the various DDL code > > should only invoke functions in pg_variable.c, which themselves can call > > function in session_variable.c if needed, and session_variable shouldn't know > > about CreateSessionVarStmt (which should probably be rename > > CreateVariableStmt?) or VariableRelationId. After an off-list bikeshedding > > session with Pavel, we came up with SessionVariableCreatePostprocess() and > > SessionVariableDropPostprocess() for the functions in session_variable.c called > > by pg_variable.c when handling CREATE VARIABLE and DROP VARIABLE commands. > > > > I'm attaching a new patchset with this change and some more (see below). I'm > > not sending .txt files as this is rebased on top on the recent GUC refactoring > > patch. It won't change the cfbot outcome though, as I also add new regression > > tests that are for now failing (see below). I tried to keep the changes in > > extra "FIXUP" patches when possible, but the API changes in the first patch > > cause conflicts in the next one, so the big session variable patch has to > > contain the needed changes. > > > > In this patchset, I also changed the following: > > > > - global pass on the comments in session_variable > > - removed now useless sessionvars_types > > - added missing prototypes for static functions (for consistency), and moved > > all the static functions before the static function > > - minor other nitpicking / stylistic changes > > > > Here are the problems I found: > > > > - IdentifyVariable() > > > > /* > > * Lock relation. This will also accept any pending invalidation > > * messages. If we got back InvalidOid, indicating not found, then > > * there's nothing to lock, but we accept invalidation messages > > * anyway, to flush any negative catcache entries that may be > > * lingering. > > */ > > + if (!OidIsValid(varid)) > > + AcceptInvalidationMessages(); > > + else if (OidIsValid(varid)) > > + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); > > + > > + if (inval_count == SharedInvalidMessageCounter) > > + break; > > + > > + retry = true; > > + old_varid = varid; > > + } > > > > AFAICS it's correct, but just to be extra cautious I'd explicitly set varid to > > InvalidOid before looping, so you restart in the same condition as the first > > iteration (since varid is initialize when declared). Also, the comments should > > be modified, it's "Lock variable", not "Lock relation", same for the comment in > > the previous chunk ("we've locked the relation that used to have this > > name..."). > > > > +Datum > > +pg_debug_show_used_session_variables(PG_FUNCTION_ARGS) > > +{ > > +[...] > > + else > > + { > > + /* > > + * When session variable was removed from catalog, but still > > + * it in memory. The memory was not purged yet. > > + */ > > + nulls[1] = true; > > + nulls[2] = true; > > + nulls[4] = true; > > + values[5] = BoolGetDatum(true); > > + nulls[6] = true; > > + nulls[7] = true; > > + nulls[8] = true; > > + } > > > > I'm wondering if we could try to improve things a bit here. Maybe display the > > variable oid instead of its name as we still have that information, the type > > (using FORMAT_TYPE_ALLOW_INVALID as there's no guarantee that the type would > > still exist without the dependency) and whether the variable is valid (at least > > per its stored value). We can keep NULL for the privileges, as there's no API > > avoid erroring if the role has been dropped. > > > > +{ oid => '8488', descr => 'debug list of used session variables', > > + proname => 'pg_debug_show_used_session_variables', prorows => '1000', proretset => 't', > > + provolatile => 's', prorettype => 'record', proargtypes => '', > > + proallargtypes => '{oid,text,text,oid,text,bool,bool,bool,bool}', > > + proargmodes => '{o,o,o,o,o,o,o,o,o}', > > + proargnames => '{varid,schema,name,typid,typname,removed,has_value,can_read,can_write}', > > > > Since we change READ / WRITE acl for SELECT / UPDATE, we should rename the > > column can_select and can_update. > > > > +static void > > +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) > > +{ > > + [...] > > + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) > > + { > > + if (hashvalue == 0 || svar->hashvalue == hashvalue) > > + { > > + [...] > > + xact_recheck_varids = list_append_unique_oid(xact_recheck_varids, > > + svar->varid); > > > > This has a pretty terrible complexity. It can degenerate badly, and there > > isn't any CHECK_FOR_INTERRUPTS so you could easily lock a backend for quite > > some time. > > > > I think we should just keep appending oids, and do a list_sort(list, > > list_oid_cmp) and list_deduplicate_oid(list) before processing the list, in > > sync_sessionvars_all() and AtPreEOXact_SessionVariable_on_xact_actions(). > > > > Maybe while at it we could reuse sync_sessionvars_all in > > AtPreEOXact_SessionVariable_on_xact_actions (with a way to ask > > for the lxid check or not), rather than duplicating the whole logic twice? > > > > +/* > > + * Fast drop of the complete content of all session variables hash table. > > + * This is code for DISCARD VARIABLES command. This command > > + * cannot be run inside transaction, so we don't need to handle > > + * end of transaction actions. > > + */ > > +void > > +ResetSessionVariables(void) > > +{ > > + /* Destroy hash table and reset related memory context */ > > + if (sessionvars) > > + { > > + hash_destroy(sessionvars); > > + sessionvars = NULL; > > + > > + hash_destroy(sessionvars_types); > > + sessionvars_types = NULL; > > + } > > + > > + /* Release memory allocated by session variables */ > > + if (SVariableMemoryContext != NULL) > > + MemoryContextReset(SVariableMemoryContext); > > + > > + /* > > + * There are not any session variables left, so simply trim xact > > + * action list, and other lists. > > + */ > > + list_free_deep(xact_on_commit_actions); > > + xact_on_commit_actions = NIL; > > + > > + /* We should clean xact_reset_varids */ > > + list_free(xact_reset_varids); > > + xact_reset_varids = NIL; > > + > > + /* we should clean xact_recheck_varids */ > > + list_free(xact_recheck_varids); > > + xact_recheck_varids = NIL; > > +} > > > > The initial comment is wrong. This function is used for both DISCARD VARIABLES > > and DISCARD ALL, but only DISCARD ALL isn't allowed in a transaction (I fixed > > the comment in the attached patchset). > > We should allow DISCARD VARIABLES in a transaction, therefore it needs some > > more thinking on which list can be freed, and in which context it should hold > > its data. AFAICS the only problematic case is ON COMMIT DROP, but an extra > > check wouldn't hurt. For instance: > > > > rjuju=# BEGIN; > > BEGIN > > > > rjuju=# CREATE TEMP VARIABLE v AS int ON COMMIT DROP; > > CREATE VARIABLE > > > > rjuju=# DISCARD VARIABLES ; > > DISCARD VARIABLES > > > > rjuju=# COMMIT; > > COMMIT > > > > rjuju=# \dV > > List of variables > > Schema | Name | Type | Collation | Nullable | Mutable | Default | Owner | Transactional end action > > -----------+------+---------+-----------+----------+---------+---------+-------+-------------------------- > > pg_temp_3 | v | integer | | t | t | | rjuju | ON COMMIT DROP > > (1 row) > > > > Note that I still think that keeping a single List for both SVariableXActAction > > helps for readability, even if it means cherry-picking which items should be > > removed on DISCARD VARIABLES (which shouldn't be a very frequent operation > > anyway). > > > > Also, xact_recheck_varids is allocated in SVariableMemoryContext, so DISCARD > > VARIABLE will crash if there's any pending recheck action. > > > > There's only one regression test for DISCARD VARIABLE, which clearly wasn't > > enough. There should be one for the ON COMMIT DROP (which can be added in > > normal regression test), one one with all action list populated (that need to > > be in isolation tester). Both are added in the patchset in a suggestion patch, > > and for now the first test fails and the second crashes. > > > > > > - set_session_variable() is documented to either succeed or not change the > > currently set value. While it's globally true, I see 2 things that could be > > problematic: > > > > - free_session_variable_value() could technically fail. However, I don't see > > how it could be happening unless there's a memory corruption, so this would > > result in either an abort, or a backend in a very bad state. Anyway, since > > pfree() can clearly ereport(ERROR) we should probably do something about > > it. That being said, I don't really see the point of trying to preserve a > > value that looks like random pointer, which will probably cause a segfault > > the next time it's used. Maybe add a PG_TRY block around the call and mark > > it as invalid (and set freeval to false) if that happens? > > > > - the final elog(DEBUG1) can also fail. It also seems highly unlikely, so > > maybe accept that this exception is ok? For now I'm adding such a comment > > in a suggestion patch. > > > > - prepare_variable_for_reading() and SetSessionVariable(): > > > > + /* Ensure so all entries in sessionvars hash table are valid */ > > + sync_sessionvars_all(); > > + > > + /* Protect used session variable against drop until transaction end */ > > + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); > > > > It's possible that a session variable is dropped after calling > > sync_sessionvars_all(), and we would receive the sinval when acquiring the lock > > on VariableRelationId but not process it until the next sync_sessionvars_all > > call. I think we should acquire the lock first and then call > > sync_sessionvars_all. I did that in the suggestion patch. > > Rebased patcshet against recent conflicts, thanks to Pavel for the reminder. > > While sending a new patch, I realized that I forgot mentionning this in > execMain.c: > > @@ -200,6 +201,61 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) > Assert(queryDesc->sourceText != NULL); > estate->es_sourceText = queryDesc->sourceText; > > + /* > + * The executor doesn't work with session variables directly. Values of > + * related session variables are copied to dedicated array, and this array > + * is passed to executor. > + */ > + if (queryDesc->num_session_variables > 0) > + { > + /* > + * When paralel access to query parameters (including related session > + * variables) is required, then related session variables are restored > + * (deserilized) in queryDesc already. So just push pointer of this > + * array to executor's estate. > + */ > + estate->es_session_variables = queryDesc->session_variables; > + estate->es_num_session_variables = queryDesc->num_session_variables; > + } > + else if (queryDesc->plannedstmt->sessionVariables) > + { > + ListCell *lc; > + int nSessionVariables; > + int i = 0; > + > + /* > + * In this case, the query uses session variables, but we have to > + * prepare the array with passed values (of used session variables) > + * first. > + */ > + nSessionVariables = list_length(queryDesc->plannedstmt->sessionVariables); > + > + /* Create the array used for passing values of used session variables */ > + estate->es_session_variables = (SessionVariableValue *) > + palloc(nSessionVariables * sizeof(SessionVariableValue)); > + > + /* Fill the array */ > + [...] > + > + estate->es_num_session_variables = nSessionVariables; > + } > > I haven't looked at that part yet, but the comments are a bit obscure. IIUC > the first branch is for parallel workers only, if the main backend provided the > array, and the 2nd chunk is for the main backend. If so, it could be made > clearer, and maybe add an assert about IsParallelWorker() (or > !IsParallelWorker()) as needed? Full list of changes: - rebased against multiple conflicts since last version - fixed the meson build - fixed the ON COMMIT DROP problem and the crash on RESET VARIABLES - fixed some copy/pasto in the expected isolation tests (visible now that it works) - added the asserts and tried to clarify the comments for the session variable handling in QueryDesc (I still haven't really read that part) - did the mentioned modifications on pg_debug_show_used_session_variables, and used CStringGetTextDatum macro to simplify the code Note that while waiting for the CI to finish I noticed that the commit message for 0001 still mentions the READ/WRITE acl. The commit messages will probably need a bit of rewording too once everything else is fixed, but this one could be changed already.
Вложения
Hi,
The patch has rotten again, sending an updated version. Also, after
talking with Pavel, he can't work on this patch before a few days so
I'm adding some extra fixup patches for the things I reported in the
last few emails, so that the cfbot can hopefully turn green.
Note that while waiting for the CI to finish I noticed that the commit
message for 0001 still mentions the READ/WRITE acl. The commit
messages will probably need a bit of rewording too once everything
else is fixed, but this one could be changed already.
Вложения
- v20221012-0010-documentation.patch
- v20221012-0006-enhancing-psql-for-session-variables.patch
- v20221012-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221012-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221012-0008-regress-tests-for-session-variables.patch
- v20221012-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221012-0005-DISCARD-VARIABLES-command.patch
- v20221012-0003-LET-command.patch
- v20221012-0002-session-variables.patch
- v20221012-0001-catalog-support-for-session-variables.patch
Hine 25. 9. 2022 v 8:56 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:Hi,
The patch has rotten again, sending an updated version. Also, after
talking with Pavel, he can't work on this patch before a few days so
I'm adding some extra fixup patches for the things I reported in the
last few emails, so that the cfbot can hopefully turn green.
Note that while waiting for the CI to finish I noticed that the commit
message for 0001 still mentions the READ/WRITE acl. The commit
messages will probably need a bit of rewording too once everything
else is fixed, but this one could be changed already.I fixed the commit message of 0001 patch. Fixed shadowed variables too.There is a partially open issue, where I and Julien are not sure about a solution, and we would like to ask for the community's opinion. I'll send this query in separate mail.
RegardsPavel
Вложения
- v20221013-1-0010-documentation.patch
- v20221013-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221013-1-0008-regress-tests-for-session-variables.patch
- v20221013-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221013-1-0006-enhancing-psql-for-session-variables.patch
- v20221013-1-0005-DISCARD-VARIABLES-command.patch
- v20221013-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221013-1-0003-LET-command.patch
- v20221013-1-0002-session-variables.patch
- v20221013-1-0001-catalog-support-for-session-variables.patch
Hi, On Thu, Oct 13, 2022 at 07:41:32AM +0200, Pavel Stehule wrote: > > > I fixed the commit message of 0001 patch. Fixed shadowed variables too. Thanks! > > > > There is a partially open issue, where I and Julien are not sure about a > > solution, and we would like to ask for the community's opinion. I'll send > > this query in separate mail. > > > > rebased with simplified code related to usage of pfree function If anyone is curious the discussion happend at [1]. I looked at the patchset, this time focusing on the LET command. Here at the comments I have for now: - gram.y @@ -11918,6 +11920,7 @@ ExplainableStmt: | CreateMatViewStmt | RefreshMatViewStmt | ExecuteStmt /* by default all are $$=$1 */ + | LetStmt ; (and other similar places) the comment should be kept to the last statement Also, having LetStmt as an ExplainableStmt means it's allowed in a CTE: cte_list: common_table_expr { $$ = list_make1($1); } | cte_list ',' common_table_expr { $$ = lappend($1, $3); } ; common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')' opt_search_clause opt_cycle_clause And doing so hits this assert in transformWithClause: if (!IsA(cte->ctequery, SelectStmt)) { /* must be a data-modifying statement */ Assert(IsA(cte->ctequery, InsertStmt) || IsA(cte->ctequery, UpdateStmt) || IsA(cte->ctequery, DeleteStmt)); pstate->p_hasModifyingCTE = true; } and I'm assuming it would also fail on this in transformLetStmt: + /* There can't be any outer WITH to worry about */ + Assert(pstate->p_ctenamespace == NIL); I guess it makes sense to be able to explain a LetStmt (or using it in a prepared statement), so it should be properly handled in transformSelectStmt. Also, I don't see any test for a prepared LET statement, this should also be covered. - transformLetStmt: + varid = IdentifyVariable(names, &attrname, ¬_unique); It would be nice to have a comment saying that the lock is acquired here + /* The grammar should have produced a SELECT */ + if (!IsA(selectQuery, Query) || + selectQuery->commandType != CMD_SELECT) + elog(ERROR, "unexpected non-SELECT command in LET command"); I'm wondering if this should be an Assert instead, as the grammar shouldn't produce anything else no matter what how hard a user try. + /* don't allow multicolumn result */ + if (list_length(exprList) != 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("expression is not scalar value"), + parser_errposition(pstate, + exprLocation((Node *) exprList)))); This isn't covered by any regression test and it probably should. It can be reached with something like LET myvar = (null::pg_class).*; The error message could also use a bit of improvement. I see that a_expr allows a select statement in parens, but this leads to a sublink which already has all the required protection to gurantee a single column, and a single row at most during execution. This one returns for non-scalar case: subquery must return only one column Maybe use something similar for it, like "expression must return only one column"? Similarly the error message in svariableStartupReceiver could be made more consistent with the related errors: + if (++outcols > 1) + elog(ERROR, "svariable DestReceiver can take only one attribute"); While on svariableReceiver, I see that the current code assumes that the caller did everything right. That's the case right now, but it should still be made more robust in case future code (or extensions) is added. I'm thinking: - svariableState.rows. Currently not really used, should check that one and only one row is received in svariableReceiveSlot and svariableShutdownReceiver (if no row is received the variable won't be reset which should probably always happen once you setup an svariableReceiver) - svariableState.typid, typmod and typlen should be double checked with the given varid in svariableStartupReceiver. - svariableState.varid should be initialized with InvalidOid to avoid undefined behavior is caller forgets to set it. I'm also wondering if SetVariableDestReceiverParams() should have an assert like LockHeldByMe() for the given varid, and maybe an assert that the varid is a session variable, to avoid running a possibly expensive execution that will fail when receiving the slot. I think the function would be better named SetVariableDestReceiverVarid() or something like that. +void +ExecuteLetStmt(ParseState *pstate, + LetStmt *stmt, + ParamListInfo params, + QueryEnvironment *queryEnv, + QueryCompletion *qc) +{ + [...] + /* run the plan to completion */ + ExecutorRun(queryDesc, ForwardScanDirection, 2L, true); Why 2 rows? I'm assuming it's an attempt to detect queries that returns more than 1 row, but it should be documented. Note that as mentioned above the dest receiver currently doesn't check it, so this definitely needs to be fixed. - IdentifyVariable: *attrname can be set even is no variable is identified. I guess that's ok as it avoids useless code, but it should probably be documented in the function header. Also, the API doesn't look ideal. AFAICS the only reason this function doesn't error out in case of ambiguous name is that transformColumnRef may check if a given name shadows a variable when session_variables_ambiguity_warning is set. But since IdentifyVariable returns InvalidOid if the given list of identifiers is ambiguous, it seems that the shadow detection can fail to detect a shadowed reference if multiple variable would shadow the name: # CREATE TYPE ab AS (a integer, b integer); CREATE TYPE # CREATE VARIABLE v_ab AS ab; CREATE VARIABLE # CREATE TABLE v_ab (a integer, b integer); CREATE TABLE # SET session_variables_ambiguity_warning = 1; SET # sELECT v_ab.a FROM v_ab; WARNING: 42702: session variable "v_ab.a" is shadowed LINE 1: select v_ab.a from v_ab; ^ DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. a --- (0 rows) # CREATE SCHEMA v_ab; CREATE SCHEMA # CREATE VARIABLE v_ab.a AS integer; CREATE VARIABLE # SELECT v_ab.a FROM v_ab; a --- (0 rows) Note that a bit later in transformColumnRef(), not_unique is checked only if the returned varid is valid, which isn't correct as InvalidOid is currently returned if not_unique is set. I think that the error should be raised in IdentifyVariable rather than having every caller check it. I'm not sure how to perfectly handle the session_variables_ambiguity_warning though. Maybe make not_unique optional, and error out if not_unique is null. If not null, set it as necessary and return one of the oid. The only use would be for shadowing detection, and in that case it won't be possible to check if a warning can be avoided as it would be if no amgibuity is found, but that's probably ok. Or maybe instead LookupVariable should have an extra argument to only match variable with a composite type if caller asks to. This would avoid scenarios like: CREATE VARIABLE myvar AS int; SELECT myvar.blabla; ERROR: 42809: type integer is not composite Is that really ok to match a variable here rather than complaining about a missing FROM-clause? + indirection_start = list_length(names) - (attrname ? 1 : 0); + indirection = list_copy_tail(stmt->target, indirection_start); + [...] + if (indirection != NULL) + { + bool targetIsArray; + char *targetName; + + targetName = get_session_variable_name(varid); + targetIsArray = OidIsValid(get_element_type(typid)); + + pstate->p_hasSessionVariables = true; + + coerced_expr = (Expr *) + transformAssignmentIndirection(pstate, + (Node *) param, + targetName, + targetIsArray, + typid, + typmod, + InvalidOid, + indirection, + list_head(indirection), + (Node *) expr, + COERCION_PLPGSQL, + stmt->location); + } I'm not sure why you use this approach rather than just having something like "ListCell *indirection_head", set it to a non-NULL value when needed, and use that (with names) instead. Note that it's also not correct to compare a List to NULL, use NIL instead. - expr_kind_allows_session_variables Even if that's a bit annoying, I think it's better to explicitly put all values there rather than having a default clause. For instance, EXPR_KIND_CYCLE_MARK is currently allowing session variables, which doesn't look ok. It's probably just an error from when the patchset was rebased, but this probably wouldn't happen if you get an error for an unmatched value if you add a new expr kind (which doesn't happen that often). [1] https://www.postgresql.org/message-id/CAFj8pRB2+pVBFsidS-AzhHdZid40OTUspWfXS0vgahHmaWosZQ@mail.gmail.com
> On Thu, Oct 13, 2022 at 07:41:32AM +0200, Pavel Stehule wrote: > rebased with simplified code related to usage of pfree function Thanks for the patch, great work! I've got a couple of questions, although I haven't fully finished reviewing yet (so more to come): * I'm curious about ALTER VARIABLE. Current implementation allows altering only the name, schema or the owner -- why not e.g. immutability? * psql tab completion implementation mentions that CREATE VARIABLE could be used inside CREATE SCHEMA: /* CREATE VARIABLE --- is allowed inside CREATE SCHEMA, so use TailMatches */ /* Complete CREATE VARIABLE <name> with AS */ else if (TailMatches("IMMUTABLE")) Is that correct? It doesn't like it works, and from what I see it requires some modifications in transformCreateSchemaStmt and schema_stmt. * psql describe mentions the following: /* * Most functions in this file are content to print an empty table when * there are no matching objects. We intentionally deviate from that * here, but only in !quiet mode, for historical reasons. */ I guess it's taken from listTables, and the extended versions says "because of the possibility that the user is confused about what the two pattern arguments mean". Are those historical reasons apply to variables as well?
> On Thu, Oct 13, 2022 at 07:41:32AM +0200, Pavel Stehule wrote:
> rebased with simplified code related to usage of pfree function
Thanks for the patch, great work!
I've got a couple of questions, although I haven't fully finished reviewing yet
(so more to come):
* I'm curious about ALTER VARIABLE. Current implementation allows altering only
the name, schema or the owner -- why not e.g. immutability?
* psql tab completion implementation mentions that CREATE VARIABLE could be
used inside CREATE SCHEMA:
/* CREATE VARIABLE --- is allowed inside CREATE SCHEMA, so use TailMatches */
/* Complete CREATE VARIABLE <name> with AS */
else if (TailMatches("IMMUTABLE"))
Is that correct? It doesn't like it works, and from what I see it requires
some modifications in transformCreateSchemaStmt and schema_stmt.
* psql describe mentions the following:
/*
* Most functions in this file are content to print an empty table when
* there are no matching objects. We intentionally deviate from that
* here, but only in !quiet mode, for historical reasons.
*/
I guess it's taken from listTables, and the extended versions says "because
of the possibility that the user is confused about what the two pattern
arguments mean". Are those historical reasons apply to variables as well?
Did not find any relations.
(2022-10-30 19:48:16) postgres=# \dV
Did not find any session variables.
Hi,
On Thu, Oct 13, 2022 at 07:41:32AM +0200, Pavel Stehule wrote:
>
> > I fixed the commit message of 0001 patch. Fixed shadowed variables too.
Thanks!
> >
> > There is a partially open issue, where I and Julien are not sure about a
> > solution, and we would like to ask for the community's opinion. I'll send
> > this query in separate mail.
> >
>
> rebased with simplified code related to usage of pfree function
If anyone is curious the discussion happend at [1].
I looked at the patchset, this time focusing on the LET command. Here at the
comments I have for now:
- gram.y
@@ -11918,6 +11920,7 @@ ExplainableStmt:
| CreateMatViewStmt
| RefreshMatViewStmt
| ExecuteStmt /* by default all are $$=$1 */
+ | LetStmt
;
(and other similar places) the comment should be kept to the last statement
Also, having LetStmt as an ExplainableStmt means it's allowed in a CTE:
cte_list:
common_table_expr { $$ = list_make1($1); }
| cte_list ',' common_table_expr { $$ = lappend($1, $3); }
;
common_table_expr: name opt_name_list AS opt_materialized '(' PreparableStmt ')' opt_search_clause opt_cycle_clause
And doing so hits this assert in transformWithClause:
if (!IsA(cte->ctequery, SelectStmt))
{
/* must be a data-modifying statement */
Assert(IsA(cte->ctequery, InsertStmt) ||
IsA(cte->ctequery, UpdateStmt) ||
IsA(cte->ctequery, DeleteStmt));
pstate->p_hasModifyingCTE = true;
}
and I'm assuming it would also fail on this in transformLetStmt:
+ /* There can't be any outer WITH to worry about */
+ Assert(pstate->p_ctenamespace == NIL);
I guess it makes sense to be able to explain a LetStmt (or using it in a
prepared statement), so it should be properly handled in transformSelectStmt.
Also, I don't see any test for a prepared LET statement, this should also be
covered.
- transformLetStmt:
+ varid = IdentifyVariable(names, &attrname, ¬_unique);
It would be nice to have a comment saying that the lock is acquired here
+ /* The grammar should have produced a SELECT */
+ if (!IsA(selectQuery, Query) ||
+ selectQuery->commandType != CMD_SELECT)
+ elog(ERROR, "unexpected non-SELECT command in LET command");
I'm wondering if this should be an Assert instead, as the grammar shouldn't
produce anything else no matter what how hard a user try.
+ /* don't allow multicolumn result */
+ if (list_length(exprList) != 1)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("expression is not scalar value"),
+ parser_errposition(pstate,
+ exprLocation((Node *) exprList))));
This isn't covered by any regression test and it probably should. It can be
reached with something like
LET myvar = (null::pg_class).*;
The error message could also use a bit of improvement.
I see that a_expr allows a select statement in parens, but this leads to a
sublink which already has all the required protection to gurantee a single
column, and a single row at most during execution. This one returns for
non-scalar case:
subquery must return only one column
Maybe use something similar for it, like "expression must return only one
column"? Similarly the error message in svariableStartupReceiver could be made
more consistent with the related errors:
+ if (++outcols > 1)
+ elog(ERROR, "svariable DestReceiver can take only one attribute");
While on svariableReceiver, I see that the current code assumes that the caller
did everything right. That's the case right now, but it should still be made
more robust in case future code (or extensions) is added. I'm thinking:
- svariableState.rows. Currently not really used, should check that one and
only one row is received in svariableReceiveSlot and
svariableShutdownReceiver (if no row is received the variable won't be reset
which should probably always happen once you setup an svariableReceiver)
- svariableState.typid, typmod and typlen should be double checked with the
given varid in svariableStartupReceiver.
- svariableState.varid should be initialized with InvalidOid to avoid undefined
behavior is caller forgets to set it.
I'm also wondering if SetVariableDestReceiverParams() should have an assert
like LockHeldByMe() for the given varid,
and maybe an assert that the varid is
a session variable, to avoid running a possibly expensive execution that will
fail when receiving the slot. I think the function would be better named
SetVariableDestReceiverVarid() or something like that.
+void
+ExecuteLetStmt(ParseState *pstate,
+ LetStmt *stmt,
+ ParamListInfo params,
+ QueryEnvironment *queryEnv,
+ QueryCompletion *qc)
+{
+ [...]
+ /* run the plan to completion */
+ ExecutorRun(queryDesc, ForwardScanDirection, 2L, true);
Why 2 rows? I'm assuming it's an attempt to detect queries that returns more
than 1 row, but it should be documented. Note that as mentioned above the dest
receiver currently doesn't check it, so this definitely needs to be fixed.
- IdentifyVariable:
*attrname can be set even is no variable is identified. I guess that's ok as
it avoids useless code, but it should probably be documented in the function
header.
Also, the API doesn't look ideal. AFAICS the only reason this function doesn't
error out in case of ambiguous name is that transformColumnRef may check if a
given name shadows a variable when session_variables_ambiguity_warning is set.
But since IdentifyVariable returns InvalidOid if the given list of identifiers
is ambiguous, it seems that the shadow detection can fail to detect a shadowed
reference if multiple variable would shadow the name:
# CREATE TYPE ab AS (a integer, b integer);
CREATE TYPE
# CREATE VARIABLE v_ab AS ab;
CREATE VARIABLE
# CREATE TABLE v_ab (a integer, b integer);
CREATE TABLE
# SET session_variables_ambiguity_warning = 1;
SET
# sELECT v_ab.a FROM v_ab;
WARNING: 42702: session variable "v_ab.a" is shadowed
LINE 1: select v_ab.a from v_ab;
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
a
---
(0 rows)
# CREATE SCHEMA v_ab;
CREATE SCHEMA
# CREATE VARIABLE v_ab.a AS integer;
CREATE VARIABLE
# SELECT v_ab.a FROM v_ab;
a
---
(0 rows)
Note that a bit later in transformColumnRef(), not_unique is checked only if
the returned varid is valid, which isn't correct as InvalidOid is currently
returned if not_unique is set.
I think that the error should be raised in IdentifyVariable rather than having
every caller check it. I'm not sure how to perfectly handle the
session_variables_ambiguity_warning though. Maybe make not_unique optional,
and error out if not_unique is null. If not null, set it as necessary and
return one of the oid. The only use would be for shadowing detection, and in
that case it won't be possible to check if a warning can be avoided as it would
be if no amgibuity is found, but that's probably ok.
Or maybe instead LookupVariable should have an extra argument to only match
variable with a composite type if caller asks to. This would avoid scenarios
like:
CREATE VARIABLE myvar AS int;
SELECT myvar.blabla;
ERROR: 42809: type integer is not composite
Is that really ok to match a variable here rather than complaining about a
missing FROM-clause?
+ indirection_start = list_length(names) - (attrname ? 1 : 0);
+ indirection = list_copy_tail(stmt->target, indirection_start);
+ [...]
+ if (indirection != NULL)
+ {
+ bool targetIsArray;
+ char *targetName;
+
+ targetName = get_session_variable_name(varid);
+ targetIsArray = OidIsValid(get_element_type(typid));
+
+ pstate->p_hasSessionVariables = true;
+
+ coerced_expr = (Expr *)
+ transformAssignmentIndirection(pstate,
+ (Node *) param,
+ targetName,
+ targetIsArray,
+ typid,
+ typmod,
+ InvalidOid,
+ indirection,
+ list_head(indirection),
+ (Node *) expr,
+ COERCION_PLPGSQL,
+ stmt->location);
+ }
I'm not sure why you use this approach rather than just having something like
"ListCell *indirection_head", set it to a non-NULL value when needed, and use
that (with names) instead. Note that it's also not correct to compare a List
to NULL, use NIL instead.
- expr_kind_allows_session_variables
Even if that's a bit annoying, I think it's better to explicitly put all values
there rather than having a default clause.
For instance, EXPR_KIND_CYCLE_MARK is currently allowing session variables,
which doesn't look ok. It's probably just an error from when the patchset was
rebased, but this probably wouldn't happen if you get an error for an unmatched
value if you add a new expr kind (which doesn't happen that often).
[1] https://www.postgresql.org/message-id/CAFj8pRB2+pVBFsidS-AzhHdZid40OTUspWfXS0vgahHmaWosZQ@mail.gmail.com
Вложения
- 0007-possibility-to-dump-session-variables-by-pg_dump.patch
- 0010-documentation.patch
- 0008-regress-tests-for-session-variables.patch
- 0006-enhancing-psql-for-session-variables.patch
- 0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- 0005-DISCARD-VARIABLES-command.patch
- 0004-support-of-LET-command-in-PLpgSQL.patch
- 0003-LET-command.patch
- 0002-session-variables.patch
- 0001-catalog-support-for-session-variables.patch
Also, the API doesn't look ideal. AFAICS the only reason this function doesn't
error out in case of ambiguous name is that transformColumnRef may check if a
given name shadows a variable when session_variables_ambiguity_warning is set.
But since IdentifyVariable returns InvalidOid if the given list of identifiers
is ambiguous, it seems that the shadow detection can fail to detect a shadowed
reference if multiple variable would shadow the name:
# CREATE TYPE ab AS (a integer, b integer);
CREATE TYPE
# CREATE VARIABLE v_ab AS ab;
CREATE VARIABLE
# CREATE TABLE v_ab (a integer, b integer);
CREATE TABLE
# SET session_variables_ambiguity_warning = 1;
SET
# sELECT v_ab.a FROM v_ab;
WARNING: 42702: session variable "v_ab.a" is shadowed
LINE 1: select v_ab.a from v_ab;
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
a
---
(0 rows)
# CREATE SCHEMA v_ab;
CREATE SCHEMA
# CREATE VARIABLE v_ab.a AS integer;
CREATE VARIABLE
# SELECT v_ab.a FROM v_ab;
a
---
(0 rows)
Note that a bit later in transformColumnRef(), not_unique is checked only if
the returned varid is valid, which isn't correct as InvalidOid is currently
returned if not_unique is set.
I think that the error should be raised in IdentifyVariable rather than having
every caller check it. I'm not sure how to perfectly handle the
session_variables_ambiguity_warning though. Maybe make not_unique optional,
and error out if not_unique is null. If not null, set it as necessary and
return one of the oid. The only use would be for shadowing detection, and in
that case it won't be possible to check if a warning can be avoided as it would
be if no amgibuity is found, but that's probably ok.
Or maybe instead LookupVariable should have an extra argument to only match
variable with a composite type if caller asks to. This would avoid scenarios
like:
CREATE VARIABLE myvar AS int;
SELECT myvar.blabla;
ERROR: 42809: type integer is not composite
Is that really ok to match a variable here rather than complaining about a
missing FROM-clause?
+ indirection_start = list_length(names) - (attrname ? 1 : 0);
+ indirection = list_copy_tail(stmt->target, indirection_start);
+ [...]
+ if (indirection != NULL)
+ {
+ bool targetIsArray;
+ char *targetName;
+
+ targetName = get_session_variable_name(varid);
+ targetIsArray = OidIsValid(get_element_type(typid));
+
+ pstate->p_hasSessionVariables = true;
+
+ coerced_expr = (Expr *)
+ transformAssignmentIndirection(pstate,
+ (Node *) param,
+ targetName,
+ targetIsArray,
+ typid,
+ typmod,
+ InvalidOid,
+ indirection,
+ list_head(indirection),
+ (Node *) expr,
+ COERCION_PLPGSQL,
+ stmt->location);
+ }
I'm not sure why you use this approach rather than just having something like
"ListCell *indirection_head", set it to a non-NULL value when needed, and use
that (with names) instead. Note that it's also not correct to compare a List
to NULL, use NIL instead.
- expr_kind_allows_session_variables
Even if that's a bit annoying, I think it's better to explicitly put all values
there rather than having a default clause.
For instance, EXPR_KIND_CYCLE_MARK is currently allowing session variables,
which doesn't look ok. It's probably just an error from when the patchset was
rebased, but this probably wouldn't happen if you get an error for an unmatched
value if you add a new expr kind (which doesn't happen that often).
[1] https://www.postgresql.org/message-id/CAFj8pRB2+pVBFsidS-AzhHdZid40OTUspWfXS0vgahHmaWosZQ@mail.gmail.com
Вложения
- v20221103-1-0008-regress-tests-for-session-variables.patch
- v20221103-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221103-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221103-1-0006-enhancing-psql-for-session-variables.patch
- v20221103-1-0010-documentation.patch
- v20221103-1-0005-DISCARD-VARIABLES-command.patch
- v20221103-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221103-1-0003-LET-command.patch
- v20221103-1-0002-session-variables.patch
- v20221103-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221104-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221104-1-0010-documentation.patch
- v20221104-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221104-1-0006-enhancing-psql-for-session-variables.patch
- v20221104-1-0008-regress-tests-for-session-variables.patch
- v20221104-1-0005-DISCARD-VARIABLES-command.patch
- v20221104-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221104-1-0003-LET-command.patch
- v20221104-1-0002-session-variables.patch
- v20221104-1-0001-catalog-support-for-session-variables.patch
> On Fri, Nov 04, 2022 at 05:58:06AM +0100, Pavel Stehule wrote: > Hi > > fix clang warning I've stumbled upon something that looks weird to me (inspired by the example from tests): =# create variable v2 as int; =# let v2 = 3; =# create view vv2 as select coalesce(v2, 0) + 1000 as result =# select * from vv2; result -------- 1003 =# set force_parallel_mode to on; =# select * from vv2; result -------- 1000 In the second select the actual work is done from a worker backend. Since values of session variables are stored in the backend local memory, it's not being shared with the worker and the value is not found in the hash map. Does this suppose to be like that?
Hi, On Fri, Nov 04, 2022 at 03:07:48PM +0100, Dmitry Dolgov wrote: > > On Fri, Nov 04, 2022 at 05:58:06AM +0100, Pavel Stehule wrote: > > Hi > > > > fix clang warning > > I've stumbled upon something that looks weird to me (inspired by the > example from tests): > > =# create variable v2 as int; > =# let v2 = 3; > =# create view vv2 as select coalesce(v2, 0) + 1000 as result > > =# select * from vv2; > result > -------- > 1003 > > =# set force_parallel_mode to on; > =# select * from vv2; > result > -------- > 1000 > > In the second select the actual work is done from a worker backend. > Since values of session variables are stored in the backend local > memory, it's not being shared with the worker and the value is not found > in the hash map. Does this suppose to be like that? There's code to serialize and restore all used variables for parallel workers (see code about PARAM_VARIABLE and queryDesc->num_session_variables / queryDesc->plannedstmt->sessionVariables). I haven't reviewed that part yet, but it's supposed to be working. Blind guess would be that it's missing something in expression walker.
> On Fri, Nov 04, 2022 at 05:58:06AM +0100, Pavel Stehule wrote:
> Hi
>
> fix clang warning
I've stumbled upon something that looks weird to me (inspired by the
example from tests):
=# create variable v2 as int;
=# let v2 = 3;
=# create view vv2 as select coalesce(v2, 0) + 1000 as result
=# select * from vv2;
result
--------
1003
=# set force_parallel_mode to on;
=# select * from vv2;
result
--------
1000
In the second select the actual work is done from a worker backend.
Since values of session variables are stored in the backend local
memory, it's not being shared with the worker and the value is not found
in the hash map. Does this suppose to be like that?
> On Fri, Nov 04, 2022 at 03:17:18PM +0100, Pavel Stehule wrote: > > I've stumbled upon something that looks weird to me (inspired by the > > example from tests): > > > > =# create variable v2 as int; > > =# let v2 = 3; > > =# create view vv2 as select coalesce(v2, 0) + 1000 as result > > > > =# select * from vv2; > > result > > -------- > > 1003 > > > > =# set force_parallel_mode to on; > > =# select * from vv2; > > result > > -------- > > 1000 > > > > In the second select the actual work is done from a worker backend. > > Since values of session variables are stored in the backend local > > memory, it's not being shared with the worker and the value is not found > > in the hash map. Does this suppose to be like that? > > > > It looks like a bug, but parallel queries should be supported. > > The value of the variable is passed as parameter to the worker backend. But > probably somewhere the original reference was not replaced by parameter > > On Fri, Nov 04, 2022 at 10:17:13PM +0800, Julien Rouhaud wrote: > Hi, > > There's code to serialize and restore all used variables for parallel workers > (see code about PARAM_VARIABLE and queryDesc->num_session_variables / > queryDesc->plannedstmt->sessionVariables). I haven't reviewed that part yet, > but it's supposed to be working. Blind guess would be that it's missing > something in expression walker. I see, thanks. I'll take a deeper look, my initial assumption was due to the fact that in the worker case create_sessionvars_hashtables is getting called for every query.
Hi, I did a quick initial review of this patch series - attached is a version with "review" commits for some of the parts. The current patch seems in pretty good shape, most of what I noticed are minor issues. I plan to do a more thorough review later. A quick overview of the issues: 0001 ---- - AtPreEOXact_SessionVariable_on_xact_actions name seems unnecessarily complicated and redundant, and mismatching nearby functions. Why not call it AtEOXact_SessionVariable, similar to AtEOXact_LargeObject? - some whitespace / ordering cleanup - I'm not sure why find_composite_type_dependencies needs the extra "else if" branch (instead of just doing "if" as before) - NamesFromList and IdentifyVariable seem introduced unnecessarily early, as they are only used in 0002 and 0003 parts (in the original patch series). Not sure if the plan is to squash everything into a single patch, or commit individual patches. - AFAIK patches don't need to modify typedefs.list. 0002 ---- - some whitespace / ordering cleanup - moving setting hasSessionVariables right after similar fields - SessionVariableCreatePostprocess prototype is redundant (2x) - I'd probably rename pg_debug_show_used_session_variables to pg_session_variables (assuming we want to keep this view) 0003 ---- - I'd rename svariableState to SVariableState, to keep the naming consistent with other similar/related typedefs. - some whitespace / ordering cleanup 0007 ---- - minor wording change Aside from that, I tried running this under valgrind, and that produces this report: ==250595== Conditional jump or move depends on uninitialised value(s) ==250595== at 0x731A48: sync_sessionvars_all (session_variable.c:513) ==250595== by 0x7321A6: prepare_variable_for_reading (session_variable.c:727) ==250595== by 0x7320BA: CopySessionVariable (session_variable.c:898) ==250595== by 0x7BC3BF: standard_ExecutorStart (execMain.c:252) ==250595== by 0x7BC042: ExecutorStart (execMain.c:146) ==250595== by 0xA89283: PortalStart (pquery.c:520) ==250595== by 0xA84E8D: exec_simple_query (postgres.c:1199) ==250595== by 0xA8425B: PostgresMain (postgres.c:4551) ==250595== by 0x998B03: BackendRun (postmaster.c:4482) ==250595== by 0x9980EC: BackendStartup (postmaster.c:4210) ==250595== by 0x996F0D: ServerLoop (postmaster.c:1804) ==250595== by 0x9948CA: PostmasterMain (postmaster.c:1476) ==250595== by 0x8526B6: main (main.c:197) ==250595== Uninitialised value was created by a heap allocation ==250595== at 0xCD86F0: MemoryContextAllocExtended (mcxt.c:1138) ==250595== by 0xC9FA1F: DynaHashAlloc (dynahash.c:292) ==250595== by 0xC9FEC1: element_alloc (dynahash.c:1715) ==250595== by 0xCA102A: get_hash_entry (dynahash.c:1324) ==250595== by 0xCA0879: hash_search_with_hash_value (dynahash.c:1097) ==250595== by 0xCA0432: hash_search (dynahash.c:958) ==250595== by 0x731614: SetSessionVariable (session_variable.c:846) ==250595== by 0x82FEED: svariableReceiveSlot (svariableReceiver.c:138) ==250595== by 0x7BD277: ExecutePlan (execMain.c:1726) ==250595== by 0x7BD0C5: standard_ExecutorRun (execMain.c:422) ==250595== by 0x7BCE63: ExecutorRun (execMain.c:366) ==250595== by 0x7332F0: ExecuteLetStmt (session_variable.c:1310) ==250595== by 0xA8CC15: standard_ProcessUtility (utility.c:1076) ==250595== by 0xA8BC72: ProcessUtility (utility.c:533) ==250595== by 0xA8B2B9: PortalRunUtility (pquery.c:1161) ==250595== by 0xA8A454: PortalRunMulti (pquery.c:1318) ==250595== by 0xA89A16: PortalRun (pquery.c:794) ==250595== by 0xA84F9E: exec_simple_query (postgres.c:1238) ==250595== by 0xA8425B: PostgresMain (postgres.c:4551) ==250595== by 0x998B03: BackendRun (postmaster.c:4482) ==250595== Which I think means this: if (filter_lxid && svar->drop_lxid == MyProc->lxid) continue; accesses drop_lxid, which was not initialized in init_session_variable. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Вложения
- 0001-catalog-support-for-session-variables-20221105.patch
- 0002-review-comments-20221105.patch
- 0003-session-variables-20221105.patch
- 0004-review-comments-20221105.patch
- 0005-LET-command-20221105.patch
- 0006-review-commnents-20221105.patch
- 0007-support-of-LET-command-in-PLpgSQL-20221105.patch
- 0008-DISCARD-VARIABLES-command-20221105.patch
- 0009-enhancing-psql-for-session-variables-20221105.patch
- 0010-possibility-to-dump-session-variables-by-pg-20221105.patch
- 0011-review-comments-20221105.patch
- 0012-regress-tests-for-session-variables-20221105.patch
- 0013-this-patch-changes-error-message-column-doe-20221105.patch
- 0014-documentation-20221105.patch
- 0015-review-comments-20221105.patch
Hi, On Sat, Nov 05, 2022 at 05:04:31PM +0100, Tomas Vondra wrote: > > I did a quick initial review of this patch series - attached is a > version with "review" commits for some of the parts. The current patch > seems in pretty good shape, most of what I noticed are minor issues. I > plan to do a more thorough review later. Thanks! I agree with all of your comments, just a few answers below > - NamesFromList and IdentifyVariable seem introduced unnecessarily > early, as they are only used in 0002 and 0003 parts (in the original > patch series). Not sure if the plan is to squash everything into a > single patch, or commit individual patches. The split was mostly done to make the patch easier to review, as it adds quite a bit of infrastructure. There have been some previous comments to have a more logical separation and fix similar issues, but there are still probably other oddities like that laying around. I personally didn't focus much on it as I don't know if the future committer will choose to squash everything or not. > - AFAIK patches don't need to modify typedefs.list. I think this was discussed a year or so ago, and my understanding is that the general rule is that it's now welcome, if not recommended, to maintain typedefs.list in each patchset. > Which I think means this: > > if (filter_lxid && svar->drop_lxid == MyProc->lxid) > continue; > > accesses drop_lxid, which was not initialized in init_session_variable. Agreed.
> On Fri, Nov 04, 2022 at 03:17:18PM +0100, Pavel Stehule wrote:
> > I've stumbled upon something that looks weird to me (inspired by the
> > example from tests):
> >
> > =# create variable v2 as int;
> > =# let v2 = 3;
> > =# create view vv2 as select coalesce(v2, 0) + 1000 as result
> >
> > =# select * from vv2;
> > result
> > --------
> > 1003
> >
> > =# set force_parallel_mode to on;
> > =# select * from vv2;
> > result
> > --------
> > 1000
> >
> > In the second select the actual work is done from a worker backend.
> > Since values of session variables are stored in the backend local
> > memory, it's not being shared with the worker and the value is not found
> > in the hash map. Does this suppose to be like that?
> >
>
> It looks like a bug, but parallel queries should be supported.
>
> The value of the variable is passed as parameter to the worker backend. But
> probably somewhere the original reference was not replaced by parameter
>
> On Fri, Nov 04, 2022 at 10:17:13PM +0800, Julien Rouhaud wrote:
> Hi,
>
> There's code to serialize and restore all used variables for parallel workers
> (see code about PARAM_VARIABLE and queryDesc->num_session_variables /
> queryDesc->plannedstmt->sessionVariables). I haven't reviewed that part yet,
> but it's supposed to be working. Blind guess would be that it's missing
> something in expression walker.
I see, thanks. I'll take a deeper look, my initial assumption was due to
the fact that in the worker case create_sessionvars_hashtables is
getting called for every query.
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -1275,6 +1275,9 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
/* If subquery had any RLS conditions, now main query does too */
parse->hasRowSecurity |= subquery->hasRowSecurity;
+ /* If subquery had session variables, now main query does too */
+ parse->hasSessionVariables |= subquery->hasSessionVariables;
+
Hi,
I did a quick initial review of this patch series - attached is a
version with "review" commits for some of the parts. The current patch
seems in pretty good shape, most of what I noticed are minor issues. I
plan to do a more thorough review later.
A quick overview of the issues:
0001
----
- AtPreEOXact_SessionVariable_on_xact_actions name seems unnecessarily
complicated and redundant, and mismatching nearby functions. Why not
call it AtEOXact_SessionVariable, similar to AtEOXact_LargeObject?
- some whitespace / ordering cleanup
- I'm not sure why find_composite_type_dependencies needs the extra
"else if" branch (instead of just doing "if" as before)
- NamesFromList and IdentifyVariable seem introduced unnecessarily
early, as they are only used in 0002 and 0003 parts (in the original
patch series). Not sure if the plan is to squash everything into a
single patch, or commit individual patches.
- AFAIK patches don't need to modify typedefs.list.
0002
----
- some whitespace / ordering cleanup
- moving setting hasSessionVariables right after similar fields
- SessionVariableCreatePostprocess prototype is redundant (2x)
- I'd probably rename pg_debug_show_used_session_variables to
pg_session_variables (assuming we want to keep this view)
0003
----
- I'd rename svariableState to SVariableState, to keep the naming
consistent with other similar/related typedefs.
- some whitespace / ordering cleanup
0007
----
- minor wording change
Aside from that, I tried running this under valgrind, and that produces
this report:
==250595== Conditional jump or move depends on uninitialised value(s)
==250595== at 0x731A48: sync_sessionvars_all (session_variable.c:513)
==250595== by 0x7321A6: prepare_variable_for_reading
(session_variable.c:727)
==250595== by 0x7320BA: CopySessionVariable (session_variable.c:898)
==250595== by 0x7BC3BF: standard_ExecutorStart (execMain.c:252)
==250595== by 0x7BC042: ExecutorStart (execMain.c:146)
==250595== by 0xA89283: PortalStart (pquery.c:520)
==250595== by 0xA84E8D: exec_simple_query (postgres.c:1199)
==250595== by 0xA8425B: PostgresMain (postgres.c:4551)
==250595== by 0x998B03: BackendRun (postmaster.c:4482)
==250595== by 0x9980EC: BackendStartup (postmaster.c:4210)
==250595== by 0x996F0D: ServerLoop (postmaster.c:1804)
==250595== by 0x9948CA: PostmasterMain (postmaster.c:1476)
==250595== by 0x8526B6: main (main.c:197)
==250595== Uninitialised value was created by a heap allocation
==250595== at 0xCD86F0: MemoryContextAllocExtended (mcxt.c:1138)
==250595== by 0xC9FA1F: DynaHashAlloc (dynahash.c:292)
==250595== by 0xC9FEC1: element_alloc (dynahash.c:1715)
==250595== by 0xCA102A: get_hash_entry (dynahash.c:1324)
==250595== by 0xCA0879: hash_search_with_hash_value (dynahash.c:1097)
==250595== by 0xCA0432: hash_search (dynahash.c:958)
==250595== by 0x731614: SetSessionVariable (session_variable.c:846)
==250595== by 0x82FEED: svariableReceiveSlot (svariableReceiver.c:138)
==250595== by 0x7BD277: ExecutePlan (execMain.c:1726)
==250595== by 0x7BD0C5: standard_ExecutorRun (execMain.c:422)
==250595== by 0x7BCE63: ExecutorRun (execMain.c:366)
==250595== by 0x7332F0: ExecuteLetStmt (session_variable.c:1310)
==250595== by 0xA8CC15: standard_ProcessUtility (utility.c:1076)
==250595== by 0xA8BC72: ProcessUtility (utility.c:533)
==250595== by 0xA8B2B9: PortalRunUtility (pquery.c:1161)
==250595== by 0xA8A454: PortalRunMulti (pquery.c:1318)
==250595== by 0xA89A16: PortalRun (pquery.c:794)
==250595== by 0xA84F9E: exec_simple_query (postgres.c:1238)
==250595== by 0xA8425B: PostgresMain (postgres.c:4551)
==250595== by 0x998B03: BackendRun (postmaster.c:4482)
==250595==
Which I think means this:
if (filter_lxid && svar->drop_lxid == MyProc->lxid)
continue;
accesses drop_lxid, which was not initialized in init_session_variable.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Вложения
- v20221113-1-0010-documentation.patch
- v20221113-1-0008-regress-tests-for-session-variables.patch
- v20221113-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221113-1-0006-enhancing-psql-for-session-variables.patch
- v20221113-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221113-1-0005-DISCARD-VARIABLES-command.patch
- v20221113-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221113-1-0003-LET-command.patch
- v20221113-1-0002-session-variables.patch
- v20221113-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221113-2-0010-documentation.patch
- v20221113-2-0006-enhancing-psql-for-session-variables.patch
- v20221113-2-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221113-2-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221113-2-0008-regress-tests-for-session-variables.patch
- v20221113-2-0005-DISCARD-VARIABLES-command.patch
- v20221113-2-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221113-2-0003-LET-command.patch
- v20221113-2-0002-session-variables.patch
- v20221113-2-0001-catalog-support-for-session-variables.patch
On 13.11.2022 20:59, Pavel Stehule wrote: > fresh rebase Hello, Sorry, I haven't been following this thread, but I'd like to report a memory management bug. I couldn't apply the latest patches, so I tested with v20221104-1-* patches applied atop of commit b0284bfb1db. postgres=# create variable s text default 'abc'; create function f() returns text as $$ begin return g(s); end; $$ language plpgsql; create function g(t text) returns text as $$ begin let s = 'BOOM!'; return t; end; $$ language plpgsql; select f(); CREATE VARIABLE CREATE FUNCTION CREATE FUNCTION server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. LOG: server process (PID 55307) was terminated by signal 11: Segmentation fault DETAIL: Failed process was running: select f(); I believe it's a use-after-free error, triggered by assigning a new value to s in g(), thus making t a dangling pointer. After reconnecting I get a scary error: postgres=# select f(); ERROR: compressed pglz data is corrupt Best regards, -- Sergey Shinderuk https://postgrespro.com/
On 13.11.2022 20:59, Pavel Stehule wrote:
> fresh rebase
Hello,
Sorry, I haven't been following this thread, but I'd like to report a
memory management bug. I couldn't apply the latest patches, so I tested
with v20221104-1-* patches applied atop of commit b0284bfb1db.
postgres=# create variable s text default 'abc';
create function f() returns text as $$
begin
return g(s);
end;
$$ language plpgsql;
create function g(t text) returns text as $$
begin
let s = 'BOOM!';
return t;
end;
$$ language plpgsql;
select f();
CREATE VARIABLE
CREATE FUNCTION
CREATE FUNCTION
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: server process (PID 55307) was terminated by signal 11:
Segmentation fault
DETAIL: Failed process was running: select f();
I believe it's a use-after-free error, triggered by assigning a new
value to s in g(), thus making t a dangling pointer.
After reconnecting I get a scary error:
postgres=# select f();
ERROR: compressed pglz data is corrupt
Best regards,
--
Sergey Shinderuk https://postgrespro.com/
On 13.11.2022 20:59, Pavel Stehule wrote:
> fresh rebase
Hello,
Sorry, I haven't been following this thread, but I'd like to report a
memory management bug. I couldn't apply the latest patches, so I tested
with v20221104-1-* patches applied atop of commit b0284bfb1db.
postgres=# create variable s text default 'abc';
create function f() returns text as $$
begin
return g(s);
end;
$$ language plpgsql;
create function g(t text) returns text as $$
begin
let s = 'BOOM!';
return t;
end;
$$ language plpgsql;
select f();
CREATE VARIABLE
CREATE FUNCTION
CREATE FUNCTION
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: server process (PID 55307) was terminated by signal 11:
Segmentation fault
DETAIL: Failed process was running: select f();
I believe it's a use-after-free error, triggered by assigning a new
value to s in g(), thus making t a dangling pointer.
After reconnecting I get a scary error:
postgres=# select f();
ERROR: compressed pglz data is corrupt
Best regards,
--
Sergey Shinderuk https://postgrespro.com/
Вложения
- v20221115-1-0010-documentation.patch
- v20221115-1-0008-regress-tests-for-session-variables.patch
- v20221115-1-0006-enhancing-psql-for-session-variables.patch
- v20221115-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221115-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221115-1-0005-DISCARD-VARIABLES-command.patch
- v20221115-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221115-1-0003-LET-command.patch
- v20221115-1-0002-session-variables.patch
- v20221115-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221118-1-0010-documentation.patch
- v20221118-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221118-1-0006-enhancing-psql-for-session-variables.patch
- v20221118-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221118-1-0008-regress-tests-for-session-variables.patch
- v20221118-1-0005-DISCARD-VARIABLES-command.patch
- v20221118-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221118-1-0003-LET-command.patch
- v20221118-1-0002-session-variables.patch
- v20221118-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221121-1-0010-documentation.patch
- v20221121-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221121-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221121-1-0006-enhancing-psql-for-session-variables.patch
- v20221121-1-0008-regress-tests-for-session-variables.patch
- v20221121-1-0005-DISCARD-VARIABLES-command.patch
- v20221121-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221121-1-0003-LET-command.patch
- v20221121-1-0002-session-variables.patch
- v20221121-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221124-1-0010-documentation.patch
- v20221124-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221124-1-0006-enhancing-psql-for-session-variables.patch
- v20221124-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221124-1-0008-regress-tests-for-session-variables.patch
- v20221124-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221124-1-0005-DISCARD-VARIABLES-command.patch
- v20221124-1-0002-session-variables.patch
- v20221124-1-0003-LET-command.patch
- v20221124-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221202-1-0010-documentation.patch
- v20221202-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221202-1-0008-regress-tests-for-session-variables.patch
- v20221202-1-0006-enhancing-psql-for-session-variables.patch
- v20221202-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221202-1-0005-DISCARD-VARIABLES-command.patch
- v20221202-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221202-1-0002-session-variables.patch
- v20221202-1-0003-LET-command.patch
- v20221202-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221206-1-0010-documentation.patch
- v20221206-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221206-1-0006-enhancing-psql-for-session-variables.patch
- v20221206-1-0008-regress-tests-for-session-variables.patch
- v20221206-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221206-1-0005-DISCARD-VARIABLES-command.patch
- v20221206-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221206-1-0003-LET-command.patch
- v20221206-1-0002-session-variables.patch
- v20221206-1-0001-catalog-support-for-session-variables.patch
Вложения
- v20221214-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221214-0008-regress-tests-for-session-variables.patch
- v20221214-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221214-0006-enhancing-psql-for-session-variables.patch
- v20221214-0010-documentation.patch
- v20221214-0005-DISCARD-VARIABLES-command.patch
- v20221214-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221214-0003-LET-command.patch
- v20221214-0002-session-variables.patch
- v20221214-0001-catalog-support-for-session-variables.patch
Op 14-12-2022 om 05:54 schreef Pavel Stehule: > Hi > > fresh rebase typo alert: v20221214-0003-LET-command.patch contains errmsg("target session varible is of type %s" ('varible' should be 'variable') Erik
Вложения
- v20221215-1-0008-regress-tests-for-session-variables.patch
- v20221215-1-0006-enhancing-psql-for-session-variables.patch
- v20221215-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221215-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221215-1-0010-documentation.patch
- v20221215-1-0005-DISCARD-VARIABLES-command.patch
- v20221215-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221215-1-0002-session-variables.patch
- v20221215-1-0003-LET-command.patch
- v20221215-1-0001-catalog-support-for-session-variables.patch
Op 14-12-2022 om 05:54 schreef Pavel Stehule:
> Hi
>
> fresh rebase
typo alert:
v20221214-0003-LET-command.patch contains
errmsg("target session varible is of type %s"
('varible' should be 'variable')
Erik
Вложения
- v20221215-2-0008-regress-tests-for-session-variables.patch
- v20221215-2-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221215-2-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221215-2-0006-enhancing-psql-for-session-variables.patch
- v20221215-2-0010-documentation.patch
- v20221215-2-0005-DISCARD-VARIABLES-command.patch
- v20221215-2-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221215-2-0003-LET-command.patch
- v20221215-2-0002-session-variables.patch
- v20221215-2-0001-catalog-support-for-session-variables.patch
Hi, I'm continuing review the patch slowly, and have one more issue plus one philosophical question. The issue have something to do with variables invalidation. Enabling debug_discard_caches = 1 (about which I've learned from this thread) and running this subset of the test suite: CREATE SCHEMA svartest; SET search_path = svartest; CREATE VARIABLE var3 AS int; CREATE OR REPLACE FUNCTION inc(int) RETURNS int AS $$ BEGIN LET svartest.var3 = COALESCE(svartest.var3 + $1, $1); RETURN var3; END; $$ LANGUAGE plpgsql; SELECT inc(1); SELECT inc(1); SELECT inc(1); crashes in my setup like this: #2 0x0000000000b432d4 in ExceptionalCondition (conditionName=0xce9b99 "n >= 0 && n < list->length", fileName=0xce9c18"list.c", lineNumber=770) at assert.c:66 #3 0x00000000007d3acd in list_delete_nth_cell (list=0x18ab248, n=-3388) at list.c:770 #4 0x00000000007d3b88 in list_delete_cell (list=0x18ab248, cell=0x18dc028) at list.c:842 #5 0x00000000006bcb52 in sync_sessionvars_all (filter_lxid=true) at session_variable.c:524 #6 0x00000000006bd4cb in SetSessionVariable (varid=16386, value=2, isNull=false) at session_variable.c:844 #7 0x00000000006bd617 in SetSessionVariableWithSecurityCheck (varid=16386, value=2, isNull=false) at session_variable.c:885 #8 0x00007f763b890698 in exec_stmt_let (estate=0x7ffcc6fd5190, stmt=0x18aa920) at pl_exec.c:5030 #9 0x00007f763b88a746 in exec_stmts (estate=0x7ffcc6fd5190, stmts=0x18aaaa0) at pl_exec.c:2116 #10 0x00007f763b88a247 in exec_stmt_block (estate=0x7ffcc6fd5190, block=0x18aabf8) at pl_exec.c:1935 #11 0x00007f763b889a49 in exec_toplevel_block (estate=0x7ffcc6fd5190, block=0x18aabf8) at pl_exec.c:1626 #12 0x00007f763b8879df in plpgsql_exec_function (func=0x18781b0, fcinfo=0x18be110, simple_eval_estate=0x0, simple_eval_resowner=0x0,procedure_resowner=0x0, atomic=true) at pl_exec.c:615 #13 0x00007f763b8a2320 in plpgsql_call_handler (fcinfo=0x18be110) at pl_handler.c:277 #14 0x0000000000721716 in ExecInterpExpr (state=0x18bde28, econtext=0x18bd3d0, isnull=0x7ffcc6fd56d7) at execExprInterp.c:730 #15 0x0000000000723642 in ExecInterpExprStillValid (state=0x18bde28, econtext=0x18bd3d0, isNull=0x7ffcc6fd56d7) at execExprInterp.c:1855 #16 0x000000000077a78b in ExecEvalExprSwitchContext (state=0x18bde28, econtext=0x18bd3d0, isNull=0x7ffcc6fd56d7) at ../../../src/include/executor/executor.h:344 #17 0x000000000077a7f4 in ExecProject (projInfo=0x18bde20) at ../../../src/include/executor/executor.h:378 #18 0x000000000077a9dc in ExecResult (pstate=0x18bd2c0) at nodeResult.c:136 #19 0x0000000000738ea0 in ExecProcNodeFirst (node=0x18bd2c0) at execProcnode.c:464 #20 0x000000000072c6e3 in ExecProcNode (node=0x18bd2c0) at ../../../src/include/executor/executor.h:262 #21 0x000000000072f426 in ExecutePlan (estate=0x18bd098, planstate=0x18bd2c0, use_parallel_mode=false, operation=CMD_SELECT,sendTuples=true, numberTuples=0, direction=ForwardScanDirection, dest=0x18b3eb8, execute_once=true)at execMain.c:1691 #22 0x000000000072cf76 in standard_ExecutorRun (queryDesc=0x189c688, direction=ForwardScanDirection, count=0, execute_once=true)at execMain.c:423 #23 0x000000000072cdb3 in ExecutorRun (queryDesc=0x189c688, direction=ForwardScanDirection, count=0, execute_once=true)at execMain.c:367 #24 0x000000000099afdc in PortalRunSelect (portal=0x1866648, forward=true, count=0, dest=0x18b3eb8) at pquery.c:927 #25 0x000000000099ac99 in PortalRun (portal=0x1866648, count=9223372036854775807, isTopLevel=true, run_once=true, dest=0x18b3eb8,altdest=0x18b3eb8, qc=0x7ffcc6fd5a70) at pquery.c:771 #26 0x000000000099487d in exec_simple_query (query_string=0x17edcc8 "SELECT inc(1);") at postgres.c:1238 It seems that sync_sessionvars_all tries to remove a cell that either doesn't belong to the xact_recheck_varids or weird in some other way: +>>> p l - xact_recheck_varids->elements $81 = -3388 The second thing I wanted to ask about is a more strategical question. Does anyone have clear understanding where this patch is going? The thread is quite large, and it's hard to catch up with all the details -- it would be great if someone could summarize the current state of things, are there any outstanding design issues or not addressed concerns? From the first look it seems some major topics the discussion is evolving are about: * Validity of the use case. Seems to be quite convincingly addressed in [1] and [2]. * Complicated logic around invalidation, concurrent create/drop etc. (I guess the issue above is falling into the same category). * Concerns that session variables could repeat some problems of temporary tables. Is there anything else? [1]: https://www.postgresql.org/message-id/CAFj8pRBT-bRQJBqkzon7tHcoFZ1byng06peZfZa0G72z46YFxg%40mail.gmail.com [2]: https://www.postgresql.org/message-id/flat/CAFj8pRBHSAHdainq5tRhN2Nns62h9-SZi0pvNq9DTe0VM6M1%3Dg%40mail.gmail.com#4faccb978d60e9b0b5d920e1a8a05bbf
Hi,
I'm continuing review the patch slowly, and have one more issue plus one
philosophical question.
The issue have something to do with variables invalidation. Enabling
debug_discard_caches = 1 (about which I've learned from this thread) and
running this subset of the test suite:
CREATE SCHEMA svartest;
SET search_path = svartest;
CREATE VARIABLE var3 AS int;
CREATE OR REPLACE FUNCTION inc(int)
RETURNS int AS $$
BEGIN
LET svartest.var3 = COALESCE(svartest.var3 + $1, $1);
RETURN var3;
END;
$$ LANGUAGE plpgsql;
SELECT inc(1);
SELECT inc(1);
SELECT inc(1);
crashes in my setup like this:
#2 0x0000000000b432d4 in ExceptionalCondition (conditionName=0xce9b99 "n >= 0 && n < list->length", fileName=0xce9c18 "list.c", lineNumber=770) at assert.c:66
#3 0x00000000007d3acd in list_delete_nth_cell (list=0x18ab248, n=-3388) at list.c:770
#4 0x00000000007d3b88 in list_delete_cell (list=0x18ab248, cell=0x18dc028) at list.c:842
#5 0x00000000006bcb52 in sync_sessionvars_all (filter_lxid=true) at session_variable.c:524
#6 0x00000000006bd4cb in SetSessionVariable (varid=16386, value=2, isNull=false) at session_variable.c:844
#7 0x00000000006bd617 in SetSessionVariableWithSecurityCheck (varid=16386, value=2, isNull=false) at session_variable.c:885
#8 0x00007f763b890698 in exec_stmt_let (estate=0x7ffcc6fd5190, stmt=0x18aa920) at pl_exec.c:5030
#9 0x00007f763b88a746 in exec_stmts (estate=0x7ffcc6fd5190, stmts=0x18aaaa0) at pl_exec.c:2116
#10 0x00007f763b88a247 in exec_stmt_block (estate=0x7ffcc6fd5190, block=0x18aabf8) at pl_exec.c:1935
#11 0x00007f763b889a49 in exec_toplevel_block (estate=0x7ffcc6fd5190, block=0x18aabf8) at pl_exec.c:1626
#12 0x00007f763b8879df in plpgsql_exec_function (func=0x18781b0, fcinfo=0x18be110, simple_eval_estate=0x0, simple_eval_resowner=0x0, procedure_resowner=0x0, atomic=true) at pl_exec.c:615
#13 0x00007f763b8a2320 in plpgsql_call_handler (fcinfo=0x18be110) at pl_handler.c:277
#14 0x0000000000721716 in ExecInterpExpr (state=0x18bde28, econtext=0x18bd3d0, isnull=0x7ffcc6fd56d7) at execExprInterp.c:730
#15 0x0000000000723642 in ExecInterpExprStillValid (state=0x18bde28, econtext=0x18bd3d0, isNull=0x7ffcc6fd56d7) at execExprInterp.c:1855
#16 0x000000000077a78b in ExecEvalExprSwitchContext (state=0x18bde28, econtext=0x18bd3d0, isNull=0x7ffcc6fd56d7) at ../../../src/include/executor/executor.h:344
#17 0x000000000077a7f4 in ExecProject (projInfo=0x18bde20) at ../../../src/include/executor/executor.h:378
#18 0x000000000077a9dc in ExecResult (pstate=0x18bd2c0) at nodeResult.c:136
#19 0x0000000000738ea0 in ExecProcNodeFirst (node=0x18bd2c0) at execProcnode.c:464
#20 0x000000000072c6e3 in ExecProcNode (node=0x18bd2c0) at ../../../src/include/executor/executor.h:262
#21 0x000000000072f426 in ExecutePlan (estate=0x18bd098, planstate=0x18bd2c0, use_parallel_mode=false, operation=CMD_SELECT, sendTuples=true, numberTuples=0, direction=ForwardScanDirection, dest=0x18b3eb8, execute_once=true) at execMain.c:1691
#22 0x000000000072cf76 in standard_ExecutorRun (queryDesc=0x189c688, direction=ForwardScanDirection, count=0, execute_once=true) at execMain.c:423
#23 0x000000000072cdb3 in ExecutorRun (queryDesc=0x189c688, direction=ForwardScanDirection, count=0, execute_once=true) at execMain.c:367
#24 0x000000000099afdc in PortalRunSelect (portal=0x1866648, forward=true, count=0, dest=0x18b3eb8) at pquery.c:927
#25 0x000000000099ac99 in PortalRun (portal=0x1866648, count=9223372036854775807, isTopLevel=true, run_once=true, dest=0x18b3eb8, altdest=0x18b3eb8, qc=0x7ffcc6fd5a70) at pquery.c:771
#26 0x000000000099487d in exec_simple_query (query_string=0x17edcc8 "SELECT inc(1);") at postgres.c:1238
It seems that sync_sessionvars_all tries to remove a cell that either doesn't
belong to the xact_recheck_varids or weird in some other way:
+>>> p l - xact_recheck_varids->elements
$81 = -3388
The second thing I wanted to ask about is a more strategical question. Does
anyone have clear understanding where this patch is going? The thread is quite
large, and it's hard to catch up with all the details -- it would be great if
someone could summarize the current state of things, are there any outstanding
design issues or not addressed concerns?
From the first look it seems some major topics the discussion is evolving are about:
* Validity of the use case. Seems to be quite convincingly addressed in [1] and
[2].
* Complicated logic around invalidation, concurrent create/drop etc. (I guess
the issue above is falling into the same category).
* Concerns that session variables could repeat some problems of temporary tables.
Is there anything else?
[1]: https://www.postgresql.org/message-id/CAFj8pRBT-bRQJBqkzon7tHcoFZ1byng06peZfZa0G72z46YFxg%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/flat/CAFj8pRBHSAHdainq5tRhN2Nns62h9-SZi0pvNq9DTe0VM6M1%3Dg%40mail.gmail.com#4faccb978d60e9b0b5d920e1a8a05bbf
> On Thu, Dec 22, 2022 at 08:45:57PM +0100, Pavel Stehule wrote: > > From the first look it seems some major topics the discussion is evolving > > are about: > > > > * Validity of the use case. Seems to be quite convincingly addressed in > > [1] and > > [2]. > > > > * Complicated logic around invalidation, concurrent create/drop etc. (I > > guess > > the issue above is falling into the same category). > > > > * Concerns that session variables could repeat some problems of temporary > > tables. > > > > Why do you think so? The variable has no mvcc support - it is just stored > value with local visibility without mvcc support. There can be little bit > similar issues like with global temporary tables. Yeah, sorry for not being precise, I mean global temporary tables. This is not my analysis, I've simply picked up it was mentioned a couple of times here. The points above are not meant to serve as an objection against the patch, but rather to figure out if there are any gaps left to address and come up with some sort of plan with "committed" as a final destination.
> On Thu, Dec 22, 2022 at 08:45:57PM +0100, Pavel Stehule wrote:
> > From the first look it seems some major topics the discussion is evolving
> > are about:
> >
> > * Validity of the use case. Seems to be quite convincingly addressed in
> > [1] and
> > [2].
> >
> > * Complicated logic around invalidation, concurrent create/drop etc. (I
> > guess
> > the issue above is falling into the same category).
> >
> > * Concerns that session variables could repeat some problems of temporary
> > tables.
> >
>
> Why do you think so? The variable has no mvcc support - it is just stored
> value with local visibility without mvcc support. There can be little bit
> similar issues like with global temporary tables.
Yeah, sorry for not being precise, I mean global temporary tables. This
is not my analysis, I've simply picked up it was mentioned a couple of
times here. The points above are not meant to serve as an objection
against the patch, but rather to figure out if there are any gaps left
to address and come up with some sort of plan with "committed" as a
final destination.
Вложения
- v20221223-1-0010-documentation.patch
- v20221223-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20221223-1-0008-regress-tests-for-session-variables.patch
- v20221223-1-0006-enhancing-psql-for-session-variables.patch
- v20221223-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20221223-1-0005-DISCARD-VARIABLES-command.patch
- v20221223-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20221223-1-0003-LET-command.patch
- v20221223-1-0002-session-variables.patch
- v20221223-1-0001-catalog-support-for-session-variables.patch
Hi, On Fri, Dec 23, 2022 at 08:38:43AM +0100, Pavel Stehule wrote: > > I am sending an updated patch, fixing the mentioned issue. Big thanks for > testing, and checking. There were multiple reviews in the last weeks which reported some issues, but unless I'm missing something I don't see any follow up from the reviewers on the changes? I will still wait a bit to see if they chime in while I keep looking at the diff since the last version of the code I reviewed. But in the meantime I already saw a couple of things that don't look right: --- a/src/backend/commands/dropcmds.c +++ b/src/backend/commands/dropcmds.c @@ -481,6 +481,11 @@ does_not_exist_skipping(ObjectType objtype, Node *object) msg = gettext_noop("publication \"%s\" does not exist, skipping"); name = strVal(object); break; + case OBJECT_VARIABLE: + msg = gettext_noop("session variable \"%s\" does not exist, skipping"); + name = NameListToString(castNode(List, object)); + break; + default: case OBJECT_COLUMN: the "default:" seems like a thinko during a rebase? +Datum +GetSessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid) +{ + SVariable svar; + + svar = prepare_variable_for_reading(varid); + Assert(svar != NULL && svar->is_valid); + + return CopySessionVariableWithTypeCheck(varid, isNull, expected_typid); + + if (expected_typid != svar->typid) + elog(ERROR, "type of variable \"%s.%s\" is different than expected", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)); + + *isNull = svar->isnull; + + return svar->value; +} there's a unconditional return in the middle of the function, which also looks like a thinko during a rebase since CopySessionVariableWithTypeCheck mostly contains the same following code. I'm also wondering if there should be additional tests based on the last scenario reported by Dmitry? (I don't see any new or similar test, but I may have missed it). > > > Why do you think so? The variable has no mvcc support - it is just stored > > > value with local visibility without mvcc support. There can be little bit > > > similar issues like with global temporary tables. > > > > Yeah, sorry for not being precise, I mean global temporary tables. This > > is not my analysis, I've simply picked up it was mentioned a couple of > > times here. The points above are not meant to serve as an objection > > against the patch, but rather to figure out if there are any gaps left > > to address and come up with some sort of plan with "committed" as a > > final destination. > > > > There are some similarities, but there are a lot of differences too. > Handling of metadata is partially similar, but session variable is almost > the value cached in session memory. It has no statistics, it is not stored > in a file. Because there is different storage, I don't think there is some > intersection on implementation level. +1
Hi,
On Fri, Dec 23, 2022 at 08:38:43AM +0100, Pavel Stehule wrote:
>
> I am sending an updated patch, fixing the mentioned issue. Big thanks for
> testing, and checking.
There were multiple reviews in the last weeks which reported some issues, but
unless I'm missing something I don't see any follow up from the reviewers on
the changes?
I will still wait a bit to see if they chime in while I keep looking at the
diff since the last version of the code I reviewed.
But in the meantime I already saw a couple of things that don't look right:
--- a/src/backend/commands/dropcmds.c
+++ b/src/backend/commands/dropcmds.c
@@ -481,6 +481,11 @@ does_not_exist_skipping(ObjectType objtype, Node *object)
msg = gettext_noop("publication \"%s\" does not exist, skipping");
name = strVal(object);
break;
+ case OBJECT_VARIABLE:
+ msg = gettext_noop("session variable \"%s\" does not exist, skipping");
+ name = NameListToString(castNode(List, object));
+ break;
+ default:
case OBJECT_COLUMN:
the "default:" seems like a thinko during a rebase?
+Datum
+GetSessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid)
+{
+ SVariable svar;
+
+ svar = prepare_variable_for_reading(varid);
+ Assert(svar != NULL && svar->is_valid);
+
+ return CopySessionVariableWithTypeCheck(varid, isNull, expected_typid);
+
+ if (expected_typid != svar->typid)
+ elog(ERROR, "type of variable \"%s.%s\" is different than expected",
+ get_namespace_name(get_session_variable_namespace(varid)),
+ get_session_variable_name(varid));
+
+ *isNull = svar->isnull;
+
+ return svar->value;
+}
there's a unconditional return in the middle of the function, which also looks
like a thinko during a rebase since CopySessionVariableWithTypeCheck mostly
contains the same following code.
I'm also wondering if there should be additional tests based on the last
scenario reported by Dmitry? (I don't see any new or similar test, but I may
have missed it).
> > > Why do you think so? The variable has no mvcc support - it is just stored
> > > value with local visibility without mvcc support. There can be little bit
> > > similar issues like with global temporary tables.
> >
> > Yeah, sorry for not being precise, I mean global temporary tables. This
> > is not my analysis, I've simply picked up it was mentioned a couple of
> > times here. The points above are not meant to serve as an objection
> > against the patch, but rather to figure out if there are any gaps left
> > to address and come up with some sort of plan with "committed" as a
> > final destination.
> >
>
> There are some similarities, but there are a lot of differences too.
> Handling of metadata is partially similar, but session variable is almost
> the value cached in session memory. It has no statistics, it is not stored
> in a file. Because there is different storage, I don't think there is some
> intersection on implementation level.
+1
Вложения
- v20230106-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230106-1-0008-regress-tests-for-session-variables.patch
- v20230106-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230106-1-0006-enhancing-psql-for-session-variables.patch
- v20230106-1-0010-documentation.patch
- v20230106-1-0005-DISCARD-VARIABLES-command.patch
- v20230106-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230106-1-0003-LET-command.patch
- v20230106-1-0002-session-variables.patch
- v20230106-1-0001-catalog-support-for-session-variables.patch
Hi, On Fri, Jan 06, 2023 at 08:02:41PM +0100, Pavel Stehule wrote: > pá 6. 1. 2023 v 5:11 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > > +Datum > > +GetSessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid > > expected_typid) > > +{ > > + SVariable svar; > > + > > + svar = prepare_variable_for_reading(varid); > > + Assert(svar != NULL && svar->is_valid); > > + > > + return CopySessionVariableWithTypeCheck(varid, isNull, > > expected_typid); > > + > > + if (expected_typid != svar->typid) > > + elog(ERROR, "type of variable \"%s.%s\" is different than > > expected", > > + > > get_namespace_name(get_session_variable_namespace(varid)), > > + get_session_variable_name(varid)); > > + > > + *isNull = svar->isnull; > > + > > + return svar->value; > > +} > > > > there's a unconditional return in the middle of the function, which also > > looks > > like a thinko during a rebase since CopySessionVariableWithTypeCheck mostly > > contains the same following code. > > > > This looks like my mistake - originally I fixed an issue related to the > usage session variable from plpgsql, and I forced a returned copy of the > session variable's value. Now, the function > GetSessionVariableWithTypeCheck is not used anyywhere. Oh I didn't check if it was used in the patchset. > It can be used only > from extensions, where is ensured so a) the value is not changed, b) and in > a lifetime of returned value is not called any query or any expression that > can change the value of this variable. I fixed this code and I enhanced > comments. I am not sure if this function should not be removed. It is not > used by backend, but it can be handy for extensions - it reduces possible > useless copy. Hmm, how safe is it for third-party code to access the stored data directly rather than a copy? If it makes extension fragile if they're not careful enough with cache invalidation, or even give them a way to mess up with the data directly, it's probably not a good idea to provide such an API. > > I'm also wondering if there should be additional tests based on the last > > scenario reported by Dmitry? (I don't see any new or similar test, but I > > may > > have missed it). > > > > The scenario reported by Dmitry is in tests already. Oh, so I missed it sorry about that. I did some testing using debug_discard_cache in the past and didn't run into this issue, so it's probably due to a more recent changes or before such a test was added. > I am not sure if I > have to repeat it with active debug_discard_cache. I expect this mode will > be activated in some testing environments. Yes, some buildfarm animal are configured to run with various debug_discard_caches setting so it's not needed to override it in some tests (it makes testing time really slow, which will be annoying for everyone including old/slow buildfarm animals). > I have no idea how to simply emulate this issue without > debug_discard_caches on 1. It is necessary to raise the sinval message > exactly when the variable is checked against system catalogue. Manually testing while setting locally debug_discard_cache should be enough. > updated patches attached Thanks!
> It can be used only
> from extensions, where is ensured so a) the value is not changed, b) and in
> a lifetime of returned value is not called any query or any expression that
> can change the value of this variable. I fixed this code and I enhanced
> comments. I am not sure if this function should not be removed. It is not
> used by backend, but it can be handy for extensions - it reduces possible
> useless copy.
Hmm, how safe is it for third-party code to access the stored data directly
rather than a copy? If it makes extension fragile if they're not careful
enough with cache invalidation, or even give them a way to mess up with the
data directly, it's probably not a good idea to provide such an API.
> > I'm also wondering if there should be additional tests based on the last
> > scenario reported by Dmitry? (I don't see any new or similar test, but I
> > may
> > have missed it).
> >
>
> The scenario reported by Dmitry is in tests already.
Oh, so I missed it sorry about that. I did some testing using
debug_discard_cache in the past and didn't run into this issue, so it's
probably due to a more recent changes or before such a test was added.
> I am not sure if I
> have to repeat it with active debug_discard_cache. I expect this mode will
> be activated in some testing environments.
Yes, some buildfarm animal are configured to run with various
debug_discard_caches setting so it's not needed to override it in some tests
(it makes testing time really slow, which will be annoying for everyone
including old/slow buildfarm animals).
> I have no idea how to simply emulate this issue without
> debug_discard_caches on 1. It is necessary to raise the sinval message
> exactly when the variable is checked against system catalogue.
Manually testing while setting locally debug_discard_cache should be enough.
> updated patches attached
Thanks!
Вложения
- v20230107-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230107-1-0010-documentation.patch
- v20230107-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230107-1-0008-regress-tests-for-session-variables.patch
- v20230107-1-0006-enhancing-psql-for-session-variables.patch
- v20230107-1-0005-DISCARD-VARIABLES-command.patch
- v20230107-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230107-1-0003-LET-command.patch
- v20230107-1-0002-session-variables.patch
- v20230107-1-0001-catalog-support-for-session-variables.patch
Hi, On Sat, Jan 07, 2023 at 08:09:27AM +0100, Pavel Stehule wrote: > > > > Hmm, how safe is it for third-party code to access the stored data directly > > rather than a copy? If it makes extension fragile if they're not careful > > enough with cache invalidation, or even give them a way to mess up with the > > data directly, it's probably not a good idea to provide such an API. > > > > ok, I removed it Another new behavior I see is the new rowtype_only parameter for LookupVariable. Has this been discussed? I can see how it can be annoying to get a "variable isn't composite" type of error when you already know that only a composite object can be used (and other might work), but it looks really scary to entirely ignore some objects that should be found in your search_path just because of their datatype. And if we ignore something like "a.b" if "a" isn't a variable of composite type, why wouldn't we apply the same "just ignore it" rule if it's indeed a composite type but doesn't have any "b" field? Your application could also start to use different object if your drop a say json variable and create a composite variable instead. It seems to be in contradiction with how the rest of the system works and looks wrong to me. Note also that LookupVariable can be quite expensive since you may have to do a lookup for every schema found in the search_path, so the sooner it stops the better. > > > updated patches attached I forgot to mention it last time but you should bump the copyright year for all new files added when you'll send a new version of the patchset.
Hi,
On Sat, Jan 07, 2023 at 08:09:27AM +0100, Pavel Stehule wrote:
> >
> > Hmm, how safe is it for third-party code to access the stored data directly
> > rather than a copy? If it makes extension fragile if they're not careful
> > enough with cache invalidation, or even give them a way to mess up with the
> > data directly, it's probably not a good idea to provide such an API.
> >
>
> ok, I removed it
Another new behavior I see is the new rowtype_only parameter for
LookupVariable. Has this been discussed?
SELECT varx.xxx;
-ERROR: missing FROM-clause entry for table "varx"
-LINE 1: SELECT varx.xxx;
- ^
+ERROR: type text is not composite
-- don't allow multi column query
CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int);
CREATE VARIABLE v1 AS vartesttp;
@@ -1421,9 +1419,7 @@
DROP TYPE ab;
CREATE VARIABLE myvar AS int;
SELECT myvar.blabla;
-ERROR: missing FROM-clause entry for table "myvar"
-LINE 1: SELECT myvar.blabla;
- ^
+ERROR: type integer is not composite
DROP VARIABLE myvar;
-- the result of view should be same in parallel mode too
CREATE VARIABLE v1 AS int;
I can see how it can be annoying to get a "variable isn't composite" type of
error when you already know that only a composite object can be used (and other
might work), but it looks really scary to entirely ignore some objects that
should be found in your search_path just because of their datatype.
And if we ignore something like "a.b" if "a" isn't a variable of composite
type, why wouldn't we apply the same "just ignore it" rule if it's indeed a
composite type but doesn't have any "b" field? Your application could also
start to use different object if your drop a say json variable and create a
composite variable instead.
It seems to be in contradiction with how the rest of the system works and looks
wrong to me. Note also that LookupVariable can be quite expensive since you
may have to do a lookup for every schema found in the search_path, so the
sooner it stops the better.
> > > updated patches attached
I forgot to mention it last time but you should bump the copyright year for all
new files added when you'll send a new version of the patchset.
Вложения
- v20230110-1-0010-documentation.patch
- v20230110-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230110-1-0006-enhancing-psql-for-session-variables.patch
- v20230110-1-0008-regress-tests-for-session-variables.patch
- v20230110-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230110-1-0005-DISCARD-VARIABLES-command.patch
- v20230110-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230110-1-0003-LET-command.patch
- v20230110-1-0002-session-variables.patch
- v20230110-1-0001-catalog-support-for-session-variables.patch
On Tue, Jan 10, 2023 at 08:35:16PM +0100, Pavel Stehule wrote: > út 10. 1. 2023 v 3:20 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal: > > > > Another new behavior I see is the new rowtype_only parameter for > > LookupVariable. Has this been discussed? > > > > I think so it was discussed about table shadowing > > without this filter, I lost the message "missing FROM-clause entry for ..." > > -- should fail > SELECT varx.xxx; > -ERROR: missing FROM-clause entry for table "varx" > -LINE 1: SELECT varx.xxx; > - ^ > +ERROR: type text is not composite > -- don't allow multi column query > CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int); > CREATE VARIABLE v1 AS vartesttp; > @@ -1421,9 +1419,7 @@ > DROP TYPE ab; > CREATE VARIABLE myvar AS int; > SELECT myvar.blabla; > -ERROR: missing FROM-clause entry for table "myvar" > -LINE 1: SELECT myvar.blabla; > - ^ > +ERROR: type integer is not composite > DROP VARIABLE myvar; > -- the result of view should be same in parallel mode too > CREATE VARIABLE v1 AS int; > > My original idea was to try to reduce possible conflicts (in old versions > of this path, a conflict was disallowed). But it is true, so these "new" > error messages are sensible too, and with eliminating rowtype_only I can > reduce code. Ok! Another problem is that the error message as-is is highly unhelpful as it's not clear at all that the problem is coming from an unsuitable variable. Maybe change makeParamSessionVariable to use lookup_rowtype_tupdesc_noerror() and emit a friendlier error message? Something like variable "X.Y" is of type Z, which is not composite > I modified the IdentifyVariable function a little bit. With new argument > noerror I am able to ensure so no error will be raised when this function > is called just for shadowing detection. I locally modified IdentifyVariable to emit WARNING reports when noerror is set to quickly see how it was used and didn't get any regression test error. This definitely needs to be covered by regression tests. Looking as session_variables.sql, the session_variables_ambiguity_warning GUC doesn't have a lot of tests in general.
On Tue, Jan 10, 2023 at 08:35:16PM +0100, Pavel Stehule wrote:
> út 10. 1. 2023 v 3:20 odesílatel Julien Rouhaud <rjuju123@gmail.com> napsal:
> >
> > Another new behavior I see is the new rowtype_only parameter for
> > LookupVariable. Has this been discussed?
> >
>
> I think so it was discussed about table shadowing
>
> without this filter, I lost the message "missing FROM-clause entry for ..."
>
> -- should fail
> SELECT varx.xxx;
> -ERROR: missing FROM-clause entry for table "varx"
> -LINE 1: SELECT varx.xxx;
> - ^
> +ERROR: type text is not composite
> -- don't allow multi column query
> CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int);
> CREATE VARIABLE v1 AS vartesttp;
> @@ -1421,9 +1419,7 @@
> DROP TYPE ab;
> CREATE VARIABLE myvar AS int;
> SELECT myvar.blabla;
> -ERROR: missing FROM-clause entry for table "myvar"
> -LINE 1: SELECT myvar.blabla;
> - ^
> +ERROR: type integer is not composite
> DROP VARIABLE myvar;
> -- the result of view should be same in parallel mode too
> CREATE VARIABLE v1 AS int;
>
> My original idea was to try to reduce possible conflicts (in old versions
> of this path, a conflict was disallowed). But it is true, so these "new"
> error messages are sensible too, and with eliminating rowtype_only I can
> reduce code.
Ok! Another problem is that the error message as-is is highly unhelpful as
it's not clear at all that the problem is coming from an unsuitable variable.
Maybe change makeParamSessionVariable to use lookup_rowtype_tupdesc_noerror()
and emit a friendlier error message? Something like
variable "X.Y" is of type Z, which is not composite
> I modified the IdentifyVariable function a little bit. With new argument
> noerror I am able to ensure so no error will be raised when this function
> is called just for shadowing detection.
I locally modified IdentifyVariable to emit WARNING reports when noerror is set
to quickly see how it was used and didn't get any regression test error. This
definitely needs to be covered by regression tests. Looking as
session_variables.sql, the session_variables_ambiguity_warning GUC doesn't have
a lot of tests in general.
Вложения
- v20230116-1-0010-documentation.patch
- v20230116-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230116-1-0006-enhancing-psql-for-session-variables.patch
- v20230116-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230116-1-0008-regress-tests-for-session-variables.patch
- v20230116-1-0005-DISCARD-VARIABLES-command.patch
- v20230116-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230116-1-0003-LET-command.patch
- v20230116-1-0002-session-variables.patch
- v20230116-1-0001-catalog-support-for-session-variables.patch
I've accumulated another collection of various questions and comments. As a side note I'm getting a good feeling about this patch, those part I've read so far looks good to me. * I've suddenly realized one could use pseudo types for variables, and it not always works. E.g.: =# create variable pseudo_array anyarray; =# select pseudo_array; pseudo_array -------------- NULL =# let pseudo_array = ARRAY[1, 2, 3]; ERROR: 42804: target session variable is of type anyarray but expression is of type integer[] LOCATION: svariableStartupReceiver, svariableReceiver.c:79 =# create variable pseudo_unknown unknown; =# select pseudo_unknown; ERROR: XX000: failed to find conversion function from unknown to text LOCATION: coerce_type, parse_coerce.c:542 Is it supposed to be like this, or something is missing? * I think it was already mentioned in the thread, there seems to be not a single usage of CHECK_FOR_INTERRUPTS in session_variable.c . But some number of loops over the sessionvars are implemented, are they always going to be small enough to not make any troubles? * sync_sessionvars_all explains why is it necessary to copy xact_recheck_varids: When we check the variables, the system cache can be invalidated, and xact_recheck_varids can be enhanced. I'm not quite following what the "enhancement" part is about? Is xact_recheck_varids could be somehow updated concurrently with the loop? * A small typo diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c --- a/src/backend/commands/session_variable.c +++ b/src/backend/commands/session_variable.c @@ -485,7 +485,7 @@ sync_sessionvars_all(bool filter_lxid) /* * When we check the variables, the system cache can be invalidated, - * and xac_recheck_varids can be enhanced. We want to iterate + * and xact_recheck_varids can be enhanced. We want to iterate NOTE: The commentaries above were made based on the previous patch version, but it looks like those aspects were not changed.
I've accumulated another collection of various questions and comments. As a
side note I'm getting a good feeling about this patch, those part I've read so
far looks good to me.
* I've suddenly realized one could use pseudo types for variables, and
it not always works. E.g.:
=# create variable pseudo_array anyarray;
=# select pseudo_array;
pseudo_array
--------------
NULL
=# let pseudo_array = ARRAY[1, 2, 3];
ERROR: 42804: target session variable is of type anyarray but expression is of type integer[]
LOCATION: svariableStartupReceiver, svariableReceiver.c:79
=# create variable pseudo_unknown unknown;
=# select pseudo_unknown;
ERROR: XX000: failed to find conversion function from unknown to text
LOCATION: coerce_type, parse_coerce.c:542
Is it supposed to be like this, or something is missing?
* I think it was already mentioned in the thread, there seems to be not a
single usage of CHECK_FOR_INTERRUPTS in session_variable.c . But some number
of loops over the sessionvars are implemented, are they always going to be
small enough to not make any troubles?
* sync_sessionvars_all explains why is it necessary to copy xact_recheck_varids:
When we check the variables, the system cache can be invalidated,
and xact_recheck_varids can be enhanced.
I'm not quite following what the "enhancement" part is about? Is
xact_recheck_varids could be somehow updated concurrently with the loop?
* A small typo
diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c
--- a/src/backend/commands/session_variable.c
+++ b/src/backend/commands/session_variable.c
@@ -485,7 +485,7 @@ sync_sessionvars_all(bool filter_lxid)
/*
* When we check the variables, the system cache can be invalidated,
- * and xac_recheck_varids can be enhanced. We want to iterate
+ * and xact_recheck_varids can be enhanced. We want to iterate
NOTE: The commentaries above were made based on the previous patch version, but
it looks like those aspects were not changed.
Вложения
- v20230122-1-0006-enhancing-psql-for-session-variables.patch
- v20230122-1-0010-documentation.patch
- v20230122-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230122-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230122-1-0008-regress-tests-for-session-variables.patch
- v20230122-1-0005-DISCARD-VARIABLES-command.patch
- v20230122-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230122-1-0003-LET-command.patch
- v20230122-1-0002-session-variables.patch
- v20230122-1-0001-catalog-support-for-session-variables.patch
> On Sun, Jan 22, 2023 at 07:47:07PM +0100, Pavel Stehule wrote: > pá 20. 1. 2023 v 21:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> > napsal: > > > * I think it was already mentioned in the thread, there seems to be not a > > single usage of CHECK_FOR_INTERRUPTS in session_variable.c . But some > > number > > of loops over the sessionvars are implemented, are they always going to > > be > > small enough to not make any troubles? > > > > The longest cycle is a cycle that rechecks actively used variables against > system catalog. No cycle depends on the content of variables. Right, but what about the cases with huge number of variables? Not saying it could be in any sense common, but possible to do. > > * sync_sessionvars_all explains why is it necessary to copy > > xact_recheck_varids: > > > > When we check the variables, the system cache can be > > invalidated, > > and xact_recheck_varids can be enhanced. > > > > I'm not quite following what the "enhancement" part is about? Is > > xact_recheck_varids could be somehow updated concurrently with the loop? > > > > Yes. pg_variable_cache_callback can be called when > is_session_variable_valid is executed. > > Maybe "extended" can be a better word instead of "enhanced"? I reformulated > this comment Yeah, "extended" sounds better. But I was mostly confused about the mechanism, if the cache callback can interrupt the execution at any moment when called, that would explain it.
Вложения
- v20230123-1-0006-enhancing-psql-for-session-variables.patch
- v20230123-1-0010-documentation.patch
- v20230123-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230123-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230123-1-0008-regress-tests-for-session-variables.patch
- v20230123-1-0005-DISCARD-VARIABLES-command.patch
- v20230123-1-0003-LET-command.patch
- v20230123-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230123-1-0002-session-variables.patch
- v20230123-1-0001-catalog-support-for-session-variables.patch
> On Sun, Jan 22, 2023 at 07:47:07PM +0100, Pavel Stehule wrote:
> pá 20. 1. 2023 v 21:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
> > * I think it was already mentioned in the thread, there seems to be not a
> > single usage of CHECK_FOR_INTERRUPTS in session_variable.c . But some
> > number
> > of loops over the sessionvars are implemented, are they always going to
> > be
> > small enough to not make any troubles?
> >
>
> The longest cycle is a cycle that rechecks actively used variables against
> system catalog. No cycle depends on the content of variables.
Right, but what about the cases with huge number of variables? Not
saying it could be in any sense common, but possible to do.
> > * sync_sessionvars_all explains why is it necessary to copy
> > xact_recheck_varids:
> >
> > When we check the variables, the system cache can be
> > invalidated,
> > and xact_recheck_varids can be enhanced.
> >
> > I'm not quite following what the "enhancement" part is about? Is
> > xact_recheck_varids could be somehow updated concurrently with the loop?
> >
>
> Yes. pg_variable_cache_callback can be called when
> is_session_variable_valid is executed.
>
> Maybe "extended" can be a better word instead of "enhanced"? I reformulated
> this comment
Yeah, "extended" sounds better. But I was mostly confused about the
mechanism, if the cache callback can interrupt the execution at any
moment when called, that would explain it.
Вложения
- v20230123-2-0010-documentation.patch
- v20230123-2-0008-regress-tests-for-session-variables.patch
- v20230123-2-0006-enhancing-psql-for-session-variables.patch
- v20230123-2-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230123-2-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230123-2-0005-DISCARD-VARIABLES-command.patch
- v20230123-2-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230123-2-0003-LET-command.patch
- v20230123-2-0002-session-variables.patch
- v20230123-2-0001-catalog-support-for-session-variables.patch
> On Mon, Jan 23, 2023 at 07:09:27PM +0100, Pavel Stehule wrote: > po 23. 1. 2023 v 15:25 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> > napsal: > > > > On Sun, Jan 22, 2023 at 07:47:07PM +0100, Pavel Stehule wrote: > > > pá 20. 1. 2023 v 21:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> > > > napsal: > > > > > > > * I think it was already mentioned in the thread, there seems to be > > not a > > > > single usage of CHECK_FOR_INTERRUPTS in session_variable.c . But some > > > > number > > > > of loops over the sessionvars are implemented, are they always going > > to > > > > be > > > > small enough to not make any troubles? > > > > > > > > > > The longest cycle is a cycle that rechecks actively used variables > > against > > > system catalog. No cycle depends on the content of variables. > > > > Right, but what about the cases with huge number of variables? Not > > saying it could be in any sense common, but possible to do. > > > > Now I tested 10K variables (with enabled assertions, without it is should > be faster) > > [...] > > I can be wrong, but from these numbers I don't think so these sync cycles > should to contain CHECK_FOR_INTERRUPTS > > What do you think? Well, there is always possibility someone will create more variables than any arbitrary limit we have tested for. But I see your point and don't have a strong opinion about this, so let's keep it as it is :) > > > > * sync_sessionvars_all explains why is it necessary to copy > > > > xact_recheck_varids: > > > > > > > > When we check the variables, the system cache can be > > > > invalidated, > > > > and xact_recheck_varids can be enhanced. > > > > > > > > I'm not quite following what the "enhancement" part is about? Is > > > > xact_recheck_varids could be somehow updated concurrently with the > > loop? > > > > > > > > > > Yes. pg_variable_cache_callback can be called when > > > is_session_variable_valid is executed. > > > > > > Maybe "extended" can be a better word instead of "enhanced"? I > > reformulated > > > this comment > > > > Yeah, "extended" sounds better. But I was mostly confused about the > > mechanism, if the cache callback can interrupt the execution at any > > moment when called, that would explain it. > > > > patch from yesterday has extended comment in this area :-) Thanks!
>
> I can be wrong, but from these numbers I don't think so these sync cycles
> should to contain CHECK_FOR_INTERRUPTS
>
> What do you think?
Well, there is always possibility someone will create more variables
than any arbitrary limit we have tested for. But I see your point and
don't have a strong opinion about this, so let's keep it as it is :)
Вложения
- v20230202-1-0010-documentation.patch
- v20230202-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230202-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230202-1-0006-enhancing-psql-for-session-variables.patch
- v20230202-1-0008-regress-tests-for-session-variables.patch
- v20230202-1-0005-DISCARD-VARIABLES-command.patch
- v20230202-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230202-1-0001-catalog-support-for-session-variables.patch
- v20230202-1-0002-session-variables.patch
- v20230202-1-0003-LET-command.patch
begin
for i in 1..10000 loop
let xx = xx + 1;
end loop;
end;
$$;
DO
Time: 4,079 ms
delete from xx01; vacuum full xx01; vacuum;
begin
for i in 1..10000 loop
update xx01 set a = a + 1;
end loop;
end;
$$;
NOTICE: 10001
DO
Time: 1678,949 ms (00:01,679)
List of relations
┌───────────┬──────┬───────┬───────┬─────────────┬───────────────┬────────┬─────────────┐
│ Schema │ Name │ Type │ Owner │ Persistence │ Access method │ Size │ Description │
╞═══════════╪══════╪═══════╪═══════╪═════════════╪═══════════════╪════════╪═════════════╡
│ pg_temp_3 │ xx01 │ table │ pavel │ temporary │ heap │ 384 kB │ │
└───────────┴──────┴───────┴───────┴─────────────┴───────────────┴────────┴─────────────┘
(1 row)
begin
for i in 1..10000 loop
perform set_config('cust.xx', (current_setting('cust.xx')::int + 1)::text, true);
end loop;
end;
$$;
NOTICE: 10001
DO
Time: 80,201 ms
Вложения
- v20230206-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230206-0008-regress-tests-for-session-variables.patch
- v20230206-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230206-0006-enhancing-psql-for-session-variables.patch
- v20230206-0010-documentation.patch
- v20230206-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230206-0005-DISCARD-VARIABLES-command.patch
- v20230206-0003-LET-command.patch
- v20230206-0002-session-variables.patch
- v20230206-0001-catalog-support-for-session-variables.patch
Вложения
- v20230228-1-0010-documentation.patch
- v20230228-1-0008-regress-tests-for-session-variables.patch
- v20230228-1-0006-enhancing-psql-for-session-variables.patch
- v20230228-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230228-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230228-1-0005-DISCARD-VARIABLES-command.patch
- v20230228-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230228-1-0003-LET-command.patch
- v20230228-1-0002-session-variables.patch
- v20230228-1-0001-catalog-support-for-session-variables.patch
> On Tue, Feb 28, 2023 at 06:12:50AM +0100, Pavel Stehule wrote: > > fresh rebase I'm continuing to review, this time going through shadowing stuff in transformColumnRef, IdentifyVariable etc. Well, that's a lot of leg work for rather little outcome :) I guess all attempts to simplify this part weren't successful? Couple of questions to it. In IdentifyVariable in the branch handling two values the commentary says: /* * a.b can mean "schema"."variable" or "variable"."field", * Check both variants, and returns InvalidOid with * not_unique flag, when both interpretations are * possible. Second node can be star. In this case, the * only allowed possibility is "variable"."*". */ I read this as "variable"."*" is a valid combination, but the very next part of this condition says differently: /* * Session variables doesn't support unboxing by star * syntax. But this syntax have to be calculated here, * because can come from non session variables related * expressions. */ Assert(IsA(field2, A_Star)); Is the first commentary not quite correct? Another question about how shadowing warning should work between namespaces. Let's say I've got two namespaces, public and test, both have a session variable with the same name, but only one has a table with the same name: -- in public create table test_agg(a int); create type for_test_agg as (a int); create variable test_agg for_test_agg; -- in test create type for_test_agg as (a int); create variable test_agg for_test_agg; Now if we will try to trigger the shadowing warning from public namespace, it would work differently: -- in public =# let test.test_agg.a = 10; =# let test_agg.a = 20; =# set session_variables_ambiguity_warning to on; -- note the value returned from the table =# select jsonb_agg(test_agg.a) from test_agg; WARNING: 42702: session variable "test_agg.a" is shadowed LINE 1: select jsonb_agg(test_agg.a) from test_agg; ^ DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. LOCATION: transformColumnRef, parse_expr.c:940 jsonb_agg ----------- [1] -- no warning, note the session variable value =# select jsonb_agg(test.test_agg.a) from test_agg; jsonb_agg ----------- [10] It happens because in the second scenario the logic inside transformColumnRef will not set up the node variable (there is no corresponding table in the "test" schema), and the following conditions covering session variables shadowing are depending on it. Is it supposed to be like this?
> On Tue, Feb 28, 2023 at 06:12:50AM +0100, Pavel Stehule wrote:
>
> fresh rebase
I'm continuing to review, this time going through shadowing stuff in
transformColumnRef, IdentifyVariable etc. Well, that's a lot of leg work
for rather little outcome :) I guess all attempts to simplify this part
weren't successful?
Couple of questions to it. In IdentifyVariable in the branch handling
two values the commentary says:
/*
* a.b can mean "schema"."variable" or "variable"."field",
* Check both variants, and returns InvalidOid with
* not_unique flag, when both interpretations are
* possible. Second node can be star. In this case, the
* only allowed possibility is "variable"."*".
*/
I read this as "variable"."*" is a valid combination, but the very next
part of this condition says differently:
/*
* Session variables doesn't support unboxing by star
* syntax. But this syntax have to be calculated here,
* because can come from non session variables related
* expressions.
*/
Assert(IsA(field2, A_Star));
Is the first commentary not quite correct?
the only allowed possibility is "variable"."*"." should be in the next comment.
Another question about how shadowing warning should work between namespaces.
Let's say I've got two namespaces, public and test, both have a session
variable with the same name, but only one has a table with the same name:
-- in public
create table test_agg(a int);
create type for_test_agg as (a int);
create variable test_agg for_test_agg;
-- in test
create type for_test_agg as (a int);
create variable test_agg for_test_agg;
Now if we will try to trigger the shadowing warning from public
namespace, it would work differently:
-- in public
=# let test.test_agg.a = 10;
=# let test_agg.a = 20;
=# set session_variables_ambiguity_warning to on;
-- note the value returned from the table
=# select jsonb_agg(test_agg.a) from test_agg;
WARNING: 42702: session variable "test_agg.a" is shadowed
LINE 1: select jsonb_agg(test_agg.a) from test_agg;
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
LOCATION: transformColumnRef, parse_expr.c:940
jsonb_agg
-----------
[1]
-- no warning, note the session variable value
=# select jsonb_agg(test.test_agg.a) from test_agg;
jsonb_agg
-----------
[10]
It happens because in the second scenario the logic inside transformColumnRef
will not set up the node variable (there is no corresponding table in the
"test" schema), and the following conditions covering session variables
shadowing are depending on it. Is it supposed to be like this?
Вложения
- v20230308-0008-regress-tests-for-session-variables.patch
- v20230308-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230308-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230308-0010-documentation.patch
- v20230308-0006-enhancing-psql-for-session-variables.patch
- v20230308-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230308-0005-DISCARD-VARIABLES-command.patch
- v20230308-0003-LET-command.patch
- v20230308-0001-catalog-support-for-session-variables.patch
- v20230308-0002-session-variables.patch
> On Wed, Mar 08, 2023 at 08:31:07AM +0100, Pavel Stehule wrote: > pá 3. 3. 2023 v 21:19 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> > napsal: > > > > On Tue, Feb 28, 2023 at 06:12:50AM +0100, Pavel Stehule wrote: > > > > > > fresh rebase > > > > I'm continuing to review, this time going through shadowing stuff in > > transformColumnRef, IdentifyVariable etc. Well, that's a lot of leg work > > for rather little outcome :) I guess all attempts to simplify this part > > weren't successful? > > > > Originally I wrote it in the strategy "reduce false alarms". But when I > think about it, it may not be good in this case. Usually the changes are > done first on some developer environment, and good practice is to disallow > same (possibly confusing) identifiers. So I am not against making this > warning more aggressive with some possibility of false alarms. With > blocking reduction of alarms the differences in regress was zero. So I > reduced this part. Great, thank you. > > Couple of questions to it. In IdentifyVariable in the branch handling > > two values the commentary says: > > > > /* > > * a.b can mean "schema"."variable" or "variable"."field", > > * Check both variants, and returns InvalidOid with > > * not_unique flag, when both interpretations are > > * possible. Second node can be star. In this case, the > > * only allowed possibility is "variable"."*". > > */ > > > > I read this as "variable"."*" is a valid combination, but the very next > > part of this condition says differently: > > > > > > > > > /* > > * Session variables doesn't support unboxing by star > > * syntax. But this syntax have to be calculated here, > > * because can come from non session variables related > > * expressions. > > */ > > Assert(IsA(field2, A_Star)); > > > > Is the first commentary not quite correct? > > > > I think it is correct, but maybe I was not good at describing this issue. > The sentence "Second node can be a star. In this case, the > the only allowed possibility is "variable"."*"." should be in the next > comment. > > In this part we process a list of identifiers, and we try to map these > identifiers to some semantics. The parser should ensure that > all fields of lists are strings or the last field is a star. In this case > the semantic "schema".* is nonsense, and the only possible semantic > is "variable".*. It is valid semantics, but unsupported now. Unboxing is > available by syntax (var).* > > I changed the comment Thanks. Just to clarify, by "unsupported" you mean unsupported in the current patch implementation right? From what I understand value unboxing could be done without parentheses in a non-top level select query. As a side note, I'm not sure if this branch is exercised in any tests. I've replaced returning InvalidOid with actually doing LookupVariable(NULL, a, true) in this case, and all the tests are still passing. > > Another question about how shadowing warning should work between > > namespaces. > > Let's say I've got two namespaces, public and test, both have a session > > variable with the same name, but only one has a table with the same name: > > > > -- in public > > create table test_agg(a int); > > create type for_test_agg as (a int); > > create variable test_agg for_test_agg; > > > > -- in test > > create type for_test_agg as (a int); > > create variable test_agg for_test_agg; > > > > Now if we will try to trigger the shadowing warning from public > > namespace, it would work differently: > > > > -- in public > > =# let test.test_agg.a = 10; > > =# let test_agg.a = 20; > > =# set session_variables_ambiguity_warning to on; > > > > -- note the value returned from the table > > =# select jsonb_agg(test_agg.a) from test_agg; > > WARNING: 42702: session variable "test_agg.a" is shadowed > > LINE 1: select jsonb_agg(test_agg.a) from test_agg; > > ^ > > DETAIL: Session variables can be shadowed by columns, routine's > > variables and routine's arguments with the same name. > > LOCATION: transformColumnRef, parse_expr.c:940 > > jsonb_agg > > ----------- > > [1] > > > > -- no warning, note the session variable value > > =# select jsonb_agg(test.test_agg.a) from test_agg; > > jsonb_agg > > ----------- > > [10] > > > > It happens because in the second scenario the logic inside > > transformColumnRef > > will not set up the node variable (there is no corresponding table in the > > "test" schema), and the following conditions covering session variables > > shadowing are depending on it. Is it supposed to be like this? > > > > I am sorry, I don't understand what you want to describe. Session variables > are shadowed by relations, ever. It is design. In the first case, the > variable is shadowed and a warning is raised. In the second case, > "test"."test_agg"."a" is a fully unique qualified identifier, and then the > variable is used, and then it is not shadowed. Yeah, there was a misunderstanding on my side, sorry. For whatever reason I thought shadowing between schemas is a reasonable thing, but as you pointed out it doesn't really make sense.
> On Wed, Mar 08, 2023 at 08:31:07AM +0100, Pavel Stehule wrote:
> pá 3. 3. 2023 v 21:19 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
> > > On Tue, Feb 28, 2023 at 06:12:50AM +0100, Pavel Stehule wrote:
> > >
> > > fresh rebase
> >
> > I'm continuing to review, this time going through shadowing stuff in
> > transformColumnRef, IdentifyVariable etc. Well, that's a lot of leg work
> > for rather little outcome :) I guess all attempts to simplify this part
> > weren't successful?
> >
>
> Originally I wrote it in the strategy "reduce false alarms". But when I
> think about it, it may not be good in this case. Usually the changes are
> done first on some developer environment, and good practice is to disallow
> same (possibly confusing) identifiers. So I am not against making this
> warning more aggressive with some possibility of false alarms. With
> blocking reduction of alarms the differences in regress was zero. So I
> reduced this part.
Great, thank you.
> > Couple of questions to it. In IdentifyVariable in the branch handling
> > two values the commentary says:
> >
> > /*
> > * a.b can mean "schema"."variable" or "variable"."field",
> > * Check both variants, and returns InvalidOid with
> > * not_unique flag, when both interpretations are
> > * possible. Second node can be star. In this case, the
> > * only allowed possibility is "variable"."*".
> > */
> >
> > I read this as "variable"."*" is a valid combination, but the very next
> > part of this condition says differently:
> >
>
>
>
> >
> > /*
> > * Session variables doesn't support unboxing by star
> > * syntax. But this syntax have to be calculated here,
> > * because can come from non session variables related
> > * expressions.
> > */
> > Assert(IsA(field2, A_Star));
> >
> > Is the first commentary not quite correct?
> >
>
> I think it is correct, but maybe I was not good at describing this issue.
> The sentence "Second node can be a star. In this case, the
> the only allowed possibility is "variable"."*"." should be in the next
> comment.
>
> In this part we process a list of identifiers, and we try to map these
> identifiers to some semantics. The parser should ensure that
> all fields of lists are strings or the last field is a star. In this case
> the semantic "schema".* is nonsense, and the only possible semantic
> is "variable".*. It is valid semantics, but unsupported now. Unboxing is
> available by syntax (var).*
>
> I changed the comment
Thanks. Just to clarify, by "unsupported" you mean unsupported in the
current patch implementation right? From what I understand value
unboxing could be done without parentheses in a non-top level select
query.
As a side note, I'm not sure if this branch is exercised in any tests.
I've replaced returning InvalidOid with actually doing LookupVariable(NULL, a, true)
in this case, and all the tests are still passing.
> > Another question about how shadowing warning should work between
> > namespaces.
> > Let's say I've got two namespaces, public and test, both have a session
> > variable with the same name, but only one has a table with the same name:
> >
> > -- in public
> > create table test_agg(a int);
> > create type for_test_agg as (a int);
> > create variable test_agg for_test_agg;
> >
> > -- in test
> > create type for_test_agg as (a int);
> > create variable test_agg for_test_agg;
> >
> > Now if we will try to trigger the shadowing warning from public
> > namespace, it would work differently:
> >
> > -- in public
> > =# let test.test_agg.a = 10;
> > =# let test_agg.a = 20;
> > =# set session_variables_ambiguity_warning to on;
> >
> > -- note the value returned from the table
> > =# select jsonb_agg(test_agg.a) from test_agg;
> > WARNING: 42702: session variable "test_agg.a" is shadowed
> > LINE 1: select jsonb_agg(test_agg.a) from test_agg;
> > ^
> > DETAIL: Session variables can be shadowed by columns, routine's
> > variables and routine's arguments with the same name.
> > LOCATION: transformColumnRef, parse_expr.c:940
> > jsonb_agg
> > -----------
> > [1]
> >
> > -- no warning, note the session variable value
> > =# select jsonb_agg(test.test_agg.a) from test_agg;
> > jsonb_agg
> > -----------
> > [10]
> >
> > It happens because in the second scenario the logic inside
> > transformColumnRef
> > will not set up the node variable (there is no corresponding table in the
> > "test" schema), and the following conditions covering session variables
> > shadowing are depending on it. Is it supposed to be like this?
> >
>
> I am sorry, I don't understand what you want to describe. Session variables
> are shadowed by relations, ever. It is design. In the first case, the
> variable is shadowed and a warning is raised. In the second case,
> "test"."test_agg"."a" is a fully unique qualified identifier, and then the
> variable is used, and then it is not shadowed.
Yeah, there was a misunderstanding on my side, sorry. For whatever
reason I thought shadowing between schemas is a reasonable thing, but as
you pointed out it doesn't really make sense.
Вложения
- v20230317-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230317-1-0010-documentation.patch
- v20230317-1-0006-enhancing-psql-for-session-variables.patch
- v20230317-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230317-1-0008-regress-tests-for-session-variables.patch
- v20230317-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230317-1-0005-DISCARD-VARIABLES-command.patch
- v20230317-1-0003-LET-command.patch
- v20230317-1-0002-session-variables.patch
- v20230317-1-0001-catalog-support-for-session-variables.patch
On 17.03.23 21:50, Pavel Stehule wrote: > rebase + fix-update pg_dump tests I have a few comments on the code: 0001 ExecGrant_Variable() could probably use ExecGrant_common(). The additions to syscache.c should be formatted to the new style. in pg_variable.h: - create_lsn ought to have a "var" prefix. - typo: "typmode for variable's type" - What is the purpose of struct Variable? It seems very similar to FormData_pg_variable. At least a comment would be useful. Preserve the trailing comma in ParseExprKind. 0002 expr_kind_allows_session_variables() should have some explanation about criteria for determining which expression kinds should allow variables. Usually, we handle EXPR_KIND_* switches without default case, so we get notified what needs to be changed if a new enum symbol is added. 0010 The material from the tutorial (advanced.sgml) might be better in ddl.sgml. In catalogs.sgml, the columns don't match the ones actually defined in pg_variable.h in patch 0001 (e.g., create_lsn is missing and the order doesn't match). (The order of columns in pg_variable.h didn't immediately make sense to me either, so maybe there is a middle ground to be found.) session_variables_ambiguity_warning: There needs to be more information about this. The current explanation is basically just, "warn if your query is confusing". Why do I want that? Why would I not want that? What is the alternative? What are some examples? Shouldn't there be a standard behavior without a need to configure anything? In allfiles.sgml, dropVariable should be before dropView.
On 17.03.23 21:50, Pavel Stehule wrote: > Hi > > rebase + fix-update pg_dump tests > > Regards > > Pavel > I have spent several hours studying the code and the past discussions. The problem I see in general is that everyone who reviews and tests the patches finds more problems, behavioral, weird internal errors, crashes. These are then immediately fixed, and the cycle starts again. I don't have the sense that this process has arrived at a steady state yet. The other issue is that by its nature this patch adds a lot of code in a lot of places. Large patches are more likely to be successful if they add a lot of code in one place or smaller amounts of code in a lot of places. But this patch does both and it's just overwhelming. There is so much new internal functionality and terminology. Variables can be created, registered, initialized, stored, copied, prepared, set, freed, removed, released, synced, dropped, and more. I don't know if anyone has actually reviewed all that in detail. Has any effort been made to make this simpler, smaller, reduce scope, refactoring, find commonalities with other features, try to manage the complexity somehow? I'm not making a comment on the details of the functionality itself. I just think on the coding level it's not gotten to a satisfying situation yet.
On 17.03.23 21:50, Pavel Stehule wrote:
> Hi
>
> rebase + fix-update pg_dump tests
>
> Regards
>
> Pavel
>
I have spent several hours studying the code and the past discussions.
The problem I see in general is that everyone who reviews and tests the
patches finds more problems, behavioral, weird internal errors, crashes.
These are then immediately fixed, and the cycle starts again. I don't
have the sense that this process has arrived at a steady state yet.
The other issue is that by its nature this patch adds a lot of code in a
lot of places. Large patches are more likely to be successful if they
add a lot of code in one place or smaller amounts of code in a lot of
places. But this patch does both and it's just overwhelming. There is
so much new internal functionality and terminology. Variables can be
created, registered, initialized, stored, copied, prepared, set, freed,
removed, released, synced, dropped, and more. I don't know if anyone
has actually reviewed all that in detail.
Has any effort been made to make this simpler, smaller, reduce scope,
refactoring, find commonalities with other features, try to manage the
complexity somehow?
I'm not making a comment on the details of the functionality itself. I
just think on the coding level it's not gotten to a satisfying situation
yet.
Hičt 23. 3. 2023 v 16:33 odesílatel Peter Eisentraut <peter.eisentraut@enterprisedb.com> napsal:On 17.03.23 21:50, Pavel Stehule wrote:
> Hi
>
> rebase + fix-update pg_dump tests
>
> Regards
>
> Pavel
>
I have spent several hours studying the code and the past discussions.
The problem I see in general is that everyone who reviews and tests the
patches finds more problems, behavioral, weird internal errors, crashes.
These are then immediately fixed, and the cycle starts again. I don't
have the sense that this process has arrived at a steady state yet.
The other issue is that by its nature this patch adds a lot of code in a
lot of places. Large patches are more likely to be successful if they
add a lot of code in one place or smaller amounts of code in a lot of
places. But this patch does both and it's just overwhelming. There is
so much new internal functionality and terminology. Variables can be
created, registered, initialized, stored, copied, prepared, set, freed,
removed, released, synced, dropped, and more. I don't know if anyone
has actually reviewed all that in detail.
Has any effort been made to make this simpler, smaller, reduce scope,
refactoring, find commonalities with other features, try to manage the
complexity somehow?
I'm not making a comment on the details of the functionality itself. I
just think on the coding level it's not gotten to a satisfying situation
yet.I agree that this patch is large, but almost all code is simple. Complex code is "only" in 0002-session-variables.patch (113KB/438KB).Now, I have no idea how the functionality can be sensibly reduced or divided (no without significant performance loss). I see two difficult points in this code:1. when to clean memory. The code implements cleaning very accurately - and this is unique in Postgres. Partially I implement some functionality of storage manager. Probably no code from Postgres can be reused, because there is not any support for global temporary objects. Cleaning based on sinval messages processing is difficult, but there is nothing else. The code is a little bit more complex, because there are three types of session variables: a) session variables, b) temp session variables, c) session variables with transaction scope. Maybe @c can be removed, and maybe we don't need to support not null default (this can simplify initialization). What do you think about it?2. how to pass a variable's value to the executor. The implementation is based on extending the Param node, but it cannot reuse query params buffers and implements own.But it is hard to simplify code, because we want to support usage variables in queries, and usage in PL/pgSQL expressions too. And both are processed differently.
RegardsPavel
On 17.03.23 21:50, Pavel Stehule wrote:
> rebase + fix-update pg_dump tests
I have a few comments on the code:
0001
ExecGrant_Variable() could probably use ExecGrant_common().
The additions to syscache.c should be formatted to the new style.
in pg_variable.h:
- create_lsn ought to have a "var" prefix.
- typo: "typmode for variable's type"
- What is the purpose of struct Variable? It seems very similar to
FormData_pg_variable. At least a comment would be useful.
/*
* The Variable struct is based on FormData_pg_variable struct. Against
* FormData_pg_variable it can hold node of deserialized expression used
* for calculation of default value.
*/
Preserve the trailing comma in ParseExprKind.
0002
expr_kind_allows_session_variables() should have some explanation
about criteria for determining which expression kinds should allow
variables.
* Returns true, when expression of kind allows using of
* session variables.
+ *
+ * The session's variables can be used everywhere where
+ * can be used external parameters. Session variables
+ * are not allowed in DDL. Session's variables cannot be
+ * used in constraints.
+ *
+ * The identifier can be parsed as an session variable
+ * only in expression's kinds where session's variables
+ * are allowed. This is the primary usage of this function.
+ *
+ * Second usage of this function is for decision if
+ * an error message "column does not exist" or "column
+ * or variable does not exist" should be printed. When
+ * we are in expression, where session variables cannot
+ * be used, we raise the first form or error message.
*/
Usually, we handle EXPR_KIND_* switches without default case, so we
get notified what needs to be changed if a new enum symbol is added.
0010
The material from the tutorial (advanced.sgml) might be better in
ddl.sgml.
In catalogs.sgml, the columns don't match the ones actually defined in
pg_variable.h in patch 0001 (e.g., create_lsn is missing and the order
doesn't match).
(The order of columns in pg_variable.h didn't immediately make sense to
me either, so maybe there is a middle ground to be found.)
-- When ALIGNOF_DOUBLE==4 (e.g. AIX), the C ABI may impose 8-byte alignment on
-- some of the C types that correspond to TYPALIGN_DOUBLE SQL types. To ensure
-- catalog C struct layout matches catalog tuple layout, arrange for the tuple
-- offset of each fixed-width, attalign='d' catalog column to be divisible by 8
-- unconditionally. Keep such columns before the first NameData column of the
-- catalog, since packagers can override NAMEDATALEN to an odd number.
session_variables_ambiguity_warning: There needs to be more
information about this. The current explanation is basically just,
"warn if your query is confusing". Why do I want that? Why would I
not want that? What is the alternative? What are some examples?
Shouldn't there be a standard behavior without a need to configure
anything?
+ The session variables can be shadowed by column references in a query. This
+ is an expected feature. The existing queries should not be broken by creating
+ any session variable, because session variables are shadowed always if the
+ identifier is ambiguous. The variables should be named without possibility
+ to collision with identifiers of other database objects (column names or
+ record field names). The warnings enabled by setting <varname>session_variables_ambiguity_warning</varname>
+ should help with finding identifier's collisions.
+<programlisting>
+CREATE TABLE foo(a int);
+INSERT INTO foo VALUES(10);
+CREATE VARIABLE a int;
+LET a = 100;
+SELECT a FROM foo;
+</programlisting>
+
+<screen>
+ a
+----
+ 10
+(1 row)
+</screen>
+
+<programlisting>
+SET session_variables_ambiguity_warning TO on;
+SELECT a FROM foo;
+</programlisting>
+
+<screen>
+WARNING: session variable "a" is shadowed
+LINE 1: SELECT a FROM foo;
+ ^
+DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
+ a
+----
+ 10
+(1 row)
+</screen>
+ </para>
+ <para>
+ This feature can significantly increase size of logs, and then it is
+ disabled by default, but for testing or development environments it
+ should be enabled.
In allfiles.sgml, dropVariable should be before dropView.
Вложения
- v20230326-1-0010-documentation.patch
- v20230326-1-0006-enhancing-psql-for-session-variables.patch
- v20230326-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230326-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230326-1-0008-regress-tests-for-session-variables.patch
- v20230326-1-0005-DISCARD-VARIABLES-command.patch
- v20230326-1-0003-LET-command.patch
- v20230326-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230326-1-0001-catalog-support-for-session-variables.patch
- v20230326-1-0002-session-variables.patch
Hi, I just have a few minor wording improvements for the various comments / documentation you quoted. On Sun, Mar 26, 2023 at 08:53:49AM +0200, Pavel Stehule wrote: > út 21. 3. 2023 v 17:18 odesílatel Peter Eisentraut < > peter.eisentraut@enterprisedb.com> napsal: > > > - What is the purpose of struct Variable? It seems very similar to > > FormData_pg_variable. At least a comment would be useful. > > > > I wrote comment there: > > > /* > * The Variable struct is based on FormData_pg_variable struct. Against > * FormData_pg_variable it can hold node of deserialized expression used > * for calculation of default value. > */ Did you mean "Unlike" rather than "Against"? > > 0002 > > > > expr_kind_allows_session_variables() should have some explanation > > about criteria for determining which expression kinds should allow > > variables. > > > > I wrote comment there: > > /* > * Returns true, when expression of kind allows using of > * session variables. > + * The session's variables can be used everywhere where > + * can be used external parameters. Session variables > + * are not allowed in DDL. Session's variables cannot be > + * used in constraints. > + * > + * The identifier can be parsed as an session variable > + * only in expression's kinds where session's variables > + * are allowed. This is the primary usage of this function. > + * > + * Second usage of this function is for decision if > + * an error message "column does not exist" or "column > + * or variable does not exist" should be printed. When > + * we are in expression, where session variables cannot > + * be used, we raise the first form or error message. > */ Maybe /* * Returns true if the given expression kind is valid for session variables * Session variables can be used everywhere where external parameters can be * used. Session variables are not allowed in DDL commands or in constraints. * * An identifier can be parsed as a session variable only for expression kinds * where session variables are allowed. This is the primary usage of this * function. * * Second usage of this function is to decide whether "column does not exist" or * "column or variable does not exist" error message should be printed. * When we are in an expression where session variables cannot be used, we raise * the first form or error message. */ > > session_variables_ambiguity_warning: There needs to be more > > information about this. The current explanation is basically just, > > "warn if your query is confusing". Why do I want that? Why would I > > not want that? What is the alternative? What are some examples? > > Shouldn't there be a standard behavior without a need to configure > > anything? > > > > I enhanced this entry: > > + <para> > + The session variables can be shadowed by column references in a > query. This > + is an expected feature. The existing queries should not be broken > by creating > + any session variable, because session variables are shadowed > always if the > + identifier is ambiguous. The variables should be named without > possibility > + to collision with identifiers of other database objects (column > names or > + record field names). The warnings enabled by setting > <varname>session_variables_ambiguity_warning</varname> > + should help with finding identifier's collisions. Maybe Session variables can be shadowed by column references in a query, this is an expected behavior. Previously working queries shouldn't error out by creating any session variable, so session variables are always shadowed if an identifier is ambiguous. Variables should be referenced using an unambiguous identifier without any possibility for a collision with identifier of other database objects (column names or record fields names). The warning messages emitted when enabling <varname>session_variables_ambiguity_warning</varname> can help finding such identifier collision. > + </para> > + <para> > + This feature can significantly increase size of logs, and then it > is > + disabled by default, but for testing or development environments it > + should be enabled. Maybe This feature can significantly increase log size, so it's disabled by default. For testing or development environments it's recommended to enable it if you use session variables.
> On Fri, Mar 24, 2023 at 08:04:08AM +0100, Pavel Stehule wrote: > čt 23. 3. 2023 v 19:54 odesílatel Pavel Stehule <pavel.stehule@gmail.com> > napsal: > > > čt 23. 3. 2023 v 16:33 odesílatel Peter Eisentraut < > > peter.eisentraut@enterprisedb.com> napsal: > > > >> The other issue is that by its nature this patch adds a lot of code in a > >> lot of places. Large patches are more likely to be successful if they > >> add a lot of code in one place or smaller amounts of code in a lot of > >> places. But this patch does both and it's just overwhelming. There is > >> so much new internal functionality and terminology. Variables can be > >> created, registered, initialized, stored, copied, prepared, set, freed, > >> removed, released, synced, dropped, and more. I don't know if anyone > >> has actually reviewed all that in detail. > >> > >> Has any effort been made to make this simpler, smaller, reduce scope, > >> refactoring, find commonalities with other features, try to manage the > >> complexity somehow? > >> > > I agree that this patch is large, but almost all code is simple. Complex > > code is "only" in 0002-session-variables.patch (113KB/438KB). > > > > Now, I have no idea how the functionality can be sensibly reduced or > > divided (no without significant performance loss). I see two difficult > > points in this code: > > > > 1. when to clean memory. The code implements cleaning very accurately - > > and this is unique in Postgres. Partially I implement some functionality of > > storage manager. Probably no code from Postgres can be reused, because > > there is not any support for global temporary objects. Cleaning based on > > sinval messages processing is difficult, but there is nothing else. The > > code is a little bit more complex, because there are three types of session > > variables: a) session variables, b) temp session variables, c) session > > variables with transaction scope. Maybe @c can be removed, and maybe we > > don't need to support not null default (this can simplify initialization). > > What do you think about it? > > > > 2. how to pass a variable's value to the executor. The implementation is > > based on extending the Param node, but it cannot reuse query params buffers > > and implements own. > > But it is hard to simplify code, because we want to support usage > > variables in queries, and usage in PL/pgSQL expressions too. And both are > > processed differently. > > > > Maybe I can divide the patch 0002-session-variables to three sections - > related to memory management, planning and execution? I agree, the patch scale is a bit overwhelming. It's worth noting that due to the nature of this change certain heavy lifting has to be done in any case, plus I've got an impression that some part of the patch are quite solid (although I haven't reviewed everything, did anyone achieve that milestone?). But still, it would be of great help to simplify the current implementation, and I'm afraid the only way of doing this is to make trades-off about functionality vs change size & complexity. Maybe instead splitting the patch into implementation components, it's possible to split it feature-by-feature, where every single patch would represent an independent (to a certain degree) functionality? I have in mind something like: catalog changes; base implementation; ACL support; xact actions implementation (on commit drop, etc); variables with default value; shadowing; etc. If such approach is possible, it will give us: flexibility to apply only a subset of the whole patch series; some understanding how much complexity is coming from each feature. What do you think about this idea? I also recall somewhere earlier in the thread Pavel has mentioned that a transactional version of session variables patch would be actually simpler, and he has plans to implement it later on. Is there another trade-off on the table we could think of, transactional vs non-transactional session variables?
> On Sun, Mar 26, 2023 at 07:32:05PM +0800, Julien Rouhaud wrote: > Hi, > > I just have a few minor wording improvements for the various comments / > documentation you quoted. Talking about documentation I've noticed that the implementation contains few limitations, that are not mentioned in the docs. Examples are WITH queries: WITH x AS (LET public.svar = 100) SELECT * FROM x; ERROR: LET not supported in WITH query and using with set-returning functions (haven't found any related tests). Another small note is about this change in the rowsecurity: /* - * For SELECT, UPDATE and DELETE, add security quals to enforce the USING - * policies. These security quals control access to existing table rows. - * Restrictive policies are combined together using AND, and permissive - * policies are combined together using OR. + * For SELECT, LET, UPDATE and DELETE, add security quals to enforce the + * USING policies. These security quals control access to existing table + * rows. Restrictive policies are combined together using AND, and + * permissive policies are combined together using OR. */ From this commentary one may think that LET command supports row level security, but I don't see it being implemented. A wrong commentary?
Hi,
I just have a few minor wording improvements for the various comments /
documentation you quoted.
On Sun, Mar 26, 2023 at 08:53:49AM +0200, Pavel Stehule wrote:
> út 21. 3. 2023 v 17:18 odesílatel Peter Eisentraut <
> peter.eisentraut@enterprisedb.com> napsal:
>
> > - What is the purpose of struct Variable? It seems very similar to
> > FormData_pg_variable. At least a comment would be useful.
> >
>
> I wrote comment there:
>
>
> /*
> * The Variable struct is based on FormData_pg_variable struct. Against
> * FormData_pg_variable it can hold node of deserialized expression used
> * for calculation of default value.
> */
Did you mean "Unlike" rather than "Against"?
> > 0002
> >
> > expr_kind_allows_session_variables() should have some explanation
> > about criteria for determining which expression kinds should allow
> > variables.
> >
>
> I wrote comment there:
>
> /*
> * Returns true, when expression of kind allows using of
> * session variables.
> + * The session's variables can be used everywhere where
> + * can be used external parameters. Session variables
> + * are not allowed in DDL. Session's variables cannot be
> + * used in constraints.
> + *
> + * The identifier can be parsed as an session variable
> + * only in expression's kinds where session's variables
> + * are allowed. This is the primary usage of this function.
> + *
> + * Second usage of this function is for decision if
> + * an error message "column does not exist" or "column
> + * or variable does not exist" should be printed. When
> + * we are in expression, where session variables cannot
> + * be used, we raise the first form or error message.
> */
Maybe
/*
* Returns true if the given expression kind is valid for session variables
* Session variables can be used everywhere where external parameters can be
* used. Session variables are not allowed in DDL commands or in constraints.
*
* An identifier can be parsed as a session variable only for expression kinds
* where session variables are allowed. This is the primary usage of this
* function.
*
* Second usage of this function is to decide whether "column does not exist" or
* "column or variable does not exist" error message should be printed.
* When we are in an expression where session variables cannot be used, we raise
* the first form or error message.
*/
> > session_variables_ambiguity_warning: There needs to be more
> > information about this. The current explanation is basically just,
> > "warn if your query is confusing". Why do I want that? Why would I
> > not want that? What is the alternative? What are some examples?
> > Shouldn't there be a standard behavior without a need to configure
> > anything?
> >
>
> I enhanced this entry:
>
> + <para>
> + The session variables can be shadowed by column references in a
> query. This
> + is an expected feature. The existing queries should not be broken
> by creating
> + any session variable, because session variables are shadowed
> always if the
> + identifier is ambiguous. The variables should be named without
> possibility
> + to collision with identifiers of other database objects (column
> names or
> + record field names). The warnings enabled by setting
> <varname>session_variables_ambiguity_warning</varname>
> + should help with finding identifier's collisions.
Maybe
Session variables can be shadowed by column references in a query, this is an
expected behavior. Previously working queries shouldn't error out by creating
any session variable, so session variables are always shadowed if an identifier
is ambiguous. Variables should be referenced using an unambiguous identifier
without any possibility for a collision with identifier of other database
objects (column names or record fields names). The warning messages emitted
when enabling <varname>session_variables_ambiguity_warning</varname> can help
finding such identifier collision.
> + </para>
> + <para>
> + This feature can significantly increase size of logs, and then it
> is
> + disabled by default, but for testing or development environments it
> + should be enabled.
Maybe
This feature can significantly increase log size, so it's disabled by default.
For testing or development environments it's recommended to enable it if you
use session variables.
> On Sun, Mar 26, 2023 at 07:32:05PM +0800, Julien Rouhaud wrote:
> Hi,
>
> I just have a few minor wording improvements for the various comments /
> documentation you quoted.
Talking about documentation I've noticed that the implementation
contains few limitations, that are not mentioned in the docs. Examples
are WITH queries:
WITH x AS (LET public.svar = 100) SELECT * FROM x;
ERROR: LET not supported in WITH query
and using with set-returning functions (haven't found any related tests).
+-- should be ok
+LET public.svar = generate_series(1, 1);
+-- should fail
+LET public.svar = generate_series(1, 2);
+ERROR: expression returned more than one row
+LET public.svar = generate_series(1, 0);
+ERROR: expression returned no rows
+DROP VARIABLE public.svar;
Another small note is about this change in the rowsecurity:
/*
- * For SELECT, UPDATE and DELETE, add security quals to enforce the USING
- * policies. These security quals control access to existing table rows.
- * Restrictive policies are combined together using AND, and permissive
- * policies are combined together using OR.
+ * For SELECT, LET, UPDATE and DELETE, add security quals to enforce the
+ * USING policies. These security quals control access to existing table
+ * rows. Restrictive policies are combined together using AND, and
+ * permissive policies are combined together using OR.
*/
From this commentary one may think that LET command supports row level
security, but I don't see it being implemented. A wrong commentary?
manager text,
company text,
contact_email text
);
t1role xxx t1role@xxx.org
t2role yyy t2role@yyy.org
\.
GRANT SELECT,INSERT ON TABLE public.accounts TO t2role;
GRANT ALL ON VARIABLE public.v TO t2role;
Assertions: on
psql (16devel)
Type "help" for help.
(2023-03-28 21:32:33) postgres=# set role to t1role;
SET
(2023-03-28 21:32:40) postgres=# select * from accounts ;
┌─────────┬─────────┬────────────────┐
│ manager │ company │ contact_email │
╞═════════╪═════════╪════════════════╡
│ t1role │ xxx │ t1role@xxx.org │
└─────────┴─────────┴────────────────┘
(1 row)
(2023-03-28 21:32:45) postgres=# let v = (select company from accounts);
LET
(2023-03-28 21:32:58) postgres=# select v;
┌─────┐
│ v │
╞═════╡
│ xxx │
└─────┘
(1 row)
(2023-03-28 21:33:03) postgres=# set role to default;
SET
(2023-03-28 21:33:12) postgres=# set role to t2role;
SET
(2023-03-28 21:33:19) postgres=# select * from accounts ;
┌─────────┬─────────┬────────────────┐
│ manager │ company │ contact_email │
╞═════════╪═════════╪════════════════╡
│ t2role │ yyy │ t2role@yyy.org │
└─────────┴─────────┴────────────────┘
(1 row)
(2023-03-28 21:33:22) postgres=# let v = (select company from accounts);
LET
(2023-03-28 21:33:26) postgres=# select v;
┌─────┐
│ v │
╞═════╡
│ yyy │
└─────┘
(1 row)
Hi,
I just have a few minor wording improvements for the various comments /
documentation you quoted.
On Sun, Mar 26, 2023 at 08:53:49AM +0200, Pavel Stehule wrote:
> út 21. 3. 2023 v 17:18 odesílatel Peter Eisentraut <
> peter.eisentraut@enterprisedb.com> napsal:
>
> > - What is the purpose of struct Variable? It seems very similar to
> > FormData_pg_variable. At least a comment would be useful.
> >
>
> I wrote comment there:
>
>
> /*
> * The Variable struct is based on FormData_pg_variable struct. Against
> * FormData_pg_variable it can hold node of deserialized expression used
> * for calculation of default value.
> */
Did you mean "Unlike" rather than "Against"?
> > 0002
> >
> > expr_kind_allows_session_variables() should have some explanation
> > about criteria for determining which expression kinds should allow
> > variables.
> >
>
> I wrote comment there:
>
> /*
> * Returns true, when expression of kind allows using of
> * session variables.
> + * The session's variables can be used everywhere where
> + * can be used external parameters. Session variables
> + * are not allowed in DDL. Session's variables cannot be
> + * used in constraints.
> + *
> + * The identifier can be parsed as an session variable
> + * only in expression's kinds where session's variables
> + * are allowed. This is the primary usage of this function.
> + *
> + * Second usage of this function is for decision if
> + * an error message "column does not exist" or "column
> + * or variable does not exist" should be printed. When
> + * we are in expression, where session variables cannot
> + * be used, we raise the first form or error message.
> */
Maybe
/*
* Returns true if the given expression kind is valid for session variables
* Session variables can be used everywhere where external parameters can be
* used. Session variables are not allowed in DDL commands or in constraints.
*
* An identifier can be parsed as a session variable only for expression kinds
* where session variables are allowed. This is the primary usage of this
* function.
*
* Second usage of this function is to decide whether "column does not exist" or
* "column or variable does not exist" error message should be printed.
* When we are in an expression where session variables cannot be used, we raise
* the first form or error message.
*/
> > session_variables_ambiguity_warning: There needs to be more
> > information about this. The current explanation is basically just,
> > "warn if your query is confusing". Why do I want that? Why would I
> > not want that? What is the alternative? What are some examples?
> > Shouldn't there be a standard behavior without a need to configure
> > anything?
> >
>
> I enhanced this entry:
>
> + <para>
> + The session variables can be shadowed by column references in a
> query. This
> + is an expected feature. The existing queries should not be broken
> by creating
> + any session variable, because session variables are shadowed
> always if the
> + identifier is ambiguous. The variables should be named without
> possibility
> + to collision with identifiers of other database objects (column
> names or
> + record field names). The warnings enabled by setting
> <varname>session_variables_ambiguity_warning</varname>
> + should help with finding identifier's collisions.
Maybe
Session variables can be shadowed by column references in a query, this is an
expected behavior. Previously working queries shouldn't error out by creating
any session variable, so session variables are always shadowed if an identifier
is ambiguous. Variables should be referenced using an unambiguous identifier
without any possibility for a collision with identifier of other database
objects (column names or record fields names). The warning messages emitted
when enabling <varname>session_variables_ambiguity_warning</varname> can help
finding such identifier collision.
> + </para>
> + <para>
> + This feature can significantly increase size of logs, and then it
> is
> + disabled by default, but for testing or development environments it
> + should be enabled.
Maybe
This feature can significantly increase log size, so it's disabled by default.
For testing or development environments it's recommended to enable it if you
use session variables.
Вложения
- v20230329-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230329-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230329-1-0008-regress-tests-for-session-variables.patch
- v20230329-1-0010-documentation.patch
- v20230329-1-0006-enhancing-psql-for-session-variables.patch
- v20230329-1-0005-DISCARD-VARIABLES-command.patch
- v20230329-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230329-1-0003-LET-command.patch
- v20230329-1-0002-session-variables.patch
- v20230329-1-0001-catalog-support-for-session-variables.patch
On 24.03.23 08:04, Pavel Stehule wrote: > Maybe I can divide the patch 0002-session-variables to three sections - > related to memory management, planning and execution? Personally, I find the existing split not helpful. There is no value (to me) in putting code, documentation, and tests in three separate patches. This is in fact counter-helpful (to me). Things like the DISCARD command (0005) and the error messages changes (0009) can be separate patches, but most of the rest should probably be a single patch. I know you have been asked earlier in the thread to provide smaller patches, so don't change it just for me, but this is my opinion.
> On Fri, Mar 24, 2023 at 08:04:08AM +0100, Pavel Stehule wrote:
> čt 23. 3. 2023 v 19:54 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
> > čt 23. 3. 2023 v 16:33 odesílatel Peter Eisentraut <
> > peter.eisentraut@enterprisedb.com> napsal:
> >
> >> The other issue is that by its nature this patch adds a lot of code in a
> >> lot of places. Large patches are more likely to be successful if they
> >> add a lot of code in one place or smaller amounts of code in a lot of
> >> places. But this patch does both and it's just overwhelming. There is
> >> so much new internal functionality and terminology. Variables can be
> >> created, registered, initialized, stored, copied, prepared, set, freed,
> >> removed, released, synced, dropped, and more. I don't know if anyone
> >> has actually reviewed all that in detail.
> >>
> >> Has any effort been made to make this simpler, smaller, reduce scope,
> >> refactoring, find commonalities with other features, try to manage the
> >> complexity somehow?
> >>
> > I agree that this patch is large, but almost all code is simple. Complex
> > code is "only" in 0002-session-variables.patch (113KB/438KB).
> >
> > Now, I have no idea how the functionality can be sensibly reduced or
> > divided (no without significant performance loss). I see two difficult
> > points in this code:
> >
> > 1. when to clean memory. The code implements cleaning very accurately -
> > and this is unique in Postgres. Partially I implement some functionality of
> > storage manager. Probably no code from Postgres can be reused, because
> > there is not any support for global temporary objects. Cleaning based on
> > sinval messages processing is difficult, but there is nothing else. The
> > code is a little bit more complex, because there are three types of session
> > variables: a) session variables, b) temp session variables, c) session
> > variables with transaction scope. Maybe @c can be removed, and maybe we
> > don't need to support not null default (this can simplify initialization).
> > What do you think about it?
> >
> > 2. how to pass a variable's value to the executor. The implementation is
> > based on extending the Param node, but it cannot reuse query params buffers
> > and implements own.
> > But it is hard to simplify code, because we want to support usage
> > variables in queries, and usage in PL/pgSQL expressions too. And both are
> > processed differently.
> >
>
> Maybe I can divide the patch 0002-session-variables to three sections -
> related to memory management, planning and execution?
I agree, the patch scale is a bit overwhelming. It's worth noting that
due to the nature of this change certain heavy lifting has to be done in
any case, plus I've got an impression that some part of the patch are
quite solid (although I haven't reviewed everything, did anyone achieve
that milestone?). But still, it would be of great help to simplify the
current implementation, and I'm afraid the only way of doing this is to
make trades-off about functionality vs change size & complexity.
Maybe instead splitting the patch into implementation components, it's
possible to split it feature-by-feature, where every single patch would
represent an independent (to a certain degree) functionality? I have in
mind something like: catalog changes; base implementation; ACL support;
xact actions implementation (on commit drop, etc); variables with
default value; shadowing; etc. If such approach is possible, it will
give us: flexibility to apply only a subset of the whole patch series;
some understanding how much complexity is coming from each feature. What
do you think about this idea?
I also recall somewhere earlier in the thread Pavel has mentioned that a
transactional version of session variables patch would be actually
simpler, and he has plans to implement it later on. Is there another
trade-off on the table we could think of, transactional vs
non-transactional session variables?
On 24.03.23 08:04, Pavel Stehule wrote:
> Maybe I can divide the patch 0002-session-variables to three sections -
> related to memory management, planning and execution?
Personally, I find the existing split not helpful. There is no value
(to me) in putting code, documentation, and tests in three separate
patches. This is in fact counter-helpful (to me). Things like the
DISCARD command (0005) and the error messages changes (0009) can be
separate patches, but most of the rest should probably be a single patch.
I know you have been asked earlier in the thread to provide smaller
patches, so don't change it just for me, but this is my opinion.
On 30.03.23 10:49, Pavel Stehule wrote: > If I reorganize the patch to the following structure, can be it useful > for you? > > 1. really basic functionality (no temporary variables, no def > expressions, no memory cleaning) > SELECT variable > LET should be supported + doc, + related tests. > > 2. support for temporary variables (session, transaction scope), > memory cleaning at the end of transaction > > 3. PL/pgSQL support > 4. pg_dump > 5. shadowing warning > 6. ... others ... That seems like an ok approach. The pg_dump support should probably go into the first patch, so it's self-contained.
> On Tue, Mar 28, 2023 at 09:34:20PM +0200, Pavel Stehule wrote: > Hi > > > Talking about documentation I've noticed that the implementation > > contains few limitations, that are not mentioned in the docs. Examples > > are WITH queries: > > > > WITH x AS (LET public.svar = 100) SELECT * FROM x; > > ERROR: LET not supported in WITH query > > > > The LET statement doesn't support the RETURNING clause, so using inside > CTE does not make any sense. > > Do you have some tips, where this behaviour should be mentioned? Yeah, you're right, it's probably not worth adding. I usually find it a good idea to explicitly mention any limitations, but WITH docs are actually have one line about statements without the RETURNING clause, plus indeed for LET it makes even less sense. > > and using with set-returning functions (haven't found any related tests). > > > > There it is: > > +CREATE VARIABLE public.svar AS int; > +-- should be ok > +LET public.svar = generate_series(1, 1); > +-- should fail > +LET public.svar = generate_series(1, 2); > +ERROR: expression returned more than one row > +LET public.svar = generate_series(1, 0); > +ERROR: expression returned no rows > +DROP VARIABLE public.svar; Oh, interesting. I was looking for another error message from parse_func.c: set-returning functions are not allowed in LET assignment expression Is this one you've posted somehow different? > > Another small note is about this change in the rowsecurity: > > > > /* > > - * For SELECT, UPDATE and DELETE, add security quals to enforce > > the USING > > - * policies. These security quals control access to existing > > table rows. > > - * Restrictive policies are combined together using AND, and > > permissive > > - * policies are combined together using OR. > > + * For SELECT, LET, UPDATE and DELETE, add security quals to > > enforce the > > + * USING policies. These security quals control access to > > existing table > > + * rows. Restrictive policies are combined together using AND, and > > + * permissive policies are combined together using OR. > > */ > > > > From this commentary one may think that LET command supports row level > > security, but I don't see it being implemented. A wrong commentary? > > > > I don't think so. The row level security should be supported. I tested it > on example from doc: > > [...] > > (2023-03-28 21:32:33) postgres=# set role to t1role; > SET > (2023-03-28 21:32:40) postgres=# select * from accounts ; > ┌─────────┬─────────┬────────────────┐ > │ manager │ company │ contact_email │ > ╞═════════╪═════════╪════════════════╡ > │ t1role │ xxx │ t1role@xxx.org │ > └─────────┴─────────┴────────────────┘ > (1 row) > > (2023-03-28 21:32:45) postgres=# let v = (select company from accounts); > LET > (2023-03-28 21:32:58) postgres=# select v; > ┌─────┐ > │ v │ > ╞═════╡ > │ xxx │ > └─────┘ > (1 row) > > (2023-03-28 21:33:03) postgres=# set role to default; > SET > (2023-03-28 21:33:12) postgres=# set role to t2role; > SET > (2023-03-28 21:33:19) postgres=# select * from accounts ; > ┌─────────┬─────────┬────────────────┐ > │ manager │ company │ contact_email │ > ╞═════════╪═════════╪════════════════╡ > │ t2role │ yyy │ t2role@yyy.org │ > └─────────┴─────────┴────────────────┘ > (1 row) > > (2023-03-28 21:33:22) postgres=# let v = (select company from accounts); > LET > (2023-03-28 21:33:26) postgres=# select v; > ┌─────┐ > │ v │ > ╞═════╡ > │ yyy │ > └─────┘ > (1 row) Hm, but isn't the row level security enforced here on the select level, not when assigning some value via LET? Plus, it seems the comment originally refer to the command types (CMD_SELECT, etc), and there is no CMD_LET and no need for it, right? I'm just trying to understand if there was anything special done for session variables in this regard, and if not, the commentary change seems to be not needed (I know, I know, it's totally nitpicking).
> On Tue, Mar 28, 2023 at 09:34:20PM +0200, Pavel Stehule wrote:
> Hi
>
> > Talking about documentation I've noticed that the implementation
> > contains few limitations, that are not mentioned in the docs. Examples
> > are WITH queries:
> >
> > WITH x AS (LET public.svar = 100) SELECT * FROM x;
> > ERROR: LET not supported in WITH query
> >
>
> The LET statement doesn't support the RETURNING clause, so using inside
> CTE does not make any sense.
>
> Do you have some tips, where this behaviour should be mentioned?
Yeah, you're right, it's probably not worth adding. I usually find it a
good idea to explicitly mention any limitations, but WITH docs are
actually have one line about statements without the RETURNING clause,
plus indeed for LET it makes even less sense.
> > and using with set-returning functions (haven't found any related tests).
> >
>
> There it is:
>
> +CREATE VARIABLE public.svar AS int;
> +-- should be ok
> +LET public.svar = generate_series(1, 1);
> +-- should fail
> +LET public.svar = generate_series(1, 2);
> +ERROR: expression returned more than one row
> +LET public.svar = generate_series(1, 0);
> +ERROR: expression returned no rows
> +DROP VARIABLE public.svar;
Oh, interesting. I was looking for another error message from
parse_func.c:
set-returning functions are not allowed in LET assignment expression
Is this one you've posted somehow different?
CREATE VARIABLE
(2023-04-01 06:26:02) postgres=# let xxx[generate_series(1,3)] = 10;
ERROR: set-returning functions are not allowed in LET assignment expression
LINE 1: let xxx[generate_series(1,3)] = 10;
^
- err = _("set-returning functions are not allowed in LET assignment expression");
+ err = _("set-returning functions are not allowed in LET target expression");
break;
> > Another small note is about this change in the rowsecurity:
> >
> > /*
> > - * For SELECT, UPDATE and DELETE, add security quals to enforce
> > the USING
> > - * policies. These security quals control access to existing
> > table rows.
> > - * Restrictive policies are combined together using AND, and
> > permissive
> > - * policies are combined together using OR.
> > + * For SELECT, LET, UPDATE and DELETE, add security quals to
> > enforce the
> > + * USING policies. These security quals control access to
> > existing table
> > + * rows. Restrictive policies are combined together using AND, and
> > + * permissive policies are combined together using OR.
> > */
> >
> > From this commentary one may think that LET command supports row level
> > security, but I don't see it being implemented. A wrong commentary?
> >
>
> I don't think so. The row level security should be supported. I tested it
> on example from doc:
>
> [...]
>
> (2023-03-28 21:32:33) postgres=# set role to t1role;
> SET
> (2023-03-28 21:32:40) postgres=# select * from accounts ;
> ┌─────────┬─────────┬────────────────┐
> │ manager │ company │ contact_email │
> ╞═════════╪═════════╪════════════════╡
> │ t1role │ xxx │ t1role@xxx.org │
> └─────────┴─────────┴────────────────┘
> (1 row)
>
> (2023-03-28 21:32:45) postgres=# let v = (select company from accounts);
> LET
> (2023-03-28 21:32:58) postgres=# select v;
> ┌─────┐
> │ v │
> ╞═════╡
> │ xxx │
> └─────┘
> (1 row)
>
> (2023-03-28 21:33:03) postgres=# set role to default;
> SET
> (2023-03-28 21:33:12) postgres=# set role to t2role;
> SET
> (2023-03-28 21:33:19) postgres=# select * from accounts ;
> ┌─────────┬─────────┬────────────────┐
> │ manager │ company │ contact_email │
> ╞═════════╪═════════╪════════════════╡
> │ t2role │ yyy │ t2role@yyy.org │
> └─────────┴─────────┴────────────────┘
> (1 row)
>
> (2023-03-28 21:33:22) postgres=# let v = (select company from accounts);
> LET
> (2023-03-28 21:33:26) postgres=# select v;
> ┌─────┐
> │ v │
> ╞═════╡
> │ yyy │
> └─────┘
> (1 row)
Hm, but isn't the row level security enforced here on the select level,
not when assigning some value via LET? Plus, it seems the comment
originally refer to the command types (CMD_SELECT, etc), and there is no
CMD_LET and no need for it, right?
I'm just trying to understand if there was anything special done for
session variables in this regard, and if not, the commentary change
seems to be not needed (I know, I know, it's totally nitpicking).
Вложения
- v20230401-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230401-1-0010-documentation.patch
- v20230401-1-0006-enhancing-psql-for-session-variables.patch
- v20230401-1-0008-regress-tests-for-session-variables.patch
- v20230401-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230401-1-0005-DISCARD-VARIABLES-command.patch
- v20230401-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230401-1-0003-LET-command.patch
- v20230401-1-0002-session-variables.patch
On Sun, 26 Mar 2023 at 07:34, Julien Rouhaud <rjuju123@gmail.com> wrote: > > This feature can significantly increase log size, so it's disabled by default. > For testing or development environments it's recommended to enable it if you > use session variables. I think it's generally not practical to have warnings for valid DML. Effectively warnings in DML are errors since they make the syntax just unusable. I suppose it's feasible to have it as a debugging option that defaults to off but I'm not sure it's really useful. I suppose it raises the question of whether session variables should be in pg_class and be in the same namespace as tables so that collisions are impossible. I haven't looked at the code to see if that's feasible or reasonable. But this feels a bit like what happened with sequences where they used to be a wholly special thing and later we realized everything was simpler if they were just a kind of relation. -- greg
On Sun, 26 Mar 2023 at 07:34, Julien Rouhaud <rjuju123@gmail.com> wrote:
>
> This feature can significantly increase log size, so it's disabled by default.
> For testing or development environments it's recommended to enable it if you
> use session variables.
I think it's generally not practical to have warnings for valid DML.
Effectively warnings in DML are errors since they make the syntax just
unusable. I suppose it's feasible to have it as a debugging option
that defaults to off but I'm not sure it's really useful.
I suppose it raises the question of whether session variables should
be in pg_class and be in the same namespace as tables so that
collisions are impossible. I haven't looked at the code to see if
that's feasible or reasonable. But this feels a bit like what happened
with sequences where they used to be a wholly special thing and later
we realized everything was simpler if they were just a kind of
relation.
--
greg
On Thu, Apr 6, 2023 at 1:58 AM Pavel Stehule <pavel.stehule@gmail.com> wrote: > > st 5. 4. 2023 v 19:20 odesílatel Greg Stark <stark@mit.edu> napsal: >> >> On Sun, 26 Mar 2023 at 07:34, Julien Rouhaud <rjuju123@gmail.com> wrote: >> > >> > This feature can significantly increase log size, so it's disabled by default. >> > For testing or development environments it's recommended to enable it if you >> > use session variables. >> >> I think it's generally not practical to have warnings for valid DML. >> Effectively warnings in DML are errors since they make the syntax just >> unusable. I suppose it's feasible to have it as a debugging option >> that defaults to off but I'm not sure it's really useful. > > > It is a tool that should help with collision detection. Without it, it can be pretty hard to detect it. It is similarto plpgsql's extra warnings. Another example is escape_string_warning, which can also emit warning for valid DML. I once had to fix some random framework that a previous employer was using, in order to move to a more recent pg version and have standard_conforming_strings on, and having escape_string_warning was quite helpful.
st 5. 4. 2023 v 19:20 odesílatel Greg Stark <stark@mit.edu> napsal:On Sun, 26 Mar 2023 at 07:34, Julien Rouhaud <rjuju123@gmail.com> wrote:
>
> This feature can significantly increase log size, so it's disabled by default.
> For testing or development environments it's recommended to enable it if you
> use session variables.
I think it's generally not practical to have warnings for valid DML.
Effectively warnings in DML are errors since they make the syntax just
unusable. I suppose it's feasible to have it as a debugging option
that defaults to off but I'm not sure it's really useful.It is a tool that should help with collision detection. Without it, it can be pretty hard to detect it. It is similar to plpgsql's extra warnings.
I suppose it raises the question of whether session variables should
be in pg_class and be in the same namespace as tables so that
collisions are impossible. I haven't looked at the code to see if
that's feasible or reasonable. But this feels a bit like what happened
with sequences where they used to be a wholly special thing and later
we realized everything was simpler if they were just a kind of
relation.The first patch did it. But at the end, it doesn't reduce conflicts, because usually the conflicts are between variables and table's attributes (columns).examplecreate variable a as int;create table foo(a int);select a from foo; -- the "a" is ambiguous, variable "a" is shadowedThis is a basic case, and the unique names don't help. The variables are more aggressive in namespace than tables, because they don't require be in FROM clause. This is the reason why we specify so variables are always shadowed. Only this behaviour is safe and robust. I cannot break any query (that doesn't use variables) by creating any variable. On second hand, an experience from Oracle's PL/SQL or from old PLpgSQL is, so unwanted shadowing can be hard to investigate (without some tools).PL/pgSQL doesn't allow conflict between PL/pgSQL variables, and SQL (now), and I think so it is best. But the scope of PLpgSQL variables is relatively small, so very strict behaviour is acceptable.The session variables are some between tables and attributes. The catalog pg_class can be enhanced about columns for variables, but it does a lot now, so I think it is not practical.
[Does an Alias work inside a procedure against a schema var?]
Does adding a schema prefix resolve it properly, so your example, I could do:
SELECT schema_var.a AS var_a, a as COL_A from t;
Again, I like the default that it is hidden, but I can envision needing both?
Regards, Kirk
examplecreate variable a as int;create table foo(a int);select a from foo; -- the "a" is ambiguous, variable "a" is shadowedThis is a basic case, and the unique names don't help. The variables are more aggressive in namespace than tables, because they don't require be in FROM clause. This is the reason why we specify so variables are always shadowed. Only this behaviour is safe and robust. I cannot break any query (that doesn't use variables) by creating any variable. On second hand, an experience from Oracle's PL/SQL or from old PLpgSQL is, so unwanted shadowing can be hard to investigate (without some tools).PL/pgSQL doesn't allow conflict between PL/pgSQL variables, and SQL (now), and I think so it is best. But the scope of PLpgSQL variables is relatively small, so very strict behaviour is acceptable.The session variables are some between tables and attributes. The catalog pg_class can be enhanced about columns for variables, but it does a lot now, so I think it is not practical.I agree about shadowing schema variables. But is there no way to fix that so that you can dereference the variable?
[Does an Alias work inside a procedure against a schema var?]
Does adding a schema prefix resolve it properly, so your example, I could do:
SELECT schema_var.a AS var_a, a as COL_A from t;
Again, I like the default that it is hidden, but I can envision needing both?
Regards, Kirk
Hine 26. 3. 2023 v 19:44 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> napsal:> On Fri, Mar 24, 2023 at 08:04:08AM +0100, Pavel Stehule wrote:
> čt 23. 3. 2023 v 19:54 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
> > čt 23. 3. 2023 v 16:33 odesílatel Peter Eisentraut <
> > peter.eisentraut@enterprisedb.com> napsal:
> >
> >> The other issue is that by its nature this patch adds a lot of code in a
> >> lot of places. Large patches are more likely to be successful if they
...
I agree, the patch scale is a bit overwhelming. It's worth noting that
due to the nature of this change certain heavy lifting has to be done in
any case, plus I've got an impression that some part of the patch are
quite solid (although I haven't reviewed everything, did anyone achieve
that milestone?). But still, it would be of great help to simplify the
current implementation, and I'm afraid the only way of doing this is to
make trades-off about functionality vs change size & complexity.There is not too much space for reduction - more - sometimes there is code reuse between features.I can reduce temporary session variables, but the same AtSubXact routines are used by memory purging routines, and if only if you drop all dependent features, then you can get some interesting number of reduced lines. I can imagine very reduced feature set like1) no temporary variables, no reset at transaction end2) without default expressions - default is null3) direct memory cleaning on drop (without possibility of saved value after reverted drop) or cleaning at session end alwaysNote - @1 and @3 shares code
Regards, Kirk
Вложения
- v20230516-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230516-1-0006-enhancing-psql-for-session-variables.patch
- v20230516-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230516-1-0008-regress-tests-for-session-variables.patch
- v20230516-1-0010-documentation.patch
- v20230516-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230516-1-0003-LET-command.patch
- v20230516-1-0005-DISCARD-VARIABLES-command.patch
- v20230516-1-0002-session-variables.patch
- v20230516-1-0001-catalog-support-for-session-variables.patch
Histill in old layout - but it can be useful for testing by someonefix build doc, fix regress tests
RegardsPavel
Вложения
- v20230517-1-0008-regress-tests-for-session-variables.patch
- v20230517-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230517-1-0006-enhancing-psql-for-session-variables.patch
- v20230517-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch
- v20230517-1-0010-documentation.patch
- v20230517-1-0005-DISCARD-VARIABLES-command.patch
- v20230517-1-0004-support-of-LET-command-in-PLpgSQL.patch
- v20230517-1-0003-LET-command.patch
- v20230517-1-0002-session-variables.patch
- v20230517-1-0001-catalog-support-for-session-variables.patch
On 30.03.23 10:49, Pavel Stehule wrote:
> If I reorganize the patch to the following structure, can be it useful
> for you?
>
> 1. really basic functionality (no temporary variables, no def
> expressions, no memory cleaning)
> SELECT variable
> LET should be supported + doc, + related tests.
>
> 2. support for temporary variables (session, transaction scope),
> memory cleaning at the end of transaction
>
> 3. PL/pgSQL support
> 4. pg_dump
> 5. shadowing warning
> 6. ... others ...
That seems like an ok approach. The pg_dump support should probably go
into the first patch, so it's self-contained.
Вложения
- v20230622-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230622-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230622-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230622-0005-implementation-of-temporary-session-variables.patch
- v20230622-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230622-0003-DISCARD-VARIABLES.patch
- v20230622-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230622-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230622-0001-Enhancing-catalog-for-support-session-variables-and-.patch
New organization1. basic CREATE VARIABLE, DROP VARIABLE, GRANT, REVOKE, ALTER, pg_dump2. basic SELECT var, LET var = value3. DISCARD VARIABLES4. cleaning memory used by dropped variables5. temporary variables + ON COMMIT DROP clause support6. ON TRANSACTION END RESET clause support7. DEFAULT expr clause support8. support NOT NULL and IMMUTABLE clauses9. use message "column or variable doesn't exists" instead "column doesn't exists"RegardsPavel
Вложения
- v20230623-0005-implementation-of-temporary-session-variables.patch
- v20230623-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230623-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230623-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230623-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230623-0003-DISCARD-VARIABLES.patch
- v20230623-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230623-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230623-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20230701-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230701-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230701-0005-implementation-of-temporary-session-variables.patch
- v20230701-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230701-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230701-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230701-0003-DISCARD-VARIABLES.patch
- v20230701-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20230701-0002-Storage-for-session-variables-and-SQL-interface.patch
+WARNING: roles created by regression test cases should have names starting with "regress_"
Вложения
- v20230706-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230706-0005-implementation-of-temporary-session-variables.patch
- v20230706-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230706-0003-DISCARD-VARIABLES.patch
- v20230706-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230706-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230706-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230706-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230706-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20230710-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230710-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230710-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230710-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230710-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20230710-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230710-0003-DISCARD-VARIABLES.patch
- v20230710-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230710-0005-implementation-of-temporary-session-variables.patch
Вложения
- v20230722-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230722-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230722-0005-implementation-of-temporary-session-variables.patch
- v20230722-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230722-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230722-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230722-0003-DISCARD-VARIABLES.patch
- v20230722-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20230722-0002-Storage-for-session-variables-and-SQL-interface.patch
Вложения
- v20230803-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230803-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230803-0005-implementation-of-temporary-session-variables.patch
- v20230803-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230803-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230803-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230803-0003-DISCARD-VARIABLES.patch
- v20230803-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230803-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> On Thu, Aug 03, 2023 at 08:15:13AM +0200, Pavel Stehule wrote: > Hi > > fresh rebase Thanks for continuing efforts. The new patch structure looks better to me (although the boundary between patches 0001 and 0002 is somewhat fuzzy, e.g. the function NameListToString is used already in the first one, but defined in the second). Couple of commentaries along the way: * Looks like it's common to use BKI_DEFAULT when defining catalog entities, something like BKI_DEFAULT(-1) for typmod, BKI_DEFAULT(0) for collation, etc. Does it make sense to put few default values into pg_variable as well? * The first patch contains: diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c @@ -2800,6 +2800,8 @@ AbortTransaction(void) AtAbort_Portals(); smgrDoPendingSyncs(false, is_parallel_worker); AtEOXact_LargeObject(false); + + /* 'false' means it's abort */ AtAbort_Notify(); AtEOXact_RelationMap(false, is_parallel_worker); AtAbort_Twophase(); What does the commentary refer to, is it needed? * I see ExplainOneQuery got a new argument: static void ExplainOneQuery(Query *query, int cursorOptions, - IntoClause *into, ExplainState *es, + IntoClause *into, Oid targetvar, ExplainState *es, const char *queryString, ParamListInfo params, QueryEnvironment *queryEnv); From what I understand it represents a potential session variable to be explained. Isn't it too specific for this interface, could it be put somewhere else? To be honest, I don't have any suggestions myself, but it feels a bit out of place here. * Session variable validity logic is not always clear, at least to me, producing following awkward pieces of code: + if (!svar->is_valid) + { + if (is_session_variable_valid(svar)) + svar->is_valid = true; I get it as there are two ways how a variable could be invalid? * It's not always easy to follow which failure modes are taken care of. E.g. + * Don't try to use possibly invalid data from svar. And we don't want to + * overwrite invalid svar immediately. The datumCopy can fail, and in this + * case, the stored value will be invalid still. I couldn't find any similar precautions, how exactly datumCopy can fail, are you referring to palloc/memcpy failures? Another confusing example was this one at the end of set_session_variable: + /* + * XXX While unlikely, an error here is possible. It wouldn't leak memory + * as the allocated chunk has already been correctly assigned to the + * session variable, but would contradict this function contract, which is + * that this function should either succeed or leave the current value + * untouched. + */ + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid), + svar->varid); It's not clear, which exactly error you're talking about, it's the last instruction in the function. Maybe it would be beneficial to have some overarching description, all in one place, about how session variables implementation handles various failures?
On Fri, Aug 11, 2023 at 05:55:26PM +0200, Dmitry Dolgov wrote: > > Another confusing example was this one at the end of set_session_variable: > > + /* > + * XXX While unlikely, an error here is possible. It wouldn't leak memory > + * as the allocated chunk has already been correctly assigned to the > + * session variable, but would contradict this function contract, which is > + * that this function should either succeed or leave the current value > + * untouched. > + */ > + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value", > + get_namespace_name(get_session_variable_namespace(svar->varid)), > + get_session_variable_name(svar->varid), > + svar->varid); > > It's not clear, which exactly error you're talking about, it's the last > instruction in the function. FTR I think I'm the one that changed that. The error I was talking about is elog() itself (in case of OOM for instance), or even one of the get_* call, if running with log_level <= DEBUG1. It's clearly really unlikely but still possible, thus this comment which also tries to explain why this elog() is not done earlier.
> On Sat, Aug 12, 2023 at 09:28:19AM +0800, Julien Rouhaud wrote: > On Fri, Aug 11, 2023 at 05:55:26PM +0200, Dmitry Dolgov wrote: > > > > Another confusing example was this one at the end of set_session_variable: > > > > + /* > > + * XXX While unlikely, an error here is possible. It wouldn't leak memory > > + * as the allocated chunk has already been correctly assigned to the > > + * session variable, but would contradict this function contract, which is > > + * that this function should either succeed or leave the current value > > + * untouched. > > + */ > > + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value", > > + get_namespace_name(get_session_variable_namespace(svar->varid)), > > + get_session_variable_name(svar->varid), > > + svar->varid); > > > > It's not clear, which exactly error you're talking about, it's the last > > instruction in the function. > > FTR I think I'm the one that changed that. The error I was talking about is > elog() itself (in case of OOM for instance), or even one of the get_* call, if > running with log_level <= DEBUG1. It's clearly really unlikely but still > possible, thus this comment which also tries to explain why this elog() is not > done earlier. I see, thanks for clarification. Absolutely nitpicking, but the crucial "that's why this elog is not done earlier" is only assumed in the comment between the lines, not stated out loud :)
On Sat, Aug 12, 2023 at 01:20:03PM +0200, Dmitry Dolgov wrote: > > On Sat, Aug 12, 2023 at 09:28:19AM +0800, Julien Rouhaud wrote: > > On Fri, Aug 11, 2023 at 05:55:26PM +0200, Dmitry Dolgov wrote: > > > > > > Another confusing example was this one at the end of set_session_variable: > > > > > > + /* > > > + * XXX While unlikely, an error here is possible. It wouldn't leak memory > > > + * as the allocated chunk has already been correctly assigned to the > > > + * session variable, but would contradict this function contract, which is > > > + * that this function should either succeed or leave the current value > > > + * untouched. > > > + */ > > > + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value", > > > + get_namespace_name(get_session_variable_namespace(svar->varid)), > > > + get_session_variable_name(svar->varid), > > > + svar->varid); > > > > > > It's not clear, which exactly error you're talking about, it's the last > > > instruction in the function. > > > > FTR I think I'm the one that changed that. The error I was talking about is > > elog() itself (in case of OOM for instance), or even one of the get_* call, if > > running with log_level <= DEBUG1. It's clearly really unlikely but still > > possible, thus this comment which also tries to explain why this elog() is not > > done earlier. > > I see, thanks for clarification. Absolutely nitpicking, but the crucial > "that's why this elog is not done earlier" is only assumed in the > comment between the lines, not stated out loud :) Well, yes although to be fair the original version of this had a prior comment that was making it much more obvious: + /* + * No error should happen after this poiht, otherwise we could leak the + * newly allocated value if any. + */ (which would maybe have been better said "Nothing that can error out should be called after that point"). After quite a lot of patch revisions it now simply says: + /* We can overwrite old variable now. No error expected */ I agree that a bit more explanation is needed, and maybe also reminding that this is because all of that is done in a persistent memory context.
> On Thu, Aug 03, 2023 at 08:15:13AM +0200, Pavel Stehule wrote:
> Hi
>
> fresh rebase
Thanks for continuing efforts. The new patch structure looks better to
me (although the boundary between patches 0001 and 0002 is somewhat
fuzzy, e.g. the function NameListToString is used already in the first
one, but defined in the second). Couple of commentaries along the way:
* Looks like it's common to use BKI_DEFAULT when defining catalog
entities, something like BKI_DEFAULT(-1) for typmod, BKI_DEFAULT(0) for
collation, etc. Does it make sense to put few default values into
pg_variable as well?
* The first patch contains:
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
@@ -2800,6 +2800,8 @@ AbortTransaction(void)
AtAbort_Portals();
smgrDoPendingSyncs(false, is_parallel_worker);
AtEOXact_LargeObject(false);
+
+ /* 'false' means it's abort */
AtAbort_Notify();
AtEOXact_RelationMap(false, is_parallel_worker);
AtAbort_Twophase();
What does the commentary refer to, is it needed?
* I see ExplainOneQuery got a new argument:
static void ExplainOneQuery(Query *query, int cursorOptions,
- IntoClause *into, ExplainState *es,
+ IntoClause *into, Oid targetvar, ExplainState *es,
const char *queryString, ParamListInfo params,
QueryEnvironment *queryEnv);
From what I understand it represents a potential session variable to be
explained. Isn't it too specific for this interface, could it be put
somewhere else? To be honest, I don't have any suggestions myself, but
it feels a bit out of place here.
* Session variable validity logic is not always clear, at least to me,
producing following awkward pieces of code:
+ if (!svar->is_valid)
+ {
+ if (is_session_variable_valid(svar))
+ svar->is_valid = true;
I get it as there are two ways how a variable could be invalid?
CREATE TEMP VARIABLE var1 AS int ON COMMIT DROP;
LET var1 = 100;
SAVEPOINT s1;
DROP VARIABLE var1;
ROLLBACK TO s1;
SELECT var1;
var1.
------
100
(1 row)
COMMIT;
* It's not always easy to follow which failure modes are taken care of. E.g.
+ * Don't try to use possibly invalid data from svar. And we don't want to
+ * overwrite invalid svar immediately. The datumCopy can fail, and in this
+ * case, the stored value will be invalid still.
I couldn't find any similar precautions, how exactly datumCopy can fail,
are you referring to palloc/memcpy failures?
Another confusing example was this one at the end of set_session_variable:
+ /*
+ * XXX While unlikely, an error here is possible. It wouldn't leak memory
+ * as the allocated chunk has already been correctly assigned to the
+ * session variable, but would contradict this function contract, which is
+ * that this function should either succeed or leave the current value
+ * untouched.
+ */
+ elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value",
+ get_namespace_name(get_session_variable_namespace(svar->varid)),
+ get_session_variable_name(svar->varid),
+ svar->varid);
It's not clear, which exactly error you're talking about, it's the last
instruction in the function.
Maybe it would be beneficial to have some overarching description, all
in one place, about how session variables implementation handles various
failures?
Вложения
- v20230823-0003-DISCARD-VARIABLES.patch
- v20230823-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230823-0005-implementation-of-temporary-session-variables.patch
- v20230823-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230823-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20230823-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230823-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
Вложения
- v20230823-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230823-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230823-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230823-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230823-0005-implementation-of-temporary-session-variables.patch
- v20230823-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230823-0003-DISCARD-VARIABLES.patch
- v20230823-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230823-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20230831-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230831-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230831-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230831-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230831-0005-implementation-of-temporary-session-variables.patch
- v20230831-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230831-0003-DISCARD-VARIABLES.patch
- v20230831-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230831-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20230928-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20230928-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20230928-0005-implementation-of-temporary-session-variables.patch
- v20230928-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20230928-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20230928-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20230928-0003-DISCARD-VARIABLES.patch
- v20230928-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20230928-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20231027-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20231027-1-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20231027-1-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20231027-1-0005-implementation-of-temporary-session-variables.patch
- v20231027-1-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20231027-1-0003-DISCARD-VARIABLES.patch
- v20231027-1-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20231027-1-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20231027-1-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> On Wed, Aug 23, 2023 at 04:02:44PM +0200, Pavel Stehule wrote: > NameListToString is already buildin function. Do you think NamesFromList? > > This is my oversight - there is just `+extern List *NamesFromList(List > *names); ` line, but sure - it should be in 0002 patch > > fixed now Right, thanks for fixing. I think there is a wrinkle with pg_session_variables function. It returns nothing if sessionvars hash table is empty, which has two consequences: * One might get confused about whether a variable is created, based on the information from the function. An expected behaviour, but could be considered a bad UX. =# CREATE VARIABLE var1 AS varchar; -- empty, is expected =# SELECT name, typname, can_select, can_update FROM pg_session_variables(); name | typname | can_select | can_update ------+---------+------------+------------ (0 rows) -- but one can't create a variable =# CREATE VARIABLE var1 AS varchar; ERROR: 42710: session variable "var1" already exists LOCATION: create_variable, pg_variable.c:102 -- yet, suddenly after a select... =# SELECT var2; var2 ------ NULL (1 row) -- ... it's not empty =# SELECT name, typname, can_select, can_update FROM pg_sessio n_variables(); name | typname | can_select | can_update ------+-------------------+------------+------------ var2 | character varying | t | t (1 row) * Running a parallel query will end up returning an empty result even after accessing the variable. -- debug_parallel_query = 1 all the time =# CREATE VARIABLE var2 AS varchar; -- empty, is expected =# SELECT name, typname, can_select, can_update FROM pg_session_variables(); name | typname | can_select | can_update ------+---------+------------+------------ (0 rows) -- but this time an access... SELECT var2; var2 ------ NULL (1 row) -- or set... =# LET var2 = 'test'; -- doesn't change the result, it's still empty =# SELECT name, typname, can_select, can_update FROM pg_session_variables(); name | typname | can_select | can_update ------+---------+------------+------------ (0 rows) Would it be a problem to make pg_session_variables inspect the catalog or something similar if needed?
> On Wed, Aug 23, 2023 at 04:02:44PM +0200, Pavel Stehule wrote:
> NameListToString is already buildin function. Do you think NamesFromList?
>
> This is my oversight - there is just `+extern List *NamesFromList(List
> *names); ` line, but sure - it should be in 0002 patch
>
> fixed now
Right, thanks for fixing.
I think there is a wrinkle with pg_session_variables function. It
returns nothing if sessionvars hash table is empty, which has two
consequences:
* One might get confused about whether a variable is created,
based on the information from the function. An expected behaviour, but
could be considered a bad UX.
=# CREATE VARIABLE var1 AS varchar;
-- empty, is expected
=# SELECT name, typname, can_select, can_update FROM pg_session_variables();
name | typname | can_select | can_update
------+---------+------------+------------
(0 rows)
-- but one can't create a variable
=# CREATE VARIABLE var1 AS varchar;
ERROR: 42710: session variable "var1" already exists
LOCATION: create_variable, pg_variable.c:102
-- yet, suddenly after a select...
=# SELECT var2;
var2
------
NULL
(1 row)
-- ... it's not empty
=# SELECT name, typname, can_select, can_update FROM pg_sessio
n_variables();
name | typname | can_select | can_update
------+-------------------+------------+------------
var2 | character varying | t | t
(1 row)
* Running a parallel query will end up returning an empty result even
after accessing the variable.
-- debug_parallel_query = 1 all the time
=# CREATE VARIABLE var2 AS varchar;
-- empty, is expected
=# SELECT name, typname, can_select, can_update FROM pg_session_variables();
name | typname | can_select | can_update
------+---------+------------+------------
(0 rows)
-- but this time an access...
SELECT var2;
var2
------
NULL
(1 row)
-- or set...
=# LET var2 = 'test';
-- doesn't change the result, it's still empty
=# SELECT name, typname, can_select, can_update FROM pg_session_variables();
name | typname | can_select | can_update
------+---------+------------+------------
(0 rows)
Would it be a problem to make pg_session_variables inspect the catalog
or something similar if needed?
Hipá 17. 11. 2023 v 20:17 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> napsal:> On Wed, Aug 23, 2023 at 04:02:44PM +0200, Pavel Stehule wrote:
> NameListToString is already buildin function. Do you think NamesFromList?
>
> This is my oversight - there is just `+extern List *NamesFromList(List
> *names); ` line, but sure - it should be in 0002 patch
>
> fixed now
Right, thanks for fixing.
I think there is a wrinkle with pg_session_variables function. It
returns nothing if sessionvars hash table is empty, which has two
consequences:
* One might get confused about whether a variable is created,
based on the information from the function. An expected behaviour, but
could be considered a bad UX.
=# CREATE VARIABLE var1 AS varchar;
-- empty, is expected
=# SELECT name, typname, can_select, can_update FROM pg_session_variables();
name | typname | can_select | can_update
------+---------+------------+------------
(0 rows)
-- but one can't create a variable
=# CREATE VARIABLE var1 AS varchar;
ERROR: 42710: session variable "var1" already exists
LOCATION: create_variable, pg_variable.c:102
-- yet, suddenly after a select...
=# SELECT var2;
var2
------
NULL
(1 row)
-- ... it's not empty
=# SELECT name, typname, can_select, can_update FROM pg_sessio
n_variables();
name | typname | can_select | can_update
------+-------------------+------------+------------
var2 | character varying | t | t
(1 row)
* Running a parallel query will end up returning an empty result even
after accessing the variable.
-- debug_parallel_query = 1 all the time
=# CREATE VARIABLE var2 AS varchar;
-- empty, is expected
=# SELECT name, typname, can_select, can_update FROM pg_session_variables();
name | typname | can_select | can_update
------+---------+------------+------------
(0 rows)
-- but this time an access...
SELECT var2;
var2
------
NULL
(1 row)
-- or set...
=# LET var2 = 'test';
-- doesn't change the result, it's still empty
=# SELECT name, typname, can_select, can_update FROM pg_session_variables();
name | typname | can_select | can_update
------+---------+------------+------------
(0 rows)
Would it be a problem to make pg_session_variables inspect the catalog
or something similar if needed?It can be very easy to build pg_session_variables based on iteration over the system catalog. But I am not sure if we want it. pg_session_variables() is designed to show the variables from session memory, and it is used for testing. Originally it was named pg_debug_session_variables. If we iterate over catalog, it means using locks, and it can have an impact on isolation tests.So maybe we can introduce a parameter for this function to show all session variables (based on catalog) or only used based on iteration over memory. Default can be "all". What do you think about it?The difference between debug_parallel_query = 1 and debug_parallel_query = 0 is strange - and I'll check it.
The difference between debug_parallel_query = 1 and debug_parallel_query = 0 is strange - and I'll check it.looks so pg_session_variables() doesn't work in debug_paralel_query mode.
> On Sat, Nov 18, 2023 at 02:19:09PM +0100, Pavel Stehule wrote: > > Would it be a problem to make pg_session_variables inspect the catalog > > or something similar if needed? > > > > It can be very easy to build pg_session_variables based on iteration over > the system catalog. But I am not sure if we want it. pg_session_variables() > is designed to show the variables from session memory, and it is used for > testing. Originally it was named pg_debug_session_variables. If we iterate > over catalog, it means using locks, and it can have an impact on isolation > tests. I see, thanks for clarification. In the end one can check the catalog directly of course, is there any other value in this function except for debugging purposes? As a side note, I'm intended to go one more time through the first few patches introducing the basic functionality, and then mark it as ready in CF. I can't break the patch in testing since quite long time, and for most parts the changes make sense to me.
> On Sat, Nov 18, 2023 at 02:19:09PM +0100, Pavel Stehule wrote:
> > Would it be a problem to make pg_session_variables inspect the catalog
> > or something similar if needed?
> >
>
> It can be very easy to build pg_session_variables based on iteration over
> the system catalog. But I am not sure if we want it. pg_session_variables()
> is designed to show the variables from session memory, and it is used for
> testing. Originally it was named pg_debug_session_variables. If we iterate
> over catalog, it means using locks, and it can have an impact on isolation
> tests.
I see, thanks for clarification. In the end one can check the catalog
directly of course, is there any other value in this function except for
debugging purposes?
As a side note, I'm intended to go one more time through the first few
patches introducing the basic functionality, and then mark it as ready
in CF. I can't break the patch in testing since quite long time, and for
most parts the changes make sense to me.
Вложения
- v20231118-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20231118-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20231118-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20231118-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20231118-0005-implementation-of-temporary-session-variables.patch
- v20231118-0003-DISCARD-VARIABLES.patch
- v20231118-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20231118-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20231118-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Hi, On Tue, Oct 17, 2023 at 08:52:13AM +0200, Pavel Stehule wrote: > > When I thought about global temporary tables, I got one maybe interesting > idea. The one significant problem of global temporary tables is place for > storing info about size or column statistics. > > I think so these data can be stored simply in session variables. Any global > temporary table can get assigned one session variable, that can hold these > data. I don't know how realistic this would be. For instance it will require to properly link the global temporary table life cycle with the session variable and I'm afraid it would require to add some hacks to make it work as needed. But this still raises the question of whether this feature could be used internally for the need of another feature. If we think it's likely, should we try to act right now and reserve the "pg_" prefix for internal use rather than do that a few years down the line and probably break some user code as it was done recently for the role names?
> On Sat, Nov 18, 2023 at 06:28:53PM +0100, Pavel Stehule wrote: > so 18. 11. 2023 v 15:54 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> > napsal: > > As a side note, I'm intended to go one more time through the first few > > patches introducing the basic functionality, and then mark it as ready > > in CF. I can't break the patch in testing since quite long time, and for > > most parts the changes make sense to me. > > I marked pg_session_variables function as PARALLEL RESTRICTED, and did > rebase So, after one week of uninterrupted evening reviews I've made it through the first four patches :) It's a decent job -- more than once, looking at the code, I thought I could construct a case when it's going to blow up, but everything was working just fine. Yet, I think the patch still has to be reshaped a bit before moving forward. I've got a couple proposals of different nature: high level changes (you probably won't like some of them, but I'm sure they're going to be useful), technical code-level improvements/comments, and few language changes. With those changes in mind I would be satisfied with the patch, and hopefully they would also make it easier for a potential committer to pick it up. # High level proposals * I would suggest reducing the scope of the patch as much as possible, and not just by trimming on the edges, but rather following Phileas Fogg's example with the steamboat Henrietta -- get rid of all non-essential parts. This will make this rather large patch more approachable for others. For that one can concentrate only on the first two patches plus the fourth one (memory cleanup after dropping variables), leaving DISCARD, ON TRANSACTION END, DEFAULT, IMMUTABLE for the follow-up in the future. Another thing in this context would be to evaluate plpgsql support for this feature. You know the use case better than me, how important it is? Is it an intrinsic part of the feature, or session variables could be still valuable enough even without plpgsql? From what I see postponing plgpsql will make everything about ~800 lines lighter (most likely more), and also allow to ignore couple of concerns about the implementation (about this later). * The new GUC session_variables_ambiguity_warning is definitely going to cause many objections, it's another knob to manage very subtle behaviour detail very few people will ever notice. I see the point behind warning about ambiguity, so probably it makes sense to bite the bullet and decide one way or another. The proposal is to warn always in potentially ambiguous situations, and if concerns are high about logging too much, maybe do the warning on lower logging levels. # Code-level observations * It feels a bit awkward to have varid assignment logic in a separate function, what about adding an argument with varid to CreateVariableDestReceiver? SetVariableDestReceiverVarid still could be used for CreateDestReceiver. /* * Initially create a DestReceiver object. */ DestReceiver * CreateVariableDestReceiver(void) /* * Set parameters for a VariableDestReceiver. * Should be called right after creating the DestReceiver. */ void SetVariableDestReceiverVarid(DestReceiver *self, Oid varid) * It's worth it to add a commentary here explaining why it's fine to use InvalidOid here: if (pstmt->commandType != CMD_UTILITY) - ExplainOnePlan(pstmt, into, es, query_string, paramLI, queryEnv, + ExplainOnePlan(pstmt, into, InvalidOid, es, query_string, paramLI, queryEnv, &planduration, (es->buffers ? &bufusage : NULL)); My understanding is that since LetStmt is CMD_UTILITY, this branch will never be visited for a session variable. * IIUC this one is introduced to exclude session variables from the normal path with EXPR_KIND_UPDATE_TARGET: + EXPR_KIND_ASSIGN_VARIABLE, /* PL/pgSQL assignment target - disallow + * session variables */ But the name doesn't sound right, maybe longer EXPR_KIND_UPDATE_TARGET_NO_VARS is better? * I'm curious about this one, which exactly part does this change cover? @@ -4888,21 +4914,43 @@ substitute_actual_parameters_mutator(Node *node, - if (param->paramkind != PARAM_EXTERN) + if (param->paramkind != PARAM_EXTERN && + param->paramkind != PARAM_VARIABLE) elog(ERROR, "unexpected paramkind: %d", (int) param->paramkind); I've commented it out, but no tests were affected. * Does it mean there could be theoretically two LET statements at the same time with different command type, one CMD_UTILITY, one CMD_SELECT? Can it cause any issues? + /* + * Inside PL/pgSQL we don't want to execute LET statement as utility + * command, because it disallow to execute expression as simple + * expression. So for PL/pgSQL we have extra path, and we return SELECT. + * Then it can be executed by exec_eval_expr. Result is dirrectly assigned + * to target session variable inside PL/pgSQL LET statement handler. This + * is extra code, extra path, but possibility to get faster execution is + * too attractive. + */ + if (stmt->plpgsql_mode) + return query; + * This probably requires more explanation, is warning the only reason for this change? + * + * The session variables should not be used as target of PL/pgSQL assign + * statement. So we should to use special parser expr kind, that disallow + * usage of session variables. This block unwanted (in this context) + * possible warning so target PL/pgSQL's variable shadows some session + * variable. */ target = transformExpr(pstate, (Node *) cref, - EXPR_KIND_UPDATE_TARGET); + EXPR_KIND_ASSIGN_VARIABLE); * It would be great to have more commentaries here: typedef struct { DestReceiver pub; Oid varid; Oid typid; int32 typmod; int typlen; int slot_offset; int rows; } SVariableState; For example, why does it make sense to have a field rows, where we interested to only know the fact that there is exactly one column? * Why there is SetSessionVariableWithSecurityCheck, but no GetSessionVariableWithSecurityCheck? Instead, object_aclcheck is done in standard_ExecutorStart, which looks a bit out of place. * pg_session_variables -- you mention it exists only for testing. What about moving it out into a separate patch for the sake of slimming down? It looks like it's used only in tests for "memory cleanup" patch, maybe they could be restructured to not require this function. * Probably it's time to drop unnecessary historical notes, like this: * Note: originally we enhanced a list xact_recheck_varids here. Unfortunately * it was not safe and a little bit too complex, because the sinval callback * function can be called when we iterate over xact_recheck_varids list. * Another issue was the possibility of being out of memory when we enhanced * the list. So now we just switch flag in related entry sessionvars hash table. * We need to iterate over hash table on every sinval message, so extra two * iteration over this hash table is not significant overhead (and we skip * entries that don't require recheck). Now we do not have any memory allocation * in the sinval handler (This note can be removed before commit). * The second patch "Storage for session variables and SQL interface", mentions DISCARD command: /* * There is no guarantee of sessionvars being initialized, even when * receiving an invalidation callback, as DISCARD [ ALL | VARIABLES ] * destroys the hash table entirely. */ This command is implemented in another patch later one, so this comment probably belong there. * This comment mentions a "direct access, without buffering": /* * Direct access to session variable (without buffering). Because * returned value can be used (without an assignement) after the * referenced session variables is updated, we have to use an copy * of stored value every time. */ *op->resvalue = GetSessionVariableWithTypeCheck(op->d.vparam.varid, op->resnull, op->d.vparam.vartype); But GetSessionVariableWithTypeCheck goes through get_session_variable and searches in the hash table. What "buffering" means in this context? * GetSessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid) Should the "WithTypeCheck" part be an argument of the GetSessionVariable? To reduce the code duplication a bit. * Just out of curiosity, why TopTransactionContext? /* * Store domain_check extra in TopTransactionContext. When we are in * other transaction, the domain_check_extra cache is not valid * anymore. */ if (svar->domain_check_extra_lxid != MyProc->lxid) svar->domain_check_extra = NULL; domain_check(svar->value, svar->isnull, svar->typid, &svar->domain_check_extra, TopTransactionContext); * In SVariableData it would be great to have more comments around freeval, domain_check_extra, domain_check_extra_lxid. * Nitpicking, but the term "shadowing" for ambiguity between a session variable and a table column might be confusing, one can imagine there is a connection between those two objects and one actively follows ("shadows") the other one. * The second patch "Storage for session variables and SQL interface" mentions in the documentation default and temporary variables: <para> The value of a session variable is local to the current session. Retrieving a variable's value returns either a <literal>NULL</literal> or a default value, unless its value has been set to something else in the current session using the <command>LET</command> command. The content of a variable is not transactional. This is the same as regular variables in PL languages. The session variables can be persistent or can be temporary. In both cases, the content of session variables is temporary and not shared (like an content of temporary tables). </para> They're implemented in the following patches, so it belongs there. * Nitpicking, maybe merge those two conditions together for readability? if (!needs_validation) return; /* * Reset, this flag here, before we start the validation. It can be set to * on by incomming sinval message. */ needs_validation = false; if (!sessionvars) return; * This one is not very clear, what is the difference between "somewhere inside a transaction" and "at the end of a transaction"? /* * This routine can be called somewhere inside transaction or at an transaction * end. When atEOX argument is false, then we are inside transaction, and we * don't want to throw entries related to session variables dropped in current * transaction. */ # Language topic Since this patch introduces a large body of documentation and commentaries, I think it would benefit from a native speaker review. I've stumbled upon few examples (attached with proposed wording, without a diff extension to not confuse the CF bot), but otherwise if anyone follows this thread, texts review is appreciated.
Вложения
Hi,
On Tue, Oct 17, 2023 at 08:52:13AM +0200, Pavel Stehule wrote:
>
> When I thought about global temporary tables, I got one maybe interesting
> idea. The one significant problem of global temporary tables is place for
> storing info about size or column statistics.
>
> I think so these data can be stored simply in session variables. Any global
> temporary table can get assigned one session variable, that can hold these
> data.
I don't know how realistic this would be. For instance it will require to
properly link the global temporary table life cycle with the session variable
and I'm afraid it would require to add some hacks to make it work as needed.
But this still raises the question of whether this feature could be used
internally for the need of another feature. If we think it's likely, should we
try to act right now and reserve the "pg_" prefix for internal use rather than
do that a few years down the line and probably break some user code as it was
done recently for the role names?
> On Sat, Nov 18, 2023 at 06:28:53PM +0100, Pavel Stehule wrote:
> so 18. 11. 2023 v 15:54 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
> > As a side note, I'm intended to go one more time through the first few
> > patches introducing the basic functionality, and then mark it as ready
> > in CF. I can't break the patch in testing since quite long time, and for
> > most parts the changes make sense to me.
>
> I marked pg_session_variables function as PARALLEL RESTRICTED, and did
> rebase
So, after one week of uninterrupted evening reviews I've made it through
the first four patches :)
It's a decent job -- more than once, looking at the code, I thought I
could construct a case when it's going to blow up, but everything was
working just fine. Yet, I think the patch still has to be reshaped a bit
before moving forward. I've got a couple proposals of different nature:
high level changes (you probably won't like some of them, but I'm sure
they're going to be useful), technical code-level improvements/comments,
and few language changes. With those changes in mind I would be
satisfied with the patch, and hopefully they would also make it easier
for a potential committer to pick it up.
# High level proposals
* I would suggest reducing the scope of the patch as much as possible,
and not just by trimming on the edges, but rather following Phileas
Fogg's example with the steamboat Henrietta -- get rid of all
non-essential parts. This will make this rather large patch more
approachable for others.
For that one can concentrate only on the first two patches plus the
fourth one (memory cleanup after dropping variables), leaving DISCARD,
ON TRANSACTION END, DEFAULT, IMMUTABLE for the follow-up in the
future.
Another thing in this context would be to evaluate plpgsql support for
this feature. You know the use case better than me, how important it
is? Is it an intrinsic part of the feature, or session variables could
be still valuable enough even without plpgsql? From what I see
postponing plgpsql will make everything about ~800 lines lighter (most
likely more), and also allow to ignore couple of concerns about the
implementation (about this later).
* The new GUC session_variables_ambiguity_warning is definitely going to
cause many objections, it's another knob to manage very subtle
behaviour detail very few people will ever notice. I see the point
behind warning about ambiguity, so probably it makes sense to bite the
bullet and decide one way or another. The proposal is to warn always
in potentially ambiguous situations, and if concerns are high about
logging too much, maybe do the warning on lower logging levels.
# Code-level observations
* It feels a bit awkward to have varid assignment logic in a separate
function, what about adding an argument with varid to
CreateVariableDestReceiver? SetVariableDestReceiverVarid still could
be used for CreateDestReceiver.
/*
* Initially create a DestReceiver object.
*/
DestReceiver *
CreateVariableDestReceiver(void)
/*
* Set parameters for a VariableDestReceiver.
* Should be called right after creating the DestReceiver.
*/
void
SetVariableDestReceiverVarid(DestReceiver *self, Oid varid)
* It's worth it to add a commentary here explaining why it's fine to use
InvalidOid here:
if (pstmt->commandType != CMD_UTILITY)
- ExplainOnePlan(pstmt, into, es, query_string, paramLI, queryEnv,
+ ExplainOnePlan(pstmt, into, InvalidOid, es, query_string, paramLI, queryEnv,
&planduration, (es->buffers ? &bufusage : NULL));
My understanding is that since LetStmt is CMD_UTILITY, this branch
will never be visited for a session variable.
* IIUC this one is introduced to exclude session variables from the normal
path with EXPR_KIND_UPDATE_TARGET:
+ EXPR_KIND_ASSIGN_VARIABLE, /* PL/pgSQL assignment target - disallow
+ * session variables */
But the name doesn't sound right, maybe longer
EXPR_KIND_UPDATE_TARGET_NO_VARS is better?
* I'm curious about this one, which exactly part does this change cover?
@@ -4888,21 +4914,43 @@ substitute_actual_parameters_mutator(Node *node,
- if (param->paramkind != PARAM_EXTERN)
+ if (param->paramkind != PARAM_EXTERN &&
+ param->paramkind != PARAM_VARIABLE)
elog(ERROR, "unexpected paramkind: %d", (int) param->paramkind);
I've commented it out, but no tests were affected.
* Does it mean there could be theoretically two LET statements at the
same time with different command type, one CMD_UTILITY, one
CMD_SELECT? Can it cause any issues?
+ /*
+ * Inside PL/pgSQL we don't want to execute LET statement as utility
+ * command, because it disallow to execute expression as simple
+ * expression. So for PL/pgSQL we have extra path, and we return SELECT.
+ * Then it can be executed by exec_eval_expr. Result is dirrectly assigned
+ * to target session variable inside PL/pgSQL LET statement handler. This
+ * is extra code, extra path, but possibility to get faster execution is
+ * too attractive.
+ */
+ if (stmt->plpgsql_mode)
+ return query;
+
* This probably requires more explanation, is warning the only reason
for this change?
+ *
+ * The session variables should not be used as target of PL/pgSQL assign
+ * statement. So we should to use special parser expr kind, that disallow
+ * usage of session variables. This block unwanted (in this context)
+ * possible warning so target PL/pgSQL's variable shadows some session
+ * variable.
*/
target = transformExpr(pstate, (Node *) cref,
- EXPR_KIND_UPDATE_TARGET);
+ EXPR_KIND_ASSIGN_VARIABLE);
* It would be great to have more commentaries here:
typedef struct
{
DestReceiver pub;
Oid varid;
Oid typid;
int32 typmod;
int typlen;
int slot_offset;
int rows;
} SVariableState;
For example, why does it make sense to have a field rows, where we
interested to only know the fact that there is exactly one column?
* Why there is SetSessionVariableWithSecurityCheck, but no
GetSessionVariableWithSecurityCheck? Instead, object_aclcheck is done
in standard_ExecutorStart, which looks a bit out of place.
* pg_session_variables -- you mention it exists only for testing. What
about moving it out into a separate patch for the sake of slimming
down? It looks like it's used only in tests for "memory cleanup"
patch, maybe they could be restructured to not require this function.
* Probably it's time to drop unnecessary historical notes, like this:
* Note: originally we enhanced a list xact_recheck_varids here. Unfortunately
* it was not safe and a little bit too complex, because the sinval callback
* function can be called when we iterate over xact_recheck_varids list.
* Another issue was the possibility of being out of memory when we enhanced
* the list. So now we just switch flag in related entry sessionvars hash table.
* We need to iterate over hash table on every sinval message, so extra two
* iteration over this hash table is not significant overhead (and we skip
* entries that don't require recheck). Now we do not have any memory allocation
* in the sinval handler (This note can be removed before commit).
* The second patch "Storage for session variables and SQL interface",
mentions DISCARD command:
/*
* There is no guarantee of sessionvars being initialized, even when
* receiving an invalidation callback, as DISCARD [ ALL | VARIABLES ]
* destroys the hash table entirely.
*/
This command is implemented in another patch later one, so this
comment probably belong there.
* This comment mentions a "direct access, without buffering":
/*
* Direct access to session variable (without buffering). Because
* returned value can be used (without an assignement) after the
* referenced session variables is updated, we have to use an copy
* of stored value every time.
*/
*op->resvalue = GetSessionVariableWithTypeCheck(op->d.vparam.varid,
op->resnull,
op->d.vparam.vartype);
But GetSessionVariableWithTypeCheck goes through get_session_variable
and searches in the hash table. What "buffering" means in this
context?
* GetSessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid)
Should the "WithTypeCheck" part be an argument of the
GetSessionVariable? To reduce the code duplication a bit.
* Just out of curiosity, why TopTransactionContext?
/*
* Store domain_check extra in TopTransactionContext. When we are in
* other transaction, the domain_check_extra cache is not valid
* anymore.
*/
if (svar->domain_check_extra_lxid != MyProc->lxid)
svar->domain_check_extra = NULL;
domain_check(svar->value, svar->isnull,
svar->typid, &svar->domain_check_extra,
TopTransactionContext);
* In SVariableData it would be great to have more comments around
freeval, domain_check_extra, domain_check_extra_lxid.
* Nitpicking, but the term "shadowing" for ambiguity between a session
variable and a table column might be confusing, one can imagine there
is a connection between those two objects and one actively follows
("shadows") the other one.
* The second patch "Storage for session variables and SQL interface"
mentions in the documentation default and temporary variables:
<para>
The value of a session variable is local to the current session. Retrieving
a variable's value returns either a <literal>NULL</literal> or a default
value, unless its value has been set to something else in the current
session using the <command>LET</command> command. The content of a variable
is not transactional. This is the same as regular variables in PL languages.
The session variables can be persistent or can be temporary. In both cases,
the content of session variables is temporary and not shared (like an
content of temporary tables).
</para>
They're implemented in the following patches, so it belongs there.
* Nitpicking, maybe merge those two conditions together for readability?
if (!needs_validation)
return;
/*
* Reset, this flag here, before we start the validation. It can be set to
* on by incomming sinval message.
*/
needs_validation = false;
if (!sessionvars)
return;
* This one is not very clear, what is the difference between "somewhere
inside a transaction" and "at the end of a transaction"?
/*
* This routine can be called somewhere inside transaction or at an transaction
* end. When atEOX argument is false, then we are inside transaction, and we
* don't want to throw entries related to session variables dropped in current
* transaction.
*/
# Language topic
Since this patch introduces a large body of documentation and
commentaries, I think it would benefit from a native speaker review.
I've stumbled upon few examples (attached with proposed wording, without
a diff extension to not confuse the CF bot), but otherwise if anyone
follows this thread, texts review is appreciated.
> On Sun, Dec 03, 2023 at 06:04:12AM +0100, Pavel Stehule wrote: > > Thank you for your review. Next two weeks I'll not too much time to work > on this patch - I have to work on some commercial work, and the week is > Prague PgConf, so my reply will be slow. But after these events I'll > concentrate on this patch. No worries, it's fine. Have fun at PGConf!
Вложения
- v20231209-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20231209-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20231209-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20231209-0005-implementation-of-temporary-session-variables.patch
- v20231209-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20231209-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20231209-0003-DISCARD-VARIABLES.patch
- v20231209-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20231209-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20231220-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20231220-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20231220-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20231220-0005-implementation-of-temporary-session-variables.patch
- v20231220-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20231220-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20231220-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20231220-0003-DISCARD-VARIABLES.patch
- v20231220-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20231229-0009-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20231229-0007-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20231229-0008-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20231229-0006-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20231229-0005-implementation-of-temporary-session-variables.patch
- v20231229-0004-memory-cleaning-after-DROP-VARIABLE.patch
- v20231229-0003-DISCARD-VARIABLES.patch
- v20231229-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20231229-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- 0001-Enhancing-catalog-for-support-session-variables-and-.patch
- 0003-EXPLAIN-LET-support.patch
- 0004-PREPARE-LET-support.patch
- 0005-plpgsql-tests.patch
- 0002-Storage-for-session-variables-and-SQL-interface.patch
- 0009-implementation-of-temporary-session-variables.patch
- 0006-function-pg_session_variables-for-cleaning-tests.patch
- 0007-DISCARD-VARIABLES.patch
- 0008-memory-cleaning-after-DROP-VARIABLE.patch
- 0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- 0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- 0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- 0013-allow-read-an-value-of-session-variable-directly-fro.patch
- 0014-allow-parallel-execution-queries-with-session-variab.patch
- 0016-GUC-session_variables_ambiguity_warning.patch
- 0015-plpgsql-implementation-for-LET-statement.patch
- 0017-this-patch-changes-error-message-column-doesn-t-exis.patch
- 0018-expression-with-session-variables-can-be-inlined.patch
Hinew update. I separated functionality to more patches.
RegardsPavel
Вложения
- v20240123-0004-PREPARE-LET-support.patch
- v20240123-0003-EXPLAIN-LET-support.patch
- v20240123-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20240123-0005-plpgsql-tests.patch
- v20240123-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240123-0006-function-pg_session_variables-for-cleaning-tests.patch
- v20240123-0009-implementation-of-temporary-session-variables.patch
- v20240123-0007-DISCARD-VARIABLES.patch
- v20240123-0008-memory-cleaning-after-DROP-VARIABLE.patch
- v20240123-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240123-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240123-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240123-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240123-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240123-0015-plpgsql-implementation-for-LET-statement.patch
- v20240123-0016-GUC-session_variables_ambiguity_warning.patch
- v20240123-0017-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240123-0018-expression-with-session-variables-can-be-inlined.patch
Вложения
- v20240124-0003-EXPLAIN-LET-support.patch
- v20240124-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20240124-0005-plpgsql-tests.patch
- v20240124-0004-PREPARE-LET-support.patch
- v20240124-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240124-0006-function-pg_session_variables-for-cleaning-tests.patch
- v20240124-0007-DISCARD-VARIABLES.patch
- v20240124-0009-implementation-of-temporary-session-variables.patch
- v20240124-0008-memory-cleaning-after-DROP-VARIABLE.patch
- v20240124-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240124-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240124-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240124-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240124-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240124-0015-plpgsql-implementation-for-LET-statement.patch
- v20240124-0016-GUC-session_variables_ambiguity_warning.patch
- v20240124-0018-expression-with-session-variables-can-be-inlined.patch
- v20240124-0017-this-patch-changes-error-message-column-doesn-t-exis.patch
Вложения
- v20240125-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240125-0004-PREPARE-LET-support.patch
- v20240125-0003-EXPLAIN-LET-support.patch
- v20240125-0005-plpgsql-tests.patch
- v20240125-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20240125-0006-function-pg_session_variables-for-cleaning-tests.patch
- v20240125-0008-memory-cleaning-after-DROP-VARIABLE.patch
- v20240125-0007-DISCARD-VARIABLES.patch
- v20240125-0009-implementation-of-temporary-session-variables.patch
- v20240125-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240125-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240125-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240125-0015-plpgsql-implementation-for-LET-statement.patch
- v20240125-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240125-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240125-0017-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240125-0016-GUC-session_variables_ambiguity_warning.patch
- v20240125-0018-expression-with-session-variables-can-be-inlined.patch
Thanks for the update, smaller patches looks promising. Off the list Pavel has mentioned that the first two patches contain a bare minimum for session variables, so I've reviewed them once more and suggest to concentrate on them first. I'm afraid the memory cleanup patch has to be added to the "bare minimum" set as well -- otherwise in my tests it was too easy to run out of memory via creating, assigning and dropping variables. Unfortunately one can't extract those three patches from the series and apply only them, the memory patch would have some conflicts. Can you maybe reshuffle the series to have those patches (1, 2 + 8) as first three? If that's possible, my proposal would be to proceed with them first. To the best of my knowledge they look good to me, except few minor details: * The documentation says in a couple of places (ddl.sgml, create_variable.sgml) that "Retrieving a session variable's value returns either a NULL or a default value", but as far as I see the default value feature is not implemented within first two patches. * Similar with mentioning immutable session variables in plpgsql.sgml . * Commentary to LookupVariable mentions a rowtype_only argument: +/* + * Returns oid of session variable specified by possibly qualified identifier. + * + * If not found, returns InvalidOid if missing_ok, else throws error. + * When rowtype_only argument is true the session variables of not + * composite types are ignored. This should to reduce possible collisions. + */ +Oid +LookupVariable(const char *nspname, + const char *varname, + bool missing_ok) but the function doesn't have it. * I've noticed an interesting result when a LET statement is used to assign a value without a subquery: create variable test as text; -- returns NULL select test; -- use repeat directly without a subquery let test = repeat("test", 100000); -- returns NULL select test; I was expecting to see an error here, is this a correct behaviour?
Thanks for the update, smaller patches looks promising.
Off the list Pavel has mentioned that the first two patches contain a
bare minimum for session variables, so I've reviewed them once more and
suggest to concentrate on them first. I'm afraid the memory cleanup
patch has to be added to the "bare minimum" set as well -- otherwise in
my tests it was too easy to run out of memory via creating, assigning
and dropping variables. Unfortunately one can't extract those three
patches from the series and apply only them, the memory patch would have
some conflicts. Can you maybe reshuffle the series to have those patches
(1, 2 + 8) as first three?
If that's possible, my proposal would be to proceed with them first. To the
best of my knowledge they look good to me, except few minor details:
* The documentation says in a couple of places (ddl.sgml,
create_variable.sgml) that "Retrieving a session variable's value
returns either a NULL or a default value", but as far as I see the
default value feature is not implemented within first two patches.
* Similar with mentioning immutable session variables in plpgsql.sgml .
* Commentary to LookupVariable mentions a rowtype_only argument:
+/*
+ * Returns oid of session variable specified by possibly qualified identifier.
+ *
+ * If not found, returns InvalidOid if missing_ok, else throws error.
+ * When rowtype_only argument is true the session variables of not
+ * composite types are ignored. This should to reduce possible collisions.
+ */
+Oid
+LookupVariable(const char *nspname,
+ const char *varname,
+ bool missing_ok)
but the function doesn't have it.
* I've noticed an interesting result when a LET statement is used to assign a
value without a subquery:
create variable test as text;
-- returns NULL
select test;
-- use repeat directly without a subquery
let test = repeat("test", 100000);
-- returns NULL
select test;
I was expecting to see an error here, is this a correct behaviour?
LET
(2024-01-28 20:32:12) postgres=# let test = repeat("test", 10);
LET
(2024-01-28 20:32:19) postgres=# select test;
┌──────────────────────┐
│ test │
╞══════════════════════╡
│ abababababababababab │
└──────────────────────┘
(1 row)
LET
(2024-01-28 20:32:48) postgres=# let test = repeat("test", 10);
LET
(2024-01-28 20:32:51) postgres=# select test;
┌──────┐
│ test │
╞══════╡
│ ∅ │
└──────┘
(1 row)
┌────────┐
│ repeat │
╞════════╡
│ ∅ │
└────────┘
(1 row)
> On Sun, Jan 28, 2024 at 08:34:40PM +0100, Pavel Stehule wrote: > > * I've noticed an interesting result when a LET statement is used to > > assign a > > value without a subquery: > > > > create variable test as text; > > -- returns NULL > > select test; > > > > -- use repeat directly without a subquery > > let test = repeat("test", 100000); > > > > -- returns NULL > > select test; > > > > I was expecting to see an error here, is this a correct behaviour? > > > > what is strange on this result? Never mind, I've got confused about the quotes here -- it was referring to the variable content, not a string.
Thanks for the update, smaller patches looks promising.
Off the list Pavel has mentioned that the first two patches contain a
bare minimum for session variables, so I've reviewed them once more and
suggest to concentrate on them first. I'm afraid the memory cleanup
patch has to be added to the "bare minimum" set as well -- otherwise in
my tests it was too easy to run out of memory via creating, assigning
and dropping variables. Unfortunately one can't extract those three
patches from the series and apply only them, the memory patch would have
some conflicts. Can you maybe reshuffle the series to have those patches
(1, 2 + 8) as first three?
If that's possible, my proposal would be to proceed with them first. To the
best of my knowledge they look good to me, except few minor details:
* The documentation says in a couple of places (ddl.sgml,
create_variable.sgml) that "Retrieving a session variable's value
returns either a NULL or a default value", but as far as I see the
default value feature is not implemented within first two patches.
* Similar with mentioning immutable session variables in plpgsql.sgml .
* Commentary to LookupVariable mentions a rowtype_only argument:
+/*
+ * Returns oid of session variable specified by possibly qualified identifier.
+ *
+ * If not found, returns InvalidOid if missing_ok, else throws error.
+ * When rowtype_only argument is true the session variables of not
+ * composite types are ignored. This should to reduce possible collisions.
+ */
+Oid
+LookupVariable(const char *nspname,
+ const char *varname,
+ bool missing_ok)
but the function doesn't have it.
* I've noticed an interesting result when a LET statement is used to assign a
value without a subquery:
create variable test as text;
-- returns NULL
select test;
-- use repeat directly without a subquery
let test = repeat("test", 100000);
-- returns NULL
select test;
I was expecting to see an error here, is this a correct behaviour?
Вложения
- v20240129-0017-expression-with-session-variables-can-be-inlined.patch
- v20240129-0016-GUC-session_variables_ambiguity_warning.patch
- v20240129-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240129-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240129-0015-plpgsql-implementation-for-LET-statement.patch
- v20240129-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240129-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240129-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240129-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240129-0009-implementation-of-temporary-session-variables.patch
- v20240129-0008-plpgsql-tests.patch
- v20240129-0007-PREPARE-LET-support.patch
- v20240129-0006-EXPLAIN-LET-support.patch
- v20240129-0004-DISCARD-VARIABLES.patch
- v20240129-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240129-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240129-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240129-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> On Mon, Jan 29, 2024 at 08:57:42AM +0100, Pavel Stehule wrote: > Hi > > ne 28. 1. 2024 v 19:00 odesílatel Dmitry Dolgov <9erthalion6@gmail.com> > napsal: > > > Thanks for the update, smaller patches looks promising. > > > > Off the list Pavel has mentioned that the first two patches contain a > > bare minimum for session variables, so I've reviewed them once more and > > suggest to concentrate on them first. I'm afraid the memory cleanup > > patch has to be added to the "bare minimum" set as well -- otherwise in > > my tests it was too easy to run out of memory via creating, assigning > > and dropping variables. Unfortunately one can't extract those three > > patches from the series and apply only them, the memory patch would have > > some conflicts. Can you maybe reshuffle the series to have those patches > > (1, 2 + 8) as first three? > > > > probably you need too > > 0006-function-pg_session_variables-for-cleaning-tests.patch and > 0007-DISCARD-VARIABLES.patch > > 6 is necessary for testing of cleaning Ok, let me take a look at those. Unless there are any objections, my plan would be to give it a final check and mark the CF item as ready for committer -- meaning the first 5 patches.
> On Mon, Jan 29, 2024 at 08:57:42AM +0100, Pavel Stehule wrote:
> Hi
>
> ne 28. 1. 2024 v 19:00 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
> > Thanks for the update, smaller patches looks promising.
> >
> > Off the list Pavel has mentioned that the first two patches contain a
> > bare minimum for session variables, so I've reviewed them once more and
> > suggest to concentrate on them first. I'm afraid the memory cleanup
> > patch has to be added to the "bare minimum" set as well -- otherwise in
> > my tests it was too easy to run out of memory via creating, assigning
> > and dropping variables. Unfortunately one can't extract those three
> > patches from the series and apply only them, the memory patch would have
> > some conflicts. Can you maybe reshuffle the series to have those patches
> > (1, 2 + 8) as first three?
> >
>
> probably you need too
>
> 0006-function-pg_session_variables-for-cleaning-tests.patch and
> 0007-DISCARD-VARIABLES.patch
>
> 6 is necessary for testing of cleaning
Ok, let me take a look at those. Unless there are any objections, my
plan would be to give it a final check and mark the CF item as ready for
committer -- meaning the first 5 patches.
Вложения
- v20240130-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240130-0016-GUC-session_variables_ambiguity_warning.patch
- v20240130-0017-expression-with-session-variables-can-be-inlined.patch
- v20240130-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240130-0015-plpgsql-implementation-for-LET-statement.patch
- v20240130-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240130-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240130-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240130-0009-implementation-of-temporary-session-variables.patch
- v20240130-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240130-0007-PREPARE-LET-support.patch
- v20240130-0008-plpgsql-tests.patch
- v20240130-0006-EXPLAIN-LET-support.patch
- v20240130-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240130-0004-DISCARD-VARIABLES.patch
- v20240130-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240130-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240130-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Yep, in this constellation the implementation holds much better (in terms of memory) in my create/let/drop testing. I've marked the CF item as ready for committer, but a note for anyone who would like to pick up it from here -- we're talking about first 5 patches here, up to the memory cleaning after DROP VARIABLE. It doesn't mean the rest is somehow not worth it, but I believe it's a good first step.
Yep, in this constellation the implementation holds much better (in
terms of memory) in my create/let/drop testing.
I've marked the CF item as ready for committer, but a note for anyone
who would like to pick up it from here -- we're talking about first 5
patches here, up to the memory cleaning after DROP VARIABLE. It doesn't
mean the rest is somehow not worth it, but I believe it's a good first
step.
Вложения
- v20240220-0016-GUC-session_variables_ambiguity_warning.patch
- v20240220-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240220-0017-expression-with-session-variables-can-be-inlined.patch
- v20240220-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240220-0015-plpgsql-implementation-for-LET-statement.patch
- v20240220-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240220-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240220-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240220-0009-implementation-of-temporary-session-variables.patch
- v20240220-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240220-0008-plpgsql-tests.patch
- v20240220-0007-PREPARE-LET-support.patch
- v20240220-0006-EXPLAIN-LET-support.patch
- v20240220-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240220-0004-DISCARD-VARIABLES.patch
- v20240220-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240220-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240220-0019-transactional-variables.patch
- v20240220-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Hi,here is new rebase of this patch set.Years ago I promised to implement support for transactional behaviour. I wrote it in patch 0019. It is based on my patch from 2020 but the memory cleaning is more readable and I believe it is correct. All other patches are without touching. The first five patches are of "should to have" type, all others (with new one) are "nice to have" type (although support for simply expr evaluation or parallel execution has strong benefits).
RegardsPavel
Вложения
- v20240227-0016-GUC-session_variables_ambiguity_warning.patch
- v20240227-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240227-0015-plpgsql-implementation-for-LET-statement.patch
- v20240227-0017-expression-with-session-variables-can-be-inlined.patch
- v20240227-0019-transactional-variables.patch
- v20240227-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240227-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240227-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240227-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240227-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240227-0008-plpgsql-tests.patch
- v20240227-0007-PREPARE-LET-support.patch
- v20240227-0006-EXPLAIN-LET-support.patch
- v20240227-0009-implementation-of-temporary-session-variables.patch
- v20240227-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240227-0004-DISCARD-VARIABLES.patch
- v20240227-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240227-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240227-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240228-0019-transactional-variables.patch
- v20240228-0017-expression-with-session-variables-can-be-inlined.patch
- v20240228-0016-GUC-session_variables_ambiguity_warning.patch
- v20240228-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240228-0015-plpgsql-implementation-for-LET-statement.patch
- v20240228-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240228-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240228-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240228-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240228-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240228-0009-implementation-of-temporary-session-variables.patch
- v20240228-0007-PREPARE-LET-support.patch
- v20240228-0006-EXPLAIN-LET-support.patch
- v20240228-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240228-0004-DISCARD-VARIABLES.patch
- v20240228-0008-plpgsql-tests.patch
- v20240228-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240228-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240228-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240304-0019-transactional-variables.patch
- v20240304-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240304-0017-expression-with-session-variables-can-be-inlined.patch
- v20240304-0015-plpgsql-implementation-for-LET-statement.patch
- v20240304-0016-GUC-session_variables_ambiguity_warning.patch
- v20240304-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240304-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240304-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240304-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240304-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240304-0009-implementation-of-temporary-session-variables.patch
- v20240304-0008-plpgsql-tests.patch
- v20240304-0007-PREPARE-LET-support.patch
- v20240304-0006-EXPLAIN-LET-support.patch
- v20240304-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240304-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240304-0004-DISCARD-VARIABLES.patch
- v20240304-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240304-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240305-0017-expression-with-session-variables-can-be-inlined.patch
- v20240305-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240305-0019-transactional-variables.patch
- v20240305-0015-plpgsql-implementation-for-LET-statement.patch
- v20240305-0016-GUC-session_variables_ambiguity_warning.patch
- v20240305-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240305-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240305-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240305-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240305-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240305-0009-implementation-of-temporary-session-variables.patch
- v20240305-0008-plpgsql-tests.patch
- v20240305-0006-EXPLAIN-LET-support.patch
- v20240305-0007-PREPARE-LET-support.patch
- v20240305-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240305-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240305-0004-DISCARD-VARIABLES.patch
- v20240305-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240305-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240311-0019-transactional-variables.patch
- v20240311-0016-GUC-session_variables_ambiguity_warning.patch
- v20240311-0017-expression-with-session-variables-can-be-inlined.patch
- v20240311-0015-plpgsql-implementation-for-LET-statement.patch
- v20240311-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240311-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240311-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240311-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240311-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240311-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240311-0009-implementation-of-temporary-session-variables.patch
- v20240311-0007-PREPARE-LET-support.patch
- v20240311-0008-plpgsql-tests.patch
- v20240311-0006-EXPLAIN-LET-support.patch
- v20240311-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240311-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240311-0004-DISCARD-VARIABLES.patch
- v20240311-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240311-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240315-0017-expression-with-session-variables-can-be-inlined.patch
- v20240315-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240315-0016-GUC-session_variables_ambiguity_warning.patch
- v20240315-0015-plpgsql-implementation-for-LET-statement.patch
- v20240315-0019-transactional-variables.patch
- v20240315-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240315-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240315-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240315-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240315-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240315-0009-implementation-of-temporary-session-variables.patch
- v20240315-0008-plpgsql-tests.patch
- v20240315-0007-PREPARE-LET-support.patch
- v20240315-0006-EXPLAIN-LET-support.patch
- v20240315-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240315-0004-DISCARD-VARIABLES.patch
- v20240315-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240315-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240315-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240318-0017-expression-with-session-variables-can-be-inlined.patch
- v20240318-0019-transactional-variables.patch
- v20240318-0015-plpgsql-implementation-for-LET-statement.patch
- v20240318-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240318-0016-GUC-session_variables_ambiguity_warning.patch
- v20240318-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240318-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240318-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240318-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240318-0009-implementation-of-temporary-session-variables.patch
- v20240318-0008-plpgsql-tests.patch
- v20240318-0007-PREPARE-LET-support.patch
- v20240318-0006-EXPLAIN-LET-support.patch
- v20240318-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240318-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240318-0004-DISCARD-VARIABLES.patch
- v20240318-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240318-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240318-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240320-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240320-0017-expression-with-session-variables-can-be-inlined.patch
- v20240320-0016-GUC-session_variables_ambiguity_warning.patch
- v20240320-0015-plpgsql-implementation-for-LET-statement.patch
- v20240320-0019-transactional-variables.patch
- v20240320-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240320-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240320-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240320-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240320-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240320-0009-implementation-of-temporary-session-variables.patch
- v20240320-0008-plpgsql-tests.patch
- v20240320-0007-PREPARE-LET-support.patch
- v20240320-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240320-0006-EXPLAIN-LET-support.patch
- v20240320-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240320-0004-DISCARD-VARIABLES.patch
- v20240320-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240320-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240320-0017-expression-with-session-variables-can-be-inlined.patch
- v20240320-0019-transactional-variables.patch
- v20240320-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240320-0015-plpgsql-implementation-for-LET-statement.patch
- v20240320-0016-GUC-session_variables_ambiguity_warning.patch
- v20240320-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240320-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240320-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240320-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240320-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240320-0009-implementation-of-temporary-session-variables.patch
- v20240320-0008-plpgsql-tests.patch
- v20240320-0007-PREPARE-LET-support.patch
- v20240320-0006-EXPLAIN-LET-support.patch
- v20240320-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240320-0004-DISCARD-VARIABLES.patch
- v20240320-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240320-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240320-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240326-0017-expression-with-session-variables-can-be-inlined.patch
- v20240326-0019-transactional-variables.patch
- v20240326-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240326-0016-GUC-session_variables_ambiguity_warning.patch
- v20240326-0015-plpgsql-implementation-for-LET-statement.patch
- v20240326-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240326-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240326-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240326-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240326-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240326-0009-implementation-of-temporary-session-variables.patch
- v20240326-0008-plpgsql-tests.patch
- v20240326-0007-PREPARE-LET-support.patch
- v20240326-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240326-0006-EXPLAIN-LET-support.patch
- v20240326-0004-DISCARD-VARIABLES.patch
- v20240326-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240326-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240326-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240331-0016-GUC-session_variables_ambiguity_warning.patch
- v20240331-0017-expression-with-session-variables-can-be-inlined.patch
- v20240331-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240331-0015-plpgsql-implementation-for-LET-statement.patch
- v20240331-0019-transactional-variables.patch
- v20240331-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240331-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240331-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240331-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240331-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240331-0009-implementation-of-temporary-session-variables.patch
- v20240331-0008-plpgsql-tests.patch
- v20240331-0006-EXPLAIN-LET-support.patch
- v20240331-0007-PREPARE-LET-support.patch
- v20240331-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240331-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240331-0004-DISCARD-VARIABLES.patch
- v20240331-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20240331-0002-Storage-for-session-variables-and-SQL-interface.patch
Вложения
- v20240402-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240402-0019-transactional-variables.patch
- v20240402-0016-GUC-session_variables_ambiguity_warning.patch
- v20240402-0017-expression-with-session-variables-can-be-inlined.patch
- v20240402-0015-plpgsql-implementation-for-LET-statement.patch
- v20240402-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240402-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240402-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240402-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240402-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240402-0008-plpgsql-tests.patch
- v20240402-0009-implementation-of-temporary-session-variables.patch
- v20240402-0007-PREPARE-LET-support.patch
- v20240402-0006-EXPLAIN-LET-support.patch
- v20240402-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240402-0004-DISCARD-VARIABLES.patch
- v20240402-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240402-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240402-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240404-0019-transactional-variables.patch
- v20240404-0016-GUC-session_variables_ambiguity_warning.patch
- v20240404-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240404-0017-expression-with-session-variables-can-be-inlined.patch
- v20240404-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240404-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240404-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240404-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240404-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240404-0015-plpgsql-implementation-for-LET-statement.patch
- v20240404-0009-implementation-of-temporary-session-variables.patch
- v20240404-0008-plpgsql-tests.patch
- v20240404-0007-PREPARE-LET-support.patch
- v20240404-0006-EXPLAIN-LET-support.patch
- v20240404-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240404-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240404-0004-DISCARD-VARIABLES.patch
- v20240404-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240404-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240411-0019-transactional-variables.patch
- v20240411-0017-expression-with-session-variables-can-be-inlined.patch
- v20240411-0016-GUC-session_variables_ambiguity_warning.patch
- v20240411-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240411-0015-plpgsql-implementation-for-LET-statement.patch
- v20240411-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240411-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240411-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240411-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240411-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240411-0008-plpgsql-tests.patch
- v20240411-0009-implementation-of-temporary-session-variables.patch
- v20240411-0006-EXPLAIN-LET-support.patch
- v20240411-0007-PREPARE-LET-support.patch
- v20240411-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240411-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240411-0004-DISCARD-VARIABLES.patch
- v20240411-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240411-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240509-0019-transactional-variables.patch
- v20240509-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240509-0016-GUC-session_variables_ambiguity_warning.patch
- v20240509-0017-expression-with-session-variables-can-be-inlined.patch
- v20240509-0015-plpgsql-implementation-for-LET-statement.patch
- v20240509-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240509-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240509-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240509-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240509-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240509-0009-implementation-of-temporary-session-variables.patch
- v20240509-0008-plpgsql-tests.patch
- v20240509-0007-PREPARE-LET-support.patch
- v20240509-0006-EXPLAIN-LET-support.patch
- v20240509-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240509-0004-DISCARD-VARIABLES.patch
- v20240509-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240509-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240509-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240517-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240517-0019-transactional-variables.patch
- v20240517-0015-plpgsql-implementation-for-LET-statement.patch
- v20240517-0016-GUC-session_variables_ambiguity_warning.patch
- v20240517-0017-expression-with-session-variables-can-be-inlined.patch
- v20240517-0014-allow-parallel-execution-queries-with-session-variab.patch
- v20240517-0013-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240517-0010-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240517-0012-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240517-0011-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240517-0009-implementation-of-temporary-session-variables.patch
- v20240517-0008-plpgsql-tests.patch
- v20240517-0007-PREPARE-LET-support.patch
- v20240517-0006-EXPLAIN-LET-support.patch
- v20240517-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240517-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240517-0004-DISCARD-VARIABLES.patch
- v20240517-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240517-0001-Enhancing-catalog-for-support-session-variables-and-.patch
On 2024-Jan-30, Dmitry Dolgov wrote: > Yep, in this constellation the implementation holds much better (in > terms of memory) in my create/let/drop testing. > > I've marked the CF item as ready for committer, but a note for anyone > who would like to pick up it from here -- we're talking about first 5 > patches here, up to the memory cleaning after DROP VARIABLE. It doesn't > mean the rest is somehow not worth it, but I believe it's a good first > step. Hmm, I think patch 16 is essential, because the point of variable shadowing is a critical aspect of how the whole thing works. So I would say that a first step would be those first five patches plus 16. I want to note that when we discussed this patch series at the dev meeting in FOSDEM, a sort-of conclusion was reached that we didn't want schema variables at all because of the fact that creating a variable would potentially change the meaning of queries by shadowing table columns. But this turns out to be incorrect: it's _variables_ that are shadowed by table columns, not the other way around. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)
On 2024-Jan-30, Dmitry Dolgov wrote:
> Yep, in this constellation the implementation holds much better (in
> terms of memory) in my create/let/drop testing.
>
> I've marked the CF item as ready for committer, but a note for anyone
> who would like to pick up it from here -- we're talking about first 5
> patches here, up to the memory cleaning after DROP VARIABLE. It doesn't
> mean the rest is somehow not worth it, but I believe it's a good first
> step.
Hmm, I think patch 16 is essential, because the point of variable shadowing
is a critical aspect of how the whole thing works. So I would say that
a first step would be those first five patches plus 16.
I want to note that when we discussed this patch series at the dev
meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
schema variables at all because of the fact that creating a variable
would potentially change the meaning of queries by shadowing table
columns. But this turns out to be incorrect: it's _variables_ that are
shadowed by table columns, not the other way around.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)
so 18. 5. 2024 v 18:31 odesílatel Alvaro Herrera <alvherre@alvh.no-ip.org> napsal:On 2024-Jan-30, Dmitry Dolgov wrote:
> Yep, in this constellation the implementation holds much better (in
> terms of memory) in my create/let/drop testing.
>
> I've marked the CF item as ready for committer, but a note for anyone
> who would like to pick up it from here -- we're talking about first 5
> patches here, up to the memory cleaning after DROP VARIABLE. It doesn't
> mean the rest is somehow not worth it, but I believe it's a good first
> step.
Hmm, I think patch 16 is essential, because the point of variable shadowing
is a critical aspect of how the whole thing works. So I would say that
a first step would be those first five patches plus 16.I'll move patch 16 to 6 position
0007-GUC-session_variables_ambiguity_warning.patch
RegardsPavel
I want to note that when we discussed this patch series at the dev
meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
schema variables at all because of the fact that creating a variable
would potentially change the meaning of queries by shadowing table
columns. But this turns out to be incorrect: it's _variables_ that are
shadowed by table columns, not the other way around.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)
Вложения
- v20240521-0019-transactional-variables.patch
- v20240521-0016-plpgsql-implementation-for-LET-statement.patch
- v20240521-0015-allow-parallel-execution-queries-with-session-variab.patch
- v20240521-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240521-0017-expression-with-session-variables-can-be-inlined.patch
- v20240521-0013-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240521-0014-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240521-0012-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240521-0011-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240521-0010-implementation-of-temporary-session-variables.patch
- v20240521-0008-EXPLAIN-LET-support.patch
- v20240521-0009-PREPARE-LET-support.patch
- v20240521-0007-GUC-session_variables_ambiguity_warning.patch
- v20240521-0006-plpgsql-tests.patch
- v20240521-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240521-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240521-0004-DISCARD-VARIABLES.patch
- v20240521-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240521-0001-Enhancing-catalog-for-support-session-variables-and-.patch
On 18.05.24 13:29, Alvaro Herrera wrote: > I want to note that when we discussed this patch series at the dev > meeting in FOSDEM, a sort-of conclusion was reached that we didn't want > schema variables at all because of the fact that creating a variable > would potentially change the meaning of queries by shadowing table > columns. But this turns out to be incorrect: it's_variables_ that are > shadowed by table columns, not the other way around. But that's still bad, because seemingly unrelated schema changes can make variables appear and disappear. For example, if you have SELECT a, b FROM table1 and then you drop column b, maybe the above query continues to work because there is also a variable b. Or maybe it now does different things because b is of a different type. This all has the potential to be very confusing.
> On Wed, May 22, 2024 at 02:37:49PM +0200, Peter Eisentraut wrote: > On 18.05.24 13:29, Alvaro Herrera wrote: > > I want to note that when we discussed this patch series at the dev > > meeting in FOSDEM, a sort-of conclusion was reached that we didn't want > > schema variables at all because of the fact that creating a variable > > would potentially change the meaning of queries by shadowing table > > columns. But this turns out to be incorrect: it's_variables_ that are > > shadowed by table columns, not the other way around. > > But that's still bad, because seemingly unrelated schema changes can make > variables appear and disappear. For example, if you have > > SELECT a, b FROM table1 > > and then you drop column b, maybe the above query continues to work because > there is also a variable b. Or maybe it now does different things because b > is of a different type. This all has the potential to be very confusing. Yeah, that's a bummer. Interestingly enough, the db2 implementation of global session variables mechanism is mentioned as similar to what we have in the patch. But weirdly, the db2 documentation just states possibility of a resolution conflict for unqualified names, nothing else. There was extensive discussion about this problem early in the thread, and one alternative is to use some sort of special syntax every time when working with a variable to clear any ambiguity [1]. It's more verbose, has to be careful to not block some useful syntax for other stuff, etc. But as Pavel said: > The different syntax disallows any collision well, it is far to what is > more usual standard in this area. And if we introduce special syntax, then > there is no way back. We cannot use :varname - this syntax is used already, > but we can use, theoretically, @var or $var. But, personally, I don't want > to use it, if there is possibility to do without it. It seems to me there is no other possibility to resolve those ambiguity issues. [1]: https://www.postgresql.org/message-id/CAFj8pRD03hwZK%2B541KDt4Eo5YuC81CBBX_P0Sa5A7g5TQFsTww%40mail.gmail.com
Peter Eisentraut <peter@eisentraut.org> writes: > On 18.05.24 13:29, Alvaro Herrera wrote: >> I want to note that when we discussed this patch series at the dev >> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want >> schema variables at all because of the fact that creating a variable >> would potentially change the meaning of queries by shadowing table >> columns. But this turns out to be incorrect: it's_variables_ that are >> shadowed by table columns, not the other way around. > But that's still bad, because seemingly unrelated schema changes can > make variables appear and disappear. For example, if you have > SELECT a, b FROM table1 > and then you drop column b, maybe the above query continues to work > because there is also a variable b. Yeah, that seems pretty dangerous. Could we make it safe enough by requiring some qualification on variable names? That is, if you mean b to be a variable, then you must write something like SELECT a, pg_variables.b FROM table1 This is still ambiguous if you use "pg_variables" as a table alias in the query, but the alias would win so the query still means what it meant before. Also, table aliases (as opposed to actual table names) don't change readily, so I don't think there's much risk of the query suddenly meaning something different than it did yesterday. regards, tom lane
On 2024-May-22, Dmitry Dolgov wrote: > Yeah, that's a bummer. Interestingly enough, the db2 implementation of > global session variables mechanism is mentioned as similar to what we > have in the patch. But weirdly, the db2 documentation just states > possibility of a resolution conflict for unqualified names, nothing > else. Perhaps the solution to all this is to avoid having the variables be implicitly present in the range table of all queries. Instead, if you need a variable's value, then you need to add the variable to the FROM clause; and if you try to read from the variable and the name conflicts with that of a column in one of the tables in the FROM clause, then you get an error that the name is ambiguous and invites to qualify it. Like, for instance, create table lefttab (a int, b int); create table righttab (c int, d int, b int); =# select b from lefttab, righttab; ERROR: column reference "b" is ambiguous LÍNEA 1: select b from lefttab, righttab; ^ but this works fine because there's no longer an ambiguity: select lefttab.b from lefttab, righttab; b ─── (0 filas) Nothing breaks if you create new variables, because your queries won't see them until you explicitly request them. And if you add add columns to either tables or variables, it's possible that some queries would start having ambiguous references, in which case they'll just stop working until you disambiguate by editing the query. Now, Pavel has been saying that variables are simple and cannot break queries (because they're always shadowed), which is why they're always implicitly visible to all queries[1]; but maybe that's a mistake. [1] https://postgr.es/m/CAFj8pRA2P7uaFGpFJxVHrHFtizBCN41J00BrEotspdD+urGBLQ@mail.gmail.com -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "La experiencia nos dice que el hombre peló millones de veces las patatas, pero era forzoso admitir la posibilidad de que en un caso entre millones, las patatas pelarían al hombre" (Ijon Tichy)
Alvaro Herrera: > Perhaps the solution to all this is to avoid having the variables be > implicitly present in the range table of all queries. Instead, if you > need a variable's value, then you need to add the variable to the FROM > clause; +1 This should make it easier to work with composite type schema variables in some cases. It could also enable schema qualifying of schema variables, or at least make it easier to do, I think. In this case variables would share the same namespace as tables and views, right? So I could not create a variable with the same name as another table. Which is a good thing, I guess. Not sure how it's currently implemented in the patch. Best, Wolfgang
On 18.05.24 13:29, Alvaro Herrera wrote:
> I want to note that when we discussed this patch series at the dev
> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
> schema variables at all because of the fact that creating a variable
> would potentially change the meaning of queries by shadowing table
> columns. But this turns out to be incorrect: it's_variables_ that are
> shadowed by table columns, not the other way around.
But that's still bad, because seemingly unrelated schema changes can
make variables appear and disappear. For example, if you have
SELECT a, b FROM table1
and then you drop column b, maybe the above query continues to work
because there is also a variable b. Or maybe it now does different
things because b is of a different type. This all has the potential to
be very confusing.
Peter Eisentraut <peter@eisentraut.org> writes:
> On 18.05.24 13:29, Alvaro Herrera wrote:
>> I want to note that when we discussed this patch series at the dev
>> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
>> schema variables at all because of the fact that creating a variable
>> would potentially change the meaning of queries by shadowing table
>> columns. But this turns out to be incorrect: it's_variables_ that are
>> shadowed by table columns, not the other way around.
> But that's still bad, because seemingly unrelated schema changes can
> make variables appear and disappear. For example, if you have
> SELECT a, b FROM table1
> and then you drop column b, maybe the above query continues to work
> because there is also a variable b.
Yeah, that seems pretty dangerous. Could we make it safe enough
by requiring some qualification on variable names? That is, if
you mean b to be a variable, then you must write something like
SELECT a, pg_variables.b FROM table1
This is still ambiguous if you use "pg_variables" as a table alias in
the query, but the alias would win so the query still means what it
meant before. Also, table aliases (as opposed to actual table names)
don't change readily, so I don't think there's much risk of the query
suddenly meaning something different than it did yesterday.
regards, tom lane
Alvaro Herrera:
> Perhaps the solution to all this is to avoid having the variables be
> implicitly present in the range table of all queries. Instead, if you
> need a variable's value, then you need to add the variable to the FROM
> clause;
+1
This should make it easier to work with composite type schema variables
in some cases. It could also enable schema qualifying of schema
variables, or at least make it easier to do, I think.
In this case variables would share the same namespace as tables and
views, right? So I could not create a variable with the same name as
another table. Which is a good thing, I guess. Not sure how it's
currently implemented in the patch.
Best,
Wolfgang
On 18.05.24 13:29, Alvaro Herrera wrote:
> I want to note that when we discussed this patch series at the dev
> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
> schema variables at all because of the fact that creating a variable
> would potentially change the meaning of queries by shadowing table
> columns. But this turns out to be incorrect: it's_variables_ that are
> shadowed by table columns, not the other way around.
But that's still bad, because seemingly unrelated schema changes can
make variables appear and disappear. For example, if you have
SELECT a, b FROM table1
and then you drop column b, maybe the above query continues to work
because there is also a variable b. Or maybe it now does different
things because b is of a different type. This all has the potential to
be very confusing.
begin
raise notice '%', (select x from foo);
end;
$$;
WARNING: session variable "x" is shadowed
LINE 1: (select x from foo)
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
QUERY: (select x from foo)
NOTICE: 110
DO
(2024-05-23 08:22:35) postgres=# do $$ declare x int default 100;
begin
raise notice '%', x;
end;
$$;
WARNING: session variable "x" is shadowed
LINE 1: x
^
DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name.
QUERY: x
NOTICE: 100
DO
> On Wed, May 22, 2024 at 02:37:49PM +0200, Peter Eisentraut wrote:
> On 18.05.24 13:29, Alvaro Herrera wrote:
> > I want to note that when we discussed this patch series at the dev
> > meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
> > schema variables at all because of the fact that creating a variable
> > would potentially change the meaning of queries by shadowing table
> > columns. But this turns out to be incorrect: it's_variables_ that are
> > shadowed by table columns, not the other way around.
>
> But that's still bad, because seemingly unrelated schema changes can make
> variables appear and disappear. For example, if you have
>
> SELECT a, b FROM table1
>
> and then you drop column b, maybe the above query continues to work because
> there is also a variable b. Or maybe it now does different things because b
> is of a different type. This all has the potential to be very confusing.
Yeah, that's a bummer. Interestingly enough, the db2 implementation of
global session variables mechanism is mentioned as similar to what we
have in the patch. But weirdly, the db2 documentation just states
possibility of a resolution conflict for unqualified names, nothing
else.
> On Wed, May 22, 2024 at 08:44:28PM +0200, Pavel Stehule wrote: > st 22. 5. 2024 v 19:25 odesílatel Tom Lane <tgl@sss.pgh.pa.us> napsal: > > > Peter Eisentraut <peter@eisentraut.org> writes: > > > On 18.05.24 13:29, Alvaro Herrera wrote: > > >> I want to note that when we discussed this patch series at the dev > > >> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want > > >> schema variables at all because of the fact that creating a variable > > >> would potentially change the meaning of queries by shadowing table > > >> columns. But this turns out to be incorrect: it's_variables_ that are > > >> shadowed by table columns, not the other way around. > > > > > But that's still bad, because seemingly unrelated schema changes can > > > make variables appear and disappear. For example, if you have > > > SELECT a, b FROM table1 > > > and then you drop column b, maybe the above query continues to work > > > because there is also a variable b. > > > > Yeah, that seems pretty dangerous. Could we make it safe enough > > by requiring some qualification on variable names? That is, if > > you mean b to be a variable, then you must write something like > > > > SELECT a, pg_variables.b FROM table1 > > > > This is still ambiguous if you use "pg_variables" as a table alias in > > the query, but the alias would win so the query still means what it > > meant before. Also, table aliases (as opposed to actual table names) > > don't change readily, so I don't think there's much risk of the query > > suddenly meaning something different than it did yesterday. > > > > With active shadowing variable warning for described example you will get a > warning before dropping. I assume you're talking about a warning, which one will get querying the table with shadowed columns. If no such query has happened yet and the column was dropped, there will be no warning. Aside that, I'm afraid dropping a warning in log does not have sufficient visibility to warn about the issue, since one needs to read those logs first. I guess what folks are looking for is more constraints out of the box, preventing any ambiguity. > Session variables are joined with schema (in my proposal). Do anybody can > do just > > CREATE SCHEMA svars; -- or what (s)he likes > CREATE VARIABLE svars.b AS int; > > SELECT a, b FROM table1 > > and if somebody can be really safe, the can write > > SELECT t.a, t.b FROM table1 t > > or > > SELECT t.a, svars.b FROM table1 t > > It can be customized in the way anybody prefers - just creating dedicated > schemas and setting search_path. Using its own schema for session variables > without enhancing search_path for this schema forces the necessity to set > only qualified names for session variables. > > Sure the naming of schemas, aliases can be unhappy wrong, and there can be > the problem. But this can be a problem today too. If I understand you correctly, you're saying that there are "best practices" how to deal with session variables to avoid any potential issues. But I think it's more user-friendly to have something that will not allow shooting yourself in the foot right out of the box. You're right, similar things could probably happen with the already existing functionality, but it doesn't give us rights to add more to it. Especially if it's going to be about a brand-new feature. As far as I can see now, it's a major design flaw that could keep the patch from being accepted. Fortunately there are few good proposals how to address this, folks are genuinely trying to help. What do you think about trying some of them out, as an alternative approach, to compare functionality and user experience? In the meantime I'm afraid I have to withdraw "Ready for committer" status, sorry. I've clearly underestimated the importance of variables shadowing, thanks Alvaro and Peter for pointing out some dangerous cases. I still believe though that the majority of the patch is in a good shape and the question about variables shadowing is the only thing that keeps it from moving forward.
> On Wed, May 22, 2024 at 08:44:28PM +0200, Pavel Stehule wrote:
> st 22. 5. 2024 v 19:25 odesílatel Tom Lane <tgl@sss.pgh.pa.us> napsal:
>
> > Peter Eisentraut <peter@eisentraut.org> writes:
> > > On 18.05.24 13:29, Alvaro Herrera wrote:
> > >> I want to note that when we discussed this patch series at the dev
> > >> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
> > >> schema variables at all because of the fact that creating a variable
> > >> would potentially change the meaning of queries by shadowing table
> > >> columns. But this turns out to be incorrect: it's_variables_ that are
> > >> shadowed by table columns, not the other way around.
> >
> > > But that's still bad, because seemingly unrelated schema changes can
> > > make variables appear and disappear. For example, if you have
> > > SELECT a, b FROM table1
> > > and then you drop column b, maybe the above query continues to work
> > > because there is also a variable b.
> >
> > Yeah, that seems pretty dangerous. Could we make it safe enough
> > by requiring some qualification on variable names? That is, if
> > you mean b to be a variable, then you must write something like
> >
> > SELECT a, pg_variables.b FROM table1
> >
> > This is still ambiguous if you use "pg_variables" as a table alias in
> > the query, but the alias would win so the query still means what it
> > meant before. Also, table aliases (as opposed to actual table names)
> > don't change readily, so I don't think there's much risk of the query
> > suddenly meaning something different than it did yesterday.
> >
>
> With active shadowing variable warning for described example you will get a
> warning before dropping.
I assume you're talking about a warning, which one will get querying the
table with shadowed columns. If no such query has happened yet and the
column was dropped, there will be no warning.
Aside that, I'm afraid dropping a warning in log does not have
sufficient visibility to warn about the issue, since one needs to read
those logs first. I guess what folks are looking for is more constraints
out of the box, preventing any ambiguity.
> Session variables are joined with schema (in my proposal). Do anybody can
> do just
>
> CREATE SCHEMA svars; -- or what (s)he likes
> CREATE VARIABLE svars.b AS int;
>
> SELECT a, b FROM table1
>
> and if somebody can be really safe, the can write
>
> SELECT t.a, t.b FROM table1 t
>
> or
>
> SELECT t.a, svars.b FROM table1 t
>
> It can be customized in the way anybody prefers - just creating dedicated
> schemas and setting search_path. Using its own schema for session variables
> without enhancing search_path for this schema forces the necessity to set
> only qualified names for session variables.
>
> Sure the naming of schemas, aliases can be unhappy wrong, and there can be
> the problem. But this can be a problem today too.
If I understand you correctly, you're saying that there are "best
practices" how to deal with session variables to avoid any potential
issues. But I think it's more user-friendly to have something that will
not allow shooting yourself in the foot right out of the box. You're
right, similar things could probably happen with the already existing
functionality, but it doesn't give us rights to add more to it.
Especially if it's going to be about a brand-new feature.
As far as I can see now, it's a major design flaw that could keep the
patch from being accepted. Fortunately there are few good proposals how
to address this, folks are genuinely trying to help. What do you think
about trying some of them out, as an alternative approach, to compare
functionality and user experience?
In the meantime I'm afraid I have to withdraw "Ready for committer"
status, sorry. I've clearly underestimated the importance of variables
shadowing, thanks Alvaro and Peter for pointing out some dangerous
cases. I still believe though that the majority of the patch is in a
good shape and the question about variables shadowing is the only thing
that keeps it from moving forward.
As far as I can see now, it's a major design flaw that could keep the
patch from being accepted. Fortunately there are few good proposals how
to address this, folks are genuinely trying to help. What do you think
about trying some of them out, as an alternative approach, to compare
functionality and user experience?It is a design flaw of SQL. The issue we talk about is the generic property of SQL, and then you cannot fix it.I thought about possibility to introduce dedicated functionsvalue(regvariable) returns any - with planner supportand possibility to force usage of this function. Another possibility is using some simple dedicated operator (syntax) for force using of variablesso theoretically this can looks like:set strict_usage_of_session_variables to on;SELECT * FROM tab WHERE a = svalue('myvar.var');orSELECT * FROM tab WHERE a = @ myvar.var;This can be really safe. Personally It is not my cup of tea, but I can live it (and this mode can be default).Theoretically we can limit usage of variables just for PL/pgSQL. It can reduce risks too, but it breaks usage variables for parametrization of DO blocks (what is my primary motivation), but it can be good enough to support migration from PL/SQL.
In the meantime I'm afraid I have to withdraw "Ready for committer"
status, sorry. I've clearly underestimated the importance of variables
shadowing, thanks Alvaro and Peter for pointing out some dangerous
cases. I still believe though that the majority of the patch is in a
good shape and the question about variables shadowing is the only thing
that keeps it from moving forward.I understand.I'll try to recapitulate my objections against proposed designsa) using syntax like MS - DECLARE command and '@@' prefix - it is dynamic, so there is not possibility of static check. It is not joined with schema, so there are possible collisions between variables and and the end the variables are named like @@mypackage_myvar - so some custom naming convention is necessary too. There is not possibility to set access rights.b) using variables like MySQL - first usage define it, and access by '@' prefix. It is simple, but without possibility of static check. There is not possibility to set access rights.c) using variables with necessity to define it in FROM clause. It is safe, but it can be less readable, when you use more variables, and it is not too readable, and user friendly, because you need to write FROM. And can be messy, because you usually will use variables in queries, and it is introduce not relations into FROM clause. But I can imagine this mode as alternative syntax, but it is very unfriendly and not intuitive (I think). More probably it doesn't fast execution in simple expression execution mode.d) my proposal - there is possibility of collisions, but consistent with naming of database objects, allows set of access rights, allows static analyze, consistent with PL/pgSQL and similar to PL/pgSQL.There is not any other possibility. Any time this is war between be user friendly, be readable, be correctly - but there is not perfect solution, because just SQL is not perfect. Almost all mentioned objections against proposed variables are valid just for tables and columns.RegardsPavel
Peter Eisentraut <peter@eisentraut.org> writes:
> On 18.05.24 13:29, Alvaro Herrera wrote:
>> I want to note that when we discussed this patch series at the dev
>> meeting in FOSDEM, a sort-of conclusion was reached that we didn't want
>> schema variables at all because of the fact that creating a variable
>> would potentially change the meaning of queries by shadowing table
>> columns. But this turns out to be incorrect: it's_variables_ that are
>> shadowed by table columns, not the other way around.
> But that's still bad, because seemingly unrelated schema changes can
> make variables appear and disappear. For example, if you have
> SELECT a, b FROM table1
> and then you drop column b, maybe the above query continues to work
> because there is also a variable b.
Yeah, that seems pretty dangerous. Could we make it safe enough
by requiring some qualification on variable names? That is, if
you mean b to be a variable, then you must write something like
SELECT a, pg_variables.b FROM table1
This is still ambiguous if you use "pg_variables" as a table alias in
the query, but the alias would win so the query still means what it
meant before. Also, table aliases (as opposed to actual table names)
don't change readily, so I don't think there's much risk of the query
suddenly meaning something different than it did yesterday.
regards, tom lane
Pavel Stehule <pavel.stehule@gmail.com> writes: > we can introduce special safe mode started by > set enable_direct_variable_read to off; > and allowing access to variables only by usage dedicated function > (supported by parser) named like variable or pg_variable Didn't we learn twenty years ago that GUCs that change query semantics are an awful idea? Pick a single access method for these things and stick to it. regards, tom lane
Pavel Stehule <pavel.stehule@gmail.com> writes:
> we can introduce special safe mode started by
> set enable_direct_variable_read to off;
> and allowing access to variables only by usage dedicated function
> (supported by parser) named like variable or pg_variable
Didn't we learn twenty years ago that GUCs that change query
semantics are an awful idea? Pick a single access method
for these things and stick to it.
regards, tom lane
Pavel Stehule: > Sure there is more possibilities, but I don't want to lost the > possibility to write code like > > CREATE TEMP VARIABLE _x; > > LET _x = 'hello'; > > DO $$ > BEGIN > RAISE NOTICE '%', _x; > END; > $$; > > So I am searching for a way to do it safely, but still intuitive and > user friendly. Maybe a middle-way between this and Alvaro's proposal could be: Whenever you have a FROM clause, a variable must be added to it to be accessible. When you don't have a FROM clause, you can access it directly. This would make the following work: RAISE NOTICE '%', _x; SELECT _x; SELECT tbl.*, _x FROM tbl, _x; SELECT tbl.*, (SELECT _x) FROM tbl, _x; SELECT tbl.*, (SELECT _x FROM _x) FROM tbl; But the following would be an error: SELECT tbl.*, _x FROM tbl; SELECT tbl.*, (SELECT _x) FROM tbl; Best, Wolfgang
Pavel Stehule:
> Sure there is more possibilities, but I don't want to lost the
> possibility to write code like
>
> CREATE TEMP VARIABLE _x;
>
> LET _x = 'hello';
>
> DO $$
> BEGIN
> RAISE NOTICE '%', _x;
> END;
> $$;
>
> So I am searching for a way to do it safely, but still intuitive and
> user friendly.
Maybe a middle-way between this and Alvaro's proposal could be:
Whenever you have a FROM clause, a variable must be added to it to be
accessible. When you don't have a FROM clause, you can access it directly.
This would make the following work:
RAISE NOTICE '%', _x;
SELECT _x;
SELECT tbl.*, _x FROM tbl, _x;
SELECT tbl.*, (SELECT _x) FROM tbl, _x;
SELECT tbl.*, (SELECT _x FROM _x) FROM tbl;
But the following would be an error:
SELECT tbl.*, _x FROM tbl;
SELECT tbl.*, (SELECT _x) FROM tbl;
Best,
Wolfgang
Pavel Stehule <pavel.stehule@gmail.com> writes:
> we can introduce special safe mode started by
> set enable_direct_variable_read to off;
> and allowing access to variables only by usage dedicated function
> (supported by parser) named like variable or pg_variable
Didn't we learn twenty years ago that GUCs that change query
semantics are an awful idea? Pick a single access method
for these things and stick to it.
regards, tom lane
> On Tue, May 28, 2024 at 05:18:02PM GMT, Pavel Stehule wrote: > > I propose another variants. First we can introduce pseudo function VAR( ). > The argument should be session variables. The name of this function can be > pgvar, globvar, ... We can talk about good name, it should not be too long, > but it is not important now. The VAR() function will be pseudo function > like COALESCE, so we can easily to set correct result type. So, the purpose of the function would be only to verify that the argument is a session variable? That seems to be a very light payload, which looks a bit awkward. Out of those options you propose I think the first one is the most straightforward one, but... > Alvaro Herrera: > > Perhaps the solution to all this is to avoid having the variables be > > implicitly present in the range table of all queries. Instead, if you > > need a variable's value, then you need to add the variable to the FROM > > clause; The more I think about this, the more I like this solution. Marking which variables are available to the query this way, and using established patterns for resolving ambiguity actually looks intuitive to me. Now I know, you've got strong objections: > I don't like this. Sure, this fixes the problem with collisions, but then > we cannot talk about variables. When some is used like a table, then it > should be a table. I can imagine memory tables, but it is a different type > of object. Table is relation, variable is just value. Variables should not > have columns, so using the same patterns for tables and variables has no > sense. Using the same catalog for variables and tables. Variables just hold > a value, and then you can use it inside a query without necessity to write > JOIN. Variables are not tables, and then it is not too confusing so they > are not transactional and don't support more rows, more columns. A FROM clause could contain a function returning a single value, nobody finds it confusing. And at least to me it's not much different from having a session variable as well, what do you think? > c) using variables with necessity to define it in FROM clause. It is safe, > but it can be less readable, when you use more variables, and it is not too > readable, and user friendly, because you need to write FROM. And can be > messy, because you usually will use variables in queries, and it is > introduce not relations into FROM clause. But I can imagine this mode as > alternative syntax, but it is very unfriendly and not intuitive (I think). The proposal from Wolfgang to have a short-cut and not add FROM in case there is no danger of ambiguity seems to resolve that. > More probably it doesn't fast execution in simple expression execution mode. Could you elaborate more, what do you mean by that? If the performance overhead is not prohibitive (which I would expect is the case), having better UX for a new feature usually beats having better performance. > It looks odd - It is not intuitive, it introduces new inconsistency inside > Postgres, or with solutions in other databases. No other database has a > similar rule, so users coming from Oracle, Db2, or MSSQL, Firebird will be > confused. Users that use PL/pgSQL will be confused. Session variables are not part of the SQL standard, and maintaining consistency with other databases is a questionable goal. Since it's a new feature, I'm not sure what you mean by inconsistency inside Postgres itself. I see that the main driving case behind this patch is to help with migrating from other databases that do have session variables. Going with variables in FROM clause, will not make a migration much harder -- some of the queries would have to modify the FROM part, and that's it, right? I could imagine it would be even easier than adding VAR() everywhere.
> On Tue, May 28, 2024 at 05:18:02PM GMT, Pavel Stehule wrote:
>
> I propose another variants. First we can introduce pseudo function VAR( ).
> The argument should be session variables. The name of this function can be
> pgvar, globvar, ... We can talk about good name, it should not be too long,
> but it is not important now. The VAR() function will be pseudo function
> like COALESCE, so we can easily to set correct result type.
So, the purpose of the function would be only to verify that the argument is a
session variable? That seems to be a very light payload, which looks a bit
awkward.
Out of those options you propose I think the first one is the
most straightforward one, but...
> Alvaro Herrera:
> > Perhaps the solution to all this is to avoid having the variables be
> > implicitly present in the range table of all queries. Instead, if you
> > need a variable's value, then you need to add the variable to the FROM
> > clause;
The more I think about this, the more I like this solution. Marking
which variables are available to the query this way, and using established
patterns for resolving ambiguity actually looks intuitive to me. Now I know,
you've got strong objections:
CREATE TYPE
(2024-05-31 12:35:26) postgres=# create function fx() returns t as $$ select 1, 2 $$ language sql;
CREATE FUNCTION
(2024-05-31 12:35:44) postgres=# select fx();
┌───────┐
│ fx │
╞═══════╡
│ (1,2) │
└───────┘
(1 row)
(2024-05-31 12:35:47) postgres=# select * from fx();
┌───┬───┐
│ a │ b │
╞═══╪═══╡
│ 1 │ 2 │
└───┴───┘
(1 row)
> I don't like this. Sure, this fixes the problem with collisions, but then
> we cannot talk about variables. When some is used like a table, then it
> should be a table. I can imagine memory tables, but it is a different type
> of object. Table is relation, variable is just value. Variables should not
> have columns, so using the same patterns for tables and variables has no
> sense. Using the same catalog for variables and tables. Variables just hold
> a value, and then you can use it inside a query without necessity to write
> JOIN. Variables are not tables, and then it is not too confusing so they
> are not transactional and don't support more rows, more columns.
A FROM clause could contain a function returning a single value, nobody
finds it confusing. And at least to me it's not much different from having a
session variable as well, what do you think?
> c) using variables with necessity to define it in FROM clause. It is safe,
> but it can be less readable, when you use more variables, and it is not too
> readable, and user friendly, because you need to write FROM. And can be
> messy, because you usually will use variables in queries, and it is
> introduce not relations into FROM clause. But I can imagine this mode as
> alternative syntax, but it is very unfriendly and not intuitive (I think).
The proposal from Wolfgang to have a short-cut and not add FROM in case there
is no danger of ambiguity seems to resolve that.
> More probably it doesn't fast execution in simple expression execution mode.
Could you elaborate more, what do you mean by that? If the performance
overhead is not prohibitive (which I would expect is the case), having better
UX for a new feature usually beats having better performance.
> It looks odd - It is not intuitive, it introduces new inconsistency inside
> Postgres, or with solutions in other databases. No other database has a
> similar rule, so users coming from Oracle, Db2, or MSSQL, Firebird will be
> confused. Users that use PL/pgSQL will be confused.
Session variables are not part of the SQL standard, and maintaining
consistency with other databases is a questionable goal. Since it's a new
feature, I'm not sure what you mean by inconsistency inside Postgres itself.
I see that the main driving case behind this patch is to help with
migrating from other databases that do have session variables. Going with
variables in FROM clause, will not make a migration much harder -- some of the
queries would have to modify the FROM part, and that's it, right? I could
imagine it would be even easier than adding VAR() everywhere.
Pavel Stehule: > 2. But my main argument is, it is not really safe - it solves Peter's > use case, but if I use a reverse example of Peter's case, I still have a > problem. > > I can have a variable x, and then I can write query like `SELECT x FROM x`; > > but if somebody creates table x(x int), then the query `SELECT x FROM x` > will be correct, but it is surely something else. So the requirement of > the usage variable inside FROM clause doesn't help. It doesn't work. But in this case you could make variables and tables share the same namespace, i.e. forbid creating a variable with the same name as an already existing table. Best, Wolfgang
Pavel Stehule:
> 2. But my main argument is, it is not really safe - it solves Peter's
> use case, but if I use a reverse example of Peter's case, I still have a
> problem.
>
> I can have a variable x, and then I can write query like `SELECT x FROM x`;
>
> but if somebody creates table x(x int), then the query `SELECT x FROM x`
> will be correct, but it is surely something else. So the requirement of
> the usage variable inside FROM clause doesn't help. It doesn't work.
But in this case you could make variables and tables share the same
namespace, i.e. forbid creating a variable with the same name as an
already existing table.
Best,
Wolfgang
Pavel Stehule: > But in this case you could make variables and tables share the same > namespace, i.e. forbid creating a variable with the same name as an > already existing table. > > > It helps, but not on 100% - there is a search path I think we can ignore the search_path for this discussion. That's not a problem of variables vs tables, but just a search path related problem. It is exactly the same thing right now, when you create a new table x(x) in a schema which happens to be earlier in your search path. The objection to the proposed approach for variables was that it would introduce *new* ambiguities, which Alvaro's suggestion avoids. Best, Wolfgang
Pavel Stehule:
> But in this case you could make variables and tables share the same
> namespace, i.e. forbid creating a variable with the same name as an
> already existing table.
>
>
> It helps, but not on 100% - there is a search path
I think we can ignore the search_path for this discussion. That's not a
problem of variables vs tables, but just a search path related problem.
It is exactly the same thing right now, when you create a new table x(x)
in a schema which happens to be earlier in your search path.
The objection to the proposed approach for variables was that it would
introduce *new* ambiguities, which Alvaro's suggestion avoids.
Best,
Wolfgang
pá 31. 5. 2024 v 13:37 odesílatel Wolfgang Walther <walther@technowledgy.de> napsal:Pavel Stehule:
> But in this case you could make variables and tables share the same
> namespace, i.e. forbid creating a variable with the same name as an
> already existing table.
>
>
> It helps, but not on 100% - there is a search path
I think we can ignore the search_path for this discussion. That's not a
problem of variables vs tables, but just a search path related problem.
It is exactly the same thing right now, when you create a new table x(x)
in a schema which happens to be earlier in your search path.I don't think it is a valid argument - search_path is there, and we cannot ignore it, because it allows just one case.And the need to use a variable in FROM clause introduces implicit unpacking or inconsistency with current work with composite's types, so I am more sure this way is not good.
The objection to the proposed approach for variables was that it would
introduce *new* ambiguities, which Alvaro's suggestion avoids.
Best,
Wolfgang
Pavel Stehule: > The session variables can be used in queries, but should be used in > PL/pgSQL expressions, and then the mandatory usage in FROM clause will > do lot of problems and unreadable code like > > DO $$ > BEGIN > RAISE NOTICE '% %', (SELECT x FROM x), (SELECT a,b FROM y); > > END > $$ > > This requirement does variables unusable in PL I already proposed earlier to only require listing them in FROM when there is actually a related FROM. In this case you could still write: RAISE NOTICE '% %', x, (SELECT a,b FROM y); (assuming only x is a variable here) Best, Wolfgang
Pavel Stehule:
> The session variables can be used in queries, but should be used in
> PL/pgSQL expressions, and then the mandatory usage in FROM clause will
> do lot of problems and unreadable code like
>
> DO $$
> BEGIN
> RAISE NOTICE '% %', (SELECT x FROM x), (SELECT a,b FROM y);
>
> END
> $$
>
> This requirement does variables unusable in PL
I already proposed earlier to only require listing them in FROM when
there is actually a related FROM.
In this case you could still write:
RAISE NOTICE '% %', x, (SELECT a,b FROM y);
(assuming only x is a variable here)
Best,
Wolfgang
In this case you could still write:
RAISE NOTICE '% %', x, (SELECT a,b FROM y);
(assuming only x is a variable here)
Best,
Wolfgang
Pavel Stehule: > When you write RAISE NOTICE '%', x, then PLpgSQL parser rewrite it to > RAISE NOTICE '%', SELECT $1 > > There is no parser just for expressions. That's why my suggestion in [1] already made a difference between: SELECT var; and SELECT col, var FROM table, var; So the "only require variable-in-FROM if FROM is used" should extend to the SQL level. That should be possible, right? Best, Wolfgang [1]: https://www.postgresql.org/message-id/e7faf42f-62b8-47f4-af5c-cb8efa3e0e20%40technowledgy.de
Pavel Stehule:
> When you write RAISE NOTICE '%', x, then PLpgSQL parser rewrite it to
> RAISE NOTICE '%', SELECT $1
>
> There is no parser just for expressions.
That's why my suggestion in [1] already made a difference between:
SELECT var;
and
SELECT col, var FROM table, var;
So the "only require variable-in-FROM if FROM is used" should extend to
the SQL level.
That should be possible, right?
On 25.05.24 12:50, Pavel Stehule wrote: > It looks odd - It is not intuitive, it introduces new inconsistency > inside Postgres, or with solutions in other databases. No other database > has a similar rule, so users coming from Oracle, Db2, or MSSQL, Firebird > will be confused. Users that use PL/pgSQL will be confused. Do you have a description of what those other systems do? Maybe you posted it already earlier?
ne 2. 6. 2024 v 23:31 odesílatel Peter Eisentraut <peter@eisentraut.org> napsal:
>
> On 25.05.24 12:50, Pavel Stehule wrote:
> > It looks odd - It is not intuitive, it introduces new inconsistency
> > inside Postgres, or with solutions in other databases. No other database
> > has a similar rule, so users coming from Oracle, Db2, or MSSQL, Firebird
> > will be confused. Users that use PL/pgSQL will be confused.
>
> Do you have a description of what those other systems do? Maybe you
> posted it already earlier?
>
I checked today
1. MySQL
MySQL knows 3 types of variables
global - the access syntax is @@varname - they are used like our GUC and only buildin system variables are supported
SET @@autocommit = off;
SELECT @@autocommit;
user defined variables - the access syntax is @varname - the behaviour is similar to psql variables, but they are server side
SET @x = 100;
SELECT @x;
local variables - only inside PL
CREATE PROCEDURE p1()
DECLARE x int;
BEGIN
SET x = 100;
SELECT x;
END;
variables has higher priority than column (like old plpgsql)
2. MSSQL
global variables - the access syntax is @@varname, they are used like GUC and little bit more - some state informations are there like @@ERROR, @@ROWCOUNT or @@IDENTITY
local variables - the access syntax is @varname, and should be declared before usage by DECLARE command. The scope is limited to batch or procedure or function, where DECLARE command was executed.
DECLARE @TestVariable AS VARCHAR(100)
SET @TestVariable = 'Think Green'
GO
PRINT @TestVariable
This script fails, because PRINT is executed in another batch. So I think so MSSQL doesn't support session variables
There are similar mechanisms like our custom GUC and usage current_setting and set_config functions. Generally, in this area is MSSQL very primitive
EXEC sp_set_session_context 'user_id', 4;
SELECT SESSION_CONTEXT(N'user_id');
3. DB2
The "user defined global variables" are similar to my proposal. The differences are different access rights "READ, WRITE" x "SELECT, UPDATE". Because PostgreSQL has SET command for GUC, I introduced LET command (DB2 uses SET)
Variables are visible in all sessions, but value is private per session. Variables are not transactional. The usage is wider than my proposal. Then can be changed by commands SET, SELECT INTO or they can be used like OUT parameters of procedures. The search path (or some like that) is used for variables too, but the variables has less priority than tables/columns.
CREATE VARIABLE myCounter INT DEFAULT 01;
SELECT EMPNO, LASTNAME, CASE WHEN myCounter = 1 THEN SALARY ELSE NULL END FROM EMPLOYEE WHERE WORKDEPT = ’A00’;
SET myCounter = 29;
There are (I think) different kinds of variables - accessed by the function GETVARIABLE('name', 'default) - it looks very similar ro our GUC and `current_setting` function. These variables can be set by connection string, are of varchar type and 10 values are allowed. Built-in session variables (configuration) can be accessed by the function GETVARIABLE too.
SQL stored procedures supports declared local variables like PL/pgSQL
4. Firebird
Firebird has something like our custom GUC. But it allow nested routines - so some functionality of session variables can be emulated with local variable and nested routines (but outer variables can be used only in Firebird 5)
The variables are accessed by syntax :varname - like psql, but if I understand to diagrams, the char ':' is optional
5. SQL/PSM
Standard introduces a concept of modules that can be joined with schemas. The variables are like PLpgSQL, but only local - the only temp tables can be defined on module levels. These tables can be accessed only from routines assigned to modules. Modules are declarative versions of our extensions (if I understand well, I didn't find any implementation). It allows you to overwrite the search patch for routines assigned in the module. Variables are not transactional, the priority - variables/columns is not specified.
6. Oracle
Oracle PL/SQL allows the use of package variables. PL/SQL is +/- ADA language - and package variables are "global" variables. They are not directly visible from SQL, but Oracle allows reduced syntax for functions without arguments, so you need to write a wrapper
CREATE OR REPLACE PACKAGE my_package
AS
FUNCTION get_a RETURN NUMBER;
END my_package;
/
CREATE OR REPLACE PACKAGE BODY my_package
AS
a NUMBER(20);
FUNCTION get_a
RETURN NUMBER
IS
BEGIN
RETURN a;
END get_a;
END my_package;
SELECT my_package.get_a FROM DUAL;
Inside SQL the higher priority has SQL, inside non SQL commands like CALL or some PL/SQL command, the higher priority has packages.
The variable should be declared by command VARIABLE and can be accessed by syntax :varname in session before usage (maybe this step is optional)
VARIABLE bv_variable_name VARCHAR2(30)
BEGIN
:bv_variable_name := 'Some Value';
END;
SELECT column_name
FROM table_name
WHERE column_name = :bv_variable_name;
You can see, the RDBMS allows different types of session variables, different implementations. Usually one system allows more implementation of session variables. There is a possibility of emulation implementation between RDBMS, but security setting is possible only in Oracle or DB2.
RegardsPavel
6. Oracle
Oracle PL/SQL allows the use of package variables. PL/SQL is +/- ADA language - and package variables are "global" variables. They are not directly visible from SQL, but Oracle allows reduced syntax for functions without arguments, so you need to write a wrapper
CREATE OR REPLACE PACKAGE my_package
AS
FUNCTION get_a RETURN NUMBER;
END my_package;
/
CREATE OR REPLACE PACKAGE BODY my_package
AS
a NUMBER(20);
FUNCTION get_a
RETURN NUMBER
IS
BEGIN
RETURN a;
END get_a;
END my_package;
SELECT my_package.get_a FROM DUAL;
Inside SQL the higher priority has SQL, inside non SQL commands like CALL or some PL/SQL command, the higher priority has packages.
Вложения
- v20240607-0017-expression-with-session-variables-can-be-inlined.patch
- v20240607-0015-allow-parallel-execution-queries-with-session-variab.patch
- v20240607-0019-transactional-variables.patch
- v20240607-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240607-0016-plpgsql-implementation-for-LET-statement.patch
- v20240607-0014-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240607-0011-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240607-0010-implementation-of-temporary-session-variables.patch
- v20240607-0009-PREPARE-LET-support.patch
- v20240607-0008-EXPLAIN-LET-support.patch
- v20240607-0007-GUC-session_variables_ambiguity_warning.patch
- v20240607-0006-plpgsql-tests.patch
- v20240607-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240607-0004-DISCARD-VARIABLES.patch
- v20240607-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240607-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240607-0012-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240607-0001-Enhancing-catalog-for-support-session-variables-and-.patch
- v20240607-0013-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
Вложения
- v20240616-0017-expression-with-session-variables-can-be-inlined.patch
- v20240616-0016-plpgsql-implementation-for-LET-statement.patch
- v20240616-0019-transactional-variables.patch
- v20240616-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240616-0015-allow-parallel-execution-queries-with-session-variab.patch
- v20240616-0013-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240616-0014-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240616-0012-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240616-0011-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240616-0010-implementation-of-temporary-session-variables.patch
- v20240616-0009-PREPARE-LET-support.patch
- v20240616-0008-EXPLAIN-LET-support.patch
- v20240616-0007-GUC-session_variables_ambiguity_warning.patch
- v20240616-0006-plpgsql-tests.patch
- v20240616-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240616-0004-DISCARD-VARIABLES.patch
- v20240616-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240616-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240616-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240619-0017-expression-with-session-variables-can-be-inlined.patch
- v20240619-0015-allow-parallel-execution-queries-with-session-variab.patch
- v20240619-0016-plpgsql-implementation-for-LET-statement.patch
- v20240619-0019-transactional-variables.patch
- v20240619-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240619-0014-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240619-0013-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240619-0011-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240619-0012-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240619-0010-implementation-of-temporary-session-variables.patch
- v20240619-0009-PREPARE-LET-support.patch
- v20240619-0008-EXPLAIN-LET-support.patch
- v20240619-0007-GUC-session_variables_ambiguity_warning.patch
- v20240619-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240619-0004-DISCARD-VARIABLES.patch
- v20240619-0006-plpgsql-tests.patch
- v20240619-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240619-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240619-0001-Enhancing-catalog-for-support-session-variables-and-.patch
Вложения
- v20240628-0017-expression-with-session-variables-can-be-inlined.patch
- v20240628-0018-this-patch-changes-error-message-column-doesn-t-exis.patch
- v20240628-0016-plpgsql-implementation-for-LET-statement.patch
- v20240628-0019-transactional-variables.patch
- v20240628-0015-allow-parallel-execution-queries-with-session-variab.patch
- v20240628-0014-allow-read-an-value-of-session-variable-directly-fro.patch
- v20240628-0013-Implementation-of-NOT-NULL-and-IMMUTABLE-clauses.patch
- v20240628-0012-Implementation-of-DEFAULT-clause-default-expressions.patch
- v20240628-0011-Implementation-ON-TRANSACTION-END-RESET-clause.patch
- v20240628-0010-implementation-of-temporary-session-variables.patch
- v20240628-0009-PREPARE-LET-support.patch
- v20240628-0007-GUC-session_variables_ambiguity_warning.patch
- v20240628-0008-EXPLAIN-LET-support.patch
- v20240628-0006-plpgsql-tests.patch
- v20240628-0005-memory-cleaning-after-DROP-VARIABLE.patch
- v20240628-0003-function-pg_session_variables-for-cleaning-tests.patch
- v20240628-0004-DISCARD-VARIABLES.patch
- v20240628-0002-Storage-for-session-variables-and-SQL-interface.patch
- v20240628-0001-Enhancing-catalog-for-support-session-variables-and-.patch