Обсуждение: On login trigger: take three
Hi hackers, Recently I have asked once again by one of our customers about login trigger in postgres. People are migrating to Postgres from Oracle and looking for Postgres analog of this Oracle feature. This topic is not new: https://www.postgresql.org/message-id/flat/1570308356720-0.post%40n3.nabble.com#4748bcb0c5fc98cec0a735dbdffb0c68 https://www.postgresql.org/message-id/flat/OSAPR01MB507373499CCCEA00EAE79875FE2D0%40OSAPR01MB5073.jpnprd01.prod.outlook.com#ed50c248be32be6955c385ca67d6cdc1 end even session connect/disconnect hooks were sometimes committed (but then reverted). As far as I understand most of the concerns were related with disconnect hook. Performing some action on session disconnect is actually much more complicated than on login. But customers are not needed it, unlike actions performed at session start. I wonder if we are really going to make some steps in this directions? The discussion above was finished with "We haven't rejected the concept altogether, AFAICT" I have tried to resurrect this patch and implement on-connect trigger on top of it. The syntax is almost the same as proposed by Takayuki: CREATE EVENT TRIGGER mytrigger AFTER CONNECTION ON mydatabase EXECUTE {PROCEDURE | FUNCTION} myproc(); I have replaced CONNECT with CONNECTION because last keyword is already recognized by Postgres and make ON clause mandatory to avoid shift-reduce conflicts. Actually specifying database name is redundant, because we can define on-connect trigger only for self database (just because triggers and functions are local to the database). It may be considered as argument against handling session start using trigger. But it seems to be the most natural mechanism for users. On connect trigger can be dropped almost in the same way as normal (on relation) trigger, but with specifying name of the database instead of relation name: DROP TRIGGER mytrigger ON mydatabase; It is possible to define arbitrary number of on-connect triggers with different names. I attached my prototype implementation of this feature. I just to be sure first that this feature will be interested to community. If so, I will continue work in it and prepare new version of the patch for the commitfest. Example -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
Hi hackers,
Recently I have asked once again by one of our customers about login
trigger in postgres. People are migrating to Postgres from Oracle and
looking for Postgres analog of this Oracle feature.
This topic is not new:
https://www.postgresql.org/message-id/flat/1570308356720-0.post%40n3.nabble.com#4748bcb0c5fc98cec0a735dbdffb0c68
https://www.postgresql.org/message-id/flat/OSAPR01MB507373499CCCEA00EAE79875FE2D0%40OSAPR01MB5073.jpnprd01.prod.outlook.com#ed50c248be32be6955c385ca67d6cdc1
end even session connect/disconnect hooks were sometimes committed (but
then reverted).
As far as I understand most of the concerns were related with disconnect
hook.
Performing some action on session disconnect is actually much more
complicated than on login.
But customers are not needed it, unlike actions performed at session start.
I wonder if we are really going to make some steps in this directions?
The discussion above was finished with "We haven't rejected the concept
altogether, AFAICT"
I have tried to resurrect this patch and implement on-connect trigger on
top of it.
The syntax is almost the same as proposed by Takayuki:
CREATE EVENT TRIGGER mytrigger
AFTER CONNECTION ON mydatabase
EXECUTE {PROCEDURE | FUNCTION} myproc();
I have replaced CONNECT with CONNECTION because last keyword is already
recognized by Postgres and
make ON clause mandatory to avoid shift-reduce conflicts.
Actually specifying database name is redundant, because we can define
on-connect trigger only for self database (just because triggers and
functions are local to the database).
It may be considered as argument against handling session start using
trigger. But it seems to be the most natural mechanism for users.
On connect trigger can be dropped almost in the same way as normal (on
relation) trigger, but with specifying name of the database instead of
relation name:
DROP TRIGGER mytrigger ON mydatabase;
It is possible to define arbitrary number of on-connect triggers with
different names.
I attached my prototype implementation of this feature.
I just to be sure first that this feature will be interested to community.
If so, I will continue work in it and prepare new version of the patch
for the commitfest.
Example
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru> > Recently I have asked once again by one of our customers about login trigger in > postgres. People are migrating to Postgres from Oracle and looking for Postgres > analog of this Oracle feature. > This topic is not new: > I attached my prototype implementation of this feature. > I just to be sure first that this feature will be interested to community. > If so, I will continue work in it and prepare new version of the patch for the > commitfest. Thanks a lot for taking on this! +10 > It may be considered as argument against handling session start using trigger. > But it seems to be the most natural mechanism for users. Yeah, it's natural, just like the Unix shells run some shell scripts in the home directory. Regards Takayuki Tsunakawa
Sorry, attached version of the patch is missing changes in one file. Here is it correct patch. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
Hičt 3. 9. 2020 v 15:43 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:Hi hackers,
Recently I have asked once again by one of our customers about login
trigger in postgres. People are migrating to Postgres from Oracle and
looking for Postgres analog of this Oracle feature.
This topic is not new:
https://www.postgresql.org/message-id/flat/1570308356720-0.post%40n3.nabble.com#4748bcb0c5fc98cec0a735dbdffb0c68
https://www.postgresql.org/message-id/flat/OSAPR01MB507373499CCCEA00EAE79875FE2D0%40OSAPR01MB5073.jpnprd01.prod.outlook.com#ed50c248be32be6955c385ca67d6cdc1
end even session connect/disconnect hooks were sometimes committed (but
then reverted).
As far as I understand most of the concerns were related with disconnect
hook.
Performing some action on session disconnect is actually much more
complicated than on login.
But customers are not needed it, unlike actions performed at session start.
I wonder if we are really going to make some steps in this directions?
The discussion above was finished with "We haven't rejected the concept
altogether, AFAICT"
I have tried to resurrect this patch and implement on-connect trigger on
top of it.
The syntax is almost the same as proposed by Takayuki:
CREATE EVENT TRIGGER mytrigger
AFTER CONNECTION ON mydatabase
EXECUTE {PROCEDURE | FUNCTION} myproc();
I have replaced CONNECT with CONNECTION because last keyword is already
recognized by Postgres and
make ON clause mandatory to avoid shift-reduce conflicts.
Actually specifying database name is redundant, because we can define
on-connect trigger only for self database (just because triggers and
functions are local to the database).
It may be considered as argument against handling session start using
trigger. But it seems to be the most natural mechanism for users.
On connect trigger can be dropped almost in the same way as normal (on
relation) trigger, but with specifying name of the database instead of
relation name:
DROP TRIGGER mytrigger ON mydatabase;
It is possible to define arbitrary number of on-connect triggers with
different names.
I attached my prototype implementation of this feature.
I just to be sure first that this feature will be interested to community.
If so, I will continue work in it and prepare new version of the patch
for the commitfest.I have a customer that requires this feature too. Now it uses a solution based on dll session autoloading. Native solution can be great.+1
I realized that on connect trigger should be implemented as EVENT TRIGGER.
So I have reimplemented my patch using event trigger and use session_start even name to make it more consistent with other events.
Now on login triggers can be created in this way:
create table connects(id serial, who text);
create function on_login_proc() returns event_trigger as $$
begin
insert into connects (who) values (current_user());
raise notice 'You are welcome!';
end;
$$ language plpgsql;
create event trigger on_login_trigger on session_start execute procedure on_login_proc();
alter event trigger on_login_trigger enable always;
Вложения
On 14.09.2020 12:44, Pavel Stehule wrote: > Hi > > I am checking last patch, and there are notices > > 1. disable_session_start_trigger should be SU_BACKEND instead SUSET > > 2. The documentation should be enhanced - there is not any note about > behave when there are unhandled exceptions, about motivation for this > event trigger > > 3. regress tests should be enhanced - the cases with exceptions are > not tested > > 4. This trigger is not executed again after RESET ALL or DISCARD ALL - > it can be a problem if somebody wants to use this trigger for > initialisation of some session objects with some pooling solutions. > > 5. The handling errors don't work well for canceling. If event trigger > waits for some event, then cancel disallow connect although connected > user is superuser > > CREATE OR REPLACE FUNCTION on_login_proc2() RETURNS EVENT_TRIGGER AS > $$ begin perform pg_sleep(10000); raise notice '%', fx1(100);raise > notice 'kuku kuku'; end $$ language plpgsql; > > probably nobody will use pg_sleep in this routine, but there can be > wait on some locks ... > > Regards > > Pavel > > > Hi Thank you very much for looking at my patch for connection triggers. I have fixed 1-3 issues in the attached patch. Concerning 4 and 5 I have some doubts: 4. Should I add some extra GUC to allow firing of session_start trigger in case of RESET ALL or DISCARD ALL ? Looks like such behavior contradicts with event name "session_start"... And do we really need it? If some pooler is using RESET ALL/DISCARD ALL to emulate session semantic then most likely it provides way to define custom actions which should be perform for session initialization. As far as I know, for example pgbouncer allows do define own on-connect hooks. 5. I do not quite understand your concern. If I define trigger procedure which is blocked (for example as in your example), then I can use pg_cancel_backend to interrupt execution of login trigger and superuser can login. What should be changed here? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
On 14.09.2020 12:44, Pavel Stehule wrote:
> Hi
>
> I am checking last patch, and there are notices
>
> 1. disable_session_start_trigger should be SU_BACKEND instead SUSET
>
> 2. The documentation should be enhanced - there is not any note about
> behave when there are unhandled exceptions, about motivation for this
> event trigger
>
> 3. regress tests should be enhanced - the cases with exceptions are
> not tested
>
> 4. This trigger is not executed again after RESET ALL or DISCARD ALL -
> it can be a problem if somebody wants to use this trigger for
> initialisation of some session objects with some pooling solutions.
>
> 5. The handling errors don't work well for canceling. If event trigger
> waits for some event, then cancel disallow connect although connected
> user is superuser
>
> CREATE OR REPLACE FUNCTION on_login_proc2() RETURNS EVENT_TRIGGER AS
> $$ begin perform pg_sleep(10000); raise notice '%', fx1(100);raise
> notice 'kuku kuku'; end $$ language plpgsql;
>
> probably nobody will use pg_sleep in this routine, but there can be
> wait on some locks ...
>
> Regards
>
> Pavel
>
>
>
Hi
Thank you very much for looking at my patch for connection triggers.
I have fixed 1-3 issues in the attached patch.
Concerning 4 and 5 I have some doubts:
4. Should I add some extra GUC to allow firing of session_start trigger
in case of RESET ALL or DISCARD ALL ?
Looks like such behavior contradicts with event name "session_start"...
And do we really need it? If some pooler is using RESET ALL/DISCARD ALL
to emulate session semantic then most likely it provides way to define
custom actions which
should be perform for session initialization. As far as I know, for
example pgbouncer allows do define own on-connect hooks.
5. I do not quite understand your concern. If I define trigger
procedure which is blocked (for example as in your example), then I can
use pg_cancel_backend to interrupt execution of login trigger and
superuser can login. What should be changed here?
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
If we introduce buildin session trigger , we should to define what is the session. Your design is much more related to the process than to session. So the correct name should be "process_start" trigger, or some should be different. I think there are two different events - process_start, and session_start, and there should be two different event triggers. Maybe the name "session_start" is just ambiguous and should be used with a different name.
I agree.
I can rename trigger to backend_start or process_start or whatever else.
5. I do not quite understand your concern. If I define trigger
procedure which is blocked (for example as in your example), then I can
use pg_cancel_backend to interrupt execution of login trigger and
superuser can login. What should be changed here?You cannot run pg_cancel_backend, because you cannot open a new session. There is a cycle.
It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC:
psql "dbname=postgres options='-c disable_session_start_trigger=true'"
On 14.09.2020 17:34, Pavel Stehule wrote:If we introduce buildin session trigger , we should to define what is the session. Your design is much more related to the process than to session. So the correct name should be "process_start" trigger, or some should be different. I think there are two different events - process_start, and session_start, and there should be two different event triggers. Maybe the name "session_start" is just ambiguous and should be used with a different name.
I agree.
I can rename trigger to backend_start or process_start or whatever else.
5. I do not quite understand your concern. If I define trigger
procedure which is blocked (for example as in your example), then I can
use pg_cancel_backend to interrupt execution of login trigger and
superuser can login. What should be changed here?You cannot run pg_cancel_backend, because you cannot open a new session. There is a cycle.
It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC:
psql "dbname=postgres options='-c disable_session_start_trigger=true'"
On Tue, Sep 15, 2020 at 2:12 AM Pavel Stehule <pavel.stehule@gmail.com> wrote: > >> >> It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC: >> >> psql "dbname=postgres options='-c disable_session_start_trigger=true'" > > > sure, I know. Just this behavior can be a very unpleasant surprise, and my question is if it can be fixed. Creating customlibpq variables can be the stop for people that use pgAdmin. > Hi, I thought in the case of using pgAdmin (assuming you can connect as superuser to a database, say the default "postgres" maintenance database, that doesn't have an EVENT TRIGGER defined for the session_start event) you could issue the query "ALTER SYSTEM SET disable_session_start_trigger TO true;" and then reload the configuration? Anyway, I am wondering if this patch is still being actively developed/improved? Regarding the last-posted patch, I'd like to give some feedback. I found that the documentation part wouldn't build because of errors in the SGML tags. There are some grammatical errors too, and some minor inconsistencies with the current documentation, and some descriptions could be improved. I think that a colon separator should be added to the NOTICE message for superuser, so it's clear exactly where the text of the underlying error message starts. Also, I think that "client_connection" is perhaps a better and more intuitive event name than "session_start", or the suggested "user_backend_start". I've therefore attached an updated patch with these suggested minor improvements, please take a look and see what you think (please compare with the original patch). Regards, Greg Nancarrow Fujitsu Australia
Вложения
On 04.12.2020 3:50, Greg Nancarrow wrote: > On Tue, Sep 15, 2020 at 2:12 AM Pavel Stehule <pavel.stehule@gmail.com> wrote: >>> It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC: >>> >>> psql "dbname=postgres options='-c disable_session_start_trigger=true'" >> >> sure, I know. Just this behavior can be a very unpleasant surprise, and my question is if it can be fixed. Creating customlibpq variables can be the stop for people that use pgAdmin. >> > Hi, > > I thought in the case of using pgAdmin (assuming you can connect as > superuser to a database, say the default "postgres" maintenance > database, that doesn't have an EVENT TRIGGER defined for the > session_start event) you could issue the query "ALTER SYSTEM SET > disable_session_start_trigger TO true;" and then reload the > configuration? As far as I understand Pavel concern was about the case when superuser defines wrong login trigger which prevents login to the system all user including himself. Right now solution of this problem is to include "options='-c disable_session_start_trigger=true'" in connection string. I do not know if it can be done with pgAdmin. > > Anyway, I am wondering if this patch is still being actively developed/improved? I do not know what else has to be improved. If you, Pavel or anybody else have some suggestions: please let me know. > > Regarding the last-posted patch, I'd like to give some feedback. I > found that the documentation part wouldn't build because of errors in > the SGML tags. There are some grammatical errors too, and some minor > inconsistencies with the current documentation, and some descriptions > could be improved. I think that a colon separator should be added to > the NOTICE message for superuser, so it's clear exactly where the text > of the underlying error message starts. Also, I think that > "client_connection" is perhaps a better and more intuitive event name > than "session_start", or the suggested "user_backend_start". > I've therefore attached an updated patch with these suggested minor > improvements, please take a look and see what you think (please > compare with the original patch). Thank you very much for detecting the problems and much more thanks for fixing them and providing your version of the patch. I have nothing against renaming "session_start" to "client_connection". -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On Fri, Dec 4, 2020 at 9:05 PM Konstantin Knizhnik <k.knizhnik@postgrespro.ru> wrote: > > As far as I understand Pavel concern was about the case when superuser > defines wrong login trigger which prevents login to the system > all user including himself. Right now solution of this problem is to > include "options='-c disable_session_start_trigger=true'" in connection > string. > I do not know if it can be done with pgAdmin. > > As an event trigger is tied to a particular database, and a GUC is global to the cluster, as long as there is one database in the cluster for which an event trigger for the "client_connection" event is NOT defined (say the default "postgres" maintenance database), then the superuser can always connect to that database, issue "ALTER SYSTEM SET disable_client_connection_trigger TO true" and reload the configuration. I tested this with pgAdmin4 and it worked fine for me, to allow login to a database for which login was previously prevented due to a badly-defined logon trigger. Pavel, is this an acceptable solution or do you still see problems with this approach? Regards, Greg Nancarrow Fujitsu Australia
On Fri, Dec 4, 2020 at 9:05 PM Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
>
> As far as I understand Pavel concern was about the case when superuser
> defines wrong login trigger which prevents login to the system
> all user including himself. Right now solution of this problem is to
> include "options='-c disable_session_start_trigger=true'" in connection
> string.
> I do not know if it can be done with pgAdmin.
> >
As an event trigger is tied to a particular database, and a GUC is
global to the cluster, as long as there is one database in the cluster
for which an event trigger for the "client_connection" event is NOT
defined (say the default "postgres" maintenance database), then the
superuser can always connect to that database, issue "ALTER SYSTEM SET
disable_client_connection_trigger TO true" and reload the
configuration. I tested this with pgAdmin4 and it worked fine for me,
to allow login to a database for which login was previously prevented
due to a badly-defined logon trigger.
Pavel, is this an acceptable solution or do you still see problems
with this approach?
Regards,
Greg Nancarrow
Fujitsu Australia
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote: > > > There are two maybe generic questions? > > 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checkedonly by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protectionagainst necessity to restart to single mode to repair the event trigger. I think so more generic solution is betterthan special disable_client_connection_trigger GUC. > > 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to blockconnection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start.This event should be started after connecting - so the exception there doesn't block connect, and should bestarted also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This featureshould not be implemented in first step, but it can be a plan for support pooled connections > I've created a separate patch to address question (1), rather than include it in the main patch, which I've adjusted accordingly. I'll leave question (2) until another time, as you suggest. See the attached patches. Regards, Greg Nancarrow Fujitsu Australia
Вложения
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.
Regards,
Greg Nancarrow
Fujitsu Australia
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.
Regards,
Greg Nancarrow
Fujitsu Australia
Hist 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.I see two possible questions?1. when you introduce this event, then the new hook is useless ?2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger
As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.
2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):
pgbench -C -S -T 100 -P 1 -M prepared postgres
without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)
with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)
As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 09.12.2020 15:34, Pavel Stehule wrote:Hist 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.I see two possible questions?1. when you introduce this event, then the new hook is useless ?2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger
As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.
2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):
pgbench -C -S -T 100 -P 1 -M prepared postgres
without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)
with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)
As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".
│ test │ WITH LT │ LT ENABLED │ LT DISABLED │
╞════════════════╪═════════╪════════════╪═════════════╡
│ ro_constant_10 │ 539 │ 877 │ 905 │
│ ro_index_10 │ 562 │ 808 │ 855 │
│ ro_constant_50 │ 527 │ 843 │ 863 │
│ ro_index_50 │ 544 │ 731 │ 742 │
└────────────────┴─────────┴────────────┴─────────────┘
(4 rows)
RETURNS event_trigger
LANGUAGE plpgsql
AS $function$
begin
if not pg_has_role(session_user, 'postgres', 'member') then
raise exception 'you are not super user';
end if;
end;
$function$;
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
st 9. 12. 2020 v 14:28 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 09.12.2020 15:34, Pavel Stehule wrote:Hist 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.I see two possible questions?1. when you introduce this event, then the new hook is useless ?2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger
As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.ok. I don't like it, but there are redundant hooks (against event triggers) already.2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):
pgbench -C -S -T 100 -P 1 -M prepared postgres
without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)
with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)
As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".This is not the worst case probably. In this patch the StartTransactionCommand is executed on every connection, although it is not necessary - and for most use cases it will not be used.I did more tests - see attachments and I see a 5% slowdown - I don't think so it is acceptable for this case. This feature is nice, and for some users important, but only really few users can use it.┌────────────────┬─────────┬────────────┬─────────────┐
│ test │ WITH LT │ LT ENABLED │ LT DISABLED │
╞════════════════╪═════════╪════════════╪═════════════╡
│ ro_constant_10 │ 539 │ 877 │ 905 │
│ ro_index_10 │ 562 │ 808 │ 855 │
│ ro_constant_50 │ 527 │ 843 │ 863 │
│ ro_index_50 │ 544 │ 731 │ 742 │
└────────────────┴─────────┴────────────┴─────────────┘
(4 rows)I tested a performance of trigger (results of first column in table):CREATE OR REPLACE FUNCTION public.foo()
RETURNS event_trigger
LANGUAGE plpgsql
AS $function$
begin
if not pg_has_role(session_user, 'postgres', 'member') then
raise exception 'you are not super user';
end if;
end;
$function$;There is an available snapshot in InitPostgres, and then there is possible to check if for the current database some connect event trigger exists.This can reduce an overhead of this patch, when there are no logon triggers.I think so implemented and used names are ok, but for this feature the performance impact should be really very minimal.There is other small issue - missing tab-complete support for CREATE TRIGGER statement in psqlRegardsPavel
Unfortunately I was not able to reproduce your results.
I just tried the case "select 1" because for this trivial query the overhead of session hooks should be the largest.
In my case variation of results was large enough.
Did you try to run test multiple times?
pgbench -j 2 -c 10 --connect -f test-ro.sql -T 60 -n postgres
disable_event_triggers = off
tps = 1812.250463 (including connections establishing)
tps = 2256.285712 (excluding connections establishing)
tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)
tps = 1830.879302 (including connections establishing)
tps = 2279.302553 (excluding connections establishing)
disable_event_triggers = on
tps = 1858.328717 (including connections establishing)
tps = 2313.661689 (excluding connections establishing)
tps = 1832.932960 (including connections establishing)
tps = 2282.074346 (excluding connections establishing)
tps = 1868.908521 (including connections establishing)
tps = 2326.559150 (excluding connections establishing)
I tried to increase run time to 1000 seconds.
Results are the following:
disable_event_triggers = off
tps = 1813.587876 (including connections establishing)
tps = 2257.844703 (excluding connections establishing)
disable_event_triggers = on
tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)
So the difference on this extreme case is 1%.
I tried your suggestion and move client_connection_hook invocation to postinit.c
It slightly improve performance:
tps = 1828.446959 (including connections establishing)
tps = 2276.142972 (excluding connections establishing)
So result is somewhere in the middle, because it allows to eliminate overhead of
starting transaction or taking snapshots but not of lookup through pg_event_trigger table (although it it empty).
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 10.12.2020 10:45, Pavel Stehule wrote:st 9. 12. 2020 v 14:28 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 09.12.2020 15:34, Pavel Stehule wrote:Hist 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.I see two possible questions?1. when you introduce this event, then the new hook is useless ?2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger
As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.ok. I don't like it, but there are redundant hooks (against event triggers) already.2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):
pgbench -C -S -T 100 -P 1 -M prepared postgres
without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)
with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)
As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".This is not the worst case probably. In this patch the StartTransactionCommand is executed on every connection, although it is not necessary - and for most use cases it will not be used.I did more tests - see attachments and I see a 5% slowdown - I don't think so it is acceptable for this case. This feature is nice, and for some users important, but only really few users can use it.┌────────────────┬─────────┬────────────┬─────────────┐
│ test │ WITH LT │ LT ENABLED │ LT DISABLED │
╞════════════════╪═════════╪════════════╪═════════════╡
│ ro_constant_10 │ 539 │ 877 │ 905 │
│ ro_index_10 │ 562 │ 808 │ 855 │
│ ro_constant_50 │ 527 │ 843 │ 863 │
│ ro_index_50 │ 544 │ 731 │ 742 │
└────────────────┴─────────┴────────────┴─────────────┘
(4 rows)I tested a performance of trigger (results of first column in table):CREATE OR REPLACE FUNCTION public.foo()
RETURNS event_trigger
LANGUAGE plpgsql
AS $function$
begin
if not pg_has_role(session_user, 'postgres', 'member') then
raise exception 'you are not super user';
end if;
end;
$function$;There is an available snapshot in InitPostgres, and then there is possible to check if for the current database some connect event trigger exists.This can reduce an overhead of this patch, when there are no logon triggers.I think so implemented and used names are ok, but for this feature the performance impact should be really very minimal.There is other small issue - missing tab-complete support for CREATE TRIGGER statement in psqlRegardsPavel
Unfortunately I was not able to reproduce your results.
I just tried the case "select 1" because for this trivial query the overhead of session hooks should be the largest.
In my case variation of results was large enough.
Did you try to run test multiple times?
pgbench -j 2 -c 10 --connect -f test-ro.sql -T 60 -n postgres
disable_event_triggers = off
tps = 1812.250463 (including connections establishing)
tps = 2256.285712 (excluding connections establishing)
tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)
tps = 1830.879302 (including connections establishing)
tps = 2279.302553 (excluding connections establishing)
disable_event_triggers = on
tps = 1858.328717 (including connections establishing)
tps = 2313.661689 (excluding connections establishing)
tps = 1832.932960 (including connections establishing)
tps = 2282.074346 (excluding connections establishing)
tps = 1868.908521 (including connections establishing)
tps = 2326.559150 (excluding connections establishing)
I tried to increase run time to 1000 seconds.
Results are the following:
disable_event_triggers = off
tps = 1813.587876 (including connections establishing)
tps = 2257.844703 (excluding connections establishing)
disable_event_triggers = on
tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)
So the difference on this extreme case is 1%.
I tried your suggestion and move client_connection_hook invocation to postinit.c
It slightly improve performance:
tps = 1828.446959 (including connections establishing)
tps = 2276.142972 (excluding connections establishing)
So result is somewhere in the middle, because it allows to eliminate overhead of
starting transaction or taking snapshots but not of lookup through pg_event_trigger table (although it it empty).
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and other things.There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.
Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.
From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.
On 10.12.2020 18:12, Pavel Stehule wrote:My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and other things.There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.
Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.
From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.
[Switching to Thread 0x7f2bff438ec0 (LWP 827497)]
ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
1025 ResourceArrayEnlarge(&(owner->catrefarr));
(gdb) bt
#0 ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
#1 0x00000000008a70f8 in SearchCatCacheInternal (cache=<optimized out>, nkeys=nkeys@entry=1, v1=v1@entry=13836, v2=v2@entry=0,
v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1273
#2 0x00000000008a7575 in SearchCatCache1 (cache=<optimized out>, v1=v1@entry=13836) at catcache.c:1167
#3 0x00000000008b7f80 in SearchSysCache1 (cacheId=cacheId@entry=21, key1=key1@entry=13836) at syscache.c:1122
#4 0x00000000005939cd in pg_database_ownercheck (db_oid=13836, roleid=16387) at aclchk.c:5114
#5 0x0000000000605b42 in EventTriggersDisabled () at event_trigger.c:650
#6 EventTriggerOnConnect () at event_trigger.c:839
#7 0x00000000007b46d7 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdd6256080, dbname=<optimized out>,
username=0xf82508 "tom") at postgres.c:4021
#8 0x0000000000741afd in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4490
#9 BackendStartup (port=<optimized out>) at postmaster.c:4212
#10 ServerLoop () at postmaster.c:1729
#11 0x0000000000742881 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xf44d00) at postmaster.c:1401
#12 0x00000000004f1ff7 in main (argc=3, argv=0xf44d00) at main.c:209
čt 10. 12. 2020 v 16:48 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 10.12.2020 18:12, Pavel Stehule wrote:My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and other things.There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.
Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.My idea was a little bit different - just inside process initialization, checking existence of event triggers, and later when it is necessary to start a transaction for trigger execution. This should to reduce useless empty transaction,
I am sure this is a problem on computers with slower CPU s, although I have I7, but because this feature is usually unused, then the performance impact should be minimal every time.
From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.It can work, but this design is not user friendly. The significant bottleneck should be forking new processes, and check of content some usually very small tables should be invisible. So if there is a possible way to implement some feature that can be enabled by default, then we should go this way. It can be pretty unfriendly if somebody writes a logon trigger and it will not work by default without any warning.When I tested last patch I found a problem (I have disabled assertions due performance testing)create role xx login;alter system set disable_event_triggers to on; -- better should be positive form - enable_event_triggers to on, offselect pg_reload_conf();psql -U xxThread 2.1 "postmaster" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2bff438ec0 (LWP 827497)]
ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
1025 ResourceArrayEnlarge(&(owner->catrefarr));
(gdb) bt
#0 ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
#1 0x00000000008a70f8 in SearchCatCacheInternal (cache=<optimized out>, nkeys=nkeys@entry=1, v1=v1@entry=13836, v2=v2@entry=0,
v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1273
#2 0x00000000008a7575 in SearchCatCache1 (cache=<optimized out>, v1=v1@entry=13836) at catcache.c:1167
#3 0x00000000008b7f80 in SearchSysCache1 (cacheId=cacheId@entry=21, key1=key1@entry=13836) at syscache.c:1122
#4 0x00000000005939cd in pg_database_ownercheck (db_oid=13836, roleid=16387) at aclchk.c:5114
#5 0x0000000000605b42 in EventTriggersDisabled () at event_trigger.c:650
#6 EventTriggerOnConnect () at event_trigger.c:839
#7 0x00000000007b46d7 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdd6256080, dbname=<optimized out>,
username=0xf82508 "tom") at postgres.c:4021
#8 0x0000000000741afd in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4490
#9 BackendStartup (port=<optimized out>) at postmaster.c:4212
#10 ServerLoop () at postmaster.c:1729
#11 0x0000000000742881 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xf44d00) at postmaster.c:1401
#12 0x00000000004f1ff7 in main (argc=3, argv=0xf44d00) at main.c:209I checked profiles, and although there is significant slowdown when there is one login trigger, it was not related to plpgsql.There is big overhead of8,98% postmaster postgres [.] guc_name_compareMaybe EventTriggerInvoke is not well optimized for very frequent execution. Overhead of plpgsql is less than 0.1%, although there is significant slowdown (when a very simple logon trigger is executed).RegardsPavel
Sorry, may be I missed something. But now I completely confused with your idea.čt 10. 12. 2020 v 16:48 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 10.12.2020 18:12, Pavel Stehule wrote:My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and other things.There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.
Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.My idea was a little bit different - just inside process initialization, checking existence of event triggers, and later when it is necessary to start a transaction for trigger execution. This should to reduce useless empty transaction,I am sure this is a problem on computers with slower CPU s, although I have I7, but because this feature is usually unused, then the performance impact should be minimal every time.
Right now processing of login hooks is done in PostgresMain.
In the previous mail you have suggested to did it in InitPostgres which is invoked from PostgresMain.
So what is the difference (except there open transaction in InitPostgres)?
So what the problem we are trying to solve now?
As far as I understand, your concern is about extra overhead during backend startup needed to check if there on-login triggers defined.
We are not speaking about trigger execution. We mostly worry about applications which are not using triggers at all. But still have to pay some small extra overhead at startup.
This overhead seems to be negligible (1% for dummy connection doing just "select 1"). But taken in account that 99.9999% applications will not use connection triggers,
even very small overhead seems to be not good.
But what can we do?
We do not want to scan pg_event_trigger table on backend startup. But how else we can skip this check, taken in account that
1. Such trigger can be registered at any moment of time
2. Triggers are registered per database, so we can not have just global flag,signaling lack of event triggers.
The only solution I see at this moment is <db,has_event_triggers> hash in shared memory.
But it seems to be overkill from my point of view.
This is why I suggested to have disable_login_event_triggers GUC set to true by default.
Please notice that event triggers are disabled by default.
From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.It can work, but this design is not user friendly. The significant bottleneck should be forking new processes, and check of content some usually very small tables should be invisible. So if there is a possible way to implement some feature that can be enabled by default, then we should go this way. It can be pretty unfriendly if somebody writes a logon trigger and it will not work by default without any warning.
You need to call "alter event trigger xxx enable always".
May be instead of GUCs we should somehow use ALTER mechanism?
But I do not understand why it is better and how it will help to solve this problem (elimination of exrta overhead when there are not triggers).
When I tested last patch I found a problem (I have disabled assertions due performance testing)create role xx login;alter system set disable_event_triggers to on; -- better should be positive form - enable_event_triggers to on, offselect pg_reload_conf();psql -U xxThread 2.1 "postmaster" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2bff438ec0 (LWP 827497)]
ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
1025 ResourceArrayEnlarge(&(owner->catrefarr));
(gdb) bt
#0 ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
#1 0x00000000008a70f8 in SearchCatCacheInternal (cache=<optimized out>, nkeys=nkeys@entry=1, v1=v1@entry=13836, v2=v2@entry=0,
v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1273
#2 0x00000000008a7575 in SearchCatCache1 (cache=<optimized out>, v1=v1@entry=13836) at catcache.c:1167
#3 0x00000000008b7f80 in SearchSysCache1 (cacheId=cacheId@entry=21, key1=key1@entry=13836) at syscache.c:1122
#4 0x00000000005939cd in pg_database_ownercheck (db_oid=13836, roleid=16387) at aclchk.c:5114
#5 0x0000000000605b42 in EventTriggersDisabled () at event_trigger.c:650
#6 EventTriggerOnConnect () at event_trigger.c:839
#7 0x00000000007b46d7 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdd6256080, dbname=<optimized out>,
username=0xf82508 "tom") at postgres.c:4021
#8 0x0000000000741afd in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4490
#9 BackendStartup (port=<optimized out>) at postmaster.c:4212
#10 ServerLoop () at postmaster.c:1729
#11 0x0000000000742881 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xf44d00) at postmaster.c:1401
#12 0x00000000004f1ff7 in main (argc=3, argv=0xf44d00) at main.c:209
pg_database_ownercheck can not be called outside transaction.
I can split replace call of EventTriggersDisabled in EventTriggerOnConnect with two separate checks:
disable_event_triggers - before start of transaction
and pg_database_ownercheck(MyDatabaseId, GetUserId()) inside transaction.
But I wonder if we need to check database ownership in this place at all?
May be just allow to alter disable_event_triggers for superusers?
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>PGC_SU_BACKEND is too strong, there should be PGC_BACKEND if this option can be used by database ownerPavel
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.
Regards,
Greg Nancarrow
Fujitsu Australia
It seems to me that current implementation of EventTriggersDisabled:
/*
* Indicates whether all event triggers are currently disabled
* for the current user.
* Event triggers are disabled when configuration parameter
* "disable_event_triggers" is true, and the current user
* is the database owner or has superuser privileges.
*/
static inline bool
EventTriggersDisabled(void)
{
return (disable_event_triggers && pg_database_ownercheck(MyDatabaseId, GetUserId()));
}
is not correct. It makes it not possible to superuser to disable triggers for all users.
Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.
So I think that EventTriggersDisabled function is not needed and can be replaced just with disable_event_triggers GUC check.
And this option should be defined with PGC_SU_BACKEND
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 09.12.2020 15:24, Pavel Stehule wrote:st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL. This feature should not be implemented in first step, but it can be a plan for support pooled connections
>PGC_SU_BACKEND is too strong, there should be PGC_BACKEND if this option can be used by database ownerPavel
I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.
Regards,
Greg Nancarrow
Fujitsu Australia
It seems to me that current implementation of EventTriggersDisabled:
/*
* Indicates whether all event triggers are currently disabled
* for the current user.
* Event triggers are disabled when configuration parameter
* "disable_event_triggers" is true, and the current user
* is the database owner or has superuser privileges.
*/
static inline bool
EventTriggersDisabled(void)
{
return (disable_event_triggers && pg_database_ownercheck(MyDatabaseId, GetUserId()));
}
is not correct. It makes it not possible to superuser to disable triggers for all users.
Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.
So I think that EventTriggersDisabled function is not needed and can be replaced just with disable_event_triggers GUC check.
And this option should be defined with PGC_SU_BACKEND-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?
Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?
Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?My idea was to reduce necessary rights to database owners. But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.yes, it is truePavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
So what's next?
I see three options:
1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).
2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".
3. Implement some mechanism for caching presence of event triggers in shared memory.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 11.12.2020 19:27, Pavel Stehule wrote:pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?My idea was to reduce necessary rights to database owners. But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.yes, it is truePavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
So what's next?
I see three options:
1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).
2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".
3. Implement some mechanism for caching presence of event triggers in shared memory.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 19:27, Pavel Stehule wrote:pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?My idea was to reduce necessary rights to database owners. But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.yes, it is truePavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
So what's next?
I see three options:
1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).
2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".
3. Implement some mechanism for caching presence of event triggers in shared memory.@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.
Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...
I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?
RegardsPavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 15.12.2020 16:18, Pavel Stehule wrote:út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 19:27, Pavel Stehule wrote:pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?My idea was to reduce necessary rights to database owners. But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.yes, it is truePavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
So what's next?
I see three options:
1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).
2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".
3. Implement some mechanism for caching presence of event triggers in shared memory.@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.
Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...
I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?
RegardsPavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 15.12.2020 16:18, Pavel Stehule wrote:út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 19:27, Pavel Stehule wrote:pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?My idea was to reduce necessary rights to database owners. But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.yes, it is truePavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
So what's next?
I see three options:
1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).
2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".
3. Implement some mechanism for caching presence of event triggers in shared memory.@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.
Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...
I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?
RegardsPavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
I like this approach more than implementation of shared hash.út 15. 12. 2020 v 15:06 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 15.12.2020 16:18, Pavel Stehule wrote:út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 19:27, Pavel Stehule wrote:pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 11.12.2020 18:40, Pavel Stehule wrote:is not correct. It makes it not possible to superuser to disable triggers for all users.pg_database_ownercheck returns true for superuser always.
Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?My idea was to reduce necessary rights to database owners. But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.Also GUCs are not associated with any database. So I do not understand why this check of database ownership is relevant at all?
What kind of protection violation we want to prevent?
It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.
But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.yes, it is truePavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
So what's next?
I see three options:
1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).
2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".
3. Implement some mechanism for caching presence of event triggers in shared memory.@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.
Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...
I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column. Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.What do you think about it?
But still I have some concerns:
1. pg_database table format has to be changed. Certainly it is not something completely unacceptable. But IMHO we should try to avoid
modification of such commonly used catalog tables as much as possible.
2. It is not so easy to maintain this flag. There can be multiple on-login triggers defined. If such trigger is dropped, we can not just clear this flag.
We should check if other triggers exist. Now assume that there are two triggers and two concurrent transactions dropping each one.
According to their snapshots them do not see changes made by other transaction. So them remove both triggers but didn't clear the flag.
Certainly we can use counter instead of flag. But I am not sure that their will be not other problems with maintaining counter.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
I like this approach more than implementation of shared hash.Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...
I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column. Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.What do you think about it?
But still I have some concerns:
1. pg_database table format has to be changed. Certainly it is not something completely unacceptable. But IMHO we should try to avoid
modification of such commonly used catalog tables as much as possible.
2. It is not so easy to maintain this flag. There can be multiple on-login triggers defined. If such trigger is dropped, we can not just clear this flag.
We should check if other triggers exist. Now assume that there are two triggers and two concurrent transactions dropping each one.
According to their snapshots them do not see changes made by other transaction. So them remove both triggers but didn't clear the flag.
Certainly we can use counter instead of flag. But I am not sure that their will be not other problems with maintaining counter.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchI like this approach more than implementation of shared hash.Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...
I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column. Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.What do you think about it?
But still I have some concerns:
1. pg_database table format has to be changed. Certainly it is not something completely unacceptable. But IMHO we should try to avoid
modification of such commonly used catalog tables as much as possible.yes, I know. Unfortunately I don't see any other and correct solution. There should be more wide discussion before this work about this topic. On second hand, this change should not break anything (if this new field will be placed on as the last field). The logon trigger really looks like a database trigger - so I think so this semantic is correct. I have no idea if it is acceptable for committers :-/. I hope so.The fact that the existence of a logon trigger can be visible from a shared database object can be an interesting feature too.
2. It is not so easy to maintain this flag. There can be multiple on-login triggers defined. If such trigger is dropped, we can not just clear this flag.
We should check if other triggers exist. Now assume that there are two triggers and two concurrent transactions dropping each one.
According to their snapshots them do not see changes made by other transaction. So them remove both triggers but didn't clear the flag.
Certainly we can use counter instead of flag. But I am not sure that their will be not other problems with maintaining counter.I don't think it is necessary. My opinion is not too strong, but if pg_class doesn't need to hold a number of triggers, then I think so pg_database doesn't need to hold this number too.
So there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
So there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
+ {
+ return;
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchSo there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).This is much better - I don't see any slowdown when logon trigger is not definedI did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is betterI don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger
RegardsPavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchSo there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).This is much better - I don't see any slowdown when logon trigger is not definedI did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is betterI don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_triggerI thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.
I prefer to implement different approach: unset dathaslogontriggers flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.
Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.
New version of the patch is attached.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
Hi, On 17.12.2020 3:31, Zhihong Yu wrote: > Hi, > For EventTriggerOnConnect(): > > + PG_CATCH(); > + { > ... > + AbortCurrentTransaction(); > + return; > > Should runlist be freed in the catch block ? No need: it is allocated in transaction memory context and removed on transaction abort. > > + gettext_noop("In case of errors in the ON > client_connection EVENT TRIGGER procedure, this parameter can be used > to disable trigger activation and provide access to the database."), > > I think the text should be on two lines (current line too long). Thank you, fixed. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 17.12.2020 9:31, Pavel Stehule wrote:st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchSo there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).This is much better - I don't see any slowdown when logon trigger is not definedI did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is betterI don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_triggerI thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.
I prefer to implement different approach: unset dathaslogontriggers flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.
Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.
New version of the patch is attached.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 17.12.2020 9:31, Pavel Stehule wrote:st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchSo there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).This is much better - I don't see any slowdown when logon trigger is not definedI did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is betterI don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_triggerI thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.
I prefer to implement different approach: unset dathaslogontriggers flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.
Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.
New version of the patch is attached.yes, it can work
RegardsPavel-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
čt 17. 12. 2020 v 19:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 17.12.2020 9:31, Pavel Stehule wrote:st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchSo there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).This is much better - I don't see any slowdown when logon trigger is not definedI did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is betterI don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_triggerI thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.
I prefer to implement different approach: unset dathaslogontriggers flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.
Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.
New version of the patch is attached.yes, it can workfor complete review the new field in pg_doc should be documented
Done.
Also I fixed some examples in documentation which involves pg_database.
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
On 20.12.2020 10:04, Pavel Stehule wrote:čt 17. 12. 2020 v 19:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 17.12.2020 9:31, Pavel Stehule wrote:st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patchSo there is still only "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.
As you suggested have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).This is much better - I don't see any slowdown when logon trigger is not definedI did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is betterI don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_triggerI thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.
I prefer to implement different approach: unset dathaslogontriggers flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.
Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.
New version of the patch is attached.yes, it can workfor complete review the new field in pg_doc should be documented
Done.
Also I fixed some examples in documentation which involves pg_database.
test fast_default ... FAILED 392 ms
test stats ... FAILED 626 ms
============== shutting down postmaster ==============
-- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
regress tests failssysviews ... FAILED 112 mstest event_trigger ... FAILED (test process exited with exit code 2) 447 ms
test fast_default ... FAILED 392 ms
test stats ... FAILED 626 ms
============== shutting down postmaster ==============
Sorry, fixed.
--
Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
On 22.12.2020 12:25, Pavel Stehule wrote:regress tests failssysviews ... FAILED 112 mstest event_trigger ... FAILED (test process exited with exit code 2) 447 ms
test fast_default ... FAILED 392 ms
test stats ... FAILED 626 ms
============== shutting down postmaster ==============
Sorry, fixed.
index 6ff783273f..7aded1848f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1008,7 +1008,7 @@ include_dir 'conf.d'
trigger when a client connects. This parameter is switched on by default.
Errors in trigger code can prevent user to login to the system.
I this case disabling this parameter in connection string can solve the problem:
- <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'".
+ <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
</para>
</listitem>
</varlistentry>
+ <term><varname>enable_client_connection_trigger</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_client_connection_trigger</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables firing the <literal>client_connection</literal>
+ trigger when a client connects. This parameter is switched on by default.
+ Errors in trigger code can prevent user to login to the system.
+ I this case disabling this parameter in connection string can solve the problem:
+ <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
index 3a43c09bf6..08f00d8fc4 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2970,7 +2970,8 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("ON");
/* Complete CREATE EVENT TRIGGER <name> ON with event_type */
else if (Matches("CREATE", "EVENT", "TRIGGER", MatchAny, "ON"))
- COMPLETE_WITH("ddl_command_start", "ddl_command_end", "sql_drop");
+ COMPLETE_WITH("ddl_command_start", "ddl_command_end",
+ "client_connection", "sql_drop");
/*
* Complete CREATE EVENT TRIGGER <name> ON <event_type>. EXECUTE FUNCTION
--Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
út 22. 12. 2020 v 12:42 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:On 22.12.2020 12:25, Pavel Stehule wrote:regress tests failssysviews ... FAILED 112 mstest event_trigger ... FAILED (test process exited with exit code 2) 447 ms
test fast_default ... FAILED 392 ms
test stats ... FAILED 626 ms
============== shutting down postmaster ==============
Sorry, fixed.no problemI had to fix docdiff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6ff783273f..7aded1848f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1008,7 +1008,7 @@ include_dir 'conf.d'
trigger when a client connects. This parameter is switched on by default.
Errors in trigger code can prevent user to login to the system.
I this case disabling this parameter in connection string can solve the problem:
- <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'".
+ <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
</para>
</listitem>
</varlistentry>I am thinking again about enable_client_connection_trigger, and although it can look useless (because error is ignored for superuser), it can be useful for some debugging and administration purposes. Probably we don't want to start the client_connection trigger from backup tools, maybe from some monitoring tools. Maybe the possibility to set this GUC can be dedicated to some special role (like pg_signal_backend).+ <varlistentry id="guc-enable-client-connection-trigger" xreflabel="enable_client_connection_trigger">
+ <term><varname>enable_client_connection_trigger</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>enable_client_connection_trigger</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Enables firing the <literal>client_connection</literal>
+ trigger when a client connects. This parameter is switched on by default.
+ Errors in trigger code can prevent user to login to the system.
+ I this case disabling this parameter in connection string can solve the problem:
+ <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
+ </para>
+ </listitem>
+ </varlistentry>There should be note, so only superuser can change this valueThere is should be tab-complete supportdiff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 3a43c09bf6..08f00d8fc4 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2970,7 +2970,8 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("ON");
/* Complete CREATE EVENT TRIGGER <name> ON with event_type */
else if (Matches("CREATE", "EVENT", "TRIGGER", MatchAny, "ON"))
- COMPLETE_WITH("ddl_command_start", "ddl_command_end", "sql_drop");
+ COMPLETE_WITH("ddl_command_start", "ddl_command_end",
+ "client_connection", "sql_drop");
/*
* Complete CREATE EVENT TRIGGER <name> ON <event_type>. EXECUTE FUNCTION
Thank you.
I have applied all your fixes in on_connect_event_trigger-12.patch.
Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
In this case GUC is most convenient way to do it.
Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вложения
Thank you.
I have applied all your fixes in on_connect_event_trigger-12.patch.
Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
In this case GUC is most convenient way to do it.
index f810789..8861f1b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1,4 +1,4 @@
-<!-- doc/src/sgml/config.sgml -->
+\<!-- doc/src/sgml/config.sgml -->
Вложения
HiThank you.
I have applied all your fixes in on_connect_event_trigger-12.patch.
Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
In this case GUC is most convenient way to do it.There was typo in patchdiff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f810789..8861f1b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1,4 +1,4 @@
-<!-- doc/src/sgml/config.sgml -->
+\<!-- doc/src/sgml/config.sgml -->
I have not any objections against functionality or design. I tested the performance, and there are no negative impacts when this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when this trigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overhead can be ignored.* make without warnings* make check-world passed* doc build passedPossible ToDo:The documentation can contain a note so usage connect triggers in environments with short life sessions and very short fast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initialization of PLpgSQL engine.I'll mark this patch as ready for committers
RegardsPavel
On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote: > > > > so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal: >> >> Hi >> >> >>> Thank you. >>> I have applied all your fixes in on_connect_event_trigger-12.patch. >>> >>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest wayto disable login triggers in case >>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTEREVENT TRIGGER DISABLE". >>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And youwant temporary disable them all, for example for testing purposes. >>> In this case GUC is most convenient way to do it. >>> >> >> There was typo in patch >> >> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml >> index f810789..8861f1b 100644 >> --- a/doc/src/sgml/config.sgml >> +++ b/doc/src/sgml/config.sgml >> @@ -1,4 +1,4 @@ >> -<!-- doc/src/sgml/config.sgml --> >> +\<!-- doc/src/sgml/config.sgml --> >> >> I have not any objections against functionality or design. I tested the performance, and there are no negative impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overheadcan be ignored. >> >> * make without warnings >> * make check-world passed >> * doc build passed >> >> Possible ToDo: >> >> The documentation can contain a note so usage connect triggers in environments with short life sessions and very shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initializationof PLpgSQL engine. >> >> I'll mark this patch as ready for committers > > > looks so this patch has not entry in commitfestapp 2021-01 > Yeah, please register this patch before the next CommitFest[1] starts, 2021-01-01 AoE[2]. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/
On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote: > > On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote: > > > > > > > > so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal: > >> > >> Hi > >> > >> > >>> Thank you. > >>> I have applied all your fixes in on_connect_event_trigger-12.patch. > >>> > >>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest wayto disable login triggers in case > >>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTEREVENT TRIGGER DISABLE". > >>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. Andyou want temporary disable them all, for example for testing purposes. > >>> In this case GUC is most convenient way to do it. > >>> > >> > >> There was typo in patch > >> > >> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml > >> index f810789..8861f1b 100644 > >> --- a/doc/src/sgml/config.sgml > >> +++ b/doc/src/sgml/config.sgml > >> @@ -1,4 +1,4 @@ > >> -<!-- doc/src/sgml/config.sgml --> > >> +\<!-- doc/src/sgml/config.sgml --> > >> > >> I have not any objections against functionality or design. I tested the performance, and there are no negative impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overheadcan be ignored. > >> > >> * make without warnings > >> * make check-world passed > >> * doc build passed > >> > >> Possible ToDo: > >> > >> The documentation can contain a note so usage connect triggers in environments with short life sessions and very shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initializationof PLpgSQL engine. > >> > >> I'll mark this patch as ready for committers > > > > > > looks so this patch has not entry in commitfestapp 2021-01 > > > > Yeah, please register this patch before the next CommitFest[1] starts, > 2021-01-01 AoE[2]. > Konstantin, did you register this patch in any CF? Even though the reviewer seems to be happy with the patch, I am afraid that we might lose track of this unless we register it. -- With Regards, Amit Kapila.
On Thu, Jan 28, 2021 at 8:17 AM Amit Kapila <amit.kapila16@gmail.com> wrote: > > On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote: > > > > On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote: > > > > > > > > > > > > so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal: > > >> > > >> Hi > > >> > > >> > > >>> Thank you. > > >>> I have applied all your fixes in on_connect_event_trigger-12.patch. > > >>> > > >>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplestway to disable login triggers in case > > >>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTEREVENT TRIGGER DISABLE". > > >>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. Andyou want temporary disable them all, for example for testing purposes. > > >>> In this case GUC is most convenient way to do it. > > >>> > > >> > > >> There was typo in patch > > >> > > >> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml > > >> index f810789..8861f1b 100644 > > >> --- a/doc/src/sgml/config.sgml > > >> +++ b/doc/src/sgml/config.sgml > > >> @@ -1,4 +1,4 @@ > > >> -<!-- doc/src/sgml/config.sgml --> > > >> +\<!-- doc/src/sgml/config.sgml --> > > >> > > >> I have not any objections against functionality or design. I tested the performance, and there are no negative impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overheadcan be ignored. > > >> > > >> * make without warnings > > >> * make check-world passed > > >> * doc build passed > > >> > > >> Possible ToDo: > > >> > > >> The documentation can contain a note so usage connect triggers in environments with short life sessions and very shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initializationof PLpgSQL engine. > > >> > > >> I'll mark this patch as ready for committers > > > > > > > > > looks so this patch has not entry in commitfestapp 2021-01 > > > > > > > Yeah, please register this patch before the next CommitFest[1] starts, > > 2021-01-01 AoE[2]. > > > > Konstantin, did you register this patch in any CF? Even though the > reviewer seems to be happy with the patch, I am afraid that we might > lose track of this unless we register it. > I see the CF entry (https://commitfest.postgresql.org/31/2900/) for this work. Sorry for the noise. -- With Regards, Amit Kapila.
On 28.01.2021 5:47, Amit Kapila wrote: > On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote: >> On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote: >>> >>> >>> so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal: >>>> Hi >>>> >>>> >>>>> Thank you. >>>>> I have applied all your fixes in on_connect_event_trigger-12.patch. >>>>> >>>>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest wayto disable login triggers in case >>>>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTEREVENT TRIGGER DISABLE". >>>>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. Andyou want temporary disable them all, for example for testing purposes. >>>>> In this case GUC is most convenient way to do it. >>>>> >>>> There was typo in patch >>>> >>>> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml >>>> index f810789..8861f1b 100644 >>>> --- a/doc/src/sgml/config.sgml >>>> +++ b/doc/src/sgml/config.sgml >>>> @@ -1,4 +1,4 @@ >>>> -<!-- doc/src/sgml/config.sgml --> >>>> +\<!-- doc/src/sgml/config.sgml --> >>>> >>>> I have not any objections against functionality or design. I tested the performance, and there are no negative impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overheadcan be ignored. >>>> >>>> * make without warnings >>>> * make check-world passed >>>> * doc build passed >>>> >>>> Possible ToDo: >>>> >>>> The documentation can contain a note so usage connect triggers in environments with short life sessions and very shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initializationof PLpgSQL engine. >>>> >>>> I'll mark this patch as ready for committers >>> >>> looks so this patch has not entry in commitfestapp 2021-01 >>> >> Yeah, please register this patch before the next CommitFest[1] starts, >> 2021-01-01 AoE[2]. >> > Konstantin, did you register this patch in any CF? Even though the > reviewer seems to be happy with the patch, I am afraid that we might > lose track of this unless we register it. > Yes, certainly: https://commitfest.postgresql.org/31/2900/ -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Вторник, 16 марта 2021, 14:32 +03:00 от Ivan Panchenko <wao@mail.ru>:
Hi,
Thank you, Konstantin, for this very good feature with numerous use cases.
Please find the modified patch attached.
I’ve added the ‘enable_client_connection_trigger’ GUC to the sample config file and also an additional example page to the docs.
Check world has passed and it is ready for committer.
Четверг, 28 января 2021, 12:04 +03:00 от Konstantin Knizhnik <k.knizhnik@postgrespro.ru>:
On 28.01.2021 5:47, Amit Kapila wrote:> On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:Yes, certainly:
>> On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>>
>>>
>>> so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>>>> Hi
>>>>
>>>>
>>>>> Thank you.
>>>>> I have applied all your fixes in on_connect_event_trigger-12.patch.
>>>>>
>>>>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
>>>>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
>>>>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
>>>>> In this case GUC is most convenient way to do it.
>>>>>
>>>> There was typo in patch
>>>>
>>>> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
>>>> index f810789..8861f1b 100644
>>>> --- a/doc/src/sgml/config.sgml
>>>> +++ b/doc/src/sgml/config.sgml
>>>> @@ -1,4 +1,4 @@
>>>> -<!-- doc/src/sgml/config.sgml -->
>>>> +\<!-- doc/src/sgml/config.sgml -->
>>>>
>>>> I have not any objections against functionality or design. I tested the performance, and there are no negative impacts when this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when this trigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overhead can be ignored.
>>>>
>>>> * make without warnings
>>>> * make check-world passed
>>>> * doc build passed
>>>>
>>>> Possible ToDo:
>>>>
>>>> The documentation can contain a note so usage connect triggers in environments with short life sessions and very short fast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initialization of PLpgSQL engine.
>>>>
>>>> I'll mark this patch as ready for committers
>>>
>>> looks so this patch has not entry in commitfestapp 2021-01
>>>
>> Yeah, please register this patch before the next CommitFest[1] starts,
>> 2021-01-01 AoE[2].
>>
> Konstantin, did you register this patch in any CF? Even though the
> reviewer seems to be happy with the patch, I am afraid that we might
> lose track of this unless we register it.
>
https://commitfest.postgresql.org/31/2900/
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Вложения
On Thu, May 20, 2021 at 2:45 PM Ivan Panchenko <wao@mail.ru> wrote: > > I have upgraded the patch for the 14th version. > I have some feedback on the patch: (1) The patch adds 3 whitespace errors ("git apply <patch-file>" reports 3 warnings) (2) doc/src/sgml/catalogs.sgml CURRENTLY: This flag is used to avoid extra lookup of pg_event_trigger table on each backend startup. SUGGEST: This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup. (3) doc/src/sgml/config.sgml CURRENTLY: Errors in trigger code can prevent user to login to the system.In this case disabling this parameter in connection string can solve the problem: SUGGEST: Errors in the trigger code can prevent a user from logging in to the system. In this case, the parameter can be disabled in the connection string, to allow the user to login: (4) doc/src/sgml/event-trigger.sgml (i) CURRENTLY: An event trigger fires whenever the event with which it is associated occurs in the database in which it is defined. Currently, the only SUGGEST: An event trigger fires whenever an associated event occurs in the database in which it is defined. Currently, the only (ii) CURRENTLY: can be useful for client connections logging, SUGGEST: can be useful for logging client connections, (5) src/backend/commands/event_trigger.c (i) There are two instances of code blocks like: xxxx = table_open(...); tuple = SearchSysCacheCopy1(...); table_close(...); These should end with: "heap_freetuple(tuple);" (ii) Typo "nuder" and grammar: CURRENTLY: There can be race condition: event trigger may be added after we have scanned pg_event_trigger table. Repeat this test nuder pg_database table lock. SUGGEST: There can be a race condition: the event trigger may be added after we have scanned the pg_event_trigger table. Repeat this test under the pg_database table lock. (6) src/backend/utils/misc/postgresql.conf.sample CURRENTLY: +#enable_client_connection_trigger = true # enables firing the client_connection trigger when a client connect SUGGEST: +#enable_client_connection_trigger = true # enables firing the client_connection trigger when a client connects Regards, Greg Nancarrow Fujitsu Australia
On Fri, May 21, 2021 at 2:46 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > On Thu, May 20, 2021 at 2:45 PM Ivan Panchenko <wao@mail.ru> wrote: > > > > I have upgraded the patch for the 14th version. > > > > I have some feedback on the patch: > I've attached an updated version of the patch. I've applied my review comments and done a bit more tidying up of documentation and comments. Also, I found that the previously-posted patch was broken by snapshot-handling changes in commit 84f5c290 (with patch applied, resulting in a coredump during regression tests) so I've added a fix for that too. Regards, Greg Nancarrow Fujitsu Australia
Вложения
On Thu, Jun 3, 2021 at 8:36 AM Greg Nancarrow <gregn4422@gmail.com> wrote: > > On Fri, May 21, 2021 at 2:46 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > > > On Thu, May 20, 2021 at 2:45 PM Ivan Panchenko <wao@mail.ru> wrote: > > > > > > I have upgraded the patch for the 14th version. > > > > > > > I have some feedback on the patch: > > > > I've attached an updated version of the patch. > I've applied my review comments and done a bit more tidying up of > documentation and comments. > Also, I found that the previously-posted patch was broken by > snapshot-handling changes in commit 84f5c290 (with patch applied, > resulting in a coredump during regression tests) so I've added a fix > for that too. CFBot shows the following failure: # poll_query_until timed out executing this query: # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM pg_catalog.pg_stat_replication WHERE application_name = 'standby_1'; # expecting this output: # t # last actual query output: # t # with stderr: # NOTICE: You are welcome! # Looks like your test exited with 29 before it could output anything. t/001_stream_rep.pl .................. Dubious, test returned 29 (wstat 7424, 0x1d00) Regards, Vignesh
On Sun, Jul 4, 2021 at 1:21 PM vignesh C <vignesh21@gmail.com> wrote: > > CFBot shows the following failure: > # poll_query_until timed out executing this query: > # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM > pg_catalog.pg_stat_replication WHERE application_name = 'standby_1'; > # expecting this output: > # t > # last actual query output: > # t > # with stderr: > # NOTICE: You are welcome! > # Looks like your test exited with 29 before it could output anything. > t/001_stream_rep.pl .................. > Dubious, test returned 29 (wstat 7424, 0x1d00) > Thanks. I found that the patch was broken by commit f452aaf7d (the part "adjust poll_query_until to insist on empty stderr as well as a stdout match"). So I had to remove a "RAISE NOTICE" (which was just an informational message) from the login trigger function, to satisfy the new poll_query_until expectations. Also, I updated a PG14 version check (now must check PG15 version). Regards, Greg Nancarrow Fujitsu Australia
Вложения
On Sun, Jul 4, 2021 at 1:21 PM vignesh C <vignesh21@gmail.com> wrote:
>
> CFBot shows the following failure:
> # poll_query_until timed out executing this query:
> # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM
> pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
> # expecting this output:
> # t
> # last actual query output:
> # t
> # with stderr:
> # NOTICE: You are welcome!
> # Looks like your test exited with 29 before it could output anything.
> t/001_stream_rep.pl ..................
> Dubious, test returned 29 (wstat 7424, 0x1d00)
>
Thanks.
I found that the patch was broken by commit f452aaf7d (the part
"adjust poll_query_until to insist on empty stderr as well as a stdout
match").
So I had to remove a "RAISE NOTICE" (which was just an informational
message) from the login trigger function, to satisfy the new
poll_query_until expectations.
Also, I updated a PG14 version check (now must check PG15 version).
Regards,
Greg Nancarrow
Fujitsu Australia
Hunk #1 FAILED at 93. 1 out of 1 hunk FAILED -- saving rejects to file src/test/regress/expected/sysviews.out.rej
I am not changing the status because it is a minor change and the patch is already "Ready for Committer".
On Mon, Jul 19, 2021 at 8:18 PM Ibrar Ahmed <ibrar.ahmad@gmail.com> wrote: > > The patch does not apply, and rebase is required. > Attached a rebased patch (minor updates to the test code). Regards, Greg Nancarrow Fujitsu Australia
Вложения
Среда, 7 июля 2021, 3:55 +03:00 от Greg Nancarrow <gregn4422@gmail.com>:
On Sun, Jul 4, 2021 at 1:21 PM vignesh C <vignesh21@gmail.com> wrote:>
> CFBot shows the following failure:
> # poll_query_until timed out executing this query:
> # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM
> pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
> # expecting this output:
> # t
> # last actual query output:
> # t
> # with stderr:
> # NOTICE: You are welcome!
> # Looks like your test exited with 29 before it could output anything.
> t/001_stream_rep.pl ..................
> Dubious, test returned 29 (wstat 7424, 0x1d00)
>
Thanks.
I found that the patch was broken by commit f452aaf7d (the part
"adjust poll_query_until to insist on empty stderr as well as a stdout
match").
So I had to remove a "RAISE NOTICE" (which was just an informational
message) from the login trigger function, to satisfy the new
poll_query_until expectations.
Does it mean that "RAISE NOTICE" should’t be used, or behaves unexpectedly in logon triggers? Should we mention this in the docs?
Also, I updated a PG14 version check (now must check PG15 version).
Regards,
Greg Nancarrow
Fujitsu Australia
On Tue, Aug 17, 2021 at 1:11 AM Ivan Panchenko <wao@mail.ru> wrote: > > Does it mean that "RAISE NOTICE" should’t be used, or behaves unexpectedly in logon triggers? Should we mention this inthe docs? > No I don't believe so, it was just that that part of the test framework (sub poll_query_until) had been changed to regard anything output to stderr as an error (so now for the test to succeed, whatever is printed to stdout must match the expected test output, and stderr must be empty). Regards, Greg Nancarrow Fujitsu Australia
> On 19 Jul 2021, at 15:25, Greg Nancarrow <gregn4422@gmail.com> wrote: > Attached a rebased patch (minor updates to the test code). I took a look at this, and while I like the proposed feature I think the patch has a bit more work required. + END IF; + +-- 2) Initialize some user session data + CREATE TEMP TABLE session_storage (x float, y integer); The example in the documentation use a mix of indentations, neither of which is the 2-space indentation used elsewhere in the docs. + /* Get the command tag. */ + tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT; This is hardcoding knowledge that currently only this trigger wont have a parsetree, and setting the tag accordingly. This should instead check the event and set CMDTAG_UNKNOWN if it isn't the expected one. + /* database has on-login triggers */ + bool dathaslogontriggers; This patch uses three different names for the same thing: client connection, logontrigger and login trigger. Since this trigger only fires after successful authentication it’s not accurate to name it client connection, as that implies it running on connections rather than logins. The nomenclature used in the tree is "login" so the patch should be adjusted everywhere to use that. +/* Hook for plugins to get control at start of session */ +client_connection_hook_type client_connection_hook = EventTriggerOnConnect; I don't see the reason for adding core functionality by hooks. Having a hook might be a useful thing on its own (to be discussed in a new thread, not hidden here), but core functionality should not depend on it IMO. + {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS, + gettext_noop("Enables the client_connection event trigger."), + gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, " ..and.. + /* + * Try to ignore error for superuser to make it possible to login even in case of errors + * during trigger execution + */ + if (!is_superuser) + PG_RE_THROW(); This patch adds two ways for superusers to bypass this event trigger in case of it being faulty, but for every other event trigger we've documented to restart in single-user mode and fixing it there. Why does this need to be different? This clearly has a bigger chance of being a footgun but I don't see that as a reason to add a GUC and a bypass that other footguns lack. + elog(NOTICE, "client_connection trigger failed with message: %s", error->message); Calling elog() in a PG_CATCH() block isn't allowed is it? + /* Runtlist is empty: clear dathaslogontriggers flag + */ s/Runtlist/Runlist/ and also commenting style. The logic for resetting the pg_database flag when firing event trigger in case the trigger has gone away is messy and a problem waiting to happen IMO. For normal triggers we don't bother with that on the grounds of it being racy, and instead perform relhastrigger removal during VACUUM. Another approach is using a counter as propose upthread, since updating that can be made safer. The current coding also isn't instructing concurrent backends to perform relcache rebuild. -- Daniel Gustafsson https://vmware.com/
+ {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
+ gettext_noop("Enables the client_connection event trigger."),
+ gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
..and..
+ /*
+ * Try to ignore error for superuser to make it possible to login even in case of errors
+ * during trigger execution
+ */
+ if (!is_superuser)
+ PG_RE_THROW();
This patch adds two ways for superusers to bypass this event trigger in case of
it being faulty, but for every other event trigger we've documented to restart
in single-user mode and fixing it there. Why does this need to be different?
This clearly has a bigger chance of being a footgun but I don't see that as a
reason to add a GUC and a bypass that other footguns lack.
> On 8 Sep 2021, at 16:02, Pavel Stehule <pavel.stehule@gmail.com> wrote: > In the time when event triggers were introduced, managed services were not too widely used like now. When we discussedthis feature we thought about environments when users have no superuser rights and have no possibility to go tosingle mode. In situations where you don't have superuser access and cannot restart in single user mode, none of the bypasses in this patch would help anyways. I understand the motivation, but continuing on even in the face of an ereport(ERROR.. ) in the hopes of being able to turn off buggy code seems pretty unsafe at best. -- Daniel Gustafsson https://vmware.com/
> On 8 Sep 2021, at 16:02, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> In the time when event triggers were introduced, managed services were not too widely used like now. When we discussed this feature we thought about environments when users have no superuser rights and have no possibility to go to single mode.
In situations where you don't have superuser access and cannot restart in
single user mode, none of the bypasses in this patch would help anyways.
I understand the motivation, but continuing on even in the face of an
ereport(ERROR.. ) in the hopes of being able to turn off buggy code seems
pretty unsafe at best.
--
Daniel Gustafsson https://vmware.com/
On Wed, Sep 8, 2021 at 10:56 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > I took a look at this, and while I like the proposed feature I think the patch > has a bit more work required. > Thanks for reviewing the patch. I am not the original patch author (who no longer seems active) but I've been contributing a bit and keeping the patch alive because I think it's a worthwhile feature. > > + > +-- 2) Initialize some user session data > + CREATE TEMP TABLE session_storage (x float, y integer); > The example in the documentation use a mix of indentations, neither of which is > the 2-space indentation used elsewhere in the docs. > Fixed, using 2-space indentation. (to be honest, the indentation seems inconsistent elsewhere in the docs e.g. I'm seeing a nearby case of 2-space on 1st indent, 6-space on 2nd indent) > > + /* Get the command tag. */ > + tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT; > This is hardcoding knowledge that currently only this trigger wont have a > parsetree, and setting the tag accordingly. This should instead check the > event and set CMDTAG_UNKNOWN if it isn't the expected one. > I updated that, but maybe not exactly how you expected? > > + /* database has on-login triggers */ > + bool dathaslogontriggers; > This patch uses three different names for the same thing: client connection, > logontrigger and login trigger. Since this trigger only fires after successful > authentication it’s not accurate to name it client connection, as that implies > it running on connections rather than logins. The nomenclature used in the > tree is "login" so the patch should be adjusted everywhere to use that. > Fixed. > > +/* Hook for plugins to get control at start of session */ > +client_connection_hook_type client_connection_hook = EventTriggerOnConnect; > I don't see the reason for adding core functionality by hooks. Having a hook > might be a useful thing on its own (to be discussed in a new thread, not hidden > here), but core functionality should not depend on it IMO. > Fair enough, I removed the hook. > > + {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS, > + gettext_noop("Enables the client_connection event trigger."), > + gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, " > ..and.. > + /* > + * Try to ignore error for superuser to make it possible to login even in case of errors > + * during trigger execution > + */ > + if (!is_superuser) > + PG_RE_THROW(); > This patch adds two ways for superusers to bypass this event trigger in case of > it being faulty, but for every other event trigger we've documented to restart > in single-user mode and fixing it there. Why does this need to be different? > This clearly has a bigger chance of being a footgun but I don't see that as a > reason to add a GUC and a bypass that other footguns lack. > OK, I removed those bypasses. We'll see what others think. > > + elog(NOTICE, "client_connection trigger failed with message: %s", error->message); > Calling elog() in a PG_CATCH() block isn't allowed is it? > I believe it is allowed (e.g. there's a case in libpq), but I removed this anyway as part of the superuser bypass. > > + /* Runtlist is empty: clear dathaslogontriggers flag > + */ > s/Runtlist/Runlist/ and also commenting style. > Fixed. > > The logic for resetting the pg_database flag when firing event trigger in case > the trigger has gone away is messy and a problem waiting to happen IMO. For > normal triggers we don't bother with that on the grounds of it being racy, and > instead perform relhastrigger removal during VACUUM. Another approach is using > a counter as propose upthread, since updating that can be made safer. The > current coding also isn't instructing concurrent backends to perform relcache > rebuild. > I think there are pros and cons of each possible approach, but I think I prefer the current way (which I have tweaked a bit) for similar reasons to those explained by the original patch author when debating whether to use reference-counting instead, in the discussion upthread (e.g. it keeps it all in one place). Also, it seems to be more inline with the documented reason why that pg_database flag was added in the first place. I have debugged a few concurrent scenarios with the current mechanism in place. If you still dislike the logic for resetting the flag, please elaborate on the issues you foresee and one of the alternative approaches can be tried. I've attached the updated patch. Regards, Greg Nancarrow Fujitsu Australia
Вложения
Hi! Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK, there is two use cases for this feature: 1 A permission / prohibition to login some users 2 Just a logging of facts of user's login Suggested patch proposes prohibition of login only by failing of login trigger and it has at least two issues: 1 In case of prohibition to login, there is no clean way to store information about unsuccessful login. Ok, it could be solved by dblink module but it seems to scary. 2 For logging purpose failing of trigger will cause impossibility to login, it could be workarounded by catching error in trigger function, but it's not so obvious for DBA. some other issues: 3 It's easy to create security hole, see attachment where non-privileged user can close access to database even for superuser. 4 Feature is not applicable for logging unsuccessful login with wrong username/password or not matched by pg_hba.conf. Oracle could operate with such cases. But I don't think that this issue is a stopper for the patch. May be, to solve that issues we could introduce return value of trigger and/or add something like IGNORE ERROR to CREATE EVENT TRIGGER command. On 15.09.2021 16:32, Greg Nancarrow wrote: > On Wed, Sep 8, 2021 at 10:56 PM Daniel Gustafsson <daniel@yesql.se> wrote: >> >> I took a look at this, and while I like the proposed feature I think the patch >> has a bit more work required. >> > > Thanks for reviewing the patch. > I am not the original patch author (who no longer seems active) but > I've been contributing a bit and keeping the patch alive because I > think it's a worthwhile feature. > >> >> + >> +-- 2) Initialize some user session data >> + CREATE TEMP TABLE session_storage (x float, y integer); >> The example in the documentation use a mix of indentations, neither of which is >> the 2-space indentation used elsewhere in the docs. >> > > Fixed, using 2-space indentation. > (to be honest, the indentation seems inconsistent elsewhere in the > docs e.g. I'm seeing a nearby case of 2-space on 1st indent, 6-space > on 2nd indent) > >> >> + /* Get the command tag. */ >> + tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT; >> This is hardcoding knowledge that currently only this trigger wont have a >> parsetree, and setting the tag accordingly. This should instead check the >> event and set CMDTAG_UNKNOWN if it isn't the expected one. >> > > I updated that, but maybe not exactly how you expected? > >> >> + /* database has on-login triggers */ >> + bool dathaslogontriggers; >> This patch uses three different names for the same thing: client connection, >> logontrigger and login trigger. Since this trigger only fires after successful >> authentication it’s not accurate to name it client connection, as that implies >> it running on connections rather than logins. The nomenclature used in the >> tree is "login" so the patch should be adjusted everywhere to use that. >> > > Fixed. > >> >> +/* Hook for plugins to get control at start of session */ >> +client_connection_hook_type client_connection_hook = EventTriggerOnConnect; >> I don't see the reason for adding core functionality by hooks. Having a hook >> might be a useful thing on its own (to be discussed in a new thread, not hidden >> here), but core functionality should not depend on it IMO. >> > > Fair enough, I removed the hook. > >> >> + {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS, >> + gettext_noop("Enables the client_connection event trigger."), >> + gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, " >> ..and.. >> + /* >> + * Try to ignore error for superuser to make it possible to login even in case of errors >> + * during trigger execution >> + */ >> + if (!is_superuser) >> + PG_RE_THROW(); >> This patch adds two ways for superusers to bypass this event trigger in case of >> it being faulty, but for every other event trigger we've documented to restart >> in single-user mode and fixing it there. Why does this need to be different? >> This clearly has a bigger chance of being a footgun but I don't see that as a >> reason to add a GUC and a bypass that other footguns lack. >> > > OK, I removed those bypasses. We'll see what others think. > >> >> + elog(NOTICE, "client_connection trigger failed with message: %s", error->message); >> Calling elog() in a PG_CATCH() block isn't allowed is it? >> > > I believe it is allowed (e.g. there's a case in libpq), but I removed > this anyway as part of the superuser bypass. > >> >> + /* Runtlist is empty: clear dathaslogontriggers flag >> + */ >> s/Runtlist/Runlist/ and also commenting style. >> > > Fixed. > >> >> The logic for resetting the pg_database flag when firing event trigger in case >> the trigger has gone away is messy and a problem waiting to happen IMO. For >> normal triggers we don't bother with that on the grounds of it being racy, and >> instead perform relhastrigger removal during VACUUM. Another approach is using >> a counter as propose upthread, since updating that can be made safer. The >> current coding also isn't instructing concurrent backends to perform relcache >> rebuild. >> > > I think there are pros and cons of each possible approach, but I think > I prefer the current way (which I have tweaked a bit) for similar > reasons to those explained by the original patch author when debating > whether to use reference-counting instead, in the discussion upthread > (e.g. it keeps it all in one place). Also, it seems to be more inline > with the documented reason why that pg_database flag was added in the > first place. I have debugged a few concurrent scenarios with the > current mechanism in place. If you still dislike the logic for > resetting the flag, please elaborate on the issues you foresee and one > of the alternative approaches can be tried. > > I've attached the updated patch. > > Regards, > Greg Nancarrow > Fujitsu Australia > -- Teodor Sigaev E-mail: teodor@sigaev.ru WWW: http://www.sigaev.ru/
Вложения
On Wed, Sep 29, 2021 at 10:14 PM Teodor Sigaev <teodor@sigaev.ru> wrote: > > Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK, > there is two use cases for this feature: > 1 A permission / prohibition to login some users > 2 Just a logging of facts of user's login > > Suggested patch proposes prohibition of login only by failing of login trigger > and it has at least two issues: > 1 In case of prohibition to login, there is no clean way to store information > about unsuccessful login. Ok, it could be solved by dblink module but it seems > to scary. It's an area that could be improved, but the patch is more intended to allow additional actions on successful login, as described by the following (taken from the doc updates in the patch): + <para> + The event trigger on the <literal>login</literal> event can be + useful for logging user logins, for verifying the connection and + assigning roles according to current circumstances, or for some session data + initialization. + </para> > 2 For logging purpose failing of trigger will cause impossibility to login, it > could be workarounded by catching error in trigger function, but it's not so > obvious for DBA. > If you look back on the past discussions on this thread, you'll see that originally the patch added a GUC for the purpose of allowing superusers to disable the event trigger, to allow login in this case. However it was argued that there was already a documented way of bypassing buggy event triggers (i.e. restart in single-user mode), so that GUC was removed. Also previously in the patch, login trigger errors for superusers were ignored in order to allow them to login in this case, but it was argued that it could well be unsafe to continue on after an error, so that too was removed. See below: > > + {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS, > + gettext_noop("Enables the client_connection event trigger."), > + gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, " > ..and.. > + /* > + * Try to ignore error for superuser to make it possible to login even in case of errors > + * during trigger execution > + */ > + if (!is_superuser) > + PG_RE_THROW(); > This patch adds two ways for superusers to bypass this event trigger in case of > it being faulty, but for every other event trigger we've documented to restart > in single-user mode and fixing it there. Why does this need to be different? > This clearly has a bigger chance of being a footgun but I don't see that as a > reason to add a GUC and a bypass that other footguns lack. > So I really don't think that a failing event trigger will cause an "impossibility to login". The patch updates the documentation to explain the use of single-user mode to fix buggy login event triggers. See below: + <para> + The <literal>login</literal> event occurs when a user logs in to the + system. + Any bugs in a trigger procedure for this event may prevent successful + login to the system. Such bugs may be fixed after first restarting the + system in single-user mode (as event triggers are disabled in this mode). + See the <xref linkend="app-postgres"/> reference page for details about + using single-user mode. + </para> + Regards, Greg Nancarrow Fujitsu Australia
> On 30 Sep 2021, at 04:15, Greg Nancarrow <gregn4422@gmail.com> wrote: > > On Wed, Sep 29, 2021 at 10:14 PM Teodor Sigaev <teodor@sigaev.ru> wrote: >> >> Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK, >> there is two use cases for this feature: >> 1 A permission / prohibition to login some users >> 2 Just a logging of facts of user's login >> >> Suggested patch proposes prohibition of login only by failing of login trigger >> and it has at least two issues: >> 1 In case of prohibition to login, there is no clean way to store information >> about unsuccessful login. Ok, it could be solved by dblink module but it seems >> to scary. > > It's an area that could be improved, but the patch is more intended to > allow additional actions on successful login, as described by the > following (taken from the doc updates in the patch): > > + <para> > + The event trigger on the <literal>login</literal> event can be > + useful for logging user logins, for verifying the connection and > + assigning roles according to current circumstances, or for some > session data > + initialization. > + </para> Running user code with potential side effects on unsuccessful logins also open up the risk for (D)DoS attacks. -- Daniel Gustafsson https://vmware.com/
On Wed, Sep 15, 2021 at 11:32 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > I've attached the updated patch. > Attached a rebased version of the patch, as it was broken by fairly recent changes (only very minor change to the previous version). Regards, Greg Nancarrow Fujitsu Australia
Вложения
Среда, 29 сентября 2021, 15:14 +03:00 от Teodor Sigaev <teodor@sigaev.ru>:
Hi!
Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK,
there is two use cases for this feature:
1 A permission / prohibition to login some users
2 Just a logging of facts of user's login
Suggested patch proposes prohibition of login only by failing of login trigger
and it has at least two issues:
1 In case of prohibition to login, there is no clean way to store information
about unsuccessful login. Ok, it could be solved by dblink module but it seems
to scary.
- logging inside the database by db_link or through background worker (like described in https://dpavlin.wordpress.com/2017/05/09/david-rader-autonomous-transactions-using-pg_background/ )
2 For logging purpose failing of trigger will cause impossibility to login, it
could be workarounded by catching error in trigger function, but it's not so
obvious for DBA.
some other issues:
3 It's easy to create security hole, see attachment where non-privileged user
can close access to database even for superuser.
4 Feature is not applicable for logging unsuccessful login with wrong
username/password or not matched by pg_hba.conf. Oracle could operate with such
cases. But I don't think that this issue is a stopper for the patch.
May be, to solve that issues we could introduce return value of trigger and/or
add something like IGNORE ERROR to CREATE EVENT TRIGGER command.
A new version of the patch is attached. It applies to the current master and contains the above mentioned GUC code, relevant tests and docs.
On 15.09.2021 16:32, Greg Nancarrow wrote:> On Wed, Sep 8, 2021 at 10:56 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>>
>> I took a look at this, and while I like the proposed feature I think the patch
>> has a bit more work required.
>>
>
> Thanks for reviewing the patch.
> I am not the original patch author (who no longer seems active) but
> I've been contributing a bit and keeping the patch alive because I
> think it's a worthwhile feature.
>
>>
>> +
>> +-- 2) Initialize some user session data
>> + CREATE TEMP TABLE session_storage (x float, y integer);
>> The example in the documentation use a mix of indentations, neither of which is
>> the 2-space indentation used elsewhere in the docs.
>>
>
> Fixed, using 2-space indentation.
> (to be honest, the indentation seems inconsistent elsewhere in the
> docs e.g. I'm seeing a nearby case of 2-space on 1st indent, 6-space
> on 2nd indent)
>
>>
>> + /* Get the command tag. */
>> + tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT;
>> This is hardcoding knowledge that currently only this trigger wont have a
>> parsetree, and setting the tag accordingly. This should instead check the
>> event and set CMDTAG_UNKNOWN if it isn't the expected one.
>>
>
> I updated that, but maybe not exactly how you expected?
>
>>
>> + /* database has on-login triggers */
>> + bool dathaslogontriggers;
>> This patch uses three different names for the same thing: client connection,
>> logontrigger and login trigger. Since this trigger only fires after successful
>> authentication it’s not accurate to name it client connection, as that implies
>> it running on connections rather than logins. The nomenclature used in the
>> tree is "login" so the patch should be adjusted everywhere to use that.
>>
>
> Fixed.
>
>>
>> +/* Hook for plugins to get control at start of session */
>> +client_connection_hook_type client_connection_hook = EventTriggerOnConnect;
>> I don't see the reason for adding core functionality by hooks. Having a hook
>> might be a useful thing on its own (to be discussed in a new thread, not hidden
>> here), but core functionality should not depend on it IMO.
>>
>
> Fair enough, I removed the hook.
>
>>
>> + {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
>> + gettext_noop("Enables the client_connection event trigger."),
>> + gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
>> ..and..
>> + /*
>> + * Try to ignore error for superuser to make it possible to login even in case of errors
>> + * during trigger execution
>> + */
>> + if (!is_superuser)
>> + PG_RE_THROW();
>> This patch adds two ways for superusers to bypass this event trigger in case of
>> it being faulty, but for every other event trigger we've documented to restart
>> in single-user mode and fixing it there. Why does this need to be different?
>> This clearly has a bigger chance of being a footgun but I don't see that as a
>> reason to add a GUC and a bypass that other footguns lack.
>>
>
> OK, I removed those bypasses. We'll see what others think.
>
>>
>> + elog(NOTICE, "client_connection trigger failed with message: %s", error->message);
>> Calling elog() in a PG_CATCH() block isn't allowed is it?
>>
>
> I believe it is allowed (e.g. there's a case in libpq), but I removed
> this anyway as part of the superuser bypass.
>
>>
>> + /* Runtlist is empty: clear dathaslogontriggers flag
>> + */
>> s/Runtlist/Runlist/ and also commenting style.
>>
>
> Fixed.
>
>>
>> The logic for resetting the pg_database flag when firing event trigger in case
>> the trigger has gone away is messy and a problem waiting to happen IMO. For
>> normal triggers we don't bother with that on the grounds of it being racy, and
>> instead perform relhastrigger removal during VACUUM. Another approach is using
>> a counter as propose upthread, since updating that can be made safer. The
>> current coding also isn't instructing concurrent backends to perform relcache
>> rebuild.
>>
>
> I think there are pros and cons of each possible approach, but I think
> I prefer the current way (which I have tweaked a bit) for similar
> reasons to those explained by the original patch author when debating
> whether to use reference-counting instead, in the discussion upthread
> (e.g. it keeps it all in one place). Also, it seems to be more inline
> with the documented reason why that pg_database flag was added in the
> first place. I have debugged a few concurrent scenarios with the
> current mechanism in place. If you still dislike the logic for
> resetting the flag, please elaborate on the issues you foresee and one
> of the alternative approaches can be tried.
>
> I've attached the updated patch.
>
> Regards,
> Greg Nancarrow
> Fujitsu Australia
>
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
Вложения
> On 3 Nov 2021, at 17:15, Ivan Panchenko <wao@mail.ru> wrote: > Среда, 29 сентября 2021, 15:14 +03:00 от Teodor Sigaev <teodor@sigaev.ru <x-msg://33/compose?To=teodor@sigaev.ru>>: > 2 For logging purpose failing of trigger will cause impossibility to login, it > could be workarounded by catching error in trigger function, but it's not so > obvious for DBA. > If the trigger contains an error, nobody can login. The database is bricked. > Previous variant of the patch proposed to fix this with going to single-user mode. > For faster recovery I propose to have also a GUC variable to turn on/off the login triggers. > It should be 'on' by default. As voiced earlier, I disagree with this and I dislike the idea of punching a hole for circumventing infrastructure intended for auditing. Use-cases for a login-trigger commonly involve audit trail logging, session initialization etc. If the login trigger bricks the production database to the extent that it needs to be restarted with the magic GUC, then it's highly likely that you *don't* want regular connections to the database for the duration of this. Any such connection won't be subject to what the trigger does which seem counter to having the trigger in the first place. This means that it's likely that the superuser fixing it will have to disable logins for everyone else while fixing, and it quicly becomes messy. With that in mind, I think single-user mode actually *helps* the users in this case, and we avoid a hole punched which in worst case can be a vector for an attack. Maybe I'm overly paranoid, but adding a backdoor of sorts for a situation which really shouldn't happen doesn't seem like a good idea. > some other issues: > 3 It's easy to create security hole, see attachment where non-privileged user > can close access to database even for superuser. > Such cases can be avoided by careful design of the login triggers and related permissions. Added such note to the documentation. If all login triggers are written carefully like that, we don't need the GUC to disable them? -- Daniel Gustafsson https://vmware.com/
> On 3 Nov 2021, at 17:15, Ivan Panchenko <wao@mail.ru> wrote:
> Среда, 29 сентября 2021, 15:14 +03:00 от Teodor Sigaev <teodor@sigaev.ru <x-msg://33/compose?To=teodor@sigaev.ru>>:
> 2 For logging purpose failing of trigger will cause impossibility to login, it
> could be workarounded by catching error in trigger function, but it's not so
> obvious for DBA.
> If the trigger contains an error, nobody can login. The database is bricked.
> Previous variant of the patch proposed to fix this with going to single-user mode.
> For faster recovery I propose to have also a GUC variable to turn on/off the login triggers.
> It should be 'on' by default.
As voiced earlier, I disagree with this and I dislike the idea of punching a
hole for circumventing infrastructure intended for auditing.
Use-cases for a login-trigger commonly involve audit trail logging, session
initialization etc. If the login trigger bricks the production database to the
extent that it needs to be restarted with the magic GUC, then it's highly
likely that you *don't* want regular connections to the database for the
duration of this. Any such connection won't be subject to what the trigger
does which seem counter to having the trigger in the first place. This means
that it's likely that the superuser fixing it will have to disable logins for
everyone else while fixing, and it quicly becomes messy.
With that in mind, I think single-user mode actually *helps* the users in this
case, and we avoid a hole punched which in worst case can be a vector for an
attack.
Maybe I'm overly paranoid, but adding a backdoor of sorts for a situation which
really shouldn't happen doesn't seem like a good idea.
On Fri, Nov 5, 2021 at 3:03 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > +1 > The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good idea,and are not needed. > If there are no objections, I plan to reinstate the previous v19 patch (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL capitalization) in the tests, as hinted at in the v20 patch, but no new functionality. Regards, Greg Nancarrow Fujitsu Australia
> On 10 Nov 2021, at 08:12, Greg Nancarrow <gregn4422@gmail.com> wrote: > > On Fri, Nov 5, 2021 at 3:03 PM Greg Nancarrow <gregn4422@gmail.com> wrote: >> >> +1 >> The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good idea,and are not needed. > > If there are no objections, I plan to reinstate the previous v19 patch > (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL > capitalization) in the tests, as hinted at in the v20 patch, but no > new functionality. No objections from me. Small nitpicks from the v19 patch: + This flag is used internally by Postgres and should not be manually changed by DBA or application. This should be <productname>PostgreSQL</productname>. + * There can be a race condition: a login event trigger may have .. + /* Fire any defined login triggers, if appropriate */ The patch say "login trigger" in most places, and "login event trigger" in a few places. We should settle for a single nomenclature, and I think "login event trigger" is the best option. -- Daniel Gustafsson https://vmware.com/
> On 10 Nov 2021, at 08:12, Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> On Fri, Nov 5, 2021 at 3:03 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>>
>> +1
>> The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good idea, and are not needed.
>
> If there are no objections, I plan to reinstate the previous v19 patch
> (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL
> capitalization) in the tests, as hinted at in the v20 patch, but no
> new functionality.
No objections from me. Small nitpicks from the v19 patch:
+ This flag is used internally by Postgres and should not be manually changed by DBA or application.
This should be <productname>PostgreSQL</productname>.
+ * There can be a race condition: a login event trigger may have
..
+ /* Fire any defined login triggers, if appropriate */
The patch say "login trigger" in most places, and "login event trigger" in a
few places. We should settle for a single nomenclature, and I think "login
event trigger" is the best option.
--
Daniel Gustafsson https://vmware.com/
On Wed, Nov 10, 2021 at 8:11 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > > If there are no objections, I plan to reinstate the previous v19 patch > > (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL > > capitalization) in the tests, as hinted at in the v20 patch, but no > > new functionality. > > No objections from me. Small nitpicks from the v19 patch: > > + This flag is used internally by Postgres and should not be manually changed by DBA or application. > This should be <productname>PostgreSQL</productname>. > > + * There can be a race condition: a login event trigger may have > .. > + /* Fire any defined login triggers, if appropriate */ > The patch say "login trigger" in most places, and "login event trigger" in a > few places. We should settle for a single nomenclature, and I think "login > event trigger" is the best option. > I've attached an updated patch, that essentially reinstates the v19 patch, but with a few improvements such as: - Updates to address nitpicks (Daniel Gustafsson) - dathaslogintriggers -> dathasloginevttriggers flag rename (too long?) and remove its restoration in pg_dump output, since it's not needed (as in v20 patch) - Some tidying of the updates to the event_trigger tests and capitalization of the test SQL Regards, Greg Nancarrow Fujitsu Australia
Вложения
> On 11 Nov 2021, at 07:37, Greg Nancarrow <gregn4422@gmail.com> wrote: > I've attached an updated patch, that essentially reinstates the v19 > patch Thanks! I've only skimmed it so far but it looks good, will do a more thorough review soon. + This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup. This should be <structname>pg_event_trigger</structname>. Sorry, missed that one at that last read-through. > - dathaslogintriggers -> dathasloginevttriggers flag rename (too > long?) I'm not crazy about this name, "evt" is commonly the abbreviation of "event trigger" (used in evtcache.c etc) so "dathasloginevt" would IMO be better. That being said, that's still not a very readable name, maybe someone else has an even better suggestion? -- Daniel Gustafsson https://vmware.com/
On Thu, Nov 11, 2021 at 8:56 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > + This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup. > This should be <structname>pg_event_trigger</structname>. Sorry, missed that > one at that last read-through. > Thanks, noted. > > - dathaslogintriggers -> dathasloginevttriggers flag rename (too > > long?) > > I'm not crazy about this name, "evt" is commonly the abbreviation of "event > trigger" (used in evtcache.c etc) so "dathasloginevt" would IMO be better. > That being said, that's still not a very readable name, maybe someone else has > an even better suggestion? > Yes you're right, in this case I was wrongly treating "evt" as an abbreviation for "event". I agree "dathasloginevt" would be better. Regards, Greg Nancarrow Fujitsu Australia
On Thu, Nov 11, 2021 at 8:56 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > + This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup. > This should be <structname>pg_event_trigger</structname>. Sorry, missed that > one at that last read-through. > Fixed. > > - dathaslogintriggers -> dathasloginevttriggers flag rename (too > > long?) > > I'm not crazy about this name, "evt" is commonly the abbreviation of "event > trigger" (used in evtcache.c etc) so "dathasloginevt" would IMO be better. > That being said, that's still not a very readable name, maybe someone else has > an even better suggestion? > Changed to "dathasloginevt", as suggested. I've attached an updated patch with these changes. I also noticed one of the Windows-based cfbots was failing with an "SSPI authentication failed for user" error, so I updated the test code for that. Regards, Greg Nancarrow Fujitsu Australia
Вложения
On Tue, Nov 16, 2021 at 4:38 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > I've attached an updated patch with these changes. > I've attached a re-based version (no functional changes from the previous) to fix cfbot failures. Regards, Greg Nancarrow Fujitsu Australia
Вложения
On Mon, Dec 6, 2021 at 12:10 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > I've attached a re-based version (no functional changes from the > previous) to fix cfbot failures. > I've attached a re-based version (no functional changes from the previous) to fix cfbot failures. Regards, Greg Nancarrow Fujitsu Australia
Вложения
On Mon, Jan 24, 2022 at 1:59 PM Greg Nancarrow <gregn4422@gmail.com> wrote: > > I've attached a re-based version (no functional changes from the > previous) to fix cfbot failures. > I've attached a re-based version (no functional changes from the previous) to fix cfbot failures. Regards, Greg Nancarrow Fujitsu Australia
Вложения
> On 15 Feb 2022, at 11:07, Greg Nancarrow <gregn4422@gmail.com> wrote: > I've attached a re-based version (no functional changes from the > previous) to fix cfbot failures. Thanks for adopting this patch, I took another look at it today and have some comments. + This flag is used internally by <productname>PostgreSQL</productname> and should not be manually changed by DBAor application. I think we should reword this to something like "This flag is used internally by <productname>PostgreSQL</productname> and should not be altered or relied upon for monitoring". We really don't want anyone touching or interpreting this value since there is no guarantee that it will be accurate. + new_record[Anum_pg_database_dathasloginevt - 1] = BoolGetDatum(datform->dathasloginevt); Since the corresponding new_record_repl valus is false, this won't do anything and can be removed. We will use the old value anyways. + if (strcmp(eventname, "login") == 0) I think this block warrants a comment on why it only applies to login triggers. + db = (Form_pg_database) GETSTRUCT(tuple); + if (!db->dathasloginevt) + { + db->dathasloginevt = true; + CatalogTupleUpdate(pg_db, &tuple->t_self, tuple); + } + else + CacheInvalidateRelcacheByTuple(tuple); This needs a CommandCounterIncrement inside the if () { .. } block right? + /* Get the command tag. */ + tag = (event == EVT_Login) ? CMDTAG_LOGIN : CreateCommandTag(parsetree); To match project style I think this should be an if-then-else block. Also, while it's tempting to move this before the assertion block in order to reuse it there, it does mean that we are calling this in a hot codepath before we know if it's required. I think we should restore the previous programming with a separate CreateCommandTag call inside the assertion block and move this one back underneath the fast-path exit. + CacheInvalidateRelcacheByTuple(tuple); + } + table_close(pg_db, RowExclusiveLock); + heap_freetuple(tuple); + } + } + CommitTransactionCommand(); Since we are commiting the transaction just after closing the table, is the relcache invalidation going to achieve much? I guess registering the event doesn't hurt much? + /* + * There can be a race condition: a login event trigger may + * have been added after the pg_event_trigger table was + * scanned, and we don't want to erroneously clear the + * dathasloginevt flag in this case. To be sure that this + * hasn't happened, repeat the scan under the pg_database + * table lock. + */ + AcceptInvalidationMessages(); + runlist = EventTriggerCommonSetup(NULL, + EVT_Login, "login", + &trigdata); It seems conceptually wrong to allocate this in the TopTransactionContext when we only generate the runlist to throw it away. At the very least I think we should free the returned list. + if (runlist == NULL) /* list is still empty, so clear the This should check for NIL and not NULL. Have you done any benchmarking on this patch? With this version I see a small slowdown on connection time of ~1.5% using pgbench for the case where there are no login event triggers defined. With a login event trigger calling an empty plpgsql function there is a ~30% slowdown (corresponding to ~1ms on my system). When there is a login event trigger defined all bets are off however, since the function called can be arbitrarily complex and it's up to the user to measure and decide - but the bare overhead we impose is still of interest of course. -- Daniel Gustafsson https://vmware.com/
On Tue, Feb 15, 2022 at 5:07 AM Greg Nancarrow <gregn4422@gmail.com> wrote: > I've attached a re-based version (no functional changes from the > previous) to fix cfbot failures. I tried this: rhaas=# create function on_login_proc() returns event_trigger as $$begin perform pg_sleep(10000000); end$$ language plpgsql; CREATE FUNCTION rhaas=# create event trigger on_login_trigger on login execute procedure on_login_proc(); When I then attempt to connect via psql, it hangs, as expected. When I press ^C, psql exits, but the backend process is not cancelled and just keeps chugging along in the background. The good news is that if I connect to another database, I can cancel all of the hung sessions using pg_cancel_backend(), and all of those processes then promptly exit, and presumably I could accomplish the same thing by sending them SIGINT directly. But it's still not great behavior. It would be easy to use up a pile of connection slots inadvertently and have to go to some trouble to get access to the server again. Since this is a psql behavior and not a server behavior, one could argue that it's unrelated to this patch, but in practice this patch seems to increase the chances of people running into problems quite a lot. -- Robert Haas EDB: http://www.enterprisedb.com
Hi, On 2022-02-15 21:07:15 +1100, Greg Nancarrow wrote: > Subject: [PATCH v25] Add a new "login" event and login event trigger support. > > The login event occurs when a user logs in to the system. I think this needs a HUGE warning in the docs about most event triggers needing to check pg_is_in_recovery() or breaking hot standby. And certainly the example given needs to include an pg_is_in_recovery() check. > + <para> > + The <literal>login</literal> event occurs when a user logs in to the > + system. > + Any bugs in a trigger procedure for this event may prevent successful > + login to the system. Such bugs may be fixed after first restarting the > + system in single-user mode (as event triggers are disabled in this mode). > + See the <xref linkend="app-postgres"/> reference page for details about > + using single-user mode. > + </para> I'm strongly against adding any new dependencies on single user mode. A saner approach might be a superuser-only GUC that can be set as part of the connection data (e.g. PGOPTIONS='-c ignore_login_event_trigger=true'). > @@ -293,6 +297,27 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO > CatalogTupleInsert(tgrel, tuple); > heap_freetuple(tuple); > > + if (strcmp(eventname, "login") == 0) > + { > + Form_pg_database db; > + Relation pg_db = table_open(DatabaseRelationId, RowExclusiveLock); > + > + /* Set dathasloginevt flag in pg_database */ > + tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); > + if (!HeapTupleIsValid(tuple)) > + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); > + db = (Form_pg_database) GETSTRUCT(tuple); > + if (!db->dathasloginevt) > + { > + db->dathasloginevt = true; > + CatalogTupleUpdate(pg_db, &tuple->t_self, tuple); > + } > + else > + CacheInvalidateRelcacheByTuple(tuple); > + table_close(pg_db, RowExclusiveLock); > + heap_freetuple(tuple); > + } > + > /* Depend on owner. */ > recordDependencyOnOwner(EventTriggerRelationId, trigoid, evtOwner); Maybe I am confused, but isn't that CacheInvalidateRelcacheByTuple call *entirely* bogus? CacheInvalidateRelcacheByTuple() expects a pg_class tuple, but you're passing in a pg_database tuple? And what is relcache invalidation even supposed to achieve here? I think this should mention that ->dathasloginevt is unset on login when appropriate. > +/* > + * Fire login event triggers. > + */ > +void > +EventTriggerOnLogin(void) > +{ > + List *runlist; > + EventTriggerData trigdata; > + > + /* > + * See EventTriggerDDLCommandStart for a discussion about why event > + * triggers are disabled in single user mode. > + */ > + if (!IsUnderPostmaster || !OidIsValid(MyDatabaseId)) > + return; > + > + StartTransactionCommand(); > + > + if (DatabaseHasLoginEventTriggers()) > + { > + runlist = EventTriggerCommonSetup(NULL, > + EVT_Login, "login", > + &trigdata); > + > + if (runlist != NIL) > + { > + /* > + * Make sure anything the main command did will be visible to the > + * event triggers. > + */ > + CommandCounterIncrement(); "Main command"? It's not clear to my why a CommandCounterIncrement() could be needed here - which previous writes do you need to make visible? > + /* > + * Runlist is empty: clear dathasloginevt flag > + */ > + Relation pg_db = table_open(DatabaseRelationId, RowExclusiveLock); > + HeapTuple tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); > + Form_pg_database db; > + > + if (!HeapTupleIsValid(tuple)) > + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); > + > + db = (Form_pg_database) GETSTRUCT(tuple); > + if (db->dathasloginevt) > + { > + /* > + * There can be a race condition: a login event trigger may > + * have been added after the pg_event_trigger table was > + * scanned, and we don't want to erroneously clear the > + * dathasloginevt flag in this case. To be sure that this > + * hasn't happened, repeat the scan under the pg_database > + * table lock. > + */ > + AcceptInvalidationMessages(); > + runlist = EventTriggerCommonSetup(NULL, > + EVT_Login, "login", > + &trigdata); This doesn't work. RowExclusiveLock doesn't conflict with another RowExclusiveLock. What is the AcceptInvalidationMessages() intending to do here? > + if (runlist == NULL) /* list is still empty, so clear the > + * flag */ > + { > + db->dathasloginevt = false; > + CatalogTupleUpdate(pg_db, &tuple->t_self, tuple); > + } > + else > + CacheInvalidateRelcacheByTuple(tuple); Copy of the bogus relcache stuff. Greetings, Andres Freund
> On 12 Mar 2022, at 03:46, Andres Freund <andres@anarazel.de> wrote: >> + <para> >> + The <literal>login</literal> event occurs when a user logs in to the >> + system. >> + Any bugs in a trigger procedure for this event may prevent successful >> + login to the system. Such bugs may be fixed after first restarting the >> + system in single-user mode (as event triggers are disabled in this mode). >> + See the <xref linkend="app-postgres"/> reference page for details about >> + using single-user mode. >> + </para> > > I'm strongly against adding any new dependencies on single user mode. > > A saner approach might be a superuser-only GUC that can be set as part of the > connection data (e.g. PGOPTIONS='-c ignore_login_event_trigger=true'). This, and similar approaches, has been discussed in this thread. I'm not a fan of holes punched with GUC's like this, but if you plan on removing single-user mode (as I recall seeing in an initdb thread) altogether then that kills the discussion. So. Since we already recommend single-user mode to handle broken event triggers, we should IMO do something to cover all of them rather than risk ending up with one disabling GUC per each event type. Right now we have this on the CREATE EVENT TRIGGER reference page: "Event triggers are disabled in single-user mode (see postgres). If an erroneous event trigger disables the database so much that you can't even drop the trigger, restart in single-user mode and you'll be able to do that." Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps? -- Daniel Gustafsson https://vmware.com/
Hi, On 2022-03-13 23:31:03 +0100, Daniel Gustafsson wrote: > > On 12 Mar 2022, at 03:46, Andres Freund <andres@anarazel.de> wrote: > > >> + <para> > >> + The <literal>login</literal> event occurs when a user logs in to the > >> + system. > >> + Any bugs in a trigger procedure for this event may prevent successful > >> + login to the system. Such bugs may be fixed after first restarting the > >> + system in single-user mode (as event triggers are disabled in this mode). > >> + See the <xref linkend="app-postgres"/> reference page for details about > >> + using single-user mode. > >> + </para> > > > > I'm strongly against adding any new dependencies on single user mode. > > > > A saner approach might be a superuser-only GUC that can be set as part of the > > connection data (e.g. PGOPTIONS='-c ignore_login_event_trigger=true'). > > This, and similar approaches, has been discussed in this thread. I'm not a fan > of holes punched with GUC's like this, but if you plan on removing single-user > mode (as I recall seeing in an initdb thread) altogether then that kills the > discussion. So. Even if we end up not removing single user mode, I think it's not OK to add new failure modes that require single user mode to resolve after not-absurd operations (I'm ok with needing single user mode if somebody does delete from pg_authid). It's too hard to reach for most. We already have GUCs like row_security, so it doesn't seem insane to add one that disables login event triggers. What is the danger that you see? > Since we already recommend single-user mode to handle broken event triggers, we > should IMO do something to cover all of them rather than risk ending up with > one disabling GUC per each event type. Right now we have this on the CREATE > EVENT TRIGGER reference page: IMO the other types of event triggers make it a heck of a lot harder to get yourself into a situation that you can't get out of... > "Event triggers are disabled in single-user mode (see postgres). If an > erroneous event trigger disables the database so much that you can't even > drop the trigger, restart in single-user mode and you'll be able to do > that." > Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps? Did you mean login instead of event? Something like it would work for me. It probably should be GUC_DISALLOW_IN_FILE? Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes: > On 2022-03-13 23:31:03 +0100, Daniel Gustafsson wrote: >> Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps? > Did you mean login instead of event? > Something like it would work for me. It probably should be > GUC_DISALLOW_IN_FILE? Why? Inserting such a setting in postgresql.conf and restarting would be the first thing I'd think of if I needed to get out of a problem. The only other way is to set it on the postmaster command line, which is going to be awkward-to-impossible with most system-provided start scripts. regards, tom lane
Hi, On 2022-03-13 19:57:08 -0400, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > On 2022-03-13 23:31:03 +0100, Daniel Gustafsson wrote: > >> Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps? > > > Did you mean login instead of event? > > > Something like it would work for me. It probably should be > > GUC_DISALLOW_IN_FILE? > > Why? Inserting such a setting in postgresql.conf and restarting > would be the first thing I'd think of if I needed to get out > of a problem. The only other way is to set it on the postmaster > command line, which is going to be awkward-to-impossible with > most system-provided start scripts. I was thinking that the way to use it would be to specify it as a client option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes: > I was thinking that the way to use it would be to specify it as a client > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. Ugh ... that would allow people (at least superusers) to bypass the login trigger at will, which seems to me to break a lot of the use-cases for the feature. I supposed we'd want this to be a PGC_POSTMASTER setting for security reasons. regards, tom lane
On 2022-03-13 20:35:44 -0400, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > I was thinking that the way to use it would be to specify it as a client > > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. > > Ugh ... that would allow people (at least superusers) to bypass > the login trigger at will, which seems to me to break a lot of > the use-cases for the feature. I supposed we'd want this to be a > PGC_POSTMASTER setting for security reasons. Shrug. This doesn't seem to add actual security to me.
> On 14 Mar 2022, at 00:33, Andres Freund <andres@anarazel.de> wrote: > We already have GUCs like row_security, so it doesn't seem insane to add one > that disables login event triggers. What is the danger that you see? My fear is that GUCs like that end up as permanent fixtures in scripts etc after having been used temporary, and then X timeunits later someone realize that the backup has never actually really worked due to a subtle issue, or something else unpleasant. The row_security GUC is kind of different IMO, as it's required for pg_dump (though it can be used in the same way as the above). -- Daniel Gustafsson https://vmware.com/
On Sun, Mar 13, 2022 at 7:34 PM Andres Freund <andres@anarazel.de> wrote: > IMO the other types of event triggers make it a heck of a lot harder to get > yourself into a situation that you can't get out of... In particular, unless something has changed since I committed this stuff originally, there's no existing type of event trigger than can prevent the superuser from logging in and running DROP EVENT TRIGGER -- or a SELECT on the system catalogs to find out what to drop. That was very much a deliberate decision on my part. I think it's fine to require dropping to single-user mode as a way of recovering from extreme situations where, for example, there are corrupted database files. If we don't need it even then, cool, but if we do, I'm not sad. But all we're talking about here is somebody maybe running a command that perhaps they should not have run. Having to take the whole system down to recover from that seems excessively painful. -- Robert Haas EDB: http://www.enterprisedb.com
> On 14 Mar 2022, at 01:08, Andres Freund <andres@anarazel.de> wrote: > I was thinking that the way to use it would be to specify it as a client > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is the previously posted v25 login event trigger patch, and 0003 adapts is to allow ignoring it (0002/0003 split only for illustrative purposes of course). Is this along the lines of what you were thinking? -- Daniel Gustafsson https://vmware.com/
Вложения
Hi, On 2022-03-14 14:47:09 +0100, Daniel Gustafsson wrote: > > On 14 Mar 2022, at 01:08, Andres Freund <andres@anarazel.de> wrote: > > > I was thinking that the way to use it would be to specify it as a client > > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. > > Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is > the previously posted v25 login event trigger patch, and 0003 adapts is to > allow ignoring it (0002/0003 split only for illustrative purposes of course). > Is this along the lines of what you were thinking? Yes. Of course there's still the bogus cache invalidation stuff etc that I commented on upthread. Greetings, Andres Freund
> On 14 Mar 2022, at 17:10, Andres Freund <andres@anarazel.de> wrote: > > Hi, > > On 2022-03-14 14:47:09 +0100, Daniel Gustafsson wrote: >>> On 14 Mar 2022, at 01:08, Andres Freund <andres@anarazel.de> wrote: >> >>> I was thinking that the way to use it would be to specify it as a client >>> option. Like PGOPTIONS='-c ignore_event_trigger=login' psql. >> >> Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is >> the previously posted v25 login event trigger patch, and 0003 adapts is to >> allow ignoring it (0002/0003 split only for illustrative purposes of course). >> Is this along the lines of what you were thinking? > > Yes. > > Of course there's still the bogus cache invalidation stuff etc that I > commented on upthread. Yeah, that was the previously posted v25 from the author (who adopted it from the original author). I took the liberty to quickly poke at the review comments you had left as well as the ones that I had found to try and progress the patch. 0001 should really go in it's own thread though to not hide it from anyone interested who isn't looking at this thread. -- Daniel Gustafsson https://vmware.com/
Вложения
Daniel Gustafsson писал 2022-03-15 23:52: > Yeah, that was the previously posted v25 from the author (who adopted > it from > the original author). I took the liberty to quickly poke at the review > comments you had left as well as the ones that I had found to try and > progress > the patch. 0001 should really go in it's own thread though to not hide > it from > anyone interested who isn't looking at this thread. Hi I got an error while running tests on Windows: 2022-03-17 17:30:02.458 MSK [6920] [unknown] LOG: no match in usermap "regress" for user "regress_user" authenticated as "vagrant@WINDOWS-2019" 2022-03-17 17:30:02.458 MSK [6920] [unknown] FATAL: SSPI authentication failed for user "regress_user" 2022-03-17 17:30:02.458 MSK [6920] [unknown] DETAIL: Connection matched pg_hba.conf line 2: "host all all 127.0.0.1/32 sspi include_realm=1 map=regress" 2022-03-17 17:30:02.526 MSK [3432] FATAL: could not receive data from WAL stream: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. I propose to make such correction: --- a/src/test/recovery/t/001_stream_rep.pl +++ b/src/test/recovery/t/001_stream_rep.pl @@ -14,7 +14,7 @@ my $node_primary = get_new_node('primary'); # and it needs proper authentication configuration. $node_primary->init( allows_streaming => 1, - auth_extra => [ '--create-role', 'repl_role' ]); + auth_extra => [ '--create-role', 'repl_role,regress_user' ]); $node_primary->start; my $backup_name = 'my_backup'; -- Andrey Sokolov Postgres Professional: http://www.postgrespro.com
> On 18 Mar 2022, at 08:24, a.sokolov@postgrespro.ru wrote: > - auth_extra => [ '--create-role', 'repl_role' ]); > + auth_extra => [ '--create-role', 'repl_role,regress_user' ]); Looking at the test in question it's not entirely clear to me what the original author really intended here, so I've changed it up a bit to actually test something and removed the need for the regress_user role. I've also fixed the silly mistake highlighted in the postgresql.conf.sample test. -- Daniel Gustafsson https://vmware.com/
Вложения
Hi, On 2022-03-18 17:03:39 +0100, Daniel Gustafsson wrote: > > On 18 Mar 2022, at 08:24, a.sokolov@postgrespro.ru wrote: > > > - auth_extra => [ '--create-role', 'repl_role' ]); > > + auth_extra => [ '--create-role', 'repl_role,regress_user' ]); > > Looking at the test in question it's not entirely clear to me what the original > author really intended here, so I've changed it up a bit to actually test > something and removed the need for the regress_user role. I've also fixed the > silly mistake highlighted in the postgresql.conf.sample test. docs fail to build: https://cirrus-ci.com/task/5556234047717376?logs=docs_build#L349 Greetings, Andres Freund
> On 22 Mar 2022, at 04:48, Andres Freund <andres@anarazel.de> wrote: > docs fail to build: https://cirrus-ci.com/task/5556234047717376?logs=docs_build#L349 Ugh, that one was on me and not the original author. Fixed. -- Daniel Gustafsson https://vmware.com/
Вложения
Daniel Gustafsson писал 2022-03-22 11:43: >> On 22 Mar 2022, at 04:48, Andres Freund <andres@anarazel.de> wrote: > >> docs fail to build: >> https://cirrus-ci.com/task/5556234047717376?logs=docs_build#L349 > > Ugh, that one was on me and not the original author. Fixed. > + data initialization. It is vital that any event trigger using the + <literal>login</literal> event checks whether or not the database is in + recovery. Does any trigger really have to contain a pg_is_in_recovery() call? In this message (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de) it was only about triggers on hot standby, which run not read-only queries -- Andrey Sokolov Postgres Professional: http://www.postgrespro.com
Hi, On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote: > + data initialization. It is vital that any event trigger using the > + <literal>login</literal> event checks whether or not the database is in > + recovery. > > Does any trigger really have to contain a pg_is_in_recovery() call? Not *any* trigger, just any trigger that writes. > In this message > (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de) > it was only about triggers on hot standby, which run not read-only queries The problem precisely is that the login triggers run on hot standby nodes, and that if they do writes, you can't login anymore. Greetings, Andres Freund
> On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote: > On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote: >> + data initialization. It is vital that any event trigger using the >> + <literal>login</literal> event checks whether or not the database is in >> + recovery. >> >> Does any trigger really have to contain a pg_is_in_recovery() call? > > Not *any* trigger, just any trigger that writes. Thats correct, the docs should be updated with something like the below I reckon. It is vital that event trigger using the <literal>login</literal> event which has side-effects checks whether or not the database is in recovery to ensure they are not performing modifications to hot standby nodes. >> In this message >> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de) >> it was only about triggers on hot standby, which run not read-only queries > > The problem precisely is that the login triggers run on hot standby nodes, and > that if they do writes, you can't login anymore. Do you think this potential foot-gun is scary enough to reject this patch? There are lots of creative ways to cause Nagios alerts from ones database, but this has the potential to do so with a small bug in userland code. Still, I kind of like the feature so I'm indecisive. -- Daniel Gustafsson https://vmware.com/
Hi, On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote: > > On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote: > > On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote: > > >> + data initialization. It is vital that any event trigger using the > >> + <literal>login</literal> event checks whether or not the database is in > >> + recovery. > >> > >> Does any trigger really have to contain a pg_is_in_recovery() call? > > > > Not *any* trigger, just any trigger that writes. > > Thats correct, the docs should be updated with something like the below I > reckon. > > It is vital that event trigger using the <literal>login</literal> event > which has side-effects checks whether or not the database is in recovery to > ensure they are not performing modifications to hot standby nodes. Maybe side-effects is a bit too general? Emitting a log message, rejecting a login, setting some GUCs, etc are all side-effects too. > >> In this message > >> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de) > >> it was only about triggers on hot standby, which run not read-only queries > > > > The problem precisely is that the login triggers run on hot standby nodes, and > > that if they do writes, you can't login anymore. > > Do you think this potential foot-gun is scary enough to reject this patch? > There are lots of creative ways to cause Nagios alerts from ones database, but > this has the potential to do so with a small bug in userland code. Still, I > kind of like the feature so I'm indecisive. It does seem like a huge footgun. But also kinda useful. So I'm really +-0. Greetings, Andres Freund
> On 28 Mar 2022, at 23:31, Andres Freund <andres@anarazel.de> wrote: > > Hi, > > On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote: >>> On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote: >>> On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote: >> >>>> + data initialization. It is vital that any event trigger using the >>>> + <literal>login</literal> event checks whether or not the database is in >>>> + recovery. >>>> >>>> Does any trigger really have to contain a pg_is_in_recovery() call? >>> >>> Not *any* trigger, just any trigger that writes. >> >> Thats correct, the docs should be updated with something like the below I >> reckon. >> >> It is vital that event trigger using the <literal>login</literal> event >> which has side-effects checks whether or not the database is in recovery to >> ensure they are not performing modifications to hot standby nodes. > > Maybe side-effects is a bit too general? Emitting a log message, rejecting a > login, setting some GUCs, etc are all side-effects too. Good point, it needs to say that modifications that cause WAL to be generated are prohibited, but in a more user-friendly readable way. Perhaps in a big red warning box. >>>> In this message >>>> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de) >>>> it was only about triggers on hot standby, which run not read-only queries >>> >>> The problem precisely is that the login triggers run on hot standby nodes, and >>> that if they do writes, you can't login anymore. >> >> Do you think this potential foot-gun is scary enough to reject this patch? >> There are lots of creative ways to cause Nagios alerts from ones database, but >> this has the potential to do so with a small bug in userland code. Still, I >> kind of like the feature so I'm indecisive. > > It does seem like a huge footgun. But also kinda useful. So I'm really +-0. Looks like we are in agreement here. I'm going to go over it again and sleep on it some more before the deadline hits. -- Daniel Gustafsson https://vmware.com/
Andres Freund <andres@anarazel.de> writes: > On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote: >> Do you think this potential foot-gun is scary enough to reject this patch? >> There are lots of creative ways to cause Nagios alerts from ones database, but >> this has the potential to do so with a small bug in userland code. Still, I >> kind of like the feature so I'm indecisive. > It does seem like a huge footgun. But also kinda useful. So I'm really +-0. An on-login trigger is *necessarily* a foot-gun; I don't see that this particular failure mode makes it any worse than it would be anyway. There has to be some not-too-difficult-to-use way to bypass a broken login trigger. Assuming we are happy with the design for doing that, might as well accept the hazards. regards, tom lane
Tue, March 29, 2022, 0:31 +03:00 from Andres Freund <andres@anarazel.de>:
Hi,
On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote:
> > On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote:
> > On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote:
>
> >> + data initialization. It is vital that any event trigger using the
> >> + <literal>login</literal> event checks whether or not the database is in
> >> + recovery.
> »
> >> Does any trigger really have to contain a pg_is_in_recovery() call?
> >
> > Not *any* trigger, just any trigger that writes.
>
> Thats correct, the docs should be updated with something like the below I
> reckon.
>
> It is vital that event trigger using the <literal>login</literal> event
> which has side-effects checks whether or not the database is in recovery to
> ensure they are not performing modifications to hot standby nodes.
Maybe side-effects is a bit too general? Emitting a log message, rejecting a
login, setting some GUCs, etc are all side-effects too.
This can be achieved by checking <function>pg_is_in_recovery</function>(), see an example below.
- single-user mode and you'll be able to do that. Even triggers can also be + single-user mode and you'll be able to do that. Event triggers can also be
It does not do anything if run on a standby. To show that it can do something on a standby to, I propose to move throwing the night exception to the beginning.
So it will be:
CREATE OR REPLACE FUNCTION init_session() RETURNS event_trigger SECURITY DEFINER LANGUAGE plpgsql AS $$ DECLARE hour integer = EXTRACT('hour' FROM current_time); rec boolean; BEGIN -- 1) Forbid logging in late: IF hour BETWEEN 2 AND 4 THEN RAISE EXCEPTION 'Login forbidden'; -- do not allow to login these hours END IF; -- The remaining stuff cannot be done on standbys, -- so ensure the database is not in recovery SELECT pg_is_in_recovery() INTO rec; IF rec THEN RETURN; END IF -- 2) Assign some roles IF hour BETWEEN 8 AND 20 THEN -- at daytime grant the day_worker role EXECUTE 'REVOKE night_worker FROM ' || quote_ident(session_user); EXECUTE 'GRANT day_worker TO ' || quote_ident(session_user); ELSE -- at other time grant the night_worker role EXECUTE 'REVOKE day_worker FROM ' || quote_ident(session_user); EXECUTE 'GRANT night_worker TO ' || quote_ident(session_user); END IF; -- 3) Initialize some user session data CREATE TEMP TABLE session_storage (x float, y integer); -- 4) Log the connection time INSERT INTO user_login_log VALUES (session_user, current_timestamp); END; $$;
\c SELECT dathasloginevt FROM pg_database WHERE datname= :'DBNAME';
dathasloginevt ---------------- f (1 row)
> >> In this message
> >> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de)
> >> it was only about triggers on hot standby, which run not read-only queries
> >
> > The problem precisely is that the login triggers run on hot standby nodes, and
> > that if they do writes, you can't login anymore.
>
> Do you think this potential foot-gun is scary enough to reject this patch?
> There are lots of creative ways to cause Nagios alerts from ones database, but
> this has the potential to do so with a small bug in userland code. Still, I
> kind of like the feature so I'm indecisive.
It does seem like a huge footgun. But also kinda useful. So I'm really +-0.
Greetings,
Andres Freund
> On 30 Mar 2022, at 13:21, Ivan Panchenko <wao@mail.ru> wrote: > Maybe side-effects is a bit too general? Emitting a log message, rejecting a > login, setting some GUCs, etc are all side-effects too. > Something like this: I've reworded the docs close to what you suggested here. > Also, please fix a typo in doc/src/sgml/ref/create_event_trigger.sgml : Done. > Regarding the trigger function example: > It does not do anything if run on a standby. To show that it can do something on a standby to, I propose to move throwingthe night exception to the beginning. Good idea, done. > Finally, let me propose to append to the regression test the following: Also a good idea, done. -- Daniel Gustafsson https://vmware.com/
Вложения
> On 29 Mar 2022, at 00:40, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Andres Freund <andres@anarazel.de> writes: >> On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote: >>> Do you think this potential foot-gun is scary enough to reject this patch? >>> There are lots of creative ways to cause Nagios alerts from ones database, but >>> this has the potential to do so with a small bug in userland code. Still, I >>> kind of like the feature so I'm indecisive. > >> It does seem like a huge footgun. But also kinda useful. So I'm really +-0. > > An on-login trigger is *necessarily* a foot-gun; I don't see that this > particular failure mode makes it any worse than it would be anyway. Agreed. > There has to be some not-too-difficult-to-use way to bypass a broken > login trigger. Assuming we are happy with the design for doing that, > might as well accept the hazards. The GUC in this patchset seems to be in line with what most in this thread have preferred, and with that in place (and single-user mode which still works for this) I think we have that covered. -- Daniel Gustafsson https://vmware.com/
Daniel Gustafsson писал 2022-03-30 16:48: >> On 30 Mar 2022, at 13:21, Ivan Panchenko <wao@mail.ru> wrote: >> Maybe side-effects is a bit too general? Emitting a log message, >> rejecting a >> login, setting some GUCs, etc are all side-effects too. >> Something like this: > > I've reworded the docs close to what you suggested here. > >> Also, please fix a typo in doc/src/sgml/ref/create_event_trigger.sgml >> : > > Done. > >> Regarding the trigger function example: >> It does not do anything if run on a standby. To show that it can do >> something on a standby to, I propose to move throwing the night >> exception to the beginning. > > Good idea, done. > >> Finally, let me propose to append to the regression test the >> following: > > Also a good idea, done. > > -- > Daniel Gustafsson https://vmware.com/ Please fix a typo in doc/src/sgml/event-trigger.sgml: "precvent" -- Andrey Sokolov Postgres Professional: http://www.postgrespro.com
> On 1 Apr 2022, at 09:16, a.sokolov@postgrespro.ru wrote: > Please fix a typo in doc/src/sgml/event-trigger.sgml: "precvent" Will do. With that fixed I think this is ready and unless I find something on another read through and test pass I hope to be able to push this before the CF closes today. -- Daniel Gustafsson https://vmware.com/
This had bitrotted a fair bit, attached is a rebase along with (mostly) documentation fixes. 0001 adds a generic GUC for ignoring event triggers and 0002 adds the login event with event trigger support, and hooks it up to the GUC such that broken triggers wont require single-user mode. Moving the CF entry back to Needs Review. -- Daniel Gustafsson https://vmware.com/
Вложения
This had bitrotted a fair bit, attached is a rebase along with (mostly)
documentation fixes. 0001 adds a generic GUC for ignoring event triggers and
0002 adds the login event with event trigger support, and hooks it up to the
GUC such that broken triggers wont require single-user mode. Moving the CF
entry back to Needs Review.
--
Daniel Gustafsson https://vmware.com/
+
+ tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
+
+ if (!HeapTupleIsValid(tuple))
On 02.09.2022 18:36, Daniel Gustafsson wrote: > This had bitrotted a fair bit, attached is a rebase along with (mostly) > documentation fixes. 0001 adds a generic GUC for ignoring event triggers and > 0002 adds the login event with event trigger support, and hooks it up to the > GUC such that broken triggers wont require single-user mode. Moving the CF > entry back to Needs Review. Hello! There is a race around setting and clearing of dathasloginevt. Steps to reproduce: 1. Create a trigger: CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$ BEGIN RAISE NOTICE 'You are welcome!'; END; $$ LANGUAGE plpgsql; CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc(); 2. Then drop it, but don't start new sessions: DROP EVENT TRIGGER on_login_trigger; 3. Create another trigger, but don't commit yet: BEGIN; CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc(); 4. Start a new session. This clears dathasloginevt. 5. Commit the transaction: COMMIT; Now we have a trigger, but it doesn't fire, because dathasloginevt=false. If two sessions create triggers concurrently, one of them will fail. Steps: 1. In the first session, start a transaction and create a trigger: BEGIN; CREATE EVENT TRIGGER on_login_trigger1 ON login EXECUTE PROCEDURE on_login_proc(); 2. In the second session, create another trigger (this query blocks): CREATE EVENT TRIGGER on_login_trigger2 ON login EXECUTE PROCEDURE on_login_proc(); 3. Commit in the first session: COMMIT; The second session fails: postgres=# CREATE EVENT TRIGGER on_login_trigger2 ON login EXECUTE PROCEDURE on_login_proc(); ERROR: tuple concurrently updated What else bothers me is that login triggers execute in an environment under user control and one has to be very careful. The example trigger from the documentation +DECLARE + hour integer = EXTRACT('hour' FROM current_time); + rec boolean; +BEGIN +-- 1. Forbid logging in between 2AM and 4AM. +IF hour BETWEEN 2 AND 4 THEN + RAISE EXCEPTION 'Login forbidden'; +END IF; can be bypassed with PGOPTIONS='-c timezone=...'. Probably this is nothing new and concerns any SECURITY DEFINER function, but still... Best regards, -- Sergey Shinderuk https://postgrespro.com/
> On 20 Sep 2022, at 15:43, Sergey Shinderuk <s.shinderuk@postgrespro.ru> wrote: > > On 02.09.2022 18:36, Daniel Gustafsson wrote: >> This had bitrotted a fair bit, attached is a rebase along with (mostly) >> documentation fixes. 0001 adds a generic GUC for ignoring event triggers and >> 0002 adds the login event with event trigger support, and hooks it up to the >> GUC such that broken triggers wont require single-user mode. Moving the CF >> entry back to Needs Review. > There is a race around setting and clearing of dathasloginevt. Thanks for the report! The whole dathasloginevt mechanism is IMO too clunky and unelegant to go ahead with, I wouldn't be surprised if there are other bugs lurking there. Since the original authors seems to have retired from the patch (I've only rebased it to try and help) I am inclined to mark it as returned with feedback. -- Daniel Gustafsson https://vmware.com/
On 20.09.2022 17:10, Daniel Gustafsson wrote: >> On 20 Sep 2022, at 15:43, Sergey Shinderuk <s.shinderuk@postgrespro.ru> wrote: >> There is a race around setting and clearing of dathasloginevt. > > Thanks for the report! The whole dathasloginevt mechanism is IMO too clunky > and unelegant to go ahead with, I wouldn't be surprised if there are other bugs > lurking there. Indeed. CREATE DATABASE doesn't copy dathasloginevt from the template database. ALTER EVENT TRIGGER ... ENABLE doesn't set dathasloginevt. In both cases triggers are enabled according to \dy output, but don't fire. The admin may not notice it immediately. -- Sergey Shinderuk https://postgrespro.com/
> On 20 Sep 2022, at 16:10, Daniel Gustafsson <daniel@yesql.se> wrote: > Since the original authors seems to have retired from the patch > (I've only rebased it to try and help) I am inclined to mark it as returned > with feedback. Doing so now since no updates have been posted for quite some time and holes have been poked in the patch. The GUC for temporarily disabling event triggers, to avoid the need for single- user mode during troubleshooting, might however be of interest so I will break that out and post to a new thread. -- Daniel Gustafsson https://vmware.com/
2022年11月4日(金) 5:23 Daniel Gustafsson <daniel@yesql.se>: > > > On 20 Sep 2022, at 16:10, Daniel Gustafsson <daniel@yesql.se> wrote: > > > Since the original authors seems to have retired from the patch > > (I've only rebased it to try and help) I am inclined to mark it as returned > > with feedback. > > Doing so now since no updates have been posted for quite some time and holes > have been poked in the patch. I see it was still "Waiting on Author" so I went ahead and changed the status. > The GUC for temporarily disabling event triggers, to avoid the need for single- > user mode during troubleshooting, might however be of interest so I will break > that out and post to a new thread. For reference this is the new thread: https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se Regards Ian Barwick
Hi hackers,
Since the original authors, as Daniel said, seems to have retired from the patch, I have allowed myself to continue the patch polishing.
Attached v32 includes fresh rebase and the following fixes:
- Copying dathasloginevt flag during DB creation from template;
- Restoring dathasloginevt flag after re-enabling a disabled trigger;
- Preserving dathasloginevt flag during not-running a trigger because of some filters (running with 'session_replication_role=replica' for example);
- Checking tags during the trigger creation;
The (en/dis)abling GUC was removed from the patch by default because it is now discussed and supported in a separate thread by Daniel Gustaffson:
https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9@yesql.se
Still the back-ported version of the Daniel’s patch is attached here too – just for convenience.
While the above changes represent the straight work continue, there is another version to consider. Most of the patch problems seem to originate from the dathasloginevt flag support. There are lots of known problems (partly fixed) and probably a lot more unfound. At the same time the flag itself is not a critical element, but just a performance optimizer for logging-in of users who are NOT using the on-login triggers.
I have made a simpler version without the flag and tried to compare the performance. The results show that life without dathasloginevt is still possible. When there is a small number of non-login event triggers, the difference is barely noticeable indeed. To show the difference, I have added 1000 sql_drop event triggers and used pgbench to continuously query the database for “select 1” with reconnects for 3 seconds. Here are the results. For each version I recorded average connection time with 0 and with 1000 event triggers:
With dathasloginevt flag: 0.609 ms VS 0.611 ms
No-flag version: 0.617 ms VS 0.727 ms
You can see that the difference is noticeable, but not that critical as we can expect for 1000 triggers (while in a vast majority of real cases there would be a much lesser number of triggers). I.e. the flagless version of the patch introduces a huge reliability raise at the cost of a small performance drop.
What do you think? Maybe it’s better to use the flagless version? Or even a small performance drop is considered as unacceptable?
The attached files include the two versions of the patch: “v32” for the updated version with flag and “v31_nf” – for the flagless version. Both versions contain “0001“ file for the patch itself and “0002” for back-ported controlling GUC.
Mikhail A. Gribkov
2022年11月4日(金) 5:23 Daniel Gustafsson <daniel@yesql.se>:
>
> > On 20 Sep 2022, at 16:10, Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > Since the original authors seems to have retired from the patch
> > (I've only rebased it to try and help) I am inclined to mark it as returned
> > with feedback.
>
> Doing so now since no updates have been posted for quite some time and holes
> have been poked in the patch.
I see it was still "Waiting on Author" so I went ahead and changed the status.
> The GUC for temporarily disabling event triggers, to avoid the need for single-
> user mode during troubleshooting, might however be of interest so I will break
> that out and post to a new thread.
For reference this is the new thread:
https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se
Regards
Ian Barwick
Вложения
Attached v33 is a new rebase of the flagless version of the patch. As there were no objections at first glance, I’ll try to post it to the upcoming commitfest, thus the brief recap of all the patch details is below.
v33-On_client_login_event_trigger
The patch introduces a trigger on login event, allowing to fire some actions right on the user connection. This can be useful for logging or connection check purposes as well as for some personalization of the environment. Usage details are described in the documentation included, but shortly usage is the same as for other triggers: create function returning event_trigger and then create event trigger on login event.
The patch is prepared to be applied to the master branch and tested when applied after e52f8b301ed54aac5162b185b43f5f1e44b6b17e commit by Thomas Munro (Date: Fri Dec 16 17:36:22 2022 +1300).
Regression tests and documentation included.
A couple of words about what and why I changed compared to the previous author's version.
First, the (en/dis)abling GUC was removed from the patch because ideologically it is a separate feature and nowadays it’s discussed and supported in a separate thread by Daniel Gustaffson:
https://www.postgresql.org/message-id/flat/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se
Second, I have removed the dathasloginevt flag. The flag was initially added to the patch for performance reasons and it did the job, although it was quite a clumsy construct causing tons of bugs and could potentially lead to tons more during further PostgreSQL evolution. I have removed the flag and found that the performance drop is not that significant.
And this is an aspect I should describe in more detail.
The possible performance drop is expected as an increased time used to login a user NOT using a login trigger.
First of all, the method of performance check:
echo 'select 1;' > ./tst.sql
pgbench -n -C -T3 -f tst.sql -U postgres postgres
The output value "average connection time" is the one I use to compare performance.
Now, what are the results.
* master branch: 0.641 ms
* patched version: 0.644 ms
No significant difference here and it is just what was expected. Based on the patch design the performance drop can be expected when there are lots of event triggers created, but not the login one. Thus I have created 1000 drop triggers (the script for creating triggers is attached too) in the database and repeated the test:
* master branch: 0.646 ms
* patched version: 0.754 ms
For 2000 triggers the patched version connection time is further increased to 0.862. Thus we have a login time rise in about 16.5% per 1000 event triggers in the database. It is a statistically noticeable value, still I don’t think it’s a critical one we should be afraid of.
N.B. The exact values of the login times slightly differ from the ones I posted in the previous email. Well, that’s the repeatability level we have. This convinces me even more that the observed level of performance drop is acceptable.
Вложения
+DECLARE
+ hour integer = EXTRACT('hour' FROM current_time);
+ rec boolean;
+BEGIN
+-- 1. Forbid logging in between 2AM and 4AM.
+IF hour BETWEEN 2 AND 4 THEN
+ RAISE EXCEPTION 'Login forbidden';
+END IF;
>can be bypassed with PGOPTIONS='-c timezone=...'. Probably this is
>nothing new and concerns any SECURITY DEFINER function, but still...
Hi hackers,
Attached v33 is a new rebase of the flagless version of the patch. As there were no objections at first glance, I’ll try to post it to the upcoming commitfest, thus the brief recap of all the patch details is below.
v33-On_client_login_event_trigger
The patch introduces a trigger on login event, allowing to fire some actions right on the user connection. This can be useful for logging or connection check purposes as well as for some personalization of the environment. Usage details are described in the documentation included, but shortly usage is the same as for other triggers: create function returning event_trigger and then create event trigger on login event.
The patch is prepared to be applied to the master branch and tested when applied after e52f8b301ed54aac5162b185b43f5f1e44b6b17e commit by Thomas Munro (Date: Fri Dec 16 17:36:22 2022 +1300).
Regression tests and documentation included.
A couple of words about what and why I changed compared to the previous author's version.
First, the (en/dis)abling GUC was removed from the patch because ideologically it is a separate feature and nowadays it’s discussed and supported in a separate thread by Daniel Gustaffson:
https://www.postgresql.org/message-id/flat/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se
Second, I have removed the dathasloginevt flag. The flag was initially added to the patch for performance reasons and it did the job, although it was quite a clumsy construct causing tons of bugs and could potentially lead to tons more during further PostgreSQL evolution. I have removed the flag and found that the performance drop is not that significant.
And this is an aspect I should describe in more detail.
The possible performance drop is expected as an increased time used to login a user NOT using a login trigger.
First of all, the method of performance check:
echo 'select 1;' > ./tst.sql
pgbench -n -C -T3 -f tst.sql -U postgres postgres
The output value "average connection time" is the one I use to compare performance.
Now, what are the results.
* master branch: 0.641 ms
* patched version: 0.644 ms
No significant difference here and it is just what was expected. Based on the patch design the performance drop can be expected when there are lots of event triggers created, but not the login one. Thus I have created 1000 drop triggers (the script for creating triggers is attached too) in the database and repeated the test:
* master branch: 0.646 ms
* patched version: 0.754 ms
For 2000 triggers the patched version connection time is further increased to 0.862. Thus we have a login time rise in about 16.5% per 1000 event triggers in the database. It is a statistically noticeable value, still I don’t think it’s a critical one we should be afraid of.
N.B. The exact values of the login times slightly differ from the ones I posted in the previous email. Well, that’s the repeatability level we have. This convinces me even more that the observed level of performance drop is acceptable.--
Mikhail A. Gribkov
Вложения
Hi Nikita,> Mikhail, I've checked the patch and previous discussion,> the condition mentioned earlier is still actual:Thanks for pointing this out! My bad, I forgot to fix the documentation example.Attached v34 has this issue fixed, as well as a couple other problems with the example code.--best regards,
Mikhail A. Gribkov
+ dbgtag = CMDTAG_LOGIN;
+ else
+ dbgtag = CreateCommandTag(parsetree);
> + dbgtag = CMDTAG_LOGIN;
> + else
> + dbgtag = CreateCommandTag(parsetree);
Moving CMDTAG_LOGIN to CreateCommandTag would change the CreateCommandTag function signature and ideology. CreateCommandTag finds a tag based on the command parse tree, while login event is a specific case when we do not have any command and the parse tree is NULL. Changing CreateCommandTag signature for these two calls looks a little bit overkill as it would lead to changing lots of other places the function is called from.
We could move this snippet to a separate function, but here are the similar concerns I think: it would make sense for a more common or a more complex snippet, but not for two cases of if-else one-liners.
On Sat, Dec 17, 2022 at 3:46 AM Mikhail Gribkov <youzhick@gmail.com> wrote:Hi Nikita,> Mikhail, I've checked the patch and previous discussion,> the condition mentioned earlier is still actual:Thanks for pointing this out! My bad, I forgot to fix the documentation example.Attached v34 has this issue fixed, as well as a couple other problems with the example code.--best regards,
Mikhail A. GribkovHi,bq. in to the system'in to' -> intobq. Any bugs in a trigger procedureAny bugs -> Any bug+ if (event == EVT_Login)
+ dbgtag = CMDTAG_LOGIN;
+ else
+ dbgtag = CreateCommandTag(parsetree);The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.Cheers
Вложения
Hi Ted,> bq. in to the system> 'in to' -> into> bq. Any bugs in a trigger procedure> Any bugs -> Any bugThanks! Fixed typos in the attached v35.> + if (event == EVT_Login)
> + dbgtag = CMDTAG_LOGIN;
> + else
> + dbgtag = CreateCommandTag(parsetree);> The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.It appears twice in fact, both cases are nearby: in the main code and under the assert-checking ifdef.
Moving CMDTAG_LOGIN to CreateCommandTag would change the CreateCommandTag function signature and ideology. CreateCommandTag finds a tag based on the command parse tree, while login event is a specific case when we do not have any command and the parse tree is NULL. Changing CreateCommandTag signature for these two calls looks a little bit overkill as it would lead to changing lots of other places the function is called from.
We could move this snippet to a separate function, but here are the similar concerns I think: it would make sense for a more common or a more complex snippet, but not for two cases of if-else one-liners.--
Hi Ted,> bq. in to the system> 'in to' -> into> bq. Any bugs in a trigger procedure> Any bugs -> Any bugThanks! Fixed typos in the attached v35.> + if (event == EVT_Login)
> + dbgtag = CMDTAG_LOGIN;
> + else
> + dbgtag = CreateCommandTag(parsetree);> The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.It appears twice in fact, both cases are nearby: in the main code and under the assert-checking ifdef.
Moving CMDTAG_LOGIN to CreateCommandTag would change the CreateCommandTag function signature and ideology. CreateCommandTag finds a tag based on the command parse tree, while login event is a specific case when we do not have any command and the parse tree is NULL. Changing CreateCommandTag signature for these two calls looks a little bit overkill as it would lead to changing lots of other places the function is called from.
We could move this snippet to a separate function, but here are the similar concerns I think: it would make sense for a more common or a more complex snippet, but not for two cases of if-else one-liners.
--On Sat, Dec 17, 2022 at 3:29 PM Ted Yu <yuzhihong@gmail.com> wrote:On Sat, Dec 17, 2022 at 3:46 AM Mikhail Gribkov <youzhick@gmail.com> wrote:Hi Nikita,> Mikhail, I've checked the patch and previous discussion,> the condition mentioned earlier is still actual:Thanks for pointing this out! My bad, I forgot to fix the documentation example.Attached v34 has this issue fixed, as well as a couple other problems with the example code.--best regards,
Mikhail A. GribkovHi,bq. in to the system'in to' -> intobq. Any bugs in a trigger procedureAny bugs -> Any bug+ if (event == EVT_Login)
+ dbgtag = CMDTAG_LOGIN;
+ else
+ dbgtag = CreateCommandTag(parsetree);The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.Cheers
HiI checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres733 tps (master), 727 tps (patched).I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.RegardsPavel
Вложения
Hi Pavel,Thanks for pointing out the tests. I completely agree that using an exception inside on-login trigger should be tested. It cannot be done via regular *.sql/*.out regress tests, thus I have added another perl test to authentication group doing this.Attached v36 patch contains this test along with the fresh rebase on master.--On Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:HiI checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres733 tps (master), 727 tps (patched).I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.
RegardsPavel
HiOn Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:HiI checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres733 tps (master), 727 tps (patched).I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.Thank youcheck-world passed without problemsbuild doc passed without problemsI think so tests are now enoughI'll mark this patch as ready for committer
RegardsPavelRegardsPavel
ne 15. 1. 2023 v 7:32 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:HiOn Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:HiI checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres733 tps (master), 727 tps (patched).I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.Thank youcheck-world passed without problemsbuild doc passed without problemsI think so tests are now enoughI'll mark this patch as ready for committerUnfortunately, I forgot one important point. There are not any tests related to backup.I miss pg_dump related tests.I mark this patch as waiting on the author.
Вложения
Hi Pavel,On Mon, Jan 16, 2023 at 9:10 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:ne 15. 1. 2023 v 7:32 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:HiOn Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:HiI checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres733 tps (master), 727 tps (patched).I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.Thank youcheck-world passed without problemsbuild doc passed without problemsI think so tests are now enoughI'll mark this patch as ready for committerUnfortunately, I forgot one important point. There are not any tests related to backup.I miss pg_dump related tests.I mark this patch as waiting on the author.Thanks for noticing this.I have added sections to pg_dump tests. Attached v37 patch contains these additions along with the fresh rebase on master.
--
Thank youmarked as ready for committerRegardsPavel
Вложения
It looks like Daniel Gustafsson, Andres, and Tom have all weighed in on this patch with at least a neutral comment (+-0 from Andres :) It looks like the main concern was breaking physical replicas and that there was consensus that as long as single-user mode worked that it was ok? So maybe it's time after 2 1/2 years to get this one committed? -- Gregory Stark As Commitfest Manager
> On 6 Mar 2023, at 21:55, Gregory Stark (as CFM) <stark.cfm@gmail.com> wrote: > > It looks like Daniel Gustafsson, Andres, and Tom have all weighed in > on this patch with at least a neutral comment (+-0 from Andres :) I think the concept of a login event trigger has merits, even though it's kind of a niche use case. > It looks like the main concern was breaking physical replicas and that > there was consensus that as long as single-user mode worked that it > was ok? Having a way to not rely on single-user mode and not causing an unacceptable performance hit (which judging by recent benchmarks might not be an issue?). I still intend to revisit this and I hope to get to it during this CF. -- Daniel Gustafsson
Hi, On 2023-03-06 15:55:01 -0500, Gregory Stark (as CFM) wrote: > It looks like Daniel Gustafsson, Andres, and Tom have all weighed in > on this patch with at least a neutral comment (+-0 from Andres :) > > It looks like the main concern was breaking physical replicas and that > there was consensus that as long as single-user mode worked that it > was ok? I don't think it's OK with just single user mode as an scape hatch. We need the GUC that was discussed as part of the thread (and I think there's a patch for that too). Greetings, Andres Freund
> On 6 Mar 2023, at 23:10, Andres Freund <andres@anarazel.de> wrote: > > Hi, > > On 2023-03-06 15:55:01 -0500, Gregory Stark (as CFM) wrote: >> It looks like Daniel Gustafsson, Andres, and Tom have all weighed in >> on this patch with at least a neutral comment (+-0 from Andres :) >> >> It looks like the main concern was breaking physical replicas and that >> there was consensus that as long as single-user mode worked that it >> was ok? > > I don't think it's OK with just single user mode as an scape hatch. We need > the GUC that was discussed as part of the thread (and I think there's a patch > for that too). This is the patch which originated from this thread: https://commitfest.postgresql.org/42/4013/ -- Daniel Gustafsson
It looks like the patch is failing a test by not dumping login_event_trigger_func? I'm guessing there's a race condition in the test but I don't know. I also don't see the tmp_test_jI6t output file being preserved in the artifacts anywhere :( https://cirrus-ci.com/task/6391161871400960?logs=test_world#L2671 [16:16:48.594] # Looks like you failed 1 test of 10350. # Running: pg_dump --no-sync --file=/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t/only_dump_measurement.sql --table-and-children=dump_test.measurement --lock-wait-timeout=180000 postgres [16:16:27.027](0.154s) ok 6765 - only_dump_measurement: pg_dump runs ..... [16:16:27.035](0.000s) not ok 6870 - only_dump_measurement: should dump CREATE FUNCTION dump_test.login_event_trigger_func [16:16:27.035](0.000s) [16:16:27.035](0.000s) # Failed test 'only_dump_measurement: should dump CREATE FUNCTION dump_test.login_event_trigger_func' # at t/002_pg_dump.pl line 4761. ..... [16:16:48.594] +++ tap check in src/bin/pg_dump +++ [16:16:48.594] [16:16:05] t/001_basic.pl ................ ok 612 ms ( 0.01 usr 0.00 sys + 0.24 cusr 0.26 csys = 0.51 CPU) [16:16:48.594] [16:16:48.594] # Failed test 'only_dump_measurement: should dump CREATE FUNCTION dump_test.login_event_trigger_func' [16:16:48.594] # at t/002_pg_dump.pl line 4761. [16:16:48.594] # Review only_dump_measurement results in /tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t
It looks like the patch is failing a test by not dumping
login_event_trigger_func? I'm guessing there's a race condition in the
test but I don't know. I also don't see the tmp_test_jI6t output file
being preserved in the artifacts anywhere :(
https://cirrus-ci.com/task/6391161871400960?logs=test_world#L2671
[16:16:48.594] # Looks like you failed 1 test of 10350.
# Running: pg_dump --no-sync
--file=/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t/only_dump_measurement.sql
--table-and-children=dump_test.measurement --lock-wait-timeout=180000
postgres
[16:16:27.027](0.154s) ok 6765 - only_dump_measurement: pg_dump runs
.....
[16:16:27.035](0.000s) not ok 6870 - only_dump_measurement: should
dump CREATE FUNCTION dump_test.login_event_trigger_func
[16:16:27.035](0.000s)
[16:16:27.035](0.000s) # Failed test 'only_dump_measurement: should
dump CREATE FUNCTION dump_test.login_event_trigger_func'
# at t/002_pg_dump.pl line 4761.
.....
[16:16:48.594] +++ tap check in src/bin/pg_dump +++
[16:16:48.594] [16:16:05] t/001_basic.pl ................ ok 612 ms (
0.01 usr 0.00 sys + 0.24 cusr 0.26 csys = 0.51 CPU)
[16:16:48.594]
[16:16:48.594] # Failed test 'only_dump_measurement: should dump
CREATE FUNCTION dump_test.login_event_trigger_func'
[16:16:48.594] # at t/002_pg_dump.pl line 4761.
[16:16:48.594] # Review only_dump_measurement results in
/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t
Вложения
On Tue, Mar 15, 2022 at 4:52 PM Daniel Gustafsson <daniel@yesql.se> wrote: > Yeah, that was the previously posted v25 from the author (who adopted it from > the original author). I took the liberty to quickly poke at the review > comments you had left as well as the ones that I had found to try and progress > the patch. 0001 should really go in it's own thread though to not hide it from > anyone interested who isn't looking at this thread. Some comments on 0001: - In general, I think we should prefer to phrase options in terms of what is done, rather than what is not done. For instance, the corresponding GUC for row-level security is row_security={on|off}, not ignore_row_security. - I think it's odd that the GUC in question doesn't accept true and false and our usual synonyms for those values. I suggest that it should, even if we want to add more possible values later. - "ignoreing" is mispleled. So is gux-ignore-event-trigger. "Even triggers" -> "Event triggers". - Perhaps the documentation for the GUC should mention that the GUC is not relevant in single-user mode because event triggers don't fire then anyway. - "Disable event triggers during the session." isn't a very good description because there is in theory nothing to prevent this from being set in postgresql.conf. Basically, I think 0001 is a good idea -- I'm much more nervous about 0002. I think we should get 0001 polished up and committed. -- Robert Haas EDB: http://www.enterprisedb.com
> On 22 Mar 2023, at 18:54, Robert Haas <robertmhaas@gmail.com> wrote: > Basically, I think 0001 is a good idea -- I'm much more nervous about > 0002. I think we should get 0001 polished up and committed. Correct me if I'm wrong, but I believe you commented on v27-0001 of the login event trigger patch series? Sorry about the confusion if so, this is a very old and lengthy thread with many twists and turns. This series was closed downthread when the original authors of login EVT left, and the 0001 GUC patch extracted into its own thread. That patch now lives at: https://commitfest.postgresql.org/42/4013/ This thread was then later revived by Mikhail Gribkov but without 0001 instead referring to the above patch for that part. The new patch for 0001 is quite different, and I welcome your eyes on that since I agree with you that it would be a good idea. -- Daniel Gustafsson
The attached v40 patch is a fresh rebase on master branch to actualize the state before the upcoming commitfest.
Nothing has changed beyond rebasing.
And just for convenience, here is a link to the exact message of the thread
describing the current approach:
https://www.postgresql.org/message-id/CAMEv5_vg4aJOoUC74XJm%2B5B7%2BTF1nT-Yhtg%2BawtBOESXG5Grfg%40mail.gmail.com
> On 22 Mar 2023, at 18:54, Robert Haas <robertmhaas@gmail.com> wrote:
> Basically, I think 0001 is a good idea -- I'm much more nervous about
> 0002. I think we should get 0001 polished up and committed.
Correct me if I'm wrong, but I believe you commented on v27-0001 of the login
event trigger patch series? Sorry about the confusion if so, this is a very
old and lengthy thread with many twists and turns. This series was closed
downthread when the original authors of login EVT left, and the 0001 GUC patch
extracted into its own thread. That patch now lives at:
https://commitfest.postgresql.org/42/4013/
This thread was then later revived by Mikhail Gribkov but without 0001 instead
referring to the above patch for that part.
The new patch for 0001 is quite different, and I welcome your eyes on that
since I agree with you that it would be a good idea.
--
Daniel Gustafsson
Вложения
Hi! On Wed, Jun 14, 2023 at 10:49 PM Mikhail Gribkov <youzhick@gmail.com> wrote: > The attached v40 patch is a fresh rebase on master branch to actualize the state before the upcoming commitfest. > Nothing has changed beyond rebasing. > > And just for convenience, here is a link to the exact message of the thread > describing the current approach: > https://www.postgresql.org/message-id/CAMEv5_vg4aJOoUC74XJm%2B5B7%2BTF1nT-Yhtg%2BawtBOESXG5Grfg%40mail.gmail.com Thank you for the update. I think the patch is interesting and demanding. The code, docs and tests seem to be quite polished already. Simultaneously, the thread is long and spread over a long time. So, it's easy to lose track. I'd like to do a short summary of design issues on this thread. 1. Initially the patch introduced "on login" triggers. It was unclear if they need to rerun on RESET ALL or DISCARD ALL [1]. The new name is "client connection" trigger, which seems fine [2]. 2. Another question is how to deal with triggers, which hangs or fails with error [1]. One possible way to workaround that is single-user mode, which is already advised to workaround the errors in other event triggers. However, in perspective single-user mode might be deleted. Also, single-user mode is considered as a worst-case scenario recovery tool, while it's very easy to block the database connections with client connection triggers. As addition/alternative to single-user mode, GUC options to disable all event triggers and/or client connection triggers. Finally, the patch for the GUC option to disable all event triggers resides in a separate thread [4]. Apparently that patch should be committed first [5]. 3. Yet another question is connection-time overhead introduced by this patch. The overhead estimate varies from no measurable overhead [6] to 5% overhead [7]. In order to overcome that, [8] has introduced a database-level flag indicating whether there are connection triggers. Later this flag was removed [9] in a hope that the possible overhead is acceptable. 4. [10] points that there is no clean way to store information about unsuccessful connections (declined by either authentication or trigger). However, this is considered out-of-scope for the current patch, and could be implemented later if needed. 5. It was also pointed out [11] that ^C in psql doesn't cancel long-running client connection triggers. That might be considered a psql problem though. 6. It has been also pointed out that [12] all triggers, which write data to the database, must check pg_is_in_recovery() to work correctly on standby. That seems to be currently reflected in the documentation. So, for me the open issues seem to be 2, 3 and 5. My plan to revive this patch is to commit the GUC patch [4], recheck the overhead and probably leave "^C in psql" problem as a separate standalone issue. Any thoughts? Links. 1. https://www.postgresql.org/message-id/CAFj8pRBdqdqvkU3mVKzoOnO+jPz-6manRV47CDEa+1jD6x6LFg%40mail.gmail.com 2. https://www.postgresql.org/message-id/CAFj8pRCxdQgHy8Mynk3hz6pFsqQ9BN6Vfgy0MJLtQBAUhWDf3w%40mail.gmail.com 3. https://www.postgresql.org/message-id/E0D5DC61-C490-45BD-A984-E8D56493EC4F%40yesql.se 4. https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9@yesql.se 5. https://www.postgresql.org/message-id/20230306221010.gszjoakt5jp7oqpd%40awork3.anarazel.de 6. https://www.postgresql.org/message-id/90760f2d-2f9c-12ab-d2c5-e8e6fb7d08de%40postgrespro.ru 7. https://www.postgresql.org/message-id/CAFj8pRChwu01VLx76nKBVyScHCsd1YnBGiKfDJ6h17g4CSnUBg%40mail.gmail.com 8. https://www.postgresql.org/message-id/4471d472-5dfc-f2b0-ad05-0ff8d0a3bb0c%40postgrespro.ru 9. https://www.postgresql.org/message-id/CAMEv5_vg4aJOoUC74XJm%2B5B7%2BTF1nT-Yhtg%2BawtBOESXG5Grfg%40mail.gmail.com 10. https://www.postgresql.org/message-id/9c897136-4755-dcfc-2d24-b12bcfe4467f%40sigaev.ru 11. https://www.postgresql.org/message-id/CA%2BTgmoZv9f1s797tihx-zXQN4AE4ZFBV5C0K%3DzngbgNu3xNNkg%40mail.gmail.com 12. https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de ------ Regards, Alexander Korotkov
> On 25 Sep 2023, at 11:13, Alexander Korotkov <aekorotkov@gmail.com> wrote: > I'd like to do a short summary of > design issues on this thread. Thanks for summarizing this long thread! > the patch for the GUC option to disable > all event triggers resides in a separate thread [4]. Apparently that > patch should be committed first [5]. I have committed the prerequisite patch for temporarily disabling EVTs via a GUC in 7750fefdb2. We should absolutely avoid creating any more dependencies on single-user mode (yes, I have changed my mind since the beginning of the thread). > 3. Yet another question is connection-time overhead introduced by this > patch. The overhead estimate varies from no measurable overhead [6] to > 5% overhead [7]. In order to overcome that, [8] has introduced a > database-level flag indicating whether there are connection triggers. > Later this flag was removed [9] in a hope that the possible overhead > is acceptable. While I disliked the flag, I do think the overhead is problematic. Last time I profiled it I found it noticeable, and it seems expensive for such a niche feature to impact such a hot path. Maybe you can think of other ways to reduce the cost here (if it indeed is an issue in the latest version of the patch, which is not one I've benchmarked)? > 5. It was also pointed out [11] that ^C in psql doesn't cancel > long-running client connection triggers. That might be considered a > psql problem though. While it is a psql problem, it's exacerbated by a slow login EVT and it breaks what I would guess is the mental model of many who press ^C in a stalled login. At the very least we should probably document the risk? -- Daniel Gustafsson
Hi, Daniel! On Mon, Sep 25, 2023 at 3:42 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > On 25 Sep 2023, at 11:13, Alexander Korotkov <aekorotkov@gmail.com> wrote: > > > I'd like to do a short summary of > > design issues on this thread. > > Thanks for summarizing this long thread! > > > the patch for the GUC option to disable > > all event triggers resides in a separate thread [4]. Apparently that > > patch should be committed first [5]. > > I have committed the prerequisite patch for temporarily disabling EVTs via a > GUC in 7750fefdb2. We should absolutely avoid creating any more dependencies > on single-user mode (yes, I have changed my mind since the beginning of the > thread). Thank you for committing 7750fefdb2. I've revised this patch according to it. I've resolved the conflicts, make use of event_triggers GUC and adjusted some comments. > > 3. Yet another question is connection-time overhead introduced by this > > patch. The overhead estimate varies from no measurable overhead [6] to > > 5% overhead [7]. In order to overcome that, [8] has introduced a > > database-level flag indicating whether there are connection triggers. > > Later this flag was removed [9] in a hope that the possible overhead > > is acceptable. > > While I disliked the flag, I do think the overhead is problematic. Last time I > profiled it I found it noticeable, and it seems expensive for such a niche > feature to impact such a hot path. Maybe you can think of other ways to reduce > the cost here (if it indeed is an issue in the latest version of the patch, > which is not one I've benchmarked)? I don't think I can reproduce the performance regression pointed out by Pavel Stehule [1]. I run a simple ";" sql script (this script doesn't even get the snapshot) on my laptop and run it multiple times with event_triggers = on and event_triggers = off; pgbench -c 10 -j 10 -M prepared -f 1.sql -P 1 -T 60 -C postgres event_triggers = on run1: 2261 run2: 2301 run3: 2281 event_triggers = off run1: 2321 run2: 2277 run3: 2267 pgbench -c 10 -j 10 -M prepared -f 1.sql -P 1 -T 60 -C postgres event_triggers = on run1: 731 run2: 740 run3: 733 event_triggers = off run1: 739 run2: 734 run3: 731 I can't confirm the measurable overhead. > > 5. It was also pointed out [11] that ^C in psql doesn't cancel > > long-running client connection triggers. That might be considered a > > psql problem though. > > While it is a psql problem, it's exacerbated by a slow login EVT and it breaks > what I would guess is the mental model of many who press ^C in a stalled login. > At the very least we should probably document the risk? Done in the attached patch. Links 1. https://www.postgresql.org/message-id/CAFj8pRChwu01VLx76nKBVyScHCsd1YnBGiKfDJ6h17g4CSnUBg%40mail.gmail.com ------ Regards, Alexander Korotkov
Вложения
> On 28 Sep 2023, at 23:50, Alexander Korotkov <aekorotkov@gmail.com> wrote: > I don't think I can reproduce the performance regression pointed out > by Pavel Stehule [1]. > I can't confirm the measurable overhead. Running the same pgbench command on my laptop looking at the average connection times, and the averaging that over five runs (low/avg/high) I see ~5% increase over master with the patched version (compiled without assertions and debug): Patched event_triggers on: 6.858 ms/7.038 ms/7.434 ms Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms Master: 6.676 ms/6.697 ms/6.760 ms This is all quite unscientific with a lot of jitter so grains of salt are to be applied, but I find it odd that you don't see any measurable effect. Are you seeing the same/similar connection times between master and with this patch applied? A few small comments on the patch: + prevent successful login to the system. Such bugs may be fixed by + restarting the system in single-user mode (as event triggers are This paragraph should be reworded to recommend the GUC instead of single-user mode (while retaining mention of single-user mode, just not as the primary option). + Also, it's recommended to evade long-running queries in s/evade/avoid/ perhaps? Thanks for working on this! -- Daniel Gustafsson
On Fri, Sep 29, 2023 at 1:15 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > On 28 Sep 2023, at 23:50, Alexander Korotkov <aekorotkov@gmail.com> wrote: > > > I don't think I can reproduce the performance regression pointed out > > by Pavel Stehule [1]. > > > I can't confirm the measurable overhead. > > > Running the same pgbench command on my laptop looking at the average connection > times, and the averaging that over five runs (low/avg/high) I see ~5% increase > over master with the patched version (compiled without assertions and debug): > > Patched event_triggers on: 6.858 ms/7.038 ms/7.434 ms > Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms > Master: 6.676 ms/6.697 ms/6.760 ms > > This is all quite unscientific with a lot of jitter so grains of salt are to be > applied, but I find it odd that you don't see any measurable effect. Are you > seeing the same/similar connection times between master and with this patch > applied? Thank you for doing experiments on your side. I've rechecked. It appears that I didn't do enough runs, thus I didn't see the overhead as more than an error. Now, I also can confirm ~5% overhead. I spent some time thinking about how to overcome this overhead, but I didn't find a brilliant option. Previously pg_database flag was proposed but then criticized as complex and error-prone. I can also imagine shmem caching mechanism. But it would require overcoming possible race conditions between shared cache invalidation and transaction commit etc. So, that would be also complex and error-prone. Any better ideas? > A few small comments on the patch: > > + prevent successful login to the system. Such bugs may be fixed by > + restarting the system in single-user mode (as event triggers are > This paragraph should be reworded to recommend the GUC instead of single-user > mode (while retaining mention of single-user mode, just not as the primary > option). > > > + Also, it's recommended to evade long-running queries in > s/evade/avoid/ perhaps? Fixed. > Thanks for working on this! Thank you as well! ------ Regards, Alexander Korotkov
Вложения
Sorry to have gone dark on this for a long time after having been asked for my input back in March. I'm not having a great time trying to keep up with email, and the threads getting split up makes it a lot worse for me. On Fri, Sep 29, 2023 at 6:15 AM Daniel Gustafsson <daniel@yesql.se> wrote: > Running the same pgbench command on my laptop looking at the average connection > times, and the averaging that over five runs (low/avg/high) I see ~5% increase > over master with the patched version (compiled without assertions and debug): > > Patched event_triggers on: 6.858 ms/7.038 ms/7.434 ms > Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms > Master: 6.676 ms/6.697 ms/6.760 ms This seems kind of crazy to me. Why does it happen? It sounds to me like we must be doing a lot of extra catalog access to find out whether there are any on-login event triggers. Like maybe a sequential scan of pg_event_trigger. Maybe we need to engineer a way to avoid that. I don't have a brilliant idea off-hand, but I feel like there should be something we can do. I think a lot of users would say that logins on PostgreSQL are too slow already. -- Robert Haas EDB: http://www.enterprisedb.com
> On 2 Oct 2023, at 20:10, Robert Haas <robertmhaas@gmail.com> wrote: > > Sorry to have gone dark on this for a long time after having been > asked for my input back in March. I'm not having a great time trying > to keep up with email, and the threads getting split up makes it a lot > worse for me. Not a problem, thanks for chiming in. > On Fri, Sep 29, 2023 at 6:15 AM Daniel Gustafsson <daniel@yesql.se> wrote: >> Running the same pgbench command on my laptop looking at the average connection >> times, and the averaging that over five runs (low/avg/high) I see ~5% increase >> over master with the patched version (compiled without assertions and debug): >> >> Patched event_triggers on: 6.858 ms/7.038 ms/7.434 ms >> Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms >> Master: 6.676 ms/6.697 ms/6.760 ms > > This seems kind of crazy to me. Why does it happen? It sounds to me > like we must be doing a lot of extra catalog access to find out > whether there are any on-login event triggers. Like maybe a sequential > scan of pg_event_trigger. That's exactly what happens, the patch is using BuildEventTriggerCache() to build the hash for EVT which is then checked for login triggers. This is clearly the bottleneck and there needs to be a fast-path. There used to be a cache flag in an earlier version of the patch but it was a but klugy, a version of that needs to be reimplemented for this patch to fly. > I think a lot of users would say that logins on PostgreSQL are too slow already. Agreed. -- Daniel Gustafsson
On Tue, Oct 3, 2023 at 9:43 AM Daniel Gustafsson <daniel@yesql.se> wrote: > That's exactly what happens, the patch is using BuildEventTriggerCache() to > build the hash for EVT which is then checked for login triggers. This is > clearly the bottleneck and there needs to be a fast-path. There used to be a > cache flag in an earlier version of the patch but it was a but klugy, a version > of that needs to be reimplemented for this patch to fly. So I haven't looked at this patch, but we basically saying that only the superuser can create login triggers, and if they do, those triggers apply to every single user on the system? That would seem to be the logical extension of the existing event trigger mechanism, but it isn't obviously as good of a fit for this case as it is for other cases where event triggers are a thing. Changing the catalog representation could be a way around this. What if you only allowed one login trigger per database, and instead of being stored in pg_event_trigger, the OID of the function gets recorded in the pg_database row? Then this would be a lot cheaper since we have to fetch the pg_database row anyway. Or change the SQL syntax to something entirely new so you can have different login triggers for different users -- and maybe users are allowed to create their own -- but the relevant ones can be found by an index scan instead of a sequential scan. I'm just spitballing here. If you think the present design is good and just want to try to speed it up, I'm not deeply opposed to that. But it's also not obvious to me how to stick a cache in front of something that's basically a full-table scan. -- Robert Haas EDB: http://www.enterprisedb.com
Hi, Robert! On Tue, Oct 3, 2023 at 5:21 PM Robert Haas <robertmhaas@gmail.com> wrote: > On Tue, Oct 3, 2023 at 9:43 AM Daniel Gustafsson <daniel@yesql.se> wrote: > > That's exactly what happens, the patch is using BuildEventTriggerCache() to > > build the hash for EVT which is then checked for login triggers. This is > > clearly the bottleneck and there needs to be a fast-path. There used to be a > > cache flag in an earlier version of the patch but it was a but klugy, a version > > of that needs to be reimplemented for this patch to fly. > > So I haven't looked at this patch, but we basically saying that only > the superuser can create login triggers, and if they do, those > triggers apply to every single user on the system? That would seem to > be the logical extension of the existing event trigger mechanism, but > it isn't obviously as good of a fit for this case as it is for other > cases where event triggers are a thing. > > Changing the catalog representation could be a way around this. What > if you only allowed one login trigger per database, and instead of > being stored in pg_event_trigger, the OID of the function gets > recorded in the pg_database row? Then this would be a lot cheaper > since we have to fetch the pg_database row anyway. Or change the SQL > syntax to something entirely new so you can have different login > triggers for different users -- and maybe users are allowed to create > their own -- but the relevant ones can be found by an index scan > instead of a sequential scan. > > I'm just spitballing here. If you think the present design is good and > just want to try to speed it up, I'm not deeply opposed to that. But > it's also not obvious to me how to stick a cache in front of something > that's basically a full-table scan. Thank you for the interesting ideas. I'd like to try to revive the version with the flag in pg_database. Will use other ideas as backup if no success. ------ Regards, Alexander Korotkov
Hi! On Tue, Oct 3, 2023 at 8:35 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > Thank you for the interesting ideas. I'd like to try to revive the > version with the flag in pg_database. Will use other ideas as backup > if no success. I've revived the patch version with pg_database.dathasloginevt flag. I took v32 version [1] and made the following changes. * Incorporate enchantments made on flagless version of patch. * Read dathasloginevt during InitPostgres() to prevent extra catalog access and even more notable StartTransactionCommand() when there are no login triggers. * Hold lock during setting of pg_database.dathasloginevt flag (v32 version actually didn't prevent race condition). * Fix AlterEventTrigger() to check event name not trigger name * Acquire conditional lock while resetting pg_database.dathasloginevt flag to prevent new database connection to hang waiting another transaction to finish. This version should be good and has no overhead. Any thoughts? Daniel, could you please re-run the performance tests? Links 1. https://www.postgresql.org/message-id/CAMEv5_vDjceLr54WUCNPPVsJs8WBWWsRW826VppNEFoLC1LAEw%40mail.gmail.com ------ Regards, Alexander Korotkov
On 2023-10-09 17:11:25 +0300, Alexander Korotkov wrote: > This version should be good and has no overhead. Any thoughts? I think you forgot to attach the patch?
On Mon, Oct 9, 2023 at 11:58 PM Andres Freund <andres@anarazel.de> wrote: > On 2023-10-09 17:11:25 +0300, Alexander Korotkov wrote: > > This version should be good and has no overhead. Any thoughts? > > I think you forgot to attach the patch? That's it! ------ Regards, Alexander Korotkov
Вложения
On Mon, Oct 9, 2023 at 10:11 AM Alexander Korotkov <aekorotkov@gmail.com> wrote: > * Hold lock during setting of pg_database.dathasloginevt flag (v32 > version actually didn't prevent race condition). So ... how does getting this flag set actually work? And how does clearing it work? In the case of row-level security, you have to explicitly enable the flag on the table level using DDL provided for that purpose. In the case of relhas{rules,triggers,subclass} the flag is set automatically as a side-effect of some other operation. I tend to consider that the latter design is somewhat messy. It's potentially even messier here, because at least when you add a rule or a trigger to a table you're expecting to take a lock on the table anyway. I don't think you necessarily expect creating a login trigger to take a lock on the database. That's a bit odd and could have visible side effects. And if you don't, then what happens is that if you create two login triggers in overlapping transactions, then (1) if there were no login triggers previously, one of the transactions fails with an internal-looking error message about a concurrent tuple update and (2) if there were login triggers previously, then it works fine. That's also a bit weird and surprising. Now the counter-argument could be that adding new DDL to enable login triggers for a database is too much cognitive burden and it's better to have the kind of weird and surprising behavior that I just discussed. I don't know that I would buy that argument, but it could be made ... and my real point here is that I don't even see these trade-offs being discussed. Apologies if they were discussed earlier and I just missed that; I confess to not having read every email message on this topic, and some of the ones I did read I read a long time ago. > This version should be good and has no overhead. Any thoughts? > Daniel, could you please re-run the performance tests? Is "no overhead" an overly bold claim here? -- Robert Haas EDB: http://www.enterprisedb.com
Hi, On 2023-10-10 08:18:46 +0300, Alexander Korotkov wrote: > @@ -968,7 +969,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > > if (!get_db_info(dbtemplate, ShareLock, > &src_dboid, &src_owner, &src_encoding, > - &src_istemplate, &src_allowconn, > + &src_istemplate, &src_allowconn, &src_hasloginevt, > &src_frozenxid, &src_minmxid, &src_deftablespace, > &src_collate, &src_ctype, &src_iculocale, &src_icurules, &src_locprovider, > &src_collversion)) This isn't your fault, but this imo has become unreadable. Think we ought to move the information about a database to a struct. > @@ -296,6 +306,13 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO > CatalogTupleInsert(tgrel, tuple); > heap_freetuple(tuple); > > + /* > + * Login event triggers have an additional flag in pg_database to allow > + * faster lookups in hot codepaths. Set the flag unless already True. > + */ > + if (strcmp(eventname, "login") == 0) > + SetDatatabaseHasLoginEventTriggers(); It's not really faster lookups, it's no lookups, right? > /* Depend on owner. */ > recordDependencyOnOwner(EventTriggerRelationId, trigoid, evtOwner); > > @@ -357,6 +374,39 @@ filter_list_to_array(List *filterlist) > return PointerGetDatum(construct_array_builtin(data, l, TEXTOID)); > } > > +/* > + * Set pg_database.dathasloginevt flag for current database indicating that > + * current database has on login triggers. > + */ > +void > +SetDatatabaseHasLoginEventTriggers(void) > +{ > + /* Set dathasloginevt flag in pg_database */ > + Form_pg_database db; > + Relation pg_db = table_open(DatabaseRelationId, RowExclusiveLock); > + HeapTuple tuple; > + > + /* > + * Use shared lock to prevent a conflit with EventTriggerOnLogin() trying > + * to reset pg_database.dathasloginevt flag. Note that we use > + * AccessShareLock allowing setters concurently. > + */ > + LockSharedObject(DatabaseRelationId, MyDatabaseId, 0, AccessShareLock); That seems like a very odd approach - how does this avoid concurrency issues with one backend setting and another unsetting the flag? And outside of that, won't this just lead to concurrently updated tuples? > + tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); > + if (!HeapTupleIsValid(tuple)) > + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); > + db = (Form_pg_database) GETSTRUCT(tuple); > + if (!db->dathasloginevt) > + { > + db->dathasloginevt = true; > + CatalogTupleUpdate(pg_db, &tuple->t_self, tuple); > + CommandCounterIncrement(); > + } > + table_close(pg_db, RowExclusiveLock); > + heap_freetuple(tuple); > +} > + > /* > * ALTER EVENT TRIGGER foo ENABLE|DISABLE|ENABLE ALWAYS|REPLICA > */ > @@ -391,6 +441,10 @@ AlterEventTrigger(AlterEventTrigStmt *stmt) > > CatalogTupleUpdate(tgrel, &tup->t_self, tup); > > + if (namestrcmp(&evtForm->evtevent, "login") == 0 && > + tgenabled != TRIGGER_DISABLED) > + SetDatatabaseHasLoginEventTriggers(); > + > InvokeObjectPostAlterHook(EventTriggerRelationId, > trigoid, 0); > > @@ -557,7 +611,7 @@ filter_event_trigger(CommandTag tag, EventTriggerCacheItem *item) > static List * > EventTriggerCommonSetup(Node *parsetree, > EventTriggerEvent event, const char *eventstr, > - EventTriggerData *trigdata) > + EventTriggerData *trigdata, bool unfiltered) > { > CommandTag tag; > List *cachelist; > @@ -582,10 +636,15 @@ EventTriggerCommonSetup(Node *parsetree, > { > CommandTag dbgtag; > > - dbgtag = CreateCommandTag(parsetree); > + if (event == EVT_Login) > + dbgtag = CMDTAG_LOGIN; > + else > + dbgtag = CreateCommandTag(parsetree); > + > if (event == EVT_DDLCommandStart || > event == EVT_DDLCommandEnd || > - event == EVT_SQLDrop) > + event == EVT_SQLDrop || > + event == EVT_Login) > { > if (!command_tag_event_trigger_ok(dbgtag)) > elog(ERROR, "unexpected command tag \"%s\"", GetCommandTagName(dbgtag)); > @@ -604,7 +663,10 @@ EventTriggerCommonSetup(Node *parsetree, > return NIL; > > /* Get the command tag. */ > - tag = CreateCommandTag(parsetree); > + if (event == EVT_Login) > + tag = CMDTAG_LOGIN; > + else > + tag = CreateCommandTag(parsetree); Seems this bit should instead be in a function, given that you have it in multiple places. > > +/* > + * Fire login event triggers if any are present. The dathasloginevt > + * pg_database flag is left when an event trigger is dropped, to avoid > + * complicating the codepath in the case of multiple event triggers. This > + * function will instead unset the flag if no trigger is defined. > + */ > +void > +EventTriggerOnLogin(void) > +{ > + List *runlist; > + EventTriggerData trigdata; > + > + /* > + * See EventTriggerDDLCommandStart for a discussion about why event > + * triggers are disabled in single user mode or via a GUC. We also need a > + * database connection (some background workers doesn't have it). > + */ > + if (!IsUnderPostmaster || !event_triggers || > + !OidIsValid(MyDatabaseId) || !MyDatabaseHasLoginEventTriggers) > + return; > + > + StartTransactionCommand(); > + runlist = EventTriggerCommonSetup(NULL, > + EVT_Login, "login", > + &trigdata, false); > + > + if (runlist != NIL) > + { > + /* > + * Event trigger execution may require an active snapshot. > + */ > + PushActiveSnapshot(GetTransactionSnapshot()); > + > + /* Run the triggers. */ > + EventTriggerInvoke(runlist, &trigdata); > + > + /* Cleanup. */ > + list_free(runlist); > + > + PopActiveSnapshot(); > + } > + /* > + * There is no active login event trigger, but our pg_database.dathasloginevt was set. > + * Try to unset this flag. We use the lock to prevent concurrent > + * SetDatatabaseHasLoginEventTriggers(), but we don't want to hang the > + * connection waiting on the lock. Thus, we are just trying to acquire > + * the lock conditionally. > + */ > + else if (ConditionalLockSharedObject(DatabaseRelationId, MyDatabaseId, > + 0, AccessExclusiveLock)) Eek. Why are we doing it this way? I think this is a seriously bad idea. Maybe it's obvious to you, but it seems much more reasonable to make the pg_database column an integer and count the number of login event triggers. When 0, then we don't need to look for login event triggers. Greetings, Andres Freund
Hi! Thank you for the review. On Tue, Oct 10, 2023 at 7:37 PM Andres Freund <andres@anarazel.de> wrote: > On 2023-10-10 08:18:46 +0300, Alexander Korotkov wrote: > > @@ -968,7 +969,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > > > > if (!get_db_info(dbtemplate, ShareLock, > > &src_dboid, &src_owner, &src_encoding, > > - &src_istemplate, &src_allowconn, > > + &src_istemplate, &src_allowconn, &src_hasloginevt, > > &src_frozenxid, &src_minmxid, &src_deftablespace, > > &src_collate, &src_ctype, &src_iculocale, &src_icurules, &src_locprovider, > > &src_collversion)) > > This isn't your fault, but this imo has become unreadable. Think we ought to > move the information about a database to a struct. Should I do this in a separate patch? > > @@ -296,6 +306,13 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO > > CatalogTupleInsert(tgrel, tuple); > > heap_freetuple(tuple); > > > > + /* > > + * Login event triggers have an additional flag in pg_database to allow > > + * faster lookups in hot codepaths. Set the flag unless already True. > > + */ > > + if (strcmp(eventname, "login") == 0) > > + SetDatatabaseHasLoginEventTriggers(); > > It's not really faster lookups, it's no lookups, right? Yes, no extra lookups. Fixed. > > /* Depend on owner. */ > > recordDependencyOnOwner(EventTriggerRelationId, trigoid, evtOwner); > > > > @@ -357,6 +374,39 @@ filter_list_to_array(List *filterlist) > > return PointerGetDatum(construct_array_builtin(data, l, TEXTOID)); > > } > > > > +/* > > + * Set pg_database.dathasloginevt flag for current database indicating that > > + * current database has on login triggers. > > + */ > > +void > > +SetDatatabaseHasLoginEventTriggers(void) > > +{ > > + /* Set dathasloginevt flag in pg_database */ > > + Form_pg_database db; > > + Relation pg_db = table_open(DatabaseRelationId, RowExclusiveLock); > > + HeapTuple tuple; > > + > > + /* > > + * Use shared lock to prevent a conflit with EventTriggerOnLogin() trying > > + * to reset pg_database.dathasloginevt flag. Note that we use > > + * AccessShareLock allowing setters concurently. > > + */ > > + LockSharedObject(DatabaseRelationId, MyDatabaseId, 0, AccessShareLock); > > That seems like a very odd approach - how does this avoid concurrency issues > with one backend setting and another unsetting the flag? And outside of that, > won't this just lead to concurrently updated tuples? When backend unsets the flag, it acquires the lock first. If lock is acquired successfully then no other backends hold it. If the concurrent backend have already inserted an event trigger then we will detect it by rechecking event list under lock. If the concurrent backend inserted/enabled event trigger and waiting for the lock, then it will set the flag once we release the lock. > > + tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); > > + if (!HeapTupleIsValid(tuple)) > > + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); > > + db = (Form_pg_database) GETSTRUCT(tuple); > > + if (!db->dathasloginevt) > > + { > > + db->dathasloginevt = true; > > + CatalogTupleUpdate(pg_db, &tuple->t_self, tuple); > > + CommandCounterIncrement(); > > + } > > + table_close(pg_db, RowExclusiveLock); > > + heap_freetuple(tuple); > > +} > > + > > /* > > * ALTER EVENT TRIGGER foo ENABLE|DISABLE|ENABLE ALWAYS|REPLICA > > */ > > @@ -391,6 +441,10 @@ AlterEventTrigger(AlterEventTrigStmt *stmt) > > > > CatalogTupleUpdate(tgrel, &tup->t_self, tup); > > > > + if (namestrcmp(&evtForm->evtevent, "login") == 0 && > > + tgenabled != TRIGGER_DISABLED) > > + SetDatatabaseHasLoginEventTriggers(); > > + > > InvokeObjectPostAlterHook(EventTriggerRelationId, > > trigoid, 0); > > > > @@ -557,7 +611,7 @@ filter_event_trigger(CommandTag tag, EventTriggerCacheItem *item) > > static List * > > EventTriggerCommonSetup(Node *parsetree, > > EventTriggerEvent event, const char *eventstr, > > - EventTriggerData *trigdata) > > + EventTriggerData *trigdata, bool unfiltered) > > { > > CommandTag tag; > > List *cachelist; > > @@ -582,10 +636,15 @@ EventTriggerCommonSetup(Node *parsetree, > > { > > CommandTag dbgtag; > > > > - dbgtag = CreateCommandTag(parsetree); > > + if (event == EVT_Login) > > + dbgtag = CMDTAG_LOGIN; > > + else > > + dbgtag = CreateCommandTag(parsetree); > > + > > if (event == EVT_DDLCommandStart || > > event == EVT_DDLCommandEnd || > > - event == EVT_SQLDrop) > > + event == EVT_SQLDrop || > > + event == EVT_Login) > > { > > if (!command_tag_event_trigger_ok(dbgtag)) > > elog(ERROR, "unexpected command tag \"%s\"", GetCommandTagName(dbgtag)); > > @@ -604,7 +663,10 @@ EventTriggerCommonSetup(Node *parsetree, > > return NIL; > > > > /* Get the command tag. */ > > - tag = CreateCommandTag(parsetree); > > + if (event == EVT_Login) > > + tag = CMDTAG_LOGIN; > > + else > > + tag = CreateCommandTag(parsetree); > > Seems this bit should instead be in a function, given that you have it in > multiple places. Done. > > +/* > > + * Fire login event triggers if any are present. The dathasloginevt > > + * pg_database flag is left when an event trigger is dropped, to avoid > > + * complicating the codepath in the case of multiple event triggers. This > > + * function will instead unset the flag if no trigger is defined. > > + */ > > +void > > +EventTriggerOnLogin(void) > > +{ > > + List *runlist; > > + EventTriggerData trigdata; > > + > > + /* > > + * See EventTriggerDDLCommandStart for a discussion about why event > > + * triggers are disabled in single user mode or via a GUC. We also need a > > + * database connection (some background workers doesn't have it). > > + */ > > + if (!IsUnderPostmaster || !event_triggers || > > + !OidIsValid(MyDatabaseId) || !MyDatabaseHasLoginEventTriggers) > > + return; > > + > > + StartTransactionCommand(); > > + runlist = EventTriggerCommonSetup(NULL, > > + EVT_Login, "login", > > + &trigdata, false); > > + > > + if (runlist != NIL) > > + { > > + /* > > + * Event trigger execution may require an active snapshot. > > + */ > > + PushActiveSnapshot(GetTransactionSnapshot()); > > + > > + /* Run the triggers. */ > > + EventTriggerInvoke(runlist, &trigdata); > > + > > + /* Cleanup. */ > > + list_free(runlist); > > + > > + PopActiveSnapshot(); > > + } > > + /* > > + * There is no active login event trigger, but our pg_database.dathasloginevt was set. > > + * Try to unset this flag. We use the lock to prevent concurrent > > + * SetDatatabaseHasLoginEventTriggers(), but we don't want to hang the > > + * connection waiting on the lock. Thus, we are just trying to acquire > > + * the lock conditionally. > > + */ > > + else if (ConditionalLockSharedObject(DatabaseRelationId, MyDatabaseId, > > + 0, AccessExclusiveLock)) > > Eek. Why are we doing it this way? I think this is a seriously bad > idea. Maybe it's obvious to you, but it seems much more reasonable to make the > pg_database column an integer and count the number of login event > triggers. When 0, then we don't need to look for login event triggers. The approach with number of login event trigger obviously will also work. But this version with lazy flag cleaning avoids need to handle every unobvious case we delete the event triggers (cascading deletion etc). ------ Regards, Alexander Korotkov
Вложения
Hi, Robert! Thank you for your feedback. On Tue, Oct 10, 2023 at 5:51 PM Robert Haas <robertmhaas@gmail.com> wrote: > > On Mon, Oct 9, 2023 at 10:11 AM Alexander Korotkov <aekorotkov@gmail.com> wrote: > > * Hold lock during setting of pg_database.dathasloginevt flag (v32 > > version actually didn't prevent race condition). > > So ... how does getting this flag set actually work? And how does > clearing it work? I hope I explained that in [1]. > In the case of row-level security, you have to explicitly enable the > flag on the table level using DDL provided for that purpose. In the > case of relhas{rules,triggers,subclass} the flag is set automatically > as a side-effect of some other operation. I tend to consider that the > latter design is somewhat messy. It's potentially even messier here, > because at least when you add a rule or a trigger to a table you're > expecting to take a lock on the table anyway. I don't think you > necessarily expect creating a login trigger to take a lock on the > database. That's a bit odd and could have visible side effects. And if > you don't, then what happens is that if you create two login triggers > in overlapping transactions, then (1) if there were no login triggers > previously, one of the transactions fails with an internal-looking > error message about a concurrent tuple update and (2) if there were > login triggers previously, then it works fine. Yep, in v43 it worked that way. One transaction has to wait for another finishing update of pg_database tuple, then fails. This is obviously ridiculous. Since overlapping setters of flag will have to wait anyway, I changed lock mode in v44 for them to AccessExclusiveLock. Now, waiting transaction then sees the updated tuple and doesn't fail. > That's also a bit weird > and surprising. Now the counter-argument could be that adding new DDL > to enable login triggers for a database is too much cognitive burden > and it's better to have the kind of weird and surprising behavior that > I just discussed. I don't know that I would buy that argument, but it > could be made ... and my real point here is that I don't even see > these trade-offs being discussed. Apologies if they were discussed > earlier and I just missed that; I confess to not having read every > email message on this topic, and some of the ones I did read I read a > long time ago. I have read the thread quite carefully. I don't think manual setting of the flag was discussed. I do think it would be extra burden for users, and I would prefer automatic flag as long as it works transparently and reliably. > > This version should be good and has no overhead. Any thoughts? > > Daniel, could you please re-run the performance tests? > > Is "no overhead" an overly bold claim here? Yes, sure. I meant no extra lookup. Hopefully that would mean no measurable overhead when is no enabled login triggers. Links 1. https://www.postgresql.org/message-id/CAPpHfdtvvozi%3Dttp8kvJQwuLrP5Q0D_5c4Pw1U67MRXcROB9yA%40mail.gmail.com ------ Regards, Alexander Korotkov
On Tue, Oct 10, 2023 at 3:43 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > Yep, in v43 it worked that way. One transaction has to wait for > another finishing update of pg_database tuple, then fails. This is > obviously ridiculous. Since overlapping setters of flag will have to > wait anyway, I changed lock mode in v44 for them to > AccessExclusiveLock. Now, waiting transaction then sees the updated > tuple and doesn't fail. Doesn't that mean that if you create the first login trigger in a database and leave the transaction open, nobody can connect to that database until the transaction ends? -- Robert Haas EDB: http://www.enterprisedb.com
On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote: > On Tue, Oct 10, 2023 at 3:43 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > > Yep, in v43 it worked that way. One transaction has to wait for > > another finishing update of pg_database tuple, then fails. This is > > obviously ridiculous. Since overlapping setters of flag will have to > > wait anyway, I changed lock mode in v44 for them to > > AccessExclusiveLock. Now, waiting transaction then sees the updated > > tuple and doesn't fail. > > Doesn't that mean that if you create the first login trigger in a > database and leave the transaction open, nobody can connect to that > database until the transaction ends? It doesn't mean that, because when trying to reset the flag v44 does conditional lock. So, if another transaction is holding the log we will just skip resetting the flag. So, the flag will be cleared on the first connection after that transaction ends. ------ Regards, Alexander Korotkov
On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote: > > Doesn't that mean that if you create the first login trigger in a > > database and leave the transaction open, nobody can connect to that > > database until the transaction ends? > > It doesn't mean that, because when trying to reset the flag v44 does > conditional lock. So, if another transaction is holding the log we > will just skip resetting the flag. So, the flag will be cleared on > the first connection after that transaction ends. But in the scenario I am describing the flag is being set, not reset. -- Robert Haas EDB: http://www.enterprisedb.com
On Fri, Oct 13, 2023 at 4:18 AM Robert Haas <robertmhaas@gmail.com> wrote: > On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote: > > > > Doesn't that mean that if you create the first login trigger in a > > > database and leave the transaction open, nobody can connect to that > > > database until the transaction ends? > > > > It doesn't mean that, because when trying to reset the flag v44 does > > conditional lock. So, if another transaction is holding the log we > > will just skip resetting the flag. So, the flag will be cleared on > > the first connection after that transaction ends. > > But in the scenario I am describing the flag is being set, not reset. Sorry, it seems I just missed some logical steps. Imagine, that transaction A created the first login trigger and hangs open. Then the new connection B sees no visible triggers yet, but dathasloginevt flag is set. Therefore, connection B tries conditional lock but just gives up because the lock is held by transaction A. Also, note that the lock has been just some lock with a custom tag. It doesn't effectively block the database. You may think about it as of custom advisory lock. ------ Regards, Alexander Korotkov
On Fri, Oct 13, 2023 at 11:26 AM Alexander Korotkov <aekorotkov@gmail.com> wrote: > On Fri, Oct 13, 2023 at 4:18 AM Robert Haas <robertmhaas@gmail.com> wrote: > > On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > > > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote: > > > > > > Doesn't that mean that if you create the first login trigger in a > > > > database and leave the transaction open, nobody can connect to that > > > > database until the transaction ends? > > > > > > It doesn't mean that, because when trying to reset the flag v44 does > > > conditional lock. So, if another transaction is holding the log we > > > will just skip resetting the flag. So, the flag will be cleared on > > > the first connection after that transaction ends. > > > > But in the scenario I am describing the flag is being set, not reset. > > Sorry, it seems I just missed some logical steps. Imagine, that > transaction A created the first login trigger and hangs open. Then > the new connection B sees no visible triggers yet, but dathasloginevt > flag is set. Therefore, connection B tries conditional lock but just > gives up because the lock is held by transaction A. > > Also, note that the lock has been just some lock with a custom tag. > It doesn't effectively block the database. You may think about it as > of custom advisory lock. I've revised the comments about the lock a bit. I've also run some tests regarding the connection time (5 runs). v45, event_triggers=on: avg=3.081ms, min=2.992ms, max=3.146ms v45, event_triggers=off: avg=3.132ms, min=3.048ms, max=3.186ms master: 3.080ms, min=3.023ms, max=3.167ms So, no measurable overhead (not surprising since no extra catalog lookup). I think this patch is in the commitable shape. Any objections? ------ Regards, Alexander Korotkov
Вложения
On Sat, Oct 14, 2023 at 2:10 AM Alexander Korotkov <aekorotkov@gmail.com> wrote: > On Fri, Oct 13, 2023 at 11:26 AM Alexander Korotkov > <aekorotkov@gmail.com> wrote: > > On Fri, Oct 13, 2023 at 4:18 AM Robert Haas <robertmhaas@gmail.com> wrote: > > > On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > > > > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote: > > > > > > > > Doesn't that mean that if you create the first login trigger in a > > > > > database and leave the transaction open, nobody can connect to that > > > > > database until the transaction ends? > > > > > > > > It doesn't mean that, because when trying to reset the flag v44 does > > > > conditional lock. So, if another transaction is holding the log we > > > > will just skip resetting the flag. So, the flag will be cleared on > > > > the first connection after that transaction ends. > > > > > > But in the scenario I am describing the flag is being set, not reset. > > > > Sorry, it seems I just missed some logical steps. Imagine, that > > transaction A created the first login trigger and hangs open. Then > > the new connection B sees no visible triggers yet, but dathasloginevt > > flag is set. Therefore, connection B tries conditional lock but just > > gives up because the lock is held by transaction A. > > > > Also, note that the lock has been just some lock with a custom tag. > > It doesn't effectively block the database. You may think about it as > > of custom advisory lock. > > I've revised the comments about the lock a bit. I've also run some > tests regarding the connection time (5 runs). > > v45, event_triggers=on: avg=3.081ms, min=2.992ms, max=3.146ms > v45, event_triggers=off: avg=3.132ms, min=3.048ms, max=3.186ms > master: 3.080ms, min=3.023ms, max=3.167ms > > So, no measurable overhead (not surprising since no extra catalog lookup). > I think this patch is in the commitable shape. Any objections? The attached revision fixes test failures spotted by commitfest.cputube.org. Also, perl scripts passed perltidy. ------ Regards, Alexander Korotkov
Вложения
On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote: > The attached revision fixes test failures spotted by > commitfest.cputube.org. Also, perl scripts passed perltidy. Still you've missed a few things. At quick glance: - The code indentation was off a bit in event_trigger.c. - 005_login_trigger.pl fails if the code is compiled with ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS because a WARNING is reported in test "create tmp objects: err equals". - 005_sspi.pl is older than the new test 005_login_trigger.pl, could you rename it with a different number? -- Michael
Вложения
On Mon, Oct 16, 2023 at 4:00 AM Michael Paquier <michael@paquier.xyz> wrote: > On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote: > > The attached revision fixes test failures spotted by > > commitfest.cputube.org. Also, perl scripts passed perltidy. > > Still you've missed a few things. At quick glance: > - The code indentation was off a bit in event_trigger.c. > - 005_login_trigger.pl fails if the code is compiled with > ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS because a WARNING is > reported in test "create tmp objects: err equals". > - 005_sspi.pl is older than the new test 005_login_trigger.pl, could > you rename it with a different number? You are very fast and sharp eye! Thank you for fixing the indentation. I just pushed fixes for the rest. ------ Regards, Alexander Korotkov
On Mon, Oct 16, 2023 at 4:00 AM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote:
> > The attached revision fixes test failures spotted by
> > commitfest.cputube.org. Also, perl scripts passed perltidy.
>
> Still you've missed a few things. At quick glance:
> - The code indentation was off a bit in event_trigger.c.
> - 005_login_trigger.pl fails if the code is compiled with
> ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS because a WARNING is
> reported in test "create tmp objects: err equals".
> - 005_sspi.pl is older than the new test 005_login_trigger.pl, could
> you rename it with a different number?
You are very fast and sharp eye!
Thank you for fixing the indentation. I just pushed fixes for the rest.
------
Regards,
Alexander Korotkov
Hi, Mikhail. On Wed, Oct 18, 2023 at 1:30 PM Mikhail Gribkov <youzhick@gmail.com> wrote: > Sorry for my long offline and thanks for the activity. So should we close the patch on the commitfest page now? I have just done this. > By the way I had one more issue with the login trigger tests (quite a rare one though). A race condition may occur on somesystems, when oidjoins test starts a moment later than normally and affects logins count for on-login trigger test. ThusI had to split event_trigger and oidjoins tests into separate parallel groups. I'll post this as an independent patchthen. Thank you for catching it. Please, post this. ------ Regards, Alexander Korotkov
Mikhail Gribkov <youzhick@gmail.com> writes: > Just for a more complete picture of the final state here. > I have posted the described fix (for avoiding race condition in the tests) > separately: > https://commitfest.postgresql.org/45/4616/ It turns out that the TAP test for this feature (006_login_trigger.pl) also has a race condition: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba&dt=2023-10-28%2003%3A33%3A28 The critical bit of the log is ack Broken pipe: write( 14, 'SELECT 1;' ) at /usr/pkg/lib/perl5/vendor_perl/5.36.0/IPC/Run/IO.pm line 550. It looks to me that what happened here is that the backend completed the authentication handshake, and then the login trigger caused a FATAL exit, and after than the connected psql session tried to send "SELECT 1" on an already-closed pipe. That failed, causing IPC::Run to panic. mamba is a fairly slow machine and doubtless has timing a bit different from what this test was created on. But I doubt there is any way to make this behavior perfectly stable across a range of machines, so I recommend just removing the test case involving a fatal exit. regards, tom lane
On Sun, Oct 29, 2023 at 6:16 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > Mikhail Gribkov <youzhick@gmail.com> writes: > > Just for a more complete picture of the final state here. > > I have posted the described fix (for avoiding race condition in the tests) > > separately: > > https://commitfest.postgresql.org/45/4616/ > > It turns out that the TAP test for this feature (006_login_trigger.pl) > also has a race condition: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba&dt=2023-10-28%2003%3A33%3A28 > > The critical bit of the log is > > ack Broken pipe: write( 14, 'SELECT 1;' ) at /usr/pkg/lib/perl5/vendor_perl/5.36.0/IPC/Run/IO.pm line 550. > > It looks to me that what happened here is that the backend completed the > authentication handshake, and then the login trigger caused a FATAL exit, > and after than the connected psql session tried to send "SELECT 1" on > an already-closed pipe. That failed, causing IPC::Run to panic. > > mamba is a fairly slow machine and doubtless has timing a bit different > from what this test was created on. But I doubt there is any way to make > this behavior perfectly stable across a range of machines, so I recommend > just removing the test case involving a fatal exit. Makes sense. Are you good with the attached patch? ------ Regards, Alexander Korotkov
Вложения
Alexander Korotkov <aekorotkov@gmail.com> writes: > On Sun, Oct 29, 2023 at 6:16 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> It looks to me that what happened here is that the backend completed the >> authentication handshake, and then the login trigger caused a FATAL exit, >> and after than the connected psql session tried to send "SELECT 1" on >> an already-closed pipe. That failed, causing IPC::Run to panic. Looking closer, what must have happened is that the psql session ended before IPC::Run could jam 'SELECT 1' into its stdin. I wonder if this could be stabilized by doing 'psql -c "SELECT 1"' and not having to write anything to the child process stdin? But I'm not sure this test case is valuable enough to put a great deal of work into. >> mamba is a fairly slow machine and doubtless has timing a bit different >> from what this test was created on. But I doubt there is any way to make >> this behavior perfectly stable across a range of machines, so I recommend >> just removing the test case involving a fatal exit. > Makes sense. Are you good with the attached patch? OK by me, though I note you misspelled "mallory" in the comment. regards, tom lane
Hello Alexander, I've discovered one more instability in the event_trigger_login test. Please look for example at case [1]: ok 213 + event_trigger 28946 ms not ok 214 - event_trigger_login 6430 ms ok 215 - fast_default 19349 ms ok 216 - tablespace 44789 ms 1..216 # 1 of 216 tests failed. --- /home/bf/bf-build/skink-master/HEAD/pgsql/src/test/regress/expected/event_trigger_login.out 2023-10-27 22:55:12.574139524 +0000 +++ /home/bf/bf-build/skink-master/HEAD/pgsql.build/src/test/regress/results/event_trigger_login.out 2024-01-03 23:49:50.177461816 +0000 @@ -40,6 +40,6 @@ SELECT dathasloginevt FROM pg_database WHERE datname= :'DBNAME'; dathasloginevt ---------------- - f + t (1 row) 2024-01-03 23:49:40.378 UTC [2235175][client backend][3/5949:0] STATEMENT: REINDEX INDEX CONCURRENTLY concur_reindex_ind; ... 2024-01-03 23:49:50.340 UTC [2260974][autovacuum worker][5/5439:18812] LOG: automatic vacuum of table "regression.pg_catalog.pg_statistic": index scans: 1 (operations logged around 23:49:50.177461816) I suspected that this failure was caused by autovacuum, and I've managed to reproduce it without Valgrind or slowing down a machine. With /tmp/extra.config: log_autovacuum_min_duration = 0 autovacuum_naptime = 1 autovacuum = on I run: for ((i=1;i<10;i++)); do echo "ITERATION $i"; \ TEMP_CONFIG=/tmp/extra.config \ TESTS=$(printf 'event_trigger_login %.0s' `seq 1000`) \ make -s check-tests || break; done and get failures on iterations 1, 2, 1... ok 693 - event_trigger_login 15 ms not ok 694 - event_trigger_login 15 ms ok 695 - event_trigger_login 21 ms Also, I've observed an anomaly after dropping a login event trigger: CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$ BEGIN RAISE NOTICE 'You are welcome!'; END; $$ LANGUAGE plpgsql; CREATE EVENT TRIGGER olt ON login EXECUTE PROCEDURE on_login_proc(); SELECT dathasloginevt FROM pg_database WHERE datname= current_database(); dathasloginevt ---------------- t (1 row) DROP EVENT TRIGGER olt; SELECT dathasloginevt FROM pg_database WHERE datname= current_database(); dathasloginevt ---------------- t (1 row) Although after reconnection (\c, as done in the event_trigger_login test), this query returns 'f'. [1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2024-01-03%2023%3A04%3A20 [2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2023-12-26%2019%3A33%3A02 Best regards, Alexander
Hi, Alexander! On Sat, Jan 13, 2024 at 6:00 PM Alexander Lakhin <exclusion@gmail.com> wrote: > I've discovered one more instability in the event_trigger_login test. > Please look for example at case [1]: > ok 213 + event_trigger 28946 ms > not ok 214 - event_trigger_login 6430 ms > ok 215 - fast_default 19349 ms > ok 216 - tablespace 44789 ms > 1..216 > # 1 of 216 tests failed. Thank you for reporting this. I'm going to take a closer look at this tomorrow. But I doubt I would find a solution other than removing the flaky parts of the test. ------ Regards, Alexander Korotkov
> On 13 Jan 2024, at 17:00, Alexander Lakhin <exclusion@gmail.com> wrote: > DROP EVENT TRIGGER olt; > SELECT dathasloginevt FROM pg_database WHERE datname= current_database(); > dathasloginevt > ---------------- > t > (1 row) > > Although after reconnection (\c, as done in the event_trigger_login test), > this query returns 'f'. This is by design in the patch. The flag isn't changed on DROP, it is only cleared on logins iff there is no login event trigger. The description in the docs is worded to indicate that it shouldn't be taken as truth for monitoring purposes: "This flag is used internally by PostgreSQL and should not be manually altered or read for monitoring purposes." -- Daniel Gustafsson
> On 13 Jan 2024, at 17:00, Alexander Lakhin <exclusion@gmail.com> wrote: > I suspected that this failure was caused by autovacuum, and I've managed to > reproduce it without Valgrind or slowing down a machine. This might be due to the fact that the cleanup codepath to remove the flag when there is no login event trigger doesn't block on locking pg_database to avoid stalling connections. There are no guarantees when the flag is cleared, so a test like this will always have potential to be flaky it seems. -- Daniel Gustafsson
On Mon, Jan 15, 2024 at 11:29 AM Daniel Gustafsson <daniel@yesql.se> wrote: > > On 13 Jan 2024, at 17:00, Alexander Lakhin <exclusion@gmail.com> wrote: > > > I suspected that this failure was caused by autovacuum, and I've managed to > > reproduce it without Valgrind or slowing down a machine. > > This might be due to the fact that the cleanup codepath to remove the flag when > there is no login event trigger doesn't block on locking pg_database to avoid > stalling connections. There are no guarantees when the flag is cleared, so a > test like this will always have potential to be flaky it seems. +1 Thank you, Daniel. I think you described everything absolutely correctly. As I wrote upthread, it doesn't seem much could be done with this, at least within a regression test. So, I just removed this query from the test. ------ Regards, Alexander Korotkov
Hello Alexander and Daniel, Please look at the following query, which triggers an assertion failure on updating the field dathasloginevt for an entry in pg_database: SELECT format('CREATE DATABASE db1 LOCALE_PROVIDER icu ICU_LOCALE en ENCODING utf8 ICU_RULES ''' || repeat(' ', 200000) || ''' TEMPLATE template0;') \gexec \c db1 - CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$ BEGIN RAISE NOTICE 'You are welcome!'; END; $$ LANGUAGE plpgsql; CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc(); DROP EVENT TRIGGER on_login_trigger; \c \connect: connection to server on socket "/tmp/.s.PGSQL.5432" failed: server closed the connection unexpectedly The stack trace of the assertion failure is: ... #5 0x000055c8699b9b8d in ExceptionalCondition ( conditionName=conditionName@entry=0x55c869a1f7c0 "HaveRegisteredOrActiveSnapshot()", fileName=fileName@entry=0x55c869a1f4c6 "toast_internals.c", lineNumber=lineNumber@entry=669) at assert.c:66 #6 0x000055c86945df0a in init_toast_snapshot (...) at toast_internals.c:669 #7 0x000055c86945dfbe in toast_delete_datum (...) at toast_internals.c:429 #8 0x000055c8694fd1da in toast_tuple_cleanup (...) at toast_helper.c:309 #9 0x000055c8694b55a1 in heap_toast_insert_or_update (...) at heaptoast.c:333 #10 0x000055c8694a8c6c in heap_update (... at heapam.c:3604 #11 0x000055c8694a96cb in simple_heap_update (...) at heapam.c:4062 #12 0x000055c869555b7b in CatalogTupleUpdate (...) at indexing.c:322 #13 0x000055c8695f0725 in EventTriggerOnLogin () at event_trigger.c:957 ... Funnily enough, when Tom Lane was wondering, whether pg_database's toast table could pose a risk [1], I came to the conclusion that it's impossible, but that was before the login triggers introduction... [1] https://www.postgresql.org/message-id/1284094.1695479962%40sss.pgh.pa.us Best regards, Alexander
On Tue, Jan 23, 2024 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote: > Please look at the following query, which triggers an assertion failure on > updating the field dathasloginevt for an entry in pg_database: > SELECT format('CREATE DATABASE db1 LOCALE_PROVIDER icu ICU_LOCALE en ENCODING utf8 > ICU_RULES ''' || repeat(' ', 200000) || ''' TEMPLATE template0;') > \gexec > \c db1 - > > CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$ > BEGIN > RAISE NOTICE 'You are welcome!'; > END; > $$ LANGUAGE plpgsql; > > CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc(); > DROP EVENT TRIGGER on_login_trigger; > > \c > > \connect: connection to server on socket "/tmp/.s.PGSQL.5432" failed: server closed the connection unexpectedly > > The stack trace of the assertion failure is: > ... > #5 0x000055c8699b9b8d in ExceptionalCondition ( > conditionName=conditionName@entry=0x55c869a1f7c0 "HaveRegisteredOrActiveSnapshot()", > fileName=fileName@entry=0x55c869a1f4c6 "toast_internals.c", lineNumber=lineNumber@entry=669) at assert.c:66 > #6 0x000055c86945df0a in init_toast_snapshot (...) at toast_internals.c:669 > #7 0x000055c86945dfbe in toast_delete_datum (...) at toast_internals.c:429 > #8 0x000055c8694fd1da in toast_tuple_cleanup (...) at toast_helper.c:309 > #9 0x000055c8694b55a1 in heap_toast_insert_or_update (...) at heaptoast.c:333 > #10 0x000055c8694a8c6c in heap_update (... at heapam.c:3604 > #11 0x000055c8694a96cb in simple_heap_update (...) at heapam.c:4062 > #12 0x000055c869555b7b in CatalogTupleUpdate (...) at indexing.c:322 > #13 0x000055c8695f0725 in EventTriggerOnLogin () at event_trigger.c:957 > ... > > Funnily enough, when Tom Lane was wondering, whether pg_database's toast > table could pose a risk [1], I came to the conclusion that it's impossible, > but that was before the login triggers introduction... > > [1] https://www.postgresql.org/message-id/1284094.1695479962%40sss.pgh.pa.us Thank you for reporting. I'm looking into this... I wonder if there is a way to avoid toast update given that we don't touch the toasted field here. ------ Regards, Alexander Korotkov
On Tue, Jan 23, 2024 at 11:52 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > On Tue, Jan 23, 2024 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote: > > Please look at the following query, which triggers an assertion failure on > > updating the field dathasloginevt for an entry in pg_database: > > SELECT format('CREATE DATABASE db1 LOCALE_PROVIDER icu ICU_LOCALE en ENCODING utf8 > > ICU_RULES ''' || repeat(' ', 200000) || ''' TEMPLATE template0;') > > \gexec > > \c db1 - > > > > CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$ > > BEGIN > > RAISE NOTICE 'You are welcome!'; > > END; > > $$ LANGUAGE plpgsql; > > > > CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc(); > > DROP EVENT TRIGGER on_login_trigger; > > > > \c > > > > \connect: connection to server on socket "/tmp/.s.PGSQL.5432" failed: server closed the connection unexpectedly > > > > The stack trace of the assertion failure is: > > ... > > #5 0x000055c8699b9b8d in ExceptionalCondition ( > > conditionName=conditionName@entry=0x55c869a1f7c0 "HaveRegisteredOrActiveSnapshot()", > > fileName=fileName@entry=0x55c869a1f4c6 "toast_internals.c", lineNumber=lineNumber@entry=669) at assert.c:66 > > #6 0x000055c86945df0a in init_toast_snapshot (...) at toast_internals.c:669 > > #7 0x000055c86945dfbe in toast_delete_datum (...) at toast_internals.c:429 > > #8 0x000055c8694fd1da in toast_tuple_cleanup (...) at toast_helper.c:309 > > #9 0x000055c8694b55a1 in heap_toast_insert_or_update (...) at heaptoast.c:333 > > #10 0x000055c8694a8c6c in heap_update (... at heapam.c:3604 > > #11 0x000055c8694a96cb in simple_heap_update (...) at heapam.c:4062 > > #12 0x000055c869555b7b in CatalogTupleUpdate (...) at indexing.c:322 > > #13 0x000055c8695f0725 in EventTriggerOnLogin () at event_trigger.c:957 > > ... > > > > Funnily enough, when Tom Lane was wondering, whether pg_database's toast > > table could pose a risk [1], I came to the conclusion that it's impossible, > > but that was before the login triggers introduction... > > > > [1] https://www.postgresql.org/message-id/1284094.1695479962%40sss.pgh.pa.us > > Thank you for reporting. I'm looking into this... > I wonder if there is a way to avoid toast update given that we don't > touch the toasted field here. Usage of heap_inplace_update() seems appropriate for me here. It avoids trouble with both TOAST and row-level locks. Alexander, could you please recheck this fixes the problem. ------ Regards, Alexander Korotkov
Вложения
Hello Alexander, 05.02.2024 02:51, Alexander Korotkov wrote: > Usage of heap_inplace_update() seems appropriate for me here. It > avoids trouble with both TOAST and row-level locks. Alexander, could > you please recheck this fixes the problem. I've re-tested the last problematic scenario and can confirm that the fix works for it (though it still doesn't prevent the autovacuum issue (with 4b885d01 reverted)), but using heap_inplace_update() was considered risky in a recent discussion: https://www.postgresql.org/message-id/1596629.1698435146%40sss.pgh.pa.us So maybe it's worth to postpone such a fix till that discussion is finished or to look for another approach... Best regards, Alexander
Hi, Alexander! On Mon, Feb 5, 2024 at 7:00 PM Alexander Lakhin <exclusion@gmail.com> wrote: > 05.02.2024 02:51, Alexander Korotkov wrote: > > > Usage of heap_inplace_update() seems appropriate for me here. It > > avoids trouble with both TOAST and row-level locks. Alexander, could > > you please recheck this fixes the problem. > > I've re-tested the last problematic scenario and can confirm that the fix > works for it (though it still doesn't prevent the autovacuum issue (with > 4b885d01 reverted)), but using heap_inplace_update() was considered risky > in a recent discussion: > https://www.postgresql.org/message-id/1596629.1698435146%40sss.pgh.pa.us > So maybe it's worth to postpone such a fix till that discussion is > finished or to look for another approach... Thank you for pointing this out. I don't think there is a particular problem with this use case for the following reasons. 1) Race conditions over pg_database.dathasloginevt are protected with lock tag. 2) Unsetting pg_database.dathasloginevt of old tuple version shouldn't cause a problem. The next tuple version will be updated by further connections. However, I agree that it's better to wait for the discussion you've pointed out before introducing another use case of heap_inplace_update(). ------ Regards, Alexander Korotkov
On Mon, Oct 16, 2023 at 6:36 AM Alexander Korotkov <aekorotkov@gmail.com> wrote: > > > On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote: > > > The attached revision fixes test failures spotted by > > > commitfest.cputube.org. Also, perl scripts passed perltidy. > > You are very fast and sharp eye! > Thank you for fixing the indentation. I just pushed fixes for the rest. I noticed some typos related to this feature. While on this, the event trigger example for C programming language shown in the docs doesn't compile as-is. fmgr.h is needed. Please see the attached patch. [1] 2024-02-06 05:31:24.453 UTC [1401414] LOG: tag LOGIN, event login #include "postgres.h" #include "commands/event_trigger.h" #include "fmgr.h" PG_MODULE_MAGIC; PG_FUNCTION_INFO_V1(onlogin); Datum onlogin(PG_FUNCTION_ARGS) { EventTriggerData *trigdata; if (!CALLED_AS_EVENT_TRIGGER(fcinfo)) /* internal error */ elog(ERROR, "not fired by event trigger manager"); trigdata = (EventTriggerData *) fcinfo->context; elog(LOG, "tag %s, event %s", GetCommandTagName(trigdata->tag), trigdata->event); PG_RETURN_NULL(); } gcc -shared -fPIC -I /home/ubuntu/postgres/pg17/include/server -o login_trigger.so login_trigger.c LOAD '/home/ubuntu/postgres/login_trigger.so'; CREATE FUNCTION onlogin() RETURNS event_trigger AS '/home/ubuntu/postgres/login_trigger' LANGUAGE C; CREATE EVENT TRIGGER onlogin ON login EXECUTE FUNCTION onlogin(); -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com