Обсуждение: On login trigger: take three

Поиск
Список
Период
Сортировка

On login trigger: take three

От
Konstantin Knizhnik
Дата:
Hi hackers,

Recently I have asked once again by one of our customers about login 
trigger in postgres. People are migrating to Postgres from Oracle and  
looking for Postgres analog of this Oracle feature.
This topic is not new:

https://www.postgresql.org/message-id/flat/1570308356720-0.post%40n3.nabble.com#4748bcb0c5fc98cec0a735dbdffb0c68

https://www.postgresql.org/message-id/flat/OSAPR01MB507373499CCCEA00EAE79875FE2D0%40OSAPR01MB5073.jpnprd01.prod.outlook.com#ed50c248be32be6955c385ca67d6cdc1

end even session connect/disconnect hooks were sometimes committed (but 
then reverted).
As far as I understand most of the concerns were related with disconnect 
hook.
Performing some action on session disconnect is actually much more 
complicated than on login.
But customers are not needed it, unlike actions performed at session start.

I wonder if we are really going to make some steps in this directions?
The discussion above was finished with "We haven't rejected the concept 
altogether, AFAICT"

I have tried to resurrect this patch and implement on-connect trigger on 
top of it.
The syntax is almost the same as proposed by Takayuki:

CREATE EVENT TRIGGER mytrigger
AFTER CONNECTION ON mydatabase
EXECUTE {PROCEDURE | FUNCTION} myproc();

I have replaced CONNECT with CONNECTION because last keyword is already 
recognized by Postgres and
make ON clause mandatory to avoid shift-reduce conflicts.

Actually specifying database name is redundant, because we can define 
on-connect trigger only for self database (just because triggers and 
functions are local to the database).
It may be considered as argument against handling session start using 
trigger. But it seems to be the most natural mechanism for users.

On connect trigger can be dropped almost in the same way as normal (on 
relation) trigger, but with specifying name of the database instead of 
relation name:

DROP TRIGGER mytrigger ON mydatabase;

It is possible to define arbitrary number of on-connect triggers with 
different names.

I attached my prototype implementation of this feature.
I just to be sure first that this feature will be interested to community.
If so, I will continue work in it and prepare new version of the patch 
for the commitfest.

Example

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi

čt 3. 9. 2020 v 15:43 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:
Hi hackers,

Recently I have asked once again by one of our customers about login
trigger in postgres. People are migrating to Postgres from Oracle and 
looking for Postgres analog of this Oracle feature.
This topic is not new:

https://www.postgresql.org/message-id/flat/1570308356720-0.post%40n3.nabble.com#4748bcb0c5fc98cec0a735dbdffb0c68
https://www.postgresql.org/message-id/flat/OSAPR01MB507373499CCCEA00EAE79875FE2D0%40OSAPR01MB5073.jpnprd01.prod.outlook.com#ed50c248be32be6955c385ca67d6cdc1

end even session connect/disconnect hooks were sometimes committed (but
then reverted).
As far as I understand most of the concerns were related with disconnect
hook.
Performing some action on session disconnect is actually much more
complicated than on login.
But customers are not needed it, unlike actions performed at session start.

I wonder if we are really going to make some steps in this directions?
The discussion above was finished with "We haven't rejected the concept
altogether, AFAICT"

I have tried to resurrect this patch and implement on-connect trigger on
top of it.
The syntax is almost the same as proposed by Takayuki:

CREATE EVENT TRIGGER mytrigger
AFTER CONNECTION ON mydatabase
EXECUTE {PROCEDURE | FUNCTION} myproc();

I have replaced CONNECT with CONNECTION because last keyword is already
recognized by Postgres and
make ON clause mandatory to avoid shift-reduce conflicts.

Actually specifying database name is redundant, because we can define
on-connect trigger only for self database (just because triggers and
functions are local to the database).
It may be considered as argument against handling session start using
trigger. But it seems to be the most natural mechanism for users.

On connect trigger can be dropped almost in the same way as normal (on
relation) trigger, but with specifying name of the database instead of
relation name:

DROP TRIGGER mytrigger ON mydatabase;

It is possible to define arbitrary number of on-connect triggers with
different names.

I attached my prototype implementation of this feature.
I just to be sure first that this feature will be interested to community.
If so, I will continue work in it and prepare new version of the patch
for the commitfest.


I have a customer that requires this feature too. Now it uses a solution based on dll session autoloading.  Native solution can be great.

+1

Pavel

Example

--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

RE: On login trigger: take three

От
"tsunakawa.takay@fujitsu.com"
Дата:
From: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
> Recently I have asked once again by one of our customers about login trigger in
> postgres. People are migrating to Postgres from Oracle and looking for Postgres
> analog of this Oracle feature.
> This topic is not new:

> I attached my prototype implementation of this feature.
> I just to be sure first that this feature will be interested to community.
> If so, I will continue work in it and prepare new version of the patch for the
> commitfest.

Thanks a lot for taking on this! +10

> It may be considered as argument against handling session start using trigger.
> But it seems to be the most natural mechanism for users.

Yeah, it's natural, just like the Unix shells run some shell scripts in the home directory.


Regards
Takayuki Tsunakawa


Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:
Sorry, attached version of the patch is missing changes in one file. 
Here is it correct patch.


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


Вложения

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 03.09.2020 17:18, Pavel Stehule wrote:
Hi

čt 3. 9. 2020 v 15:43 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:
Hi hackers,

Recently I have asked once again by one of our customers about login
trigger in postgres. People are migrating to Postgres from Oracle and 
looking for Postgres analog of this Oracle feature.
This topic is not new:

https://www.postgresql.org/message-id/flat/1570308356720-0.post%40n3.nabble.com#4748bcb0c5fc98cec0a735dbdffb0c68
https://www.postgresql.org/message-id/flat/OSAPR01MB507373499CCCEA00EAE79875FE2D0%40OSAPR01MB5073.jpnprd01.prod.outlook.com#ed50c248be32be6955c385ca67d6cdc1

end even session connect/disconnect hooks were sometimes committed (but
then reverted).
As far as I understand most of the concerns were related with disconnect
hook.
Performing some action on session disconnect is actually much more
complicated than on login.
But customers are not needed it, unlike actions performed at session start.

I wonder if we are really going to make some steps in this directions?
The discussion above was finished with "We haven't rejected the concept
altogether, AFAICT"

I have tried to resurrect this patch and implement on-connect trigger on
top of it.
The syntax is almost the same as proposed by Takayuki:

CREATE EVENT TRIGGER mytrigger
AFTER CONNECTION ON mydatabase
EXECUTE {PROCEDURE | FUNCTION} myproc();

I have replaced CONNECT with CONNECTION because last keyword is already
recognized by Postgres and
make ON clause mandatory to avoid shift-reduce conflicts.

Actually specifying database name is redundant, because we can define
on-connect trigger only for self database (just because triggers and
functions are local to the database).
It may be considered as argument against handling session start using
trigger. But it seems to be the most natural mechanism for users.

On connect trigger can be dropped almost in the same way as normal (on
relation) trigger, but with specifying name of the database instead of
relation name:

DROP TRIGGER mytrigger ON mydatabase;

It is possible to define arbitrary number of on-connect triggers with
different names.

I attached my prototype implementation of this feature.
I just to be sure first that this feature will be interested to community.
If so, I will continue work in it and prepare new version of the patch
for the commitfest.


I have a customer that requires this feature too. Now it uses a solution based on dll session autoloading.  Native solution can be great.

+1

I realized that on connect trigger should be implemented as EVENT TRIGGER.
So I have reimplemented my patch using event trigger and use session_start even name to make it more consistent with other events.
Now on login triggers can be created in this way:

create table connects(id serial, who text);
create function on_login_proc() returns event_trigger as $$
begin
  insert into connects (who) values (current_user());
  raise notice 'You are welcome!';
end;
$$ language plpgsql;
create event trigger on_login_trigger on session_start execute procedure on_login_proc();
alter event trigger on_login_trigger enable always;



Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi

I am checking last patch, and there are notices

1. disable_session_start_trigger should be SU_BACKEND instead SUSET

2. The documentation should be enhanced - there is not any note about behave when there are unhandled exceptions, about motivation for this event trigger

3. regress tests should be enhanced - the cases with exceptions are not tested

4. This trigger is not executed again after RESET ALL or DISCARD ALL - it can be a problem if somebody wants to use this trigger for initialisation of some session objects with some pooling solutions.

5. The handling errors don't work well for canceling. If event trigger waits for some event, then cancel disallow connect although connected user is superuser

CREATE OR REPLACE FUNCTION on_login_proc2() RETURNS EVENT_TRIGGER AS $$ begin perform pg_sleep(10000); raise notice '%', fx1(100);raise notice 'kuku kuku'; end  $$ language plpgsql;

probably nobody will use pg_sleep in this routine, but there can be wait on some locks ...

Regards

Pavel



Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:

On 14.09.2020 12:44, Pavel Stehule wrote:
> Hi
>
> I am checking last patch, and there are notices
>
> 1. disable_session_start_trigger should be SU_BACKEND instead SUSET
>
> 2. The documentation should be enhanced - there is not any note about 
> behave when there are unhandled exceptions, about motivation for this 
> event trigger
>
> 3. regress tests should be enhanced - the cases with exceptions are 
> not tested
>
> 4. This trigger is not executed again after RESET ALL or DISCARD ALL - 
> it can be a problem if somebody wants to use this trigger for 
> initialisation of some session objects with some pooling solutions.
>
> 5. The handling errors don't work well for canceling. If event trigger 
> waits for some event, then cancel disallow connect although connected 
> user is superuser
>
> CREATE OR REPLACE FUNCTION on_login_proc2() RETURNS EVENT_TRIGGER AS 
> $$ begin perform pg_sleep(10000); raise notice '%', fx1(100);raise 
> notice 'kuku kuku'; end  $$ language plpgsql;
>
> probably nobody will use pg_sleep in this routine, but there can be 
> wait on some locks ...
>
> Regards
>
> Pavel
>
>
>

Hi
Thank you very much for looking at my patch for connection triggers.
I have fixed 1-3 issues in the attached patch.
Concerning 4 and 5 I have some doubts:

4. Should I add some extra GUC to allow firing of session_start trigger 
in case of  RESET ALL or DISCARD ALL ?
Looks like such behavior contradicts with event name "session_start"...
And do we really need it? If some pooler is using RESET ALL/DISCARD ALL 
to emulate session semantic then  most likely it provides way to define 
custom actions which
should be perform for session initialization. As far as I know, for 
example pgbouncer allows do define own on-connect hooks.

5. I do not quite understand your concern. If I define  trigger 
procedure which is  blocked (for example as in your example), then I can 
use pg_cancel_backend to interrupt execution of login trigger and 
superuser can login. What should be changed here?



-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:


po 14. 9. 2020 v 16:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 14.09.2020 12:44, Pavel Stehule wrote:
> Hi
>
> I am checking last patch, and there are notices
>
> 1. disable_session_start_trigger should be SU_BACKEND instead SUSET
>
> 2. The documentation should be enhanced - there is not any note about
> behave when there are unhandled exceptions, about motivation for this
> event trigger
>
> 3. regress tests should be enhanced - the cases with exceptions are
> not tested
>
> 4. This trigger is not executed again after RESET ALL or DISCARD ALL -
> it can be a problem if somebody wants to use this trigger for
> initialisation of some session objects with some pooling solutions.
>
> 5. The handling errors don't work well for canceling. If event trigger
> waits for some event, then cancel disallow connect although connected
> user is superuser
>
> CREATE OR REPLACE FUNCTION on_login_proc2() RETURNS EVENT_TRIGGER AS
> $$ begin perform pg_sleep(10000); raise notice '%', fx1(100);raise
> notice 'kuku kuku'; end  $$ language plpgsql;
>
> probably nobody will use pg_sleep in this routine, but there can be
> wait on some locks ...
>
> Regards
>
> Pavel
>
>
>

Hi
Thank you very much for looking at my patch for connection triggers.
I have fixed 1-3 issues in the attached patch.
Concerning 4 and 5 I have some doubts:

4. Should I add some extra GUC to allow firing of session_start trigger
in case of  RESET ALL or DISCARD ALL ?
Looks like such behavior contradicts with event name "session_start"...
And do we really need it? If some pooler is using RESET ALL/DISCARD ALL
to emulate session semantic then  most likely it provides way to define
custom actions which
should be perform for session initialization. As far as I know, for
example pgbouncer allows do define own on-connect hooks.

If we introduce buildin session trigger , we should to define what is the session. Your design is much more related to the process than to session. So the correct name should be "process_start" trigger, or some should be different. I think there are two different events - process_start, and session_start, and there should be two different event triggers. Maybe the name "session_start" is just ambiguous and should be used with a different name. 
 

5. I do not quite understand your concern. If I define  trigger
procedure which is  blocked (for example as in your example), then I can
use pg_cancel_backend to interrupt execution of login trigger and
superuser can login. What should be changed here?

You cannot run pg_cancel_backend, because you cannot open a new session. There is a cycle.

Regards

Pavel

 



--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 14.09.2020 17:34, Pavel Stehule wrote:
If we introduce buildin session trigger , we should to define what is the session. Your design is much more related to the process than to session. So the correct name should be "process_start" trigger, or some should be different. I think there are two different events - process_start, and session_start, and there should be two different event triggers. Maybe the name "session_start" is just ambiguous and should be used with a different name. 

I agree.
I can rename trigger to backend_start or process_start or whatever else.

 

5. I do not quite understand your concern. If I define  trigger
procedure which is  blocked (for example as in your example), then I can
use pg_cancel_backend to interrupt execution of login trigger and
superuser can login. What should be changed here?

You cannot run pg_cancel_backend, because you cannot open a new session. There is a cycle.

It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC:

psql "dbname=postgres options='-c disable_session_start_trigger=true'"


Re: On login trigger: take three

От
Pavel Stehule
Дата:


po 14. 9. 2020 v 17:53 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 14.09.2020 17:34, Pavel Stehule wrote:
If we introduce buildin session trigger , we should to define what is the session. Your design is much more related to the process than to session. So the correct name should be "process_start" trigger, or some should be different. I think there are two different events - process_start, and session_start, and there should be two different event triggers. Maybe the name "session_start" is just ambiguous and should be used with a different name. 

I agree.
I can rename trigger to backend_start or process_start or whatever else.

Creating a good name can be hard - it is not called for any process - so maybe "user_backend_start" ? 


 

5. I do not quite understand your concern. If I define  trigger
procedure which is  blocked (for example as in your example), then I can
use pg_cancel_backend to interrupt execution of login trigger and
superuser can login. What should be changed here?

You cannot run pg_cancel_backend, because you cannot open a new session. There is a cycle.

It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC:

psql "dbname=postgres options='-c disable_session_start_trigger=true'"

sure, I know. Just this behavior can be a very unpleasant surprise, and my question is if it can be fixed.  Creating custom libpq variables can be the stop for people that use pgAdmin.



Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Tue, Sep 15, 2020 at 2:12 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>>
>> It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC:
>>
>> psql "dbname=postgres options='-c disable_session_start_trigger=true'"
>
>
> sure, I know. Just this behavior can be a very unpleasant surprise, and my question is if it can be fixed.  Creating
customlibpq variables can be the stop for people that use pgAdmin.
 
>

Hi,

I thought in the case of using pgAdmin (assuming you can connect as
superuser to a database, say the default "postgres" maintenance
database, that doesn't have an EVENT TRIGGER defined for the
session_start event) you could issue the query "ALTER SYSTEM SET
disable_session_start_trigger TO true;"  and then reload the
configuration?

Anyway, I am wondering if this patch is still being actively developed/improved?

Regarding the last-posted patch, I'd like to give some feedback. I
found that the documentation part wouldn't build because of errors in
the SGML tags. There are some grammatical errors too, and some minor
inconsistencies with the current documentation, and some descriptions
could be improved. I think that a colon separator should be added to
the NOTICE message for superuser, so it's clear exactly where the text
of the underlying error message starts. Also, I think that
"client_connection" is perhaps a better and more intuitive event name
than "session_start", or the suggested "user_backend_start".
I've therefore attached an updated patch with these suggested minor
improvements, please take a look and see what you think (please
compare with the original patch).

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:

On 04.12.2020 3:50, Greg Nancarrow wrote:
> On Tue, Sep 15, 2020 at 2:12 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>> It is always possible to login by disabling startup triggers using disable_session_start_trigger GUC:
>>>
>>> psql "dbname=postgres options='-c disable_session_start_trigger=true'"
>>
>> sure, I know. Just this behavior can be a very unpleasant surprise, and my question is if it can be fixed.  Creating
customlibpq variables can be the stop for people that use pgAdmin.
 
>>
> Hi,
>
> I thought in the case of using pgAdmin (assuming you can connect as
> superuser to a database, say the default "postgres" maintenance
> database, that doesn't have an EVENT TRIGGER defined for the
> session_start event) you could issue the query "ALTER SYSTEM SET
> disable_session_start_trigger TO true;"  and then reload the
> configuration?
As far as I understand Pavel concern was about the case when superuser 
defines wrong login trigger which prevents login to the system
all user including himself. Right now solution of this problem is to 
include "options='-c disable_session_start_trigger=true'" in connection 
string.
I do not know if it can be done with pgAdmin.
>
> Anyway, I am wondering if this patch is still being actively developed/improved?

I do not know what else has to be improved.
If you, Pavel or anybody else have some suggestions: please let me know.
>
> Regarding the last-posted patch, I'd like to give some feedback. I
> found that the documentation part wouldn't build because of errors in
> the SGML tags. There are some grammatical errors too, and some minor
> inconsistencies with the current documentation, and some descriptions
> could be improved. I think that a colon separator should be added to
> the NOTICE message for superuser, so it's clear exactly where the text
> of the underlying error message starts. Also, I think that
> "client_connection" is perhaps a better and more intuitive event name
> than "session_start", or the suggested "user_backend_start".
> I've therefore attached an updated patch with these suggested minor
> improvements, please take a look and see what you think (please
> compare with the original patch).

Thank you very much for detecting the problems and much more thanks for 
fixing them and providing your version of the patch.
I have nothing against renaming "session_start" to "client_connection".

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company




Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Fri, Dec 4, 2020 at 9:05 PM Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
>
> As far as I understand Pavel concern was about the case when superuser
> defines wrong login trigger which prevents login to the system
> all user including himself. Right now solution of this problem is to
> include "options='-c disable_session_start_trigger=true'" in connection
> string.
> I do not know if it can be done with pgAdmin.
> >

As an event trigger is tied to a particular database, and a GUC is
global to the cluster, as long as there is one database in the cluster
for which an event trigger for the "client_connection" event is NOT
defined (say the default "postgres" maintenance database), then the
superuser can always connect to that database, issue "ALTER SYSTEM SET
disable_client_connection_trigger TO true" and reload the
configuration. I tested this with pgAdmin4 and it worked fine for me,
to allow login to a database for which login was previously prevented
due to a badly-defined logon trigger.

Pavel, is this an acceptable solution or do you still see problems
with this approach?


Regards,
Greg Nancarrow
Fujitsu Australia



Re: On login trigger: take three

От
Pavel Stehule
Дата:


út 8. 12. 2020 v 1:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Fri, Dec 4, 2020 at 9:05 PM Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
>
> As far as I understand Pavel concern was about the case when superuser
> defines wrong login trigger which prevents login to the system
> all user including himself. Right now solution of this problem is to
> include "options='-c disable_session_start_trigger=true'" in connection
> string.
> I do not know if it can be done with pgAdmin.
> >

As an event trigger is tied to a particular database, and a GUC is
global to the cluster, as long as there is one database in the cluster
for which an event trigger for the "client_connection" event is NOT
defined (say the default "postgres" maintenance database), then the
superuser can always connect to that database, issue "ALTER SYSTEM SET
disable_client_connection_trigger TO true" and reload the
configuration. I tested this with pgAdmin4 and it worked fine for me,
to allow login to a database for which login was previously prevented
due to a badly-defined logon trigger.

yes, it can work .. Maybe for this operation only database owner rights should be necessary. The super user is maybe too strong.

There are two maybe generic questions?

1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.

2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections

Regards

Pavel




Pavel, is this an acceptable solution or do you still see problems
with this approach?


Regards,
Greg Nancarrow
Fujitsu Australia

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be
checkedonly by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be
protectionagainst necessity to restart to single mode to repair the event trigger. I think so more generic solution is
betterthan special disable_client_connection_trigger GUC. 
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to
blockconnection to database. Can be interesting, and I am not sure how much work it is to introduce the second event -
session_start.This event should be started after connecting - so the exception there doesn't block connect, and should
bestarted also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This
featureshould not be implemented in first step, but it can be a plan for support pooled connections 
>

I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:


st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

PGC_SU_BACKEND is too strong, there should be PGC_BACKEND if this option can be used by database owner

Pavel


I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

Regards,
Greg Nancarrow
Fujitsu Australia

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi

st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

I see two possible questions?

1. when you introduce this event, then the new hook is useless ?

2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger

Pavel


Regards,
Greg Nancarrow
Fujitsu Australia

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 09.12.2020 15:34, Pavel Stehule wrote:
Hi

st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

I see two possible questions?

1. when you introduce this event, then the new hook is useless ?

2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger

As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.
2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):

pgbench -C -S -T 100 -P 1 -M prepared postgres

without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)

with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)

As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


st 9. 12. 2020 v 14:28 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 09.12.2020 15:34, Pavel Stehule wrote:
Hi

st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

I see two possible questions?

1. when you introduce this event, then the new hook is useless ?

2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger

As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.

ok. I don't like it, but there are redundant hooks (against event triggers) already. 


2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):

pgbench -C -S -T 100 -P 1 -M prepared postgres

without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)

with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)

As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".

This is not the worst case probably. In this patch the StartTransactionCommand is executed on every connection, although it is not necessary - and for most use cases it will not be used.

I did more tests - see attachments and I see a 5% slowdown - I don't think so it is acceptable for this case. This feature is nice, and for some users important, but only really few users can use it.

┌────────────────┬─────────┬────────────┬─────────────┐
│      test      │ WITH LT │ LT ENABLED │ LT DISABLED │
╞════════════════╪═════════╪════════════╪═════════════╡
│ ro_constant_10 │     539 │        877 │         905 │
│ ro_index_10    │     562 │        808 │         855 │
│ ro_constant_50 │     527 │        843 │         863 │
│ ro_index_50    │     544 │        731 │         742 │
└────────────────┴─────────┴────────────┴─────────────┘
(4 rows)


I tested a performance of trigger (results of first column in table):

CREATE OR REPLACE FUNCTION public.foo()
 RETURNS event_trigger
 LANGUAGE plpgsql
AS $function$
begin
  if not pg_has_role(session_user, 'postgres', 'member') then
    raise exception 'you are not super user';
  end if;
end;
$function$;

There is an available snapshot in InitPostgres, and then there is possible to check if for the current database some connect event trigger exists.This can reduce an overhead of this patch, when there are no logon triggers.

I think so implemented and used names are ok, but for this feature the performance impact should be really very minimal.

There is other small issue - missing tab-complete support for CREATE TRIGGER statement in psql

Regards

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 10.12.2020 10:45, Pavel Stehule wrote:


st 9. 12. 2020 v 14:28 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 09.12.2020 15:34, Pavel Stehule wrote:
Hi

st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

I see two possible questions?

1. when you introduce this event, then the new hook is useless ?

2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger

As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.

ok. I don't like it, but there are redundant hooks (against event triggers) already. 


2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):

pgbench -C -S -T 100 -P 1 -M prepared postgres

without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)

with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)

As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".

This is not the worst case probably. In this patch the StartTransactionCommand is executed on every connection, although it is not necessary - and for most use cases it will not be used.

I did more tests - see attachments and I see a 5% slowdown - I don't think so it is acceptable for this case. This feature is nice, and for some users important, but only really few users can use it.

┌────────────────┬─────────┬────────────┬─────────────┐
│      test      │ WITH LT │ LT ENABLED │ LT DISABLED │
╞════════════════╪═════════╪════════════╪═════════════╡
│ ro_constant_10 │     539 │        877 │         905 │
│ ro_index_10    │     562 │        808 │         855 │
│ ro_constant_50 │     527 │        843 │         863 │
│ ro_index_50    │     544 │        731 │         742 │
└────────────────┴─────────┴────────────┴─────────────┘
(4 rows)


I tested a performance of trigger (results of first column in table):

CREATE OR REPLACE FUNCTION public.foo()
 RETURNS event_trigger
 LANGUAGE plpgsql
AS $function$
begin
  if not pg_has_role(session_user, 'postgres', 'member') then
    raise exception 'you are not super user';
  end if;
end;
$function$;

There is an available snapshot in InitPostgres, and then there is possible to check if for the current database some connect event trigger exists.This can reduce an overhead of this patch, when there are no logon triggers.

I think so implemented and used names are ok, but for this feature the performance impact should be really very minimal.

There is other small issue - missing tab-complete support for CREATE TRIGGER statement in psql

Regards

Pavel


Unfortunately I was not able to reproduce your results.
I just tried the case "select 1" because for this trivial query the overhead of session hooks should be the largest.
In my case variation of results was large enough.
Did you try to run test multiple times?

pgbench -j 2 -c 10 --connect -f test-ro.sql -T 60 -n postgres

disable_event_triggers = off
tps = 1812.250463 (including connections establishing)
tps = 2256.285712 (excluding connections establishing)

tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)

tps = 1830.879302 (including connections establishing)
tps = 2279.302553 (excluding connections establishing)


disable_event_triggers = on
tps = 1858.328717 (including connections establishing)
tps = 2313.661689 (excluding connections establishing)

tps = 1832.932960 (including connections establishing)
tps = 2282.074346 (excluding connections establishing)

tps = 1868.908521 (including connections establishing)
tps = 2326.559150 (excluding connections establishing)


I tried to increase run time to 1000 seconds.
Results are the following:

disable_event_triggers = off
tps = 1813.587876 (including connections establishing)
tps = 2257.844703 (excluding connections establishing)

disable_event_triggers = on
tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)

So the difference on this extreme case is 1%.

I tried your suggestion and move client_connection_hook invocation to postinit.c
It slightly improve performance:

tps = 1828.446959 (including connections establishing)
tps = 2276.142972 (excluding connections establishing)

So result is somewhere in the middle, because it allows to eliminate overhead of
starting transaction or taking snapshots but not of lookup through pg_event_trigger table (although it it empty).

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


čt 10. 12. 2020 v 14:03 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 10.12.2020 10:45, Pavel Stehule wrote:


st 9. 12. 2020 v 14:28 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 09.12.2020 15:34, Pavel Stehule wrote:
Hi

st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

I see two possible questions?

1. when you introduce this event, then the new hook is useless ?

2. what is a performance impact for users that want not to use this feature. What is a overhead of EventTriggerOnConnect and is possible to skip this step if database has not any event trigger

As far as I understand this are questions to me rather than to Greg.
1. Do you mean client_connection_hook? It is used to implement this new event type. It can be also used for other purposes.

ok. I don't like it, but there are redundant hooks (against event triggers) already. 


2. Connection overhead is quite large. Supporting on connect hook requires traversal of event trigger relation. But this overhead is negligible comparing with overhead of establishing connection. In any case I did the following test (with local connection):

pgbench -C -S -T 100 -P 1 -M prepared postgres

without this patch:
tps = 424.287889 (including connections establishing)
tps = 952.911068 (excluding connections establishing)

with this patch (but without any connection trigger defined):
tps = 434.642947 (including connections establishing)
tps = 995.525383 (excluding connections establishing)

As you can see - there is almost now different (patched version is even faster, but it seems to be just "white noise".

This is not the worst case probably. In this patch the StartTransactionCommand is executed on every connection, although it is not necessary - and for most use cases it will not be used.

I did more tests - see attachments and I see a 5% slowdown - I don't think so it is acceptable for this case. This feature is nice, and for some users important, but only really few users can use it.

┌────────────────┬─────────┬────────────┬─────────────┐
│      test      │ WITH LT │ LT ENABLED │ LT DISABLED │
╞════════════════╪═════════╪════════════╪═════════════╡
│ ro_constant_10 │     539 │        877 │         905 │
│ ro_index_10    │     562 │        808 │         855 │
│ ro_constant_50 │     527 │        843 │         863 │
│ ro_index_50    │     544 │        731 │         742 │
└────────────────┴─────────┴────────────┴─────────────┘
(4 rows)


I tested a performance of trigger (results of first column in table):

CREATE OR REPLACE FUNCTION public.foo()
 RETURNS event_trigger
 LANGUAGE plpgsql
AS $function$
begin
  if not pg_has_role(session_user, 'postgres', 'member') then
    raise exception 'you are not super user';
  end if;
end;
$function$;

There is an available snapshot in InitPostgres, and then there is possible to check if for the current database some connect event trigger exists.This can reduce an overhead of this patch, when there are no logon triggers.

I think so implemented and used names are ok, but for this feature the performance impact should be really very minimal.

There is other small issue - missing tab-complete support for CREATE TRIGGER statement in psql

Regards

Pavel


Unfortunately I was not able to reproduce your results.
I just tried the case "select 1" because for this trivial query the overhead of session hooks should be the largest.
In my case variation of results was large enough.
Did you try to run test multiple times?

pgbench -j 2 -c 10 --connect -f test-ro.sql -T 60 -n postgres

disable_event_triggers = off
tps = 1812.250463 (including connections establishing)
tps = 2256.285712 (excluding connections establishing)

tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)

tps = 1830.879302 (including connections establishing)
tps = 2279.302553 (excluding connections establishing)


disable_event_triggers = on
tps = 1858.328717 (including connections establishing)
tps = 2313.661689 (excluding connections establishing)

tps = 1832.932960 (including connections establishing)
tps = 2282.074346 (excluding connections establishing)

tps = 1868.908521 (including connections establishing)
tps = 2326.559150 (excluding connections establishing)


I tried to increase run time to 1000 seconds.
Results are the following:

disable_event_triggers = off
tps = 1813.587876 (including connections establishing)
tps = 2257.844703 (excluding connections establishing)

disable_event_triggers = on
tps = 1838.107242 (including connections establishing)
tps = 2288.507668 (excluding connections establishing)

So the difference on this extreme case is 1%.

you have a stronger CPU (2x), and probably you are hitting different bottleneck or maybe some other is wrong. I can recheck it and I can attach profiles.



I tried your suggestion and move client_connection_hook invocation to postinit.c
It slightly improve performance:

tps = 1828.446959 (including connections establishing)
tps = 2276.142972 (excluding connections establishing)

So result is somewhere in the middle, because it allows to eliminate overhead of
starting transaction or taking snapshots but not of lookup through pg_event_trigger table (although it it empty).

My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and  other things.

There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.

Regards

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 10.12.2020 18:12, Pavel Stehule wrote:

My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and  other things.

There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.



Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory  data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.

From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.

Re: On login trigger: take three

От
Pavel Stehule
Дата:


čt 10. 12. 2020 v 16:48 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 10.12.2020 18:12, Pavel Stehule wrote:

My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and  other things.

There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.



Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory  data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.

My idea was a little bit different - just inside process initialization, checking existence of event triggers, and later when it is necessary to start a transaction for trigger execution. This should to reduce useless empty transaction,

I am sure this is a problem on computers with slower CPU s, although I have I7, but because this feature is usually unused, then the performance impact should be minimal every time.

 

From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.

It can work, but this design is not user friendly.  The significant bottleneck should be forking new processes, and check of content some usually very small tables should be invisible. So if there is a possible way to implement some feature that can be enabled by default, then we should go this way. It can be pretty unfriendly if somebody writes a logon trigger and it will not work by default without any warning.

When I tested last patch I found a problem (I have disabled assertions due performance testing)

create role xx login;
alter system set disable_event_triggers to on; -- better should be positive form - enable_event_triggers to on, off
select pg_reload_conf();

psql -U xx

Thread 2.1 "postmaster" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2bff438ec0 (LWP 827497)]
ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
1025 ResourceArrayEnlarge(&(owner->catrefarr));
(gdb) bt
#0  ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
#1  0x00000000008a70f8 in SearchCatCacheInternal (cache=<optimized out>, nkeys=nkeys@entry=1, v1=v1@entry=13836, v2=v2@entry=0,
    v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1273
#2  0x00000000008a7575 in SearchCatCache1 (cache=<optimized out>, v1=v1@entry=13836) at catcache.c:1167
#3  0x00000000008b7f80 in SearchSysCache1 (cacheId=cacheId@entry=21, key1=key1@entry=13836) at syscache.c:1122
#4  0x00000000005939cd in pg_database_ownercheck (db_oid=13836, roleid=16387) at aclchk.c:5114
#5  0x0000000000605b42 in EventTriggersDisabled () at event_trigger.c:650
#6  EventTriggerOnConnect () at event_trigger.c:839
#7  0x00000000007b46d7 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdd6256080, dbname=<optimized out>,
    username=0xf82508 "tom") at postgres.c:4021
#8  0x0000000000741afd in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4490
#9  BackendStartup (port=<optimized out>) at postmaster.c:4212
#10 ServerLoop () at postmaster.c:1729
#11 0x0000000000742881 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xf44d00) at postmaster.c:1401
#12 0x00000000004f1ff7 in main (argc=3, argv=0xf44d00) at main.c:209

I checked profiles, and although there is significant slowdown when there is one login trigger, it was not related to plpgsql.

There is big overhead of

   8,98%  postmaster  postgres              [.] guc_name_compare

Maybe EventTriggerInvoke is not well optimized for very frequent execution. Overhead of plpgsql is less than 0.1%, although there is significant slowdown (when a very simple logon trigger is executed).

Regards

Pavel


 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


čt 10. 12. 2020 v 19:09 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:


čt 10. 12. 2020 v 16:48 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 10.12.2020 18:12, Pavel Stehule wrote:

My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and  other things.

There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.



Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory  data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.

My idea was a little bit different - just inside process initialization, checking existence of event triggers, and later when it is necessary to start a transaction for trigger execution. This should to reduce useless empty transaction,

but this information can be accessed via shared memory if it is necessary. Probably it doesn't need a special hash table.  Maybe a special flag per process.


I am sure this is a problem on computers with slower CPU s, although I have I7, but because this feature is usually unused, then the performance impact should be minimal every time.

 

From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.

It can work, but this design is not user friendly.  The significant bottleneck should be forking new processes, and check of content some usually very small tables should be invisible. So if there is a possible way to implement some feature that can be enabled by default, then we should go this way. It can be pretty unfriendly if somebody writes a logon trigger and it will not work by default without any warning.

When I tested last patch I found a problem (I have disabled assertions due performance testing)

create role xx login;
alter system set disable_event_triggers to on; -- better should be positive form - enable_event_triggers to on, off
select pg_reload_conf();

psql -U xx

Thread 2.1 "postmaster" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2bff438ec0 (LWP 827497)]
ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
1025 ResourceArrayEnlarge(&(owner->catrefarr));
(gdb) bt
#0  ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
#1  0x00000000008a70f8 in SearchCatCacheInternal (cache=<optimized out>, nkeys=nkeys@entry=1, v1=v1@entry=13836, v2=v2@entry=0,
    v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1273
#2  0x00000000008a7575 in SearchCatCache1 (cache=<optimized out>, v1=v1@entry=13836) at catcache.c:1167
#3  0x00000000008b7f80 in SearchSysCache1 (cacheId=cacheId@entry=21, key1=key1@entry=13836) at syscache.c:1122
#4  0x00000000005939cd in pg_database_ownercheck (db_oid=13836, roleid=16387) at aclchk.c:5114
#5  0x0000000000605b42 in EventTriggersDisabled () at event_trigger.c:650
#6  EventTriggerOnConnect () at event_trigger.c:839
#7  0x00000000007b46d7 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdd6256080, dbname=<optimized out>,
    username=0xf82508 "tom") at postgres.c:4021
#8  0x0000000000741afd in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4490
#9  BackendStartup (port=<optimized out>) at postmaster.c:4212
#10 ServerLoop () at postmaster.c:1729
#11 0x0000000000742881 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xf44d00) at postmaster.c:1401
#12 0x00000000004f1ff7 in main (argc=3, argv=0xf44d00) at main.c:209

I checked profiles, and although there is significant slowdown when there is one login trigger, it was not related to plpgsql.

There is big overhead of

   8,98%  postmaster  postgres              [.] guc_name_compare

Maybe EventTriggerInvoke is not well optimized for very frequent execution. Overhead of plpgsql is less than 0.1%, although there is significant slowdown (when a very simple logon trigger is executed).

Regards

Pavel


 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 10.12.2020 21:09, Pavel Stehule wrote:


čt 10. 12. 2020 v 16:48 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 10.12.2020 18:12, Pavel Stehule wrote:

My idea was a little bit different. Inside postinit initialize some global variables with info if there are event triggers or not. And later you can use this variable to start transactions and  other things.

There will be two access to pg_event_trigger, but it can eliminate useless and probably more expensive start_transaction and end_transaction.



Do you mean some variable in shared memory or GUCs?
It was my first idea - to use some flag in shared memory to make it possible fast check that there are not event triggers.
But this flag should be sent per database. May be I missed something, but there is no any per-database shared memory  data structure in Postgres.
Certainly it is possible to organize some hash db->event_info, but it makes this patch several times more complex.

My idea was a little bit different - just inside process initialization, checking existence of event triggers, and later when it is necessary to start a transaction for trigger execution. This should to reduce useless empty transaction,

I am sure this is a problem on computers with slower CPU s, although I have I7, but because this feature is usually unused, then the performance impact should be minimal every time.

Sorry, may be I missed something. But now I completely confused with your idea.
Right now processing of login hooks is done in PostgresMain.
In the previous mail you have suggested to did it in InitPostgres which is invoked from PostgresMain.
So what is the difference (except there open transaction in InitPostgres)?

So what the problem we are trying to solve now?
As far as I understand, your concern is about extra overhead during backend startup needed to check if there on-login triggers defined.
We are not speaking about trigger execution. We mostly worry about applications which are not using triggers at all. But still have to pay some small extra overhead at startup.
This overhead seems to be negligible (1% for dummy connection doing just "select 1"). But taken in account that 99.9999% applications will not use connection triggers,
even very small overhead seems to be not good.

But what can we do?
We do not want to scan pg_event_trigger table on backend startup. But how else we can skip this check, taken in account that
1. Such trigger can be registered at any moment of time
2. Triggers are registered per database, so we can not have just global flag,signaling lack of event triggers.

The only solution I see at this moment is <db,has_event_triggers> hash in shared memory.
But it seems to be overkill from my point of view.

This is why I suggested to have disable_login_event_triggers GUC  set to true by default.



 

From my point of view it is better to have separate GUC disabling just client connection events and switch it on by default.
So only those who need this events with switch it on, other users will not pay any extra cost for it.

It can work, but this design is not user friendly.  The significant bottleneck should be forking new processes, and check of content some usually very small tables should be invisible. So if there is a possible way to implement some feature that can be enabled by default, then we should go this way. It can be pretty unfriendly if somebody writes a logon trigger and it will not work by default without any warning.
Please notice that event triggers are disabled by default.
You need to call "alter event trigger xxx enable always".
May be instead of GUCs we should somehow use ALTER mechanism?
But I do not understand why it is better and how it will help to solve this problem (elimination of exrta overhead when there are not triggers).


When I tested last patch I found a problem (I have disabled assertions due performance testing)

create role xx login;
alter system set disable_event_triggers to on; -- better should be positive form - enable_event_triggers to on, off
select pg_reload_conf();

psql -U xx

Thread 2.1 "postmaster" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2bff438ec0 (LWP 827497)]
ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
1025 ResourceArrayEnlarge(&(owner->catrefarr));
(gdb) bt
#0  ResourceOwnerEnlargeCatCacheRefs (owner=0x0) at resowner.c:1025
#1  0x00000000008a70f8 in SearchCatCacheInternal (cache=<optimized out>, nkeys=nkeys@entry=1, v1=v1@entry=13836, v2=v2@entry=0,
    v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1273
#2  0x00000000008a7575 in SearchCatCache1 (cache=<optimized out>, v1=v1@entry=13836) at catcache.c:1167
#3  0x00000000008b7f80 in SearchSysCache1 (cacheId=cacheId@entry=21, key1=key1@entry=13836) at syscache.c:1122
#4  0x00000000005939cd in pg_database_ownercheck (db_oid=13836, roleid=16387) at aclchk.c:5114
#5  0x0000000000605b42 in EventTriggersDisabled () at event_trigger.c:650
#6  EventTriggerOnConnect () at event_trigger.c:839
#7  0x00000000007b46d7 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdd6256080, dbname=<optimized out>,
    username=0xf82508 "tom") at postgres.c:4021
#8  0x0000000000741afd in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4490
#9  BackendStartup (port=<optimized out>) at postmaster.c:4212
#10 ServerLoop () at postmaster.c:1729
#11 0x0000000000742881 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0xf44d00) at postmaster.c:1401
#12 0x00000000004f1ff7 in main (argc=3, argv=0xf44d00) at main.c:209



pg_database_ownercheck can not be called outside transaction.
I can split replace call of EventTriggersDisabled in EventTriggerOnConnect with two separate checks:
disable_event_triggers - before start of transaction
and pg_database_ownercheck(MyDatabaseId, GetUserId()) inside transaction.

But I wonder if we need to check database ownership in this place at all?
May be just allow to alter disable_event_triggers for superusers?

 

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 09.12.2020 15:24, Pavel Stehule wrote:


st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

PGC_SU_BACKEND is too strong, there should be PGC_BACKEND if this option can be used by database owner

Pavel


I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

Regards,
Greg Nancarrow
Fujitsu Australia

It seems to me that current implementation of EventTriggersDisabled:


 /*
 * Indicates whether all event triggers are currently disabled
 * for the current user.
 * Event triggers are disabled when configuration parameter
 * "disable_event_triggers" is true, and the current user
 * is the database owner or has superuser privileges.
 */
static inline bool
EventTriggersDisabled(void)
{
    return (disable_event_triggers && pg_database_ownercheck(MyDatabaseId, GetUserId()));
}


is not correct. It makes it not possible to superuser to disable triggers for all users.
Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

So I think that EventTriggersDisabled function is not needed and can be replaced just with disable_event_triggers GUC check.
And this option should be defined with PGC_SU_BACKEND




-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


pá 11. 12. 2020 v 16:07 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 09.12.2020 15:24, Pavel Stehule wrote:


st 9. 12. 2020 v 13:17 odesílatel Greg Nancarrow <gregn4422@gmail.com> napsal:
On Tue, Dec 8, 2020 at 3:26 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> There are two maybe generic questions?
>
> 1. Maybe we can introduce more generic GUC for all event triggers like disable_event_triggers? This GUC can be checked only by the database owner or super user. It can be an alternative ALTER TABLE DISABLE TRIGGER ALL. It can be protection against necessity to restart to single mode to repair the event trigger. I think so more generic solution is better than special disable_client_connection_trigger GUC.
>
> 2. I have no objection against client_connection. It is probably better for the mentioned purpose - possibility to block connection to database. Can be interesting, and I am not sure how much work it is to introduce the second event - session_start. This event should be started after connecting - so the exception there doesn't block connect, and should be started also after the new statement "DISCARD SESSION", that will be started automatically after DISCARD ALL.  This feature should not be implemented in first step, but it can be a plan for support pooled connections
>

PGC_SU_BACKEND is too strong, there should be PGC_BACKEND if this option can be used by database owner

Pavel


I've created a separate patch to address question (1), rather than
include it in the main patch, which I've adjusted accordingly. I'll
leave question (2) until another time, as you suggest.
See the attached patches.

Regards,
Greg Nancarrow
Fujitsu Australia

It seems to me that current implementation of EventTriggersDisabled:


 /*
 * Indicates whether all event triggers are currently disabled
 * for the current user.
 * Event triggers are disabled when configuration parameter
 * "disable_event_triggers" is true, and the current user
 * is the database owner or has superuser privileges.
 */
static inline bool
EventTriggersDisabled(void)
{
    return (disable_event_triggers && pg_database_ownercheck(MyDatabaseId, GetUserId()));
}


is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.



So I think that EventTriggersDisabled function is not needed and can be replaced just with disable_event_triggers GUC check.
And this option should be defined with PGC_SU_BACKEND




-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?


Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 11.12.2020 19:27, Pavel Stehule wrote:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

So what's next?
I see three options:

1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are  disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).

2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".

3. Implement some mechanism for caching presence of event triggers in shared memory.


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 19:27, Pavel Stehule wrote:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

So what's next?
I see three options:

1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are  disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).

2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".

3. Implement some mechanism for caching presence of event triggers in shared memory.

@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.

Regards

Pavel



-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 15.12.2020 16:18, Pavel Stehule wrote:


út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 19:27, Pavel Stehule wrote:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

So what's next?
I see three options:

1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are  disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).

2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".

3. Implement some mechanism for caching presence of event triggers in shared memory.

@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.


Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...

I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?






Regards

Pavel



-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


út 15. 12. 2020 v 15:06 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 15.12.2020 16:18, Pavel Stehule wrote:


út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 19:27, Pavel Stehule wrote:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

So what's next?
I see three options:

1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are  disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).

2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".

3. Implement some mechanism for caching presence of event triggers in shared memory.

@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.


Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...

I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?

 This is the worst case but tested on a not too bad CPU (i7). In virtual environments with overloaded CPU the effect can be worse.

 






Regards

Pavel



-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


út 15. 12. 2020 v 15:06 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 15.12.2020 16:18, Pavel Stehule wrote:


út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 19:27, Pavel Stehule wrote:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

So what's next?
I see three options:

1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are  disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).

2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".

3. Implement some mechanism for caching presence of event triggers in shared memory.

@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.


Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...

I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?



I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column.  Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.

What do you think about it?

Pavel
 




Regards

Pavel



-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 15.12.2020 18:25, Pavel Stehule wrote:


út 15. 12. 2020 v 15:06 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 15.12.2020 16:18, Pavel Stehule wrote:


út 15. 12. 2020 v 14:12 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 19:27, Pavel Stehule wrote:


pá 11. 12. 2020 v 17:05 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 11.12.2020 18:40, Pavel Stehule wrote:

is not correct. It makes it not possible to superuser to disable triggers for all users.

pg_database_ownercheck returns true for superuser always.

Sorry, but I consider different case: when normal user is connected to the database.
In this case pg_database_ownercheck returns false and trigger is not disabled, isn't it?

My idea was to reduce necessary rights to database owners.  But you have a true, so only superuser can create event trigger, so this feature cannot be used in DBaaS environments, and then my original idea was wrong.



Also GUCs are not associated with any database. So I do not understand why  this check of database ownership is relevant at all?

What kind of protection violation we want to prevent?

It seems to be obvious that normal user should not be able to prevent trigger execution because this triggers may be used to enforce some security policies.
If trigger was created by user itself, then it can drop or disable it using ALTER statement. GUC is not needed for it.

when you cannot connect to the database, then you cannot do ALTER. In DBaaS environments lot of users has not superuser rights.


But only superusers can set login triggers, right?
So only superuser can make a mistake in this trigger. But he have enough rights to recover this error. Normal users are not able to define on connection triggers and
should not have rights to disable them.

yes, it is true

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

So what's next?
I see three options:

1. Do not introduce GUC for disabling all event triggers (especially taken in account that them are  disabled by default).
Return back to the patch on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch with "disable_client_connection_trigger"
and make it true by default (to eliminate any overhead for users which are not using on logintriggers).

2. Have two GUCS: "disable_client_connection_trigger" and "disable_event_triggers".

3. Implement some mechanism for caching presence of event triggers in shared memory.

@3 is the best design (the things should work well by default), @2 is a little bit chaotic and @1 looks like a workaround.


Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...

I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?



I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column.  Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.

What do you think about it?

I like this approach more than implementation of shared hash.
But still I have some concerns:

1. pg_database table format has to be changed. Certainly it is not something  completely unacceptable. But IMHO we should try to avoid
modification of such commonly used catalog tables as much as possible.

2. It is not so easy to maintain this flag. There can be multiple on-login triggers defined. If such trigger is dropped, we can not just clear this flag.
We should check if other triggers exist. Now assume that there are two triggers and two concurrent transactions dropping each one.
According to their snapshots them do not see changes made by other transaction. So them remove both triggers but didn't clear the flag.
Certainly we can use counter instead of flag. But I am not sure that their will be not other problems with maintaining counter.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:

Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...

I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?



I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column.  Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.

What do you think about it?

I like this approach more than implementation of shared hash.
But still I have some concerns:

1. pg_database table format has to be changed. Certainly it is not something  completely unacceptable. But IMHO we should try to avoid
modification of such commonly used catalog tables as much as possible.

yes, I know. Unfortunately I  don't see any other and correct solution. There should be more wide discussion before this work about this topic. On second hand, this change should not break anything (if this new field will be placed on as the last field). The logon trigger really looks like a database trigger - so I think so this semantic is correct. I have no idea if it is acceptable for committers :-/. I hope so. 

The fact that the existence of a logon trigger can be visible from a shared database object can be an interesting feature too.


2. It is not so easy to maintain this flag. There can be multiple on-login triggers defined. If such trigger is dropped, we can not just clear this flag.
We should check if other triggers exist. Now assume that there are two triggers and two concurrent transactions dropping each one.
According to their snapshots them do not see changes made by other transaction. So them remove both triggers but didn't clear the flag.
Certainly we can use counter instead of flag. But I am not sure that their will be not other problems with maintaining counter.

I don't think it is necessary.  My opinion is not too strong, but if pg_class doesn't need to hold a number of triggers, then I think so pg_database doesn't need to hold this number too.


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 15.12.2020 20:16, Pavel Stehule wrote:

Please notice that we still need GUC to disable on-login triggers: to make it possible for superuser who did mistake and defined incorrect on-login trigger to
login to the system.
Do we need GUC to disable all other event triggers? May be I am wrong, but I do not see much need in such GUC: error in any of such event triggers is non fatal
and can be easily reverted.
So the only question is whether "disable_client_connection_trigger" should be true by default or not...

I agree with you that @2 is a little bit chaotic and @1 looks like a workaround.
But from my point of view @3 is not the best solution but overkill: maintaining yet another shared hash just to save few milliseconds on login seems to be too high price.
Actually there are many things which are loaded by new backend from the database on start: for example - catalog.
This is why launch of new backend is an expensive operation.
Certainly if we execute "select 1", then system catalog is not needed...
But does anybody start new backend just to execute "select 1" and exit?



I understand so the implementation of a new shared cache can be a lot of work. The best way is enhancing pg_database about one column with information about the login triggers (dathaslogontriggers). In init time these data are in syscache, and can be easily checked. Some like pg_attribute have an atthasdef column.  Same fields has pg_class - relhasrules, relhastriggers, ... Then the overhead of this design should be really zero.

What do you think about it?

I like this approach more than implementation of shared hash.
But still I have some concerns:

1. pg_database table format has to be changed. Certainly it is not something  completely unacceptable. But IMHO we should try to avoid
modification of such commonly used catalog tables as much as possible.

yes, I know. Unfortunately I  don't see any other and correct solution. There should be more wide discussion before this work about this topic. On second hand, this change should not break anything (if this new field will be placed on as the last field). The logon trigger really looks like a database trigger - so I think so this semantic is correct. I have no idea if it is acceptable for committers :-/. I hope so. 

The fact that the existence of a logon trigger can be visible from a shared database object can be an interesting feature too.


2. It is not so easy to maintain this flag. There can be multiple on-login triggers defined. If such trigger is dropped, we can not just clear this flag.
We should check if other triggers exist. Now assume that there are two triggers and two concurrent transactions dropping each one.
According to their snapshots them do not see changes made by other transaction. So them remove both triggers but didn't clear the flag.
Certainly we can use counter instead of flag. But I am not sure that their will be not other problems with maintaining counter.

I don't think it is necessary.  My opinion is not too strong, but if pg_class doesn't need to hold a number of triggers, then I think so pg_database doesn't need to hold this number too.

Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).



-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

Regards

Pavel





-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Zhihong Yu
Дата:
Hi,
For EventTriggerOnConnect():

+           PG_CATCH();
+           {
...
+               AbortCurrentTransaction();
+               return;

Should runlist be freed in the catch block ?

+           gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, this parameter can be used to disable trigger activation and provide access to the database."),

I think the text should be on two lines (current line too long).

Cheers

Re: On login trigger: take three

От
Pavel Stehule
Дата:


st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

I thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.

Pavel


Regards

Pavel





-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 17.12.2020 9:31, Pavel Stehule wrote:


st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

I thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.




I prefer to implement different approach: unset dathaslogontriggers  flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.

Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.


New version of the patch is attached.




-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:
Hi,

On 17.12.2020 3:31, Zhihong Yu wrote:
> Hi,
> For EventTriggerOnConnect():
>
> +           PG_CATCH();
> +           {
> ...
> +               AbortCurrentTransaction();
> +               return;
>
> Should runlist be freed in the catch block ?

No need: it is allocated in transaction memory context and removed on 
transaction abort.

>
> +           gettext_noop("In case of errors in the ON 
> client_connection EVENT TRIGGER procedure, this parameter can be used 
> to disable trigger activation and provide access to the database."),
>
> I think the text should be on two lines (current line too long).

Thank you, fixed.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company




Re: On login trigger: take three

От
Pavel Stehule
Дата:


čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 17.12.2020 9:31, Pavel Stehule wrote:


st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

I thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.




I prefer to implement different approach: unset dathaslogontriggers  flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.

Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.


New version of the patch is attached.

yes, it can work

Regards

Pavel







-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Pavel Stehule
Дата:


čt 17. 12. 2020 v 19:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:


čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 17.12.2020 9:31, Pavel Stehule wrote:


st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

I thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.




I prefer to implement different approach: unset dathaslogontriggers  flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.

Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.


New version of the patch is attached.

yes, it can work

for complete review the new field in pg_doc should be documented

Regards

Pavel


Regards

Pavel







-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 20.12.2020 10:04, Pavel Stehule wrote:


čt 17. 12. 2020 v 19:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:


čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 17.12.2020 9:31, Pavel Stehule wrote:


st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

I thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.




I prefer to implement different approach: unset dathaslogontriggers  flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.

Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.


New version of the patch is attached.

yes, it can work

for complete review the new field in pg_doc should be documented


Done.
Also I fixed some examples in documentation which involves pg_database.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi

po 21. 12. 2020 v 11:06 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 20.12.2020 10:04, Pavel Stehule wrote:


čt 17. 12. 2020 v 19:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:


čt 17. 12. 2020 v 14:04 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 17.12.2020 9:31, Pavel Stehule wrote:


st 16. 12. 2020 v 20:38 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Attached please find new versoin of the patch based on on_connect_event_trigger_WITH_SUGGESTED_UPDATES.patch
So there is still only  "disable_client_connection_trigger" GUC? because we need possibility to disable client connect triggers and there is no such need for other event types.

As you suggested  have added "dathaslogontriggers" flag which is set when client connection trigger is created.
This flag is never cleaned (to avoid visibility issues mentioned in my previous mail).

This is much better - I don't see any slowdown when logon trigger is not defined

I did some benchmarks and looks so starting language handler is relatively expensive - it is about 25% and starting handler like event trigger has about 35%. But this problem can be solved later and elsewhere.

I prefer the inverse form of disable_connection_trigger. Almost all GUC are in positive form - so enable_connection_triggger is better

I don't think so current handling dathaslogontriggers is good for production usage. The protection against race condition can be solved by lock on pg_event_trigger

I thought about it, and probably the counter of connect triggers will be better there. The implementation will be simpler and more robust.




I prefer to implement different approach: unset dathaslogontriggers  flag in event trigger itself when no triggers are returned by EventTriggerCommonSetup.
I am using double checking to prevent race condition.
The main reason for this approach is that dropping of triggers is not done in event_trigger.c: it is done by generic code for all resources.
It seems to be there is no right place where decrementing of trigger counters can be inserted.
Also I prefer to keep all logon-trigger specific code in one file.

Also, as you suggested, I renamed disable_connection_trigger to enable_connection_trigger.


New version of the patch is attached.

yes, it can work

for complete review the new field in pg_doc should be documented


Done.
Also I fixed some examples in documentation which involves pg_database.

regress tests fails

     sysviews                     ... FAILED      112 ms
test event_trigger                ... FAILED (test process exited with exit code 2)      447 ms
test fast_default                 ... FAILED      392 ms
test stats                        ... FAILED      626 ms
============== shutting down postmaster               ==============

Regards

Pavel


-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 22.12.2020 12:25, Pavel Stehule wrote:

regress tests fails

     sysviews                     ... FAILED      112 ms
test event_trigger                ... FAILED (test process exited with exit code 2)      447 ms
test fast_default                 ... FAILED      392 ms
test stats                        ... FAILED      626 ms
============== shutting down postmaster               ==============


Sorry, fixed.


--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:


út 22. 12. 2020 v 12:42 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 22.12.2020 12:25, Pavel Stehule wrote:

regress tests fails

     sysviews                     ... FAILED      112 ms
test event_trigger                ... FAILED (test process exited with exit code 2)      447 ms
test fast_default                 ... FAILED      392 ms
test stats                        ... FAILED      626 ms
============== shutting down postmaster               ==============


Sorry, fixed.

no problem

I had to fix doc

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6ff783273f..7aded1848f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1008,7 +1008,7 @@ include_dir 'conf.d'
         trigger when a client connects. This parameter is switched on by default.
         Errors in trigger code can prevent user to login to the system.
         I this case disabling this parameter in connection string can solve the problem:
-        <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'".
+        <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
        </para>
       </listitem>
      </varlistentry>

I am thinking again about enable_client_connection_trigger, and although it can look useless (because error is ignored for superuser), it can be useful for some debugging and administration purposes. Probably we don't want to start the client_connection trigger from backup tools, maybe from some monitoring tools. Maybe the possibility to set this GUC can be dedicated to some special role (like pg_signal_backend).

+     <varlistentry id="guc-enable-client-connection-trigger" xreflabel="enable_client_connection_trigger">
+      <term><varname>enable_client_connection_trigger</varname> (<type>boolean</type>)
+      <indexterm>
+       <primary><varname>enable_client_connection_trigger</varname> configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Enables firing the <literal>client_connection</literal>
+        trigger when a client connects. This parameter is switched on by default.
+        Errors in trigger code can prevent user to login to the system.
+        I this case disabling this parameter in connection string can solve the problem:
+        <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
+       </para>
+      </listitem>
+     </varlistentry>

There should be note, so only superuser can change this value

There is should be tab-complete support

diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 3a43c09bf6..08f00d8fc4 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2970,7 +2970,8 @@ psql_completion(const char *text, int start, int end)
        COMPLETE_WITH("ON");
    /* Complete CREATE EVENT TRIGGER <name> ON with event_type */
    else if (Matches("CREATE", "EVENT", "TRIGGER", MatchAny, "ON"))
-       COMPLETE_WITH("ddl_command_start", "ddl_command_end", "sql_drop");
+       COMPLETE_WITH("ddl_command_start", "ddl_command_end",
+                     "client_connection", "sql_drop");
 
    /*
     * Complete CREATE EVENT TRIGGER <name> ON <event_type>.  EXECUTE FUNCTION

Regards

Pavel





--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:


On 22.12.2020 21:19, Pavel Stehule wrote:


út 22. 12. 2020 v 12:42 odesílatel Konstantin Knizhnik <k.knizhnik@postgrespro.ru> napsal:


On 22.12.2020 12:25, Pavel Stehule wrote:

regress tests fails

     sysviews                     ... FAILED      112 ms
test event_trigger                ... FAILED (test process exited with exit code 2)      447 ms
test fast_default                 ... FAILED      392 ms
test stats                        ... FAILED      626 ms
============== shutting down postmaster               ==============


Sorry, fixed.

no problem

I had to fix doc

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6ff783273f..7aded1848f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1008,7 +1008,7 @@ include_dir 'conf.d'
         trigger when a client connects. This parameter is switched on by default.
         Errors in trigger code can prevent user to login to the system.
         I this case disabling this parameter in connection string can solve the problem:
-        <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'".
+        <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
        </para>
       </listitem>
      </varlistentry>

I am thinking again about enable_client_connection_trigger, and although it can look useless (because error is ignored for superuser), it can be useful for some debugging and administration purposes. Probably we don't want to start the client_connection trigger from backup tools, maybe from some monitoring tools. Maybe the possibility to set this GUC can be dedicated to some special role (like pg_signal_backend).

+     <varlistentry id="guc-enable-client-connection-trigger" xreflabel="enable_client_connection_trigger">
+      <term><varname>enable_client_connection_trigger</varname> (<type>boolean</type>)
+      <indexterm>
+       <primary><varname>enable_client_connection_trigger</varname> configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Enables firing the <literal>client_connection</literal>
+        trigger when a client connects. This parameter is switched on by default.
+        Errors in trigger code can prevent user to login to the system.
+        I this case disabling this parameter in connection string can solve the problem:
+        <literal>psql "dbname=postgres options='-c enable_client_connection_trigger=false'"</literal>.
+       </para>
+      </listitem>
+     </varlistentry>

There should be note, so only superuser can change this value

There is should be tab-complete support

diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 3a43c09bf6..08f00d8fc4 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2970,7 +2970,8 @@ psql_completion(const char *text, int start, int end)
        COMPLETE_WITH("ON");
    /* Complete CREATE EVENT TRIGGER <name> ON with event_type */
    else if (Matches("CREATE", "EVENT", "TRIGGER", MatchAny, "ON"))
-       COMPLETE_WITH("ddl_command_start", "ddl_command_end", "sql_drop");
+       COMPLETE_WITH("ddl_command_start", "ddl_command_end",
+                     "client_connection", "sql_drop");
 
    /*
     * Complete CREATE EVENT TRIGGER <name> ON <event_type>.  EXECUTE FUNCTION

Thank you.
I have applied all your fixes in on_connect_event_trigger-12.patch.

Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
In this case GUC is most convenient way to do it.



Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi


Thank you.
I have applied all your fixes in on_connect_event_trigger-12.patch.

Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
In this case GUC is most convenient way to do it.


There was typo in patch

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f810789..8861f1b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1,4 +1,4 @@
-<!-- doc/src/sgml/config.sgml -->
+\<!-- doc/src/sgml/config.sgml -->
 
I have not any objections against functionality or design. I tested the performance, and there are no negative impacts when this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when this trigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overhead can be ignored.

* make without warnings
* make check-world passed
* doc build passed

Possible ToDo:

The documentation can contain a note so usage connect triggers in environments with short life sessions and very short fast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initialization of PLpgSQL engine.

I'll mark this patch as ready for committers

Regards

Pavel




Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 
Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:


so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Hi


Thank you.
I have applied all your fixes in on_connect_event_trigger-12.patch.

Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
In this case GUC is most convenient way to do it.


There was typo in patch

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f810789..8861f1b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1,4 +1,4 @@
-<!-- doc/src/sgml/config.sgml -->
+\<!-- doc/src/sgml/config.sgml -->
 
I have not any objections against functionality or design. I tested the performance, and there are no negative impacts when this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when this trigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overhead can be ignored.

* make without warnings
* make check-world passed
* doc build passed

Possible ToDo:

The documentation can contain a note so usage connect triggers in environments with short life sessions and very short fast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initialization of PLpgSQL engine.

I'll mark this patch as ready for committers

looks so this patch has not entry in commitfestapp 2021-01

Regards

Pavel


Regards

Pavel




Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Re: On login trigger: take three

От
Masahiko Sawada
Дата:
On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
>
> so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>>
>> Hi
>>
>>
>>> Thank you.
>>> I have applied all your fixes in on_connect_event_trigger-12.patch.
>>>
>>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest
wayto disable login triggers in case 
>>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using
"ALTEREVENT TRIGGER DISABLE". 
>>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And
youwant temporary disable them all, for example for testing purposes. 
>>> In this case GUC is most convenient way to do it.
>>>
>>
>> There was typo in patch
>>
>> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
>> index f810789..8861f1b 100644
>> --- a/doc/src/sgml/config.sgml
>> +++ b/doc/src/sgml/config.sgml
>> @@ -1,4 +1,4 @@
>> -<!-- doc/src/sgml/config.sgml -->
>> +\<!-- doc/src/sgml/config.sgml -->
>>
>> I have not any objections against functionality or design. I tested the performance, and there are no negative
impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when
thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this
overheadcan be ignored. 
>>
>> * make without warnings
>> * make check-world passed
>> * doc build passed
>>
>> Possible ToDo:
>>
>> The documentation can contain a note so usage connect triggers in environments with short life sessions and very
shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of
initializationof PLpgSQL engine. 
>>
>> I'll mark this patch as ready for committers
>
>
> looks so this patch has not entry in commitfestapp 2021-01
>

Yeah, please register this patch before the next CommitFest[1] starts,
2021-01-01 AoE[2].

Regards,

[1] https://commitfest.postgresql.org/31/
[2] https://en.wikipedia.org/wiki/Anywhere_on_Earth

--
Masahiko Sawada
EnterpriseDB:  https://www.enterprisedb.com/



Re: On login trigger: take three

От
Amit Kapila
Дата:
On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> >
> >
> >
> > so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
> >>
> >> Hi
> >>
> >>
> >>> Thank you.
> >>> I have applied all your fixes in on_connect_event_trigger-12.patch.
> >>>
> >>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest
wayto disable login triggers in case 
> >>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using
"ALTEREVENT TRIGGER DISABLE". 
> >>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers.
Andyou want temporary disable them all, for example for testing purposes. 
> >>> In this case GUC is most convenient way to do it.
> >>>
> >>
> >> There was typo in patch
> >>
> >> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
> >> index f810789..8861f1b 100644
> >> --- a/doc/src/sgml/config.sgml
> >> +++ b/doc/src/sgml/config.sgml
> >> @@ -1,4 +1,4 @@
> >> -<!-- doc/src/sgml/config.sgml -->
> >> +\<!-- doc/src/sgml/config.sgml -->
> >>
> >> I have not any objections against functionality or design. I tested the performance, and there are no negative
impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when
thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this
overheadcan be ignored. 
> >>
> >> * make without warnings
> >> * make check-world passed
> >> * doc build passed
> >>
> >> Possible ToDo:
> >>
> >> The documentation can contain a note so usage connect triggers in environments with short life sessions and very
shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of
initializationof PLpgSQL engine. 
> >>
> >> I'll mark this patch as ready for committers
> >
> >
> > looks so this patch has not entry in commitfestapp 2021-01
> >
>
> Yeah, please register this patch before the next CommitFest[1] starts,
> 2021-01-01 AoE[2].
>

Konstantin, did you register this patch in any CF? Even though the
reviewer seems to be happy with the patch, I am afraid that we might
lose track of this unless we register it.

--
With Regards,
Amit Kapila.



Re: On login trigger: take three

От
Amit Kapila
Дата:
On Thu, Jan 28, 2021 at 8:17 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> > >
> > >
> > >
> > > so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
> > >>
> > >> Hi
> > >>
> > >>
> > >>> Thank you.
> > >>> I have applied all your fixes in on_connect_event_trigger-12.patch.
> > >>>
> > >>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and
simplestway to disable login triggers in case 
> > >>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using
"ALTEREVENT TRIGGER DISABLE". 
> > >>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers.
Andyou want temporary disable them all, for example for testing purposes. 
> > >>> In this case GUC is most convenient way to do it.
> > >>>
> > >>
> > >> There was typo in patch
> > >>
> > >> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
> > >> index f810789..8861f1b 100644
> > >> --- a/doc/src/sgml/config.sgml
> > >> +++ b/doc/src/sgml/config.sgml
> > >> @@ -1,4 +1,4 @@
> > >> -<!-- doc/src/sgml/config.sgml -->
> > >> +\<!-- doc/src/sgml/config.sgml -->
> > >>
> > >> I have not any objections against functionality or design. I tested the performance, and there are no negative
impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when
thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this
overheadcan be ignored. 
> > >>
> > >> * make without warnings
> > >> * make check-world passed
> > >> * doc build passed
> > >>
> > >> Possible ToDo:
> > >>
> > >> The documentation can contain a note so usage connect triggers in environments with short life sessions and very
shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of
initializationof PLpgSQL engine. 
> > >>
> > >> I'll mark this patch as ready for committers
> > >
> > >
> > > looks so this patch has not entry in commitfestapp 2021-01
> > >
> >
> > Yeah, please register this patch before the next CommitFest[1] starts,
> > 2021-01-01 AoE[2].
> >
>
> Konstantin, did you register this patch in any CF? Even though the
> reviewer seems to be happy with the patch, I am afraid that we might
> lose track of this unless we register it.
>

I see the CF entry (https://commitfest.postgresql.org/31/2900/) for
this work. Sorry for the noise.

--
With Regards,
Amit Kapila.



Re: On login trigger: take three

От
Konstantin Knizhnik
Дата:

On 28.01.2021 5:47, Amit Kapila wrote:
> On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>>
>>>
>>> so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>>>> Hi
>>>>
>>>>
>>>>> Thank you.
>>>>> I have applied all your fixes in on_connect_event_trigger-12.patch.
>>>>>
>>>>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest
wayto disable login triggers in case
 
>>>>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using
"ALTEREVENT TRIGGER DISABLE".
 
>>>>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers.
Andyou want temporary disable them all, for example for testing purposes.
 
>>>>> In this case GUC is most convenient way to do it.
>>>>>
>>>> There was typo in patch
>>>>
>>>> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
>>>> index f810789..8861f1b 100644
>>>> --- a/doc/src/sgml/config.sgml
>>>> +++ b/doc/src/sgml/config.sgml
>>>> @@ -1,4 +1,4 @@
>>>> -<!-- doc/src/sgml/config.sgml -->
>>>> +\<!-- doc/src/sgml/config.sgml -->
>>>>
>>>> I have not any objections against functionality or design. I tested the performance, and there are no negative
impactswhen this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when
thistrigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this
overheadcan be ignored.
 
>>>>
>>>> * make without warnings
>>>> * make check-world passed
>>>> * doc build passed
>>>>
>>>> Possible ToDo:
>>>>
>>>> The documentation can contain a note so usage connect triggers in environments with short life sessions and very
shortfast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of
initializationof PLpgSQL engine.
 
>>>>
>>>> I'll mark this patch as ready for committers
>>>
>>> looks so this patch has not entry in commitfestapp 2021-01
>>>
>> Yeah, please register this patch before the next CommitFest[1] starts,
>> 2021-01-01 AoE[2].
>>
> Konstantin, did you register this patch in any CF? Even though the
> reviewer seems to be happy with the patch, I am afraid that we might
> lose track of this unless we register it.
>
Yes, certainly:
https://commitfest.postgresql.org/31/2900/

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company




Re[3]: On login trigger: take three

От
Ivan Panchenko
Дата:
Hi,
I have upgraded the patch for the 14th version.

Вторник, 16 марта 2021, 14:32 +03:00 от Ivan Panchenko <wao@mail.ru>:
 
Hi,

Thank you, Konstantin, for this very good feature with numerous use cases.
Please find the modified patch attached.

I’ve added the ‘enable_client_connection_trigger’ GUC to the sample config file and also an additional example page to the docs.
Check world has passed and it is ready for committer.
 
Четверг, 28 января 2021, 12:04 +03:00 от Konstantin Knizhnik <k.knizhnik@postgrespro.ru>:
 


On 28.01.2021 5:47, Amit Kapila wrote:
> On Mon, Dec 28, 2020 at 5:46 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> On Sat, Dec 26, 2020 at 4:04 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>>
>>>
>>> so 26. 12. 2020 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>>>> Hi
>>>>
>>>>
>>>>> Thank you.
>>>>> I have applied all your fixes in on_connect_event_trigger-12.patch.
>>>>>
>>>>> Concerning enable_client_connection_trigger GUC, I think that it is really useful: it is the fastest and simplest way to disable login triggers in case
>>>>> of some problems with them (not only for superuser itself, but for all users). Yes, it can be also done using "ALTER EVENT TRIGGER DISABLE".
>>>>> But assume that you have a lot of databases with different login policies enforced by on-login event triggers. And you want temporary disable them all, for example for testing purposes.
>>>>> In this case GUC is most convenient way to do it.
>>>>>
>>>> There was typo in patch
>>>>
>>>> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
>>>> index f810789..8861f1b 100644
>>>> --- a/doc/src/sgml/config.sgml
>>>> +++ b/doc/src/sgml/config.sgml
>>>> @@ -1,4 +1,4 @@
>>>> -<!-- doc/src/sgml/config.sgml -->
>>>> +\<!-- doc/src/sgml/config.sgml -->
>>>>
>>>> I have not any objections against functionality or design. I tested the performance, and there are no negative impacts when this feature is not used. There is significant overhead related to plpgsql runtime initialization, but when this trigger will be used, then probably some other PLpgSQL procedures and functions will be used too, and then this overhead can be ignored.
>>>>
>>>> * make without warnings
>>>> * make check-world passed
>>>> * doc build passed
>>>>
>>>> Possible ToDo:
>>>>
>>>> The documentation can contain a note so usage connect triggers in environments with short life sessions and very short fast queries without usage PLpgSQL functions or procedures can have negative impact on performance due overhead of initialization of PLpgSQL engine.
>>>>
>>>> I'll mark this patch as ready for committers
>>>
>>> looks so this patch has not entry in commitfestapp 2021-01
>>>
>> Yeah, please register this patch before the next CommitFest[1] starts,
>> 2021-01-01 AoE[2].
>>
> Konstantin, did you register this patch in any CF? Even though the
> reviewer seems to be happy with the patch, I am afraid that we might
> lose track of this unless we register it.
>
Yes, certainly:
https://commitfest.postgresql.org/31/2900/

--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

 
 
 
 
 
 
 
 
 
Вложения

Re: Re[3]: On login trigger: take three

От
Greg Nancarrow
Дата:
On Thu, May 20, 2021 at 2:45 PM Ivan Panchenko <wao@mail.ru> wrote:
>
> I have upgraded the patch for the 14th version.
>

I have some feedback on the patch:

(1) The patch adds 3 whitespace errors ("git apply <patch-file>"
reports 3 warnings)


(2) doc/src/sgml/catalogs.sgml

CURRENTLY:
This flag is used to avoid extra lookup of pg_event_trigger table on
each backend startup.

SUGGEST:
This flag is used to avoid extra lookups on the pg_event_trigger table
during each backend startup.


(3) doc/src/sgml/config.sgml

CURRENTLY:
Errors in trigger code can prevent user to login to the system.In this
case disabling this parameter in connection string can solve the
problem:

SUGGEST:
Errors in the trigger code can prevent a user from logging in to the
system. In this case, the parameter can be disabled in the connection
string, to allow the user to login:


(4) doc/src/sgml/event-trigger.sgml

(i)

CURRENTLY:
An event trigger fires whenever the event with which it is associated
occurs in the database in which it is defined. Currently, the only

SUGGEST:
An event trigger fires whenever an associated event occurs in the
database in which it is defined. Currently, the only


(ii)

CURRENTLY:
can be useful for client connections logging,

SUGGEST:
can be useful for logging client connections,


(5) src/backend/commands/event_trigger.c

(i) There are two instances of code blocks like:

  xxxx = table_open(...);
  tuple = SearchSysCacheCopy1(...);
  table_close(...);

These should end with: "heap_freetuple(tuple);"

(ii) Typo "nuder" and grammar:

CURRENTLY:
There can be race condition: event trigger may be added after we have
scanned pg_event_trigger table. Repeat this test nuder  pg_database
table lock.

SUGGEST:
There can be a race condition: the event trigger may be added after we
have scanned the pg_event_trigger table. Repeat this test under the
pg_database table lock.


(6) src/backend/utils/misc/postgresql.conf.sample

CURRENTLY:
+#enable_client_connection_trigger = true # enables firing the
client_connection trigger when a client connect

SUGGEST:
+#enable_client_connection_trigger = true # enables firing the
client_connection trigger when a client connects


Regards,
Greg Nancarrow
Fujitsu Australia



Re: Re[3]: On login trigger: take three

От
Greg Nancarrow
Дата:
On Fri, May 21, 2021 at 2:46 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> On Thu, May 20, 2021 at 2:45 PM Ivan Panchenko <wao@mail.ru> wrote:
> >
> > I have upgraded the patch for the 14th version.
> >
>
> I have some feedback on the patch:
>

I've attached an updated version of the patch.
I've applied my review comments and done a bit more tidying up of
documentation and comments.
Also, I found that the previously-posted patch was broken by
snapshot-handling changes in commit 84f5c290 (with patch applied,
resulting in a coredump during regression tests) so I've added a fix
for that too.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: Re[3]: On login trigger: take three

От
vignesh C
Дата:
On Thu, Jun 3, 2021 at 8:36 AM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> On Fri, May 21, 2021 at 2:46 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
> >
> > On Thu, May 20, 2021 at 2:45 PM Ivan Panchenko <wao@mail.ru> wrote:
> > >
> > > I have upgraded the patch for the 14th version.
> > >
> >
> > I have some feedback on the patch:
> >
>
> I've attached an updated version of the patch.
> I've applied my review comments and done a bit more tidying up of
> documentation and comments.
> Also, I found that the previously-posted patch was broken by
> snapshot-handling changes in commit 84f5c290 (with patch applied,
> resulting in a coredump during regression tests) so I've added a fix
> for that too.

CFBot shows the following failure:
# poll_query_until timed out executing this query:
# SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM
pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
# expecting this output:
# t
# last actual query output:
# t
# with stderr:
# NOTICE: You are welcome!
# Looks like your test exited with 29 before it could output anything.
t/001_stream_rep.pl ..................
Dubious, test returned 29 (wstat 7424, 0x1d00)

Regards,
Vignesh



Re: Re[3]: On login trigger: take three

От
Greg Nancarrow
Дата:
On Sun, Jul 4, 2021 at 1:21 PM vignesh C <vignesh21@gmail.com> wrote:
>
> CFBot shows the following failure:
> # poll_query_until timed out executing this query:
> # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM
> pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
> # expecting this output:
> # t
> # last actual query output:
> # t
> # with stderr:
> # NOTICE: You are welcome!
> # Looks like your test exited with 29 before it could output anything.
> t/001_stream_rep.pl ..................
> Dubious, test returned 29 (wstat 7424, 0x1d00)
>

Thanks.
I found that the patch was broken by commit f452aaf7d (the part
"adjust poll_query_until to insist on empty stderr as well as a stdout
match").
So I had to remove a "RAISE NOTICE" (which was just an informational
message) from the login trigger function, to satisfy the new
poll_query_until expectations.
Also, I updated a PG14 version check (now must check PG15 version).

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: Re[3]: On login trigger: take three

От
Ibrar Ahmed
Дата:


On Wed, Jul 7, 2021 at 5:55 AM Greg Nancarrow <gregn4422@gmail.com> wrote:
On Sun, Jul 4, 2021 at 1:21 PM vignesh C <vignesh21@gmail.com> wrote:
>
> CFBot shows the following failure:
> # poll_query_until timed out executing this query:
> # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM
> pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
> # expecting this output:
> # t
> # last actual query output:
> # t
> # with stderr:
> # NOTICE: You are welcome!
> # Looks like your test exited with 29 before it could output anything.
> t/001_stream_rep.pl ..................
> Dubious, test returned 29 (wstat 7424, 0x1d00)
>

Thanks.
I found that the patch was broken by commit f452aaf7d (the part
"adjust poll_query_until to insist on empty stderr as well as a stdout
match").
So I had to remove a "RAISE NOTICE" (which was just an informational
message) from the login trigger function, to satisfy the new
poll_query_until expectations.
Also, I updated a PG14 version check (now must check PG15 version).

Regards,
Greg Nancarrow
Fujitsu Australia


The patch does not apply, and rebase is required. 
Hunk #1 FAILED at 93.
1 out of 1 hunk FAILED -- saving rejects to file src/test/regress/expected/sysviews.out.rej

I am not changing the status because it is a minor change and the patch is already "Ready for Committer".

--
Ibrar Ahmed

Re: Re[3]: On login trigger: take three

От
Greg Nancarrow
Дата:
On Mon, Jul 19, 2021 at 8:18 PM Ibrar Ahmed <ibrar.ahmad@gmail.com> wrote:
>
> The patch does not apply, and rebase is required.
>

Attached a rebased patch (minor updates to the test code).

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re[5]: On login trigger: take three

От
Ivan Panchenko
Дата:
 
Hi Greg,
 
Среда, 7 июля 2021, 3:55 +03:00 от Greg Nancarrow <gregn4422@gmail.com>:
 
On Sun, Jul 4, 2021 at 1:21 PM vignesh C <vignesh21@gmail.com> wrote:
>
> CFBot shows the following failure:
> # poll_query_until timed out executing this query:
> # SELECT '0/3046250' <= replay_lsn AND state = 'streaming' FROM
> pg_catalog.pg_stat_replication WHERE application_name = 'standby_1';
> # expecting this output:
> # t
> # last actual query output:
> # t
> # with stderr:
> # NOTICE: You are welcome!
> # Looks like your test exited with 29 before it could output anything.
> t/001_stream_rep.pl ..................
> Dubious, test returned 29 (wstat 7424, 0x1d00)
>

Thanks.
I found that the patch was broken by commit f452aaf7d (the part
"adjust poll_query_until to insist on empty stderr as well as a stdout
match").
So I had to remove a "RAISE NOTICE" (which was just an informational
message) from the login trigger function, to satisfy the new
poll_query_until expectations.

Does it mean that "RAISE NOTICE" should’t be used, or behaves unexpectedly in logon triggers? Should we mention this in the docs?
 
Regards,
Ivan Panchenko
Also, I updated a PG14 version check (now must check PG15 version).

Regards,
Greg Nancarrow
Fujitsu Australia
 

Re: Re[5]: On login trigger: take three

От
Greg Nancarrow
Дата:
On Tue, Aug 17, 2021 at 1:11 AM Ivan Panchenko <wao@mail.ru> wrote:
>
> Does it mean that "RAISE NOTICE" should’t be used, or behaves unexpectedly in logon triggers? Should we mention this
inthe docs? 
>

No I don't believe so, it was just that that part of the test
framework (sub poll_query_until) had been changed to regard anything
output to stderr as an error (so now for the test to succeed, whatever
is printed to stdout must match the expected test output, and stderr
must be empty).

Regards,
Greg Nancarrow
Fujitsu Australia



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 19 Jul 2021, at 15:25, Greg Nancarrow <gregn4422@gmail.com> wrote:

> Attached a rebased patch (minor updates to the test code).

I took a look at this, and while I like the proposed feature I think the patch
has a bit more work required.


+    END IF;
+
+-- 2) Initialize some user session data
+   CREATE TEMP TABLE session_storage (x float, y integer);
The example in the documentation use a mix of indentations, neither of which is
the 2-space indentation used elsewhere in the docs.


+    /* Get the command tag. */
+    tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT;
This is hardcoding knowledge that currently only this trigger wont have a
parsetree, and setting the tag accordingly.  This should instead check the
event and set CMDTAG_UNKNOWN if it isn't the expected one.


+    /* database has on-login triggers */
+    bool        dathaslogontriggers;
This patch uses three different names for the same thing: client connection,
logontrigger and login trigger.  Since this trigger only fires after successful
authentication it’s not accurate to name it client connection, as that implies
it running on connections rather than logins.  The nomenclature used in the
tree is "login" so the patch should be adjusted everywhere to use that.


+/* Hook for plugins to get control at start of session */
+client_connection_hook_type client_connection_hook = EventTriggerOnConnect;
I don't see the reason for adding core functionality by hooks.  Having a hook
might be a useful thing on its own (to be discussed in a new thread, not hidden
here), but core functionality should not depend on it IMO.


+    {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
+        gettext_noop("Enables the client_connection event trigger."),
+        gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
 ..and..
+    /*
+     * Try to ignore error for superuser to make it possible to login even in case of errors
+     * during trigger execution
+     */
+    if (!is_superuser)
+        PG_RE_THROW();
This patch adds two ways for superusers to bypass this event trigger in case of
it being faulty, but for every other event trigger we've documented to restart
in single-user mode and fixing it there.  Why does this need to be different?
This clearly has a bigger chance of being a footgun but I don't see that as a
reason to add a GUC and a bypass that other footguns lack.


+    elog(NOTICE, "client_connection trigger failed with message: %s", error->message);
Calling elog() in a PG_CATCH() block isn't allowed is it?


+    /* Runtlist is empty: clear dathaslogontriggers flag
+     */
s/Runtlist/Runlist/ and also commenting style.


The logic for resetting the pg_database flag when firing event trigger in case
the trigger has gone away is messy and a problem waiting to happen IMO.  For
normal triggers we don't bother with that on the grounds of it being racy, and
instead perform relhastrigger removal during VACUUM.  Another approach is using
a counter as propose upthread, since updating that can be made safer.  The
current coding also isn't instructing concurrent backends to perform relcache
rebuild.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Pavel Stehule
Дата:

Hi


+       {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
+               gettext_noop("Enables the client_connection event trigger."),
+               gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
 ..and..
+       /*
+        * Try to ignore error for superuser to make it possible to login even in case of errors
+        * during trigger execution
+        */
+       if (!is_superuser)
+               PG_RE_THROW();
This patch adds two ways for superusers to bypass this event trigger in case of
it being faulty, but for every other event trigger we've documented to restart
in single-user mode and fixing it there.  Why does this need to be different?
This clearly has a bigger chance of being a footgun but I don't see that as a
reason to add a GUC and a bypass that other footguns lack.



In the time when event triggers were introduced, managed services were not too widely used like now. When we discussed this feature we thought about environments when users have no superuser rights and have no possibility to go to single mode.

Personally, I prefer to introduce some bypassing for event triggers instead of removing bypass from login triggers.

Regards

Pavel



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 8 Sep 2021, at 16:02, Pavel Stehule <pavel.stehule@gmail.com> wrote:

> In the time when event triggers were introduced, managed services were not too widely used like now. When we
discussedthis feature we thought about environments when users have no superuser rights and have no possibility to go
tosingle mode. 

In situations where you don't have superuser access and cannot restart in
single user mode, none of the bypasses in this patch would help anyways.

I understand the motivation, but continuing on even in the face of an
ereport(ERROR..  ) in the hopes of being able to turn off buggy code seems
pretty unsafe at best.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Pavel Stehule
Дата:


st 8. 9. 2021 v 20:23 odesílatel Daniel Gustafsson <daniel@yesql.se> napsal:
> On 8 Sep 2021, at 16:02, Pavel Stehule <pavel.stehule@gmail.com> wrote:

> In the time when event triggers were introduced, managed services were not too widely used like now. When we discussed this feature we thought about environments when users have no superuser rights and have no possibility to go to single mode.

In situations where you don't have superuser access and cannot restart in
single user mode, none of the bypasses in this patch would help anyways.

If I remember well, it should be possible - you can set GUC in connection string, and this GUC is limited to the database owner.


I understand the motivation, but continuing on even in the face of an
ereport(ERROR..  ) in the hopes of being able to turn off buggy code seems
pretty unsafe at best.

I don't understand what you mean. I can disable the logon trigger by GUC. I cannot to enable an continue after an exception.

Regards

Pavel
 

--
Daniel Gustafsson               https://vmware.com/

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Wed, Sep 8, 2021 at 10:56 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> I took a look at this, and while I like the proposed feature I think the patch
> has a bit more work required.
>

Thanks for reviewing the patch.
I am not the original patch author (who no longer seems active) but
I've been contributing a bit and keeping the patch alive because I
think it's a worthwhile feature.

>
> +
> +-- 2) Initialize some user session data
> +   CREATE TEMP TABLE session_storage (x float, y integer);
> The example in the documentation use a mix of indentations, neither of which is
> the 2-space indentation used elsewhere in the docs.
>

Fixed, using 2-space indentation.
(to be honest, the indentation seems inconsistent elsewhere in the
docs e.g. I'm seeing a nearby case of 2-space on 1st indent, 6-space
on 2nd indent)

>
> +       /* Get the command tag. */
> +       tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT;
> This is hardcoding knowledge that currently only this trigger wont have a
> parsetree, and setting the tag accordingly.  This should instead check the
> event and set CMDTAG_UNKNOWN if it isn't the expected one.
>

I updated that, but maybe not exactly how you expected?

>
> +       /* database has on-login triggers */
> +       bool            dathaslogontriggers;
> This patch uses three different names for the same thing: client connection,
> logontrigger and login trigger.  Since this trigger only fires after successful
> authentication it’s not accurate to name it client connection, as that implies
> it running on connections rather than logins.  The nomenclature used in the
> tree is "login" so the patch should be adjusted everywhere to use that.
>

Fixed.

>
> +/* Hook for plugins to get control at start of session */
> +client_connection_hook_type client_connection_hook = EventTriggerOnConnect;
> I don't see the reason for adding core functionality by hooks.  Having a hook
> might be a useful thing on its own (to be discussed in a new thread, not hidden
> here), but core functionality should not depend on it IMO.
>

Fair enough, I removed the hook.

>
> +       {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
> +               gettext_noop("Enables the client_connection event trigger."),
> +               gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
>  ..and..
> +       /*
> +        * Try to ignore error for superuser to make it possible to login even in case of errors
> +        * during trigger execution
> +        */
> +       if (!is_superuser)
> +               PG_RE_THROW();
> This patch adds two ways for superusers to bypass this event trigger in case of
> it being faulty, but for every other event trigger we've documented to restart
> in single-user mode and fixing it there.  Why does this need to be different?
> This clearly has a bigger chance of being a footgun but I don't see that as a
> reason to add a GUC and a bypass that other footguns lack.
>

OK, I removed those bypasses. We'll see what others think.

>
> +       elog(NOTICE, "client_connection trigger failed with message: %s", error->message);
> Calling elog() in a PG_CATCH() block isn't allowed is it?
>

I believe it is allowed (e.g. there's a case in libpq), but I removed
this anyway as part of the superuser bypass.

>
> +       /* Runtlist is empty: clear dathaslogontriggers flag
> +        */
> s/Runtlist/Runlist/ and also commenting style.
>

Fixed.

>
> The logic for resetting the pg_database flag when firing event trigger in case
> the trigger has gone away is messy and a problem waiting to happen IMO.  For
> normal triggers we don't bother with that on the grounds of it being racy, and
> instead perform relhastrigger removal during VACUUM.  Another approach is using
> a counter as propose upthread, since updating that can be made safer.  The
> current coding also isn't instructing concurrent backends to perform relcache
> rebuild.
>

I think there are pros and cons of each possible approach, but I think
I prefer the current way (which I have tweaked a bit) for similar
reasons to those explained by the original patch author when debating
whether to use reference-counting instead, in the discussion upthread
(e.g. it keeps it all in one place). Also, it seems to be more inline
with the documented reason why that pg_database flag was added in the
first place. I have debugged a few concurrent scenarios with the
current mechanism in place. If you still dislike the logic for
resetting the flag, please elaborate on the issues you foresee and one
of the alternative approaches can be tried.

I've attached the updated patch.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Teodor Sigaev
Дата:
Hi!

Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK, 
there is two use cases for this feature:
1 A permission / prohibition to login some users
2 Just a logging of facts of user's login

Suggested patch proposes prohibition of login only by failing of login trigger 
and it has at least two issues:
1 In case of prohibition to login, there is no clean way to store information 
about unsuccessful  login. Ok, it could be solved by dblink module but it seems 
to scary.
2 For logging purpose failing of trigger will cause impossibility to login, it 
could be workarounded by catching error in trigger function, but it's not so 
obvious for DBA.

some other issues:
3 It's easy to create security hole, see attachment where non-privileged user 
can close access to database even for superuser.
4 Feature is not applicable for logging unsuccessful login with wrong 
username/password or not matched by pg_hba.conf. Oracle could operate with such 
cases. But I don't think that this issue is a stopper for the patch.

May be, to solve that issues we could introduce return value of trigger and/or 
add something like IGNORE ERROR to CREATE EVENT TRIGGER command.

On 15.09.2021 16:32, Greg Nancarrow wrote:
> On Wed, Sep 8, 2021 at 10:56 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>>
>> I took a look at this, and while I like the proposed feature I think the patch
>> has a bit more work required.
>>
> 
> Thanks for reviewing the patch.
> I am not the original patch author (who no longer seems active) but
> I've been contributing a bit and keeping the patch alive because I
> think it's a worthwhile feature.
> 
>>
>> +
>> +-- 2) Initialize some user session data
>> +   CREATE TEMP TABLE session_storage (x float, y integer);
>> The example in the documentation use a mix of indentations, neither of which is
>> the 2-space indentation used elsewhere in the docs.
>>
> 
> Fixed, using 2-space indentation.
> (to be honest, the indentation seems inconsistent elsewhere in the
> docs e.g. I'm seeing a nearby case of 2-space on 1st indent, 6-space
> on 2nd indent)
> 
>>
>> +       /* Get the command tag. */
>> +       tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT;
>> This is hardcoding knowledge that currently only this trigger wont have a
>> parsetree, and setting the tag accordingly.  This should instead check the
>> event and set CMDTAG_UNKNOWN if it isn't the expected one.
>>
> 
> I updated that, but maybe not exactly how you expected?
> 
>>
>> +       /* database has on-login triggers */
>> +       bool            dathaslogontriggers;
>> This patch uses three different names for the same thing: client connection,
>> logontrigger and login trigger.  Since this trigger only fires after successful
>> authentication it’s not accurate to name it client connection, as that implies
>> it running on connections rather than logins.  The nomenclature used in the
>> tree is "login" so the patch should be adjusted everywhere to use that.
>>
> 
> Fixed.
> 
>>
>> +/* Hook for plugins to get control at start of session */
>> +client_connection_hook_type client_connection_hook = EventTriggerOnConnect;
>> I don't see the reason for adding core functionality by hooks.  Having a hook
>> might be a useful thing on its own (to be discussed in a new thread, not hidden
>> here), but core functionality should not depend on it IMO.
>>
> 
> Fair enough, I removed the hook.
> 
>>
>> +       {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
>> +               gettext_noop("Enables the client_connection event trigger."),
>> +               gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
>>   ..and..
>> +       /*
>> +        * Try to ignore error for superuser to make it possible to login even in case of errors
>> +        * during trigger execution
>> +        */
>> +       if (!is_superuser)
>> +               PG_RE_THROW();
>> This patch adds two ways for superusers to bypass this event trigger in case of
>> it being faulty, but for every other event trigger we've documented to restart
>> in single-user mode and fixing it there.  Why does this need to be different?
>> This clearly has a bigger chance of being a footgun but I don't see that as a
>> reason to add a GUC and a bypass that other footguns lack.
>>
> 
> OK, I removed those bypasses. We'll see what others think.
> 
>>
>> +       elog(NOTICE, "client_connection trigger failed with message: %s", error->message);
>> Calling elog() in a PG_CATCH() block isn't allowed is it?
>>
> 
> I believe it is allowed (e.g. there's a case in libpq), but I removed
> this anyway as part of the superuser bypass.
> 
>>
>> +       /* Runtlist is empty: clear dathaslogontriggers flag
>> +        */
>> s/Runtlist/Runlist/ and also commenting style.
>>
> 
> Fixed.
> 
>>
>> The logic for resetting the pg_database flag when firing event trigger in case
>> the trigger has gone away is messy and a problem waiting to happen IMO.  For
>> normal triggers we don't bother with that on the grounds of it being racy, and
>> instead perform relhastrigger removal during VACUUM.  Another approach is using
>> a counter as propose upthread, since updating that can be made safer.  The
>> current coding also isn't instructing concurrent backends to perform relcache
>> rebuild.
>>
> 
> I think there are pros and cons of each possible approach, but I think
> I prefer the current way (which I have tweaked a bit) for similar
> reasons to those explained by the original patch author when debating
> whether to use reference-counting instead, in the discussion upthread
> (e.g. it keeps it all in one place). Also, it seems to be more inline
> with the documented reason why that pg_database flag was added in the
> first place. I have debugged a few concurrent scenarios with the
> current mechanism in place. If you still dislike the logic for
> resetting the flag, please elaborate on the issues you foresee and one
> of the alternative approaches can be tried.
> 
> I've attached the updated patch.
> 
> Regards,
> Greg Nancarrow
> Fujitsu Australia
> 

-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

Вложения

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Wed, Sep 29, 2021 at 10:14 PM Teodor Sigaev <teodor@sigaev.ru> wrote:
>
> Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK,
> there is two use cases for this feature:
> 1 A permission / prohibition to login some users
> 2 Just a logging of facts of user's login
>
> Suggested patch proposes prohibition of login only by failing of login trigger
> and it has at least two issues:
> 1 In case of prohibition to login, there is no clean way to store information
> about unsuccessful  login. Ok, it could be solved by dblink module but it seems
> to scary.

It's an area that could be improved, but the patch is more intended to
allow additional actions on successful login, as described by the
following (taken from the doc updates in the patch):

+   <para>
+    The event trigger on the <literal>login</literal> event can be
+    useful for logging user logins, for verifying the connection and
+    assigning roles according to current circumstances, or for some
session data
+    initialization.
+   </para>

> 2 For logging purpose failing of trigger will cause impossibility to login, it
> could be workarounded by catching error in trigger function, but it's not so
> obvious for DBA.
>

If you look back on the past discussions on this thread, you'll see
that originally the patch added a GUC for the purpose of allowing
superusers to disable the event trigger, to allow login in this case.
However it was argued that there was already a documented way of
bypassing buggy event triggers (i.e. restart in single-user mode), so
that GUC was removed.
Also previously in the patch, login trigger errors for superusers were
ignored in order to allow them to login in this case, but it was
argued that it could well be unsafe to continue on after an error, so
that too was removed.
See below:

>
> +       {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
> +               gettext_noop("Enables the client_connection event trigger."),
> +               gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
>  ..and..
> +       /*
> +        * Try to ignore error for superuser to make it possible to login even in case of errors
> +        * during trigger execution
> +        */
> +       if (!is_superuser)
> +               PG_RE_THROW();
> This patch adds two ways for superusers to bypass this event trigger in case of
> it being faulty, but for every other event trigger we've documented to restart
> in single-user mode and fixing it there.  Why does this need to be different?
> This clearly has a bigger chance of being a footgun but I don't see that as a
> reason to add a GUC and a bypass that other footguns lack.
>

So I really don't think that a failing event trigger will cause an
"impossibility to login".
The patch updates the documentation to explain the use of single-user
mode to fix buggy login event triggers. See below:

+   <para>
+     The <literal>login</literal> event occurs when a user logs in to the
+     system.
+     Any bugs in a trigger procedure for this event may prevent successful
+     login to the system. Such bugs may be fixed after first restarting the
+     system in single-user mode (as event triggers are disabled in this mode).
+     See the <xref linkend="app-postgres"/> reference page for details about
+     using single-user mode.
+   </para>
+

Regards,
Greg Nancarrow
Fujitsu Australia



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 30 Sep 2021, at 04:15, Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> On Wed, Sep 29, 2021 at 10:14 PM Teodor Sigaev <teodor@sigaev.ru> wrote:
>>
>> Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK,
>> there is two use cases for this feature:
>> 1 A permission / prohibition to login some users
>> 2 Just a logging of facts of user's login
>>
>> Suggested patch proposes prohibition of login only by failing of login trigger
>> and it has at least two issues:
>> 1 In case of prohibition to login, there is no clean way to store information
>> about unsuccessful  login. Ok, it could be solved by dblink module but it seems
>> to scary.
>
> It's an area that could be improved, but the patch is more intended to
> allow additional actions on successful login, as described by the
> following (taken from the doc updates in the patch):
>
> +   <para>
> +    The event trigger on the <literal>login</literal> event can be
> +    useful for logging user logins, for verifying the connection and
> +    assigning roles according to current circumstances, or for some
> session data
> +    initialization.
> +   </para>

Running user code with potential side effects on unsuccessful logins also open
up the risk for (D)DoS attacks.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Wed, Sep 15, 2021 at 11:32 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> I've attached the updated patch.
>

Attached a rebased version of the patch, as it was broken by fairly
recent changes (only very minor change to the previous version).

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re[2]: On login trigger: take three

От
Ivan Panchenko
Дата:
Dear colleagues, 
Please see my suggestions below and the updated patch attached.
 
 
 
 
 
Среда, 29 сентября 2021, 15:14 +03:00 от Teodor Sigaev <teodor@sigaev.ru>:
 
Hi!

Nice feature, but, sorry, I see some design problem in suggested feature. AFAIK,
there is two use cases for this feature:
1 A permission / prohibition to login some users
2 Just a logging of facts of user's login

Suggested patch proposes prohibition of login only by failing of login trigger
and it has at least two issues:
1 In case of prohibition to login, there is no clean way to store information
about unsuccessful login. Ok, it could be solved by dblink module but it seems
to scary.
This is a common problem of logging errors and unsuccessful transactions of any kind. It can be solved by:
- logging to external log storage (stupid logging to files or syslog or whatever you can imagine with PL/Perl (sorry))
- logging inside the database by db_link or through background worker (like described in https://dpavlin.wordpress.com/2017/05/09/david-rader-autonomous-transactions-using-pg_background/ )
- or by implementing autonomous transactions in PostgreSQL, which is already under development by some of my and Teodor’s colleagues.
 
So I propose not to invent another solution to this common problem here.
2 For logging purpose failing of trigger will cause impossibility to login, it
could be workarounded by catching error in trigger function, but it's not so
obvious for DBA.
If the trigger contains an error, nobody can login. The database is bricked.
Previous variant of the patch proposed to fix this with going to single-user mode.
For faster recovery I propose to have also a GUC variable to turn on/off the login triggers.
It should be 'on' by default.

some other issues:
3 It's easy to create security hole, see attachment where non-privileged user
can close access to database even for superuser.
Such cases can be avoided by careful design of the login triggers and related permissions. Added such note to the documentation.
4 Feature is not applicable for logging unsuccessful login with wrong
username/password or not matched by pg_hba.conf. Oracle could operate with such
cases. But I don't think that this issue is a stopper for the patch.
Yes. Btw, note that pg_hba functionality can be implemented completely inside the login trigger :) !

May be, to solve that issues we could introduce return value of trigger and/or
add something like IGNORE ERROR to CREATE EVENT TRIGGER command.
Any solutions which make syntax more complicated can lead Postgres to become Oracle (in the worst sense). I do not like this.

A new version of the patch is attached. It applies to the current master and contains the above mentioned GUC code, relevant tests and docs.
Regards,
Ivan Panchenko

On 15.09.2021 16:32, Greg Nancarrow wrote:
> On Wed, Sep 8, 2021 at 10:56 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>>
>> I took a look at this, and while I like the proposed feature I think the patch
>> has a bit more work required.
>>
>
> Thanks for reviewing the patch.
> I am not the original patch author (who no longer seems active) but
> I've been contributing a bit and keeping the patch alive because I
> think it's a worthwhile feature.
>
>>
>> +
>> +-- 2) Initialize some user session data
>> + CREATE TEMP TABLE session_storage (x float, y integer);
>> The example in the documentation use a mix of indentations, neither of which is
>> the 2-space indentation used elsewhere in the docs.
>>
>
> Fixed, using 2-space indentation.
> (to be honest, the indentation seems inconsistent elsewhere in the
> docs e.g. I'm seeing a nearby case of 2-space on 1st indent, 6-space
> on 2nd indent)
>
>>
>> + /* Get the command tag. */
>> + tag = parsetree ? CreateCommandTag(parsetree) : CMDTAG_CONNECT;
>> This is hardcoding knowledge that currently only this trigger wont have a
>> parsetree, and setting the tag accordingly. This should instead check the
>> event and set CMDTAG_UNKNOWN if it isn't the expected one.
>>
>
> I updated that, but maybe not exactly how you expected?
>
>>
>> + /* database has on-login triggers */
>> + bool dathaslogontriggers;
>> This patch uses three different names for the same thing: client connection,
>> logontrigger and login trigger. Since this trigger only fires after successful
>> authentication it’s not accurate to name it client connection, as that implies
>> it running on connections rather than logins. The nomenclature used in the
>> tree is "login" so the patch should be adjusted everywhere to use that.
>>
>
> Fixed.
>
>>
>> +/* Hook for plugins to get control at start of session */
>> +client_connection_hook_type client_connection_hook = EventTriggerOnConnect;
>> I don't see the reason for adding core functionality by hooks. Having a hook
>> might be a useful thing on its own (to be discussed in a new thread, not hidden
>> here), but core functionality should not depend on it IMO.
>>
>
> Fair enough, I removed the hook.
>
>>
>> + {"enable_client_connection_trigger", PGC_SU_BACKEND, DEVELOPER_OPTIONS,
>> + gettext_noop("Enables the client_connection event trigger."),
>> + gettext_noop("In case of errors in the ON client_connection EVENT TRIGGER procedure, "
>> ..and..
>> + /*
>> + * Try to ignore error for superuser to make it possible to login even in case of errors
>> + * during trigger execution
>> + */
>> + if (!is_superuser)
>> + PG_RE_THROW();
>> This patch adds two ways for superusers to bypass this event trigger in case of
>> it being faulty, but for every other event trigger we've documented to restart
>> in single-user mode and fixing it there. Why does this need to be different?
>> This clearly has a bigger chance of being a footgun but I don't see that as a
>> reason to add a GUC and a bypass that other footguns lack.
>>
>
> OK, I removed those bypasses. We'll see what others think.
>
>>
>> + elog(NOTICE, "client_connection trigger failed with message: %s", error->message);
>> Calling elog() in a PG_CATCH() block isn't allowed is it?
>>
>
> I believe it is allowed (e.g. there's a case in libpq), but I removed
> this anyway as part of the superuser bypass.
>
>>
>> + /* Runtlist is empty: clear dathaslogontriggers flag
>> + */
>> s/Runtlist/Runlist/ and also commenting style.
>>
>
> Fixed.
>
>>
>> The logic for resetting the pg_database flag when firing event trigger in case
>> the trigger has gone away is messy and a problem waiting to happen IMO. For
>> normal triggers we don't bother with that on the grounds of it being racy, and
>> instead perform relhastrigger removal during VACUUM. Another approach is using
>> a counter as propose upthread, since updating that can be made safer. The
>> current coding also isn't instructing concurrent backends to perform relcache
>> rebuild.
>>
>
> I think there are pros and cons of each possible approach, but I think
> I prefer the current way (which I have tweaked a bit) for similar
> reasons to those explained by the original patch author when debating
> whether to use reference-counting instead, in the discussion upthread
> (e.g. it keeps it all in one place). Also, it seems to be more inline
> with the documented reason why that pg_database flag was added in the
> first place. I have debugged a few concurrent scenarios with the
> current mechanism in place. If you still dislike the logic for
> resetting the flag, please elaborate on the issues you foresee and one
> of the alternative approaches can be tried.
>
> I've attached the updated patch.
>
> Regards,
> Greg Nancarrow
> Fujitsu Australia
>

--
Teodor Sigaev E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/
 
Вложения

Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 3 Nov 2021, at 17:15, Ivan Panchenko <wao@mail.ru> wrote:
> Среда, 29 сентября 2021, 15:14 +03:00 от Teodor Sigaev <teodor@sigaev.ru <x-msg://33/compose?To=teodor@sigaev.ru>>:
> 2 For logging purpose failing of trigger will cause impossibility to login, it
> could be workarounded by catching error in trigger function, but it's not so
> obvious for DBA.
> If the trigger contains an error, nobody can login. The database is bricked.
> Previous variant of the patch proposed to fix this with going to single-user mode.
> For faster recovery I propose to have also a GUC variable to turn on/off the login triggers.
> It should be 'on' by default.

As voiced earlier, I disagree with this and I dislike the idea of punching a
hole for circumventing infrastructure intended for auditing.

Use-cases for a login-trigger commonly involve audit trail logging, session
initialization etc.  If the login trigger bricks the production database to the
extent that it needs to be restarted with the magic GUC, then it's highly
likely that you *don't* want regular connections to the database for the
duration of this.  Any such connection won't be subject to what the trigger
does which seem counter to having the trigger in the first place.  This means
that it's likely that the superuser fixing it will have to disable logins for
everyone else while fixing, and it quicly becomes messy.

With that in mind, I think single-user mode actually *helps* the users in this
case, and we avoid a hole punched which in worst case can be a vector for an
attack.

Maybe I'm overly paranoid, but adding a backdoor of sorts for a situation which
really shouldn't happen doesn't seem like a good idea.
> some other issues:
> 3 It's easy to create security hole, see attachment where non-privileged user
> can close access to database even for superuser.
> Such cases can be avoided by careful design of the login triggers and related permissions. Added such note to the
documentation.

If all login triggers are written carefully like that, we don't need the GUC to
disable them?

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Greg Nancarrow
Дата:

On Thu, Nov 4, 2021 at 7:43 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> On 3 Nov 2021, at 17:15, Ivan Panchenko <wao@mail.ru> wrote:
> Среда, 29 сентября 2021, 15:14 +03:00 от Teodor Sigaev <teodor@sigaev.ru <x-msg://33/compose?To=teodor@sigaev.ru>>:
> 2 For logging purpose failing of trigger will cause impossibility to login, it
> could be workarounded by catching error in trigger function, but it's not so
> obvious for DBA.
> If the trigger contains an error, nobody can login. The database is bricked.
> Previous variant of the patch proposed to fix this with going to single-user mode.
> For faster recovery I propose to have also a GUC variable to turn on/off the login triggers.
> It should be 'on' by default.

As voiced earlier, I disagree with this and I dislike the idea of punching a
hole for circumventing infrastructure intended for auditing.

Use-cases for a login-trigger commonly involve audit trail logging, session
initialization etc.  If the login trigger bricks the production database to the
extent that it needs to be restarted with the magic GUC, then it's highly
likely that you *don't* want regular connections to the database for the
duration of this.  Any such connection won't be subject to what the trigger
does which seem counter to having the trigger in the first place.  This means
that it's likely that the superuser fixing it will have to disable logins for
everyone else while fixing, and it quicly becomes messy.

With that in mind, I think single-user mode actually *helps* the users in this
case, and we avoid a hole punched which in worst case can be a vector for an
attack.

Maybe I'm overly paranoid, but adding a backdoor of sorts for a situation which
really shouldn't happen doesn't seem like a good idea.


+1
The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good idea, and are not needed.


Regards,
Greg Nancarrow
Fujitsu Australia

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Fri, Nov 5, 2021 at 3:03 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> +1
> The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good
idea,and are not needed.
 
>

If there are no objections, I plan to reinstate the previous v19 patch
(as v21), perhaps with a few minor improvements and cleanups (e.g. SQL
capitalization) in the tests, as hinted at in the v20 patch, but no
new functionality.

Regards,
Greg Nancarrow
Fujitsu Australia



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 10 Nov 2021, at 08:12, Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> On Fri, Nov 5, 2021 at 3:03 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>>
>> +1
>> The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good
idea,and are not needed. 
>
> If there are no objections, I plan to reinstate the previous v19 patch
> (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL
> capitalization) in the tests, as hinted at in the v20 patch, but no
> new functionality.

No objections from me. Small nitpicks from the v19 patch:

+        This flag is used internally by Postgres and should not be manually changed by DBA or application.
This should be <productname>PostgreSQL</productname>.

+     * There can be a race condition: a login event trigger may have
..
+    /* Fire any defined login triggers, if appropriate */
The patch say "login trigger" in most places, and "login event trigger" in a
few places.  We should settle for a single nomenclature, and I think "login
event trigger" is the best option.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Pavel Stehule
Дата:


st 10. 11. 2021 v 10:11 odesílatel Daniel Gustafsson <daniel@yesql.se> napsal:
> On 10 Nov 2021, at 08:12, Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> On Fri, Nov 5, 2021 at 3:03 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>>
>> +1
>> The arguments given are pretty convincing IMHO, and I agree that the additions made in the v20 patch are not a good idea, and are not needed.
>
> If there are no objections, I plan to reinstate the previous v19 patch
> (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL
> capitalization) in the tests, as hinted at in the v20 patch, but no
> new functionality.

No objections from me. Small nitpicks from the v19 patch:

ok,

Regards

Pavel


+        This flag is used internally by Postgres and should not be manually changed by DBA or application.
This should be <productname>PostgreSQL</productname>.

+        * There can be a race condition: a login event trigger may have
..
+       /* Fire any defined login triggers, if appropriate */
The patch say "login trigger" in most places, and "login event trigger" in a
few places.  We should settle for a single nomenclature, and I think "login
event trigger" is the best option.

--
Daniel Gustafsson               https://vmware.com/

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Wed, Nov 10, 2021 at 8:11 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > If there are no objections, I plan to reinstate the previous v19 patch
> > (as v21), perhaps with a few minor improvements and cleanups (e.g. SQL
> > capitalization) in the tests, as hinted at in the v20 patch, but no
> > new functionality.
>
> No objections from me. Small nitpicks from the v19 patch:
>
> +        This flag is used internally by Postgres and should not be manually changed by DBA or application.
> This should be <productname>PostgreSQL</productname>.
>
> +        * There can be a race condition: a login event trigger may have
> ..
> +       /* Fire any defined login triggers, if appropriate */
> The patch say "login trigger" in most places, and "login event trigger" in a
> few places.  We should settle for a single nomenclature, and I think "login
> event trigger" is the best option.
>

I've attached an updated patch, that essentially reinstates the v19
patch, but with a few improvements such as:
- Updates to address nitpicks (Daniel Gustafsson)
- dathaslogintriggers -> dathasloginevttriggers flag rename (too
long?) and remove its restoration in pg_dump output, since it's not
needed (as in v20 patch)
- Some tidying of the updates to the event_trigger tests and
capitalization of the test SQL

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 11 Nov 2021, at 07:37, Greg Nancarrow <gregn4422@gmail.com> wrote:

> I've attached an updated patch, that essentially reinstates the v19
> patch

Thanks!  I've only skimmed it so far but it looks good, will do a more thorough
review soon.

+        This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup.
This should be <structname>pg_event_trigger</structname>.  Sorry, missed that
one at that last read-through.

> - dathaslogintriggers -> dathasloginevttriggers flag rename (too
> long?)

I'm not crazy about this name, "evt" is commonly the abbreviation of "event
trigger" (used in evtcache.c etc) so "dathasloginevt" would IMO be better.
That being said, that's still not a very readable name, maybe someone else has
an even better suggestion?

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Thu, Nov 11, 2021 at 8:56 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> +        This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup.
> This should be <structname>pg_event_trigger</structname>.  Sorry, missed that
> one at that last read-through.
>
Thanks, noted.

> > - dathaslogintriggers -> dathasloginevttriggers flag rename (too
> > long?)
>
> I'm not crazy about this name, "evt" is commonly the abbreviation of "event
> trigger" (used in evtcache.c etc) so "dathasloginevt" would IMO be better.
> That being said, that's still not a very readable name, maybe someone else has
> an even better suggestion?
>
Yes you're right, in this case I was wrongly treating "evt" as an
abbreviation for "event".
I agree "dathasloginevt" would be better.


Regards,
Greg Nancarrow
Fujitsu Australia



Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Thu, Nov 11, 2021 at 8:56 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> +        This flag is used to avoid extra lookups on the pg_event_trigger table during each backend startup.
> This should be <structname>pg_event_trigger</structname>.  Sorry, missed that
> one at that last read-through.
>
Fixed.

> > - dathaslogintriggers -> dathasloginevttriggers flag rename (too
> > long?)
>
> I'm not crazy about this name, "evt" is commonly the abbreviation of "event
> trigger" (used in evtcache.c etc) so "dathasloginevt" would IMO be better.
> That being said, that's still not a very readable name, maybe someone else has
> an even better suggestion?
>
Changed to "dathasloginevt", as suggested.

I've attached an updated patch with these changes.
I also noticed one of the Windows-based cfbots was failing with an
"SSPI authentication failed for user" error, so I updated the test
code for that.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Tue, Nov 16, 2021 at 4:38 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> I've attached an updated patch with these changes.
>

I've attached a re-based version (no functional changes from the
previous) to fix cfbot failures.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Mon, Dec 6, 2021 at 12:10 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> I've attached a re-based version (no functional changes from the
> previous) to fix cfbot failures.
>

I've attached a re-based version (no functional changes from the
previous) to fix cfbot failures.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Greg Nancarrow
Дата:
On Mon, Jan 24, 2022 at 1:59 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> I've attached a re-based version (no functional changes from the
> previous) to fix cfbot failures.
>

I've attached a re-based version (no functional changes from the
previous) to fix cfbot failures.

Regards,
Greg Nancarrow
Fujitsu Australia

Вложения

Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 15 Feb 2022, at 11:07, Greg Nancarrow <gregn4422@gmail.com> wrote:

> I've attached a re-based version (no functional changes from the
> previous) to fix cfbot failures.

Thanks for adopting this patch, I took another look at it today and have some
comments.

+        This flag is used internally by <productname>PostgreSQL</productname> and should not be manually changed by
DBAor application. 
I think we should reword this to something like "This flag is used internally
by <productname>PostgreSQL</productname> and should not be altered or relied
upon for monitoring".  We really don't want anyone touching or interpreting
this value since there is no guarantee that it will be accurate.


+    new_record[Anum_pg_database_dathasloginevt - 1] = BoolGetDatum(datform->dathasloginevt);
Since the corresponding new_record_repl valus is false, this won't do anything
and can be removed.  We will use the old value anyways.


+    if (strcmp(eventname, "login") == 0)
I think this block warrants a comment on why it only applies to login triggers.


+        db = (Form_pg_database) GETSTRUCT(tuple);
+        if (!db->dathasloginevt)
+        {
+            db->dathasloginevt = true;
+            CatalogTupleUpdate(pg_db, &tuple->t_self, tuple);
+        }
+        else
+            CacheInvalidateRelcacheByTuple(tuple);
This needs a CommandCounterIncrement inside the if () { .. } block right?


+    /* Get the command tag. */
+    tag = (event == EVT_Login) ? CMDTAG_LOGIN : CreateCommandTag(parsetree);
To match project style I think this should be an if-then-else block.  Also,
while it's tempting to move this before the assertion block in order to reuse
it there, it does mean that we are calling this in a hot codepath before we
know if it's required.  I think we should restore the previous programming with
a separate CreateCommandTag call inside the assertion block and move this one
back underneath the fast-path exit.


+                    CacheInvalidateRelcacheByTuple(tuple);
+            }
+            table_close(pg_db, RowExclusiveLock);
+            heap_freetuple(tuple);
+        }
+    }
+    CommitTransactionCommand();
Since we are commiting the transaction just after closing the table, is the
relcache invalidation going to achieve much?  I guess registering the event
doesn't hurt much?


+        /*
+         * There can be a race condition: a login event trigger may
+         * have been added after the pg_event_trigger table was
+         * scanned, and we don't want to erroneously clear the
+         * dathasloginevt flag in this case. To be sure that this
+         * hasn't happened, repeat the scan under the pg_database
+         * table lock.
+         */
+        AcceptInvalidationMessages();
+        runlist = EventTriggerCommonSetup(NULL,
+                          EVT_Login, "login",
+                          &trigdata);
It seems conceptually wrong to allocate this in the TopTransactionContext when
we only generate the runlist to throw it away.  At the very least I think we
should free the returned list.


+    if (runlist == NULL)    /* list is still empty, so clear the
This should check for NIL and not NULL.


Have you done any benchmarking on this patch?  With this version I see a small
slowdown on connection time of ~1.5% using pgbench for the case where there are
no login event triggers defined.  With a login event trigger calling an empty
plpgsql function there is a ~30% slowdown (corresponding to ~1ms on my system).
When there is a login event trigger defined all bets are off however, since the
function called can be arbitrarily complex and it's up to the user to measure
and decide - but the bare overhead we impose is still of interest of course.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Robert Haas
Дата:
On Tue, Feb 15, 2022 at 5:07 AM Greg Nancarrow <gregn4422@gmail.com> wrote:
> I've attached a re-based version (no functional changes from the
> previous) to fix cfbot failures.

I tried this:

rhaas=# create function on_login_proc() returns event_trigger as
$$begin perform pg_sleep(10000000); end$$ language plpgsql;
CREATE FUNCTION
rhaas=# create event trigger on_login_trigger on login execute
procedure on_login_proc();

When I then attempt to connect via psql, it hangs, as expected. When I
press ^C, psql exits, but the backend process is not cancelled and
just keeps chugging along in the background. The good news is that if
I connect to another database, I can cancel all of the hung sessions
using pg_cancel_backend(), and all of those processes then promptly
exit, and presumably I could accomplish the same thing by sending them
SIGINT directly. But it's still not great behavior. It would be easy
to use up a pile of connection slots inadvertently and have to go to
some trouble to get access to the server again. Since this is a psql
behavior and not a server behavior, one could argue that it's
unrelated to this patch, but in practice this patch seems to increase
the chances of people running into problems quite a lot.

-- 
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-02-15 21:07:15 +1100, Greg Nancarrow wrote:
> Subject: [PATCH v25] Add a new "login" event and login event trigger support.
> 
> The login event occurs when a user logs in to the system.

I think this needs a HUGE warning in the docs about most event triggers
needing to check pg_is_in_recovery() or breaking hot standby. And certainly
the example given needs to include an pg_is_in_recovery() check.


> +   <para>
> +     The <literal>login</literal> event occurs when a user logs in to the
> +     system.
> +     Any bugs in a trigger procedure for this event may prevent successful
> +     login to the system. Such bugs may be fixed after first restarting the
> +     system in single-user mode (as event triggers are disabled in this mode).
> +     See the <xref linkend="app-postgres"/> reference page for details about
> +     using single-user mode.
> +   </para>

I'm strongly against adding any new dependencies on single user mode.

A saner approach might be a superuser-only GUC that can be set as part of the
connection data (e.g. PGOPTIONS='-c ignore_login_event_trigger=true').


> @@ -293,6 +297,27 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO
>      CatalogTupleInsert(tgrel, tuple);
>      heap_freetuple(tuple);
>  
> +    if (strcmp(eventname, "login") == 0)
> +    {
> +        Form_pg_database db;
> +        Relation    pg_db = table_open(DatabaseRelationId, RowExclusiveLock);
> +
> +        /* Set dathasloginevt flag in pg_database */
> +        tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
> +        if (!HeapTupleIsValid(tuple))
> +            elog(ERROR, "cache lookup failed for database %u", MyDatabaseId);
> +        db = (Form_pg_database) GETSTRUCT(tuple);
> +        if (!db->dathasloginevt)
> +        {
> +            db->dathasloginevt = true;
> +            CatalogTupleUpdate(pg_db, &tuple->t_self, tuple);
> +        }
> +        else
> +            CacheInvalidateRelcacheByTuple(tuple);
> +        table_close(pg_db, RowExclusiveLock);
> +        heap_freetuple(tuple);
> +    }
> +
>      /* Depend on owner. */
>      recordDependencyOnOwner(EventTriggerRelationId, trigoid, evtOwner);

Maybe I am confused, but isn't that CacheInvalidateRelcacheByTuple call
*entirely* bogus? CacheInvalidateRelcacheByTuple() expects a pg_class tuple,
but you're passing in a pg_database tuple?  And what is relcache invalidation
even supposed to achieve here?


I think this should mention that ->dathasloginevt is unset on login when
appropriate.



> +/*
> + * Fire login event triggers.
> + */
> +void
> +EventTriggerOnLogin(void)
> +{
> +    List       *runlist;
> +    EventTriggerData trigdata;
> +
> +    /*
> +     * See EventTriggerDDLCommandStart for a discussion about why event
> +     * triggers are disabled in single user mode.
> +     */
> +    if (!IsUnderPostmaster || !OidIsValid(MyDatabaseId))
> +        return;
> +
> +    StartTransactionCommand();
> +
> +    if (DatabaseHasLoginEventTriggers())
> +    {
> +        runlist = EventTriggerCommonSetup(NULL,
> +                                          EVT_Login, "login",
> +                                          &trigdata);
> +
> +        if (runlist != NIL)
> +        {
> +            /*
> +             * Make sure anything the main command did will be visible to the
> +             * event triggers.
> +             */
> +            CommandCounterIncrement();

"Main command"?

It's not clear to my why a CommandCounterIncrement() could be needed here -
which previous writes do you need to make visible?



> +            /*
> +             * Runlist is empty: clear dathasloginevt flag
> +             */
> +            Relation    pg_db = table_open(DatabaseRelationId, RowExclusiveLock);
> +            HeapTuple    tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
> +            Form_pg_database db;
> +
> +            if (!HeapTupleIsValid(tuple))
> +                elog(ERROR, "cache lookup failed for database %u", MyDatabaseId);
> +
> +            db = (Form_pg_database) GETSTRUCT(tuple);
> +            if (db->dathasloginevt)
> +            {
> +                /*
> +                 * There can be a race condition: a login event trigger may
> +                 * have been added after the pg_event_trigger table was
> +                 * scanned, and we don't want to erroneously clear the
> +                 * dathasloginevt flag in this case. To be sure that this
> +                 * hasn't happened, repeat the scan under the pg_database
> +                 * table lock.
> +                 */
> +                AcceptInvalidationMessages();
> +                runlist = EventTriggerCommonSetup(NULL,
> +                                                  EVT_Login, "login",
> +                                                  &trigdata);

This doesn't work. RowExclusiveLock doesn't conflict with another
RowExclusiveLock.

What is the AcceptInvalidationMessages() intending to do here?


> +                if (runlist == NULL)    /* list is still empty, so clear the
> +                                         * flag */
> +                {
> +                    db->dathasloginevt = false;
> +                    CatalogTupleUpdate(pg_db, &tuple->t_self, tuple);
> +                }
> +                else
> +                    CacheInvalidateRelcacheByTuple(tuple);

Copy of the bogus relcache stuff.


Greetings,

Andres Freund



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 12 Mar 2022, at 03:46, Andres Freund <andres@anarazel.de> wrote:

>> +   <para>
>> +     The <literal>login</literal> event occurs when a user logs in to the
>> +     system.
>> +     Any bugs in a trigger procedure for this event may prevent successful
>> +     login to the system. Such bugs may be fixed after first restarting the
>> +     system in single-user mode (as event triggers are disabled in this mode).
>> +     See the <xref linkend="app-postgres"/> reference page for details about
>> +     using single-user mode.
>> +   </para>
>
> I'm strongly against adding any new dependencies on single user mode.
>
> A saner approach might be a superuser-only GUC that can be set as part of the
> connection data (e.g. PGOPTIONS='-c ignore_login_event_trigger=true').

This, and similar approaches, has been discussed in this thread.  I'm not a fan
of holes punched with GUC's like this, but if you plan on removing single-user
mode (as I recall seeing in an initdb thread) altogether then that kills the
discussion. So.

Since we already recommend single-user mode to handle broken event triggers, we
should IMO do something to cover all of them rather than risk ending up with
one disabling GUC per each event type.  Right now we have this on the CREATE
EVENT TRIGGER reference page:

    "Event triggers are disabled in single-user mode (see postgres).  If an
    erroneous event trigger disables the database so much that you can't even
    drop the trigger, restart in single-user mode and you'll be able to do
    that."

Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps?

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-03-13 23:31:03 +0100, Daniel Gustafsson wrote:
> > On 12 Mar 2022, at 03:46, Andres Freund <andres@anarazel.de> wrote:
> 
> >> +   <para>
> >> +     The <literal>login</literal> event occurs when a user logs in to the
> >> +     system.
> >> +     Any bugs in a trigger procedure for this event may prevent successful
> >> +     login to the system. Such bugs may be fixed after first restarting the
> >> +     system in single-user mode (as event triggers are disabled in this mode).
> >> +     See the <xref linkend="app-postgres"/> reference page for details about
> >> +     using single-user mode.
> >> +   </para>
> > 
> > I'm strongly against adding any new dependencies on single user mode.
> > 
> > A saner approach might be a superuser-only GUC that can be set as part of the
> > connection data (e.g. PGOPTIONS='-c ignore_login_event_trigger=true').
> 
> This, and similar approaches, has been discussed in this thread.  I'm not a fan
> of holes punched with GUC's like this, but if you plan on removing single-user
> mode (as I recall seeing in an initdb thread) altogether then that kills the
> discussion. So.

Even if we end up not removing single user mode, I think it's not OK to add
new failure modes that require single user mode to resolve after not-absurd
operations (I'm ok with needing single user mode if somebody does delete from
pg_authid). It's too hard to reach for most.

We already have GUCs like row_security, so it doesn't seem insane to add one
that disables login event triggers. What is the danger that you see?


> Since we already recommend single-user mode to handle broken event triggers, we
> should IMO do something to cover all of them rather than risk ending up with
> one disabling GUC per each event type.  Right now we have this on the CREATE
> EVENT TRIGGER reference page:

IMO the other types of event triggers make it a heck of a lot harder to get
yourself into a situation that you can't get out of...


>     "Event triggers are disabled in single-user mode (see postgres).  If an
>     erroneous event trigger disables the database so much that you can't even
>     drop the trigger, restart in single-user mode and you'll be able to do
>     that."
> Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps? 

Did you mean login instead of event?

Something like it would work for me. It probably should be
GUC_DISALLOW_IN_FILE?

Greetings,

Andres Freund



Re: On login trigger: take three

От
Tom Lane
Дата:
Andres Freund <andres@anarazel.de> writes:
> On 2022-03-13 23:31:03 +0100, Daniel Gustafsson wrote:
>> Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps? 

> Did you mean login instead of event?

> Something like it would work for me. It probably should be
> GUC_DISALLOW_IN_FILE?

Why?  Inserting such a setting in postgresql.conf and restarting
would be the first thing I'd think of if I needed to get out
of a problem.  The only other way is to set it on the postmaster
command line, which is going to be awkward-to-impossible with
most system-provided start scripts.

            regards, tom lane



Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-03-13 19:57:08 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-03-13 23:31:03 +0100, Daniel Gustafsson wrote:
> >> Something like a '-c ignore_event_trigger=<event|all>' GUC perhaps?
>
> > Did you mean login instead of event?
>
> > Something like it would work for me. It probably should be
> > GUC_DISALLOW_IN_FILE?
>
> Why?  Inserting such a setting in postgresql.conf and restarting
> would be the first thing I'd think of if I needed to get out
> of a problem.  The only other way is to set it on the postmaster
> command line, which is going to be awkward-to-impossible with
> most system-provided start scripts.

I was thinking that the way to use it would be to specify it as a client
option. Like PGOPTIONS='-c ignore_event_trigger=login' psql.

Greetings,

Andres Freund



Re: On login trigger: take three

От
Tom Lane
Дата:
Andres Freund <andres@anarazel.de> writes:
> I was thinking that the way to use it would be to specify it as a client
> option. Like PGOPTIONS='-c ignore_event_trigger=login' psql.

Ugh ... that would allow people (at least superusers) to bypass
the login trigger at will, which seems to me to break a lot of
the use-cases for the feature.  I supposed we'd want this to be a
PGC_POSTMASTER setting for security reasons.

            regards, tom lane



Re: On login trigger: take three

От
Andres Freund
Дата:
On 2022-03-13 20:35:44 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I was thinking that the way to use it would be to specify it as a client
> > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql.
> 
> Ugh ... that would allow people (at least superusers) to bypass
> the login trigger at will, which seems to me to break a lot of
> the use-cases for the feature.  I supposed we'd want this to be a
> PGC_POSTMASTER setting for security reasons.

Shrug. This doesn't seem to add actual security to me.



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 14 Mar 2022, at 00:33, Andres Freund <andres@anarazel.de> wrote:

> We already have GUCs like row_security, so it doesn't seem insane to add one
> that disables login event triggers. What is the danger that you see?

My fear is that GUCs like that end up as permanent fixtures in scripts etc
after having been used temporary, and then X timeunits later someone realize
that the backup has never actually really worked due to a subtle issue, or
something else unpleasant.

The row_security GUC is kind of different IMO, as it's required for pg_dump
(though it can be used in the same way as the above).

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Robert Haas
Дата:
On Sun, Mar 13, 2022 at 7:34 PM Andres Freund <andres@anarazel.de> wrote:
> IMO the other types of event triggers make it a heck of a lot harder to get
> yourself into a situation that you can't get out of...

In particular, unless something has changed since I committed this
stuff originally, there's no existing type of event trigger than can
prevent the superuser from logging in and running DROP EVENT TRIGGER
-- or a SELECT on the system catalogs to find out what to drop. That
was very much a deliberate decision on my part.

I think it's fine to require dropping to single-user mode as a way of
recovering from extreme situations where, for example, there are
corrupted database files. If we don't need it even then, cool, but if
we do, I'm not sad. But all we're talking about here is somebody maybe
running a command that perhaps they should not have run. Having to
take the whole system down to recover from that seems excessively
painful.

-- 
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 14 Mar 2022, at 01:08, Andres Freund <andres@anarazel.de> wrote:

> I was thinking that the way to use it would be to specify it as a client
> option. Like PGOPTIONS='-c ignore_event_trigger=login' psql.

Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is
the previously posted v25 login event trigger patch, and 0003 adapts is to
allow ignoring it (0002/0003 split only for illustrative purposes of course).
Is this along the lines of what you were thinking?

--
Daniel Gustafsson        https://vmware.com/


Вложения

Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-03-14 14:47:09 +0100, Daniel Gustafsson wrote:
> > On 14 Mar 2022, at 01:08, Andres Freund <andres@anarazel.de> wrote:
> 
> > I was thinking that the way to use it would be to specify it as a client
> > option. Like PGOPTIONS='-c ignore_event_trigger=login' psql.
> 
> Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is
> the previously posted v25 login event trigger patch, and 0003 adapts is to
> allow ignoring it (0002/0003 split only for illustrative purposes of course).
> Is this along the lines of what you were thinking?

Yes.

Of course there's still the bogus cache invalidation stuff etc that I
commented on upthread.

Greetings,

Andres Freund



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 14 Mar 2022, at 17:10, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2022-03-14 14:47:09 +0100, Daniel Gustafsson wrote:
>>> On 14 Mar 2022, at 01:08, Andres Freund <andres@anarazel.de> wrote:
>>
>>> I was thinking that the way to use it would be to specify it as a client
>>> option. Like PGOPTIONS='-c ignore_event_trigger=login' psql.
>>
>> Attached is a quick PoC/sketch of such a patch, where 0001 adds a guc, 0002 is
>> the previously posted v25 login event trigger patch, and 0003 adapts is to
>> allow ignoring it (0002/0003 split only for illustrative purposes of course).
>> Is this along the lines of what you were thinking?
>
> Yes.
>
> Of course there's still the bogus cache invalidation stuff etc that I
> commented on upthread.

Yeah, that was the previously posted v25 from the author (who adopted it from
the original author).  I took the liberty to quickly poke at the review
comments you had left as well as the ones that I had found to try and progress
the patch.  0001 should really go in it's own thread though to not hide it from
anyone interested who isn't looking at this thread.

--
Daniel Gustafsson        https://vmware.com/


Вложения

Re: On login trigger: take three

От
a.sokolov@postgrespro.ru
Дата:
Daniel Gustafsson писал 2022-03-15 23:52:

> Yeah, that was the previously posted v25 from the author (who adopted 
> it from
> the original author).  I took the liberty to quickly poke at the review
> comments you had left as well as the ones that I had found to try and 
> progress
> the patch.  0001 should really go in it's own thread though to not hide 
> it from
> anyone interested who isn't looking at this thread.
Hi
I got an error while running tests on Windows:
2022-03-17 17:30:02.458 MSK [6920] [unknown] LOG:  no match in usermap 
"regress" for user "regress_user" authenticated as 
"vagrant@WINDOWS-2019"
2022-03-17 17:30:02.458 MSK [6920] [unknown] FATAL:  SSPI authentication 
failed for user "regress_user"
2022-03-17 17:30:02.458 MSK [6920] [unknown] DETAIL:  Connection matched 
pg_hba.conf line 2: "host all all 127.0.0.1/32  sspi include_realm=1 
map=regress"
2022-03-17 17:30:02.526 MSK [3432] FATAL:  could not receive data from 
WAL stream: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

I propose to make such correction:
--- a/src/test/recovery/t/001_stream_rep.pl
+++ b/src/test/recovery/t/001_stream_rep.pl
@@ -14,7 +14,7 @@ my $node_primary = get_new_node('primary');
  # and it needs proper authentication configuration.
  $node_primary->init(
         allows_streaming => 1,
-       auth_extra       => [ '--create-role', 'repl_role' ]);
+       auth_extra       => [ '--create-role', 'repl_role,regress_user' 
]);
  $node_primary->start;
  my $backup_name = 'my_backup';


-- 
Andrey Sokolov
Postgres Professional: http://www.postgrespro.com



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 18 Mar 2022, at 08:24, a.sokolov@postgrespro.ru wrote:

> -       auth_extra       => [ '--create-role', 'repl_role' ]);
> +       auth_extra       => [ '--create-role', 'repl_role,regress_user' ]);

Looking at the test in question it's not entirely clear to me what the original
author really intended here, so I've changed it up a bit to actually test
something and removed the need for the regress_user role.  I've also fixed the
silly mistake highlighted in the postgresql.conf.sample test.

--
Daniel Gustafsson        https://vmware.com/


Вложения

Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-03-18 17:03:39 +0100, Daniel Gustafsson wrote:
> > On 18 Mar 2022, at 08:24, a.sokolov@postgrespro.ru wrote:
> 
> > -       auth_extra       => [ '--create-role', 'repl_role' ]);
> > +       auth_extra       => [ '--create-role', 'repl_role,regress_user' ]);
> 
> Looking at the test in question it's not entirely clear to me what the original
> author really intended here, so I've changed it up a bit to actually test
> something and removed the need for the regress_user role.  I've also fixed the
> silly mistake highlighted in the postgresql.conf.sample test.

docs fail to build: https://cirrus-ci.com/task/5556234047717376?logs=docs_build#L349

Greetings,

Andres Freund



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 22 Mar 2022, at 04:48, Andres Freund <andres@anarazel.de> wrote:

> docs fail to build: https://cirrus-ci.com/task/5556234047717376?logs=docs_build#L349

Ugh, that one was on me and not the original author. Fixed.

--
Daniel Gustafsson        https://vmware.com/


Вложения

Re: On login trigger: take three

От
a.sokolov@postgrespro.ru
Дата:
Daniel Gustafsson писал 2022-03-22 11:43:
>> On 22 Mar 2022, at 04:48, Andres Freund <andres@anarazel.de> wrote:
> 
>> docs fail to build: 
>> https://cirrus-ci.com/task/5556234047717376?logs=docs_build#L349
> 
> Ugh, that one was on me and not the original author. Fixed.
> 

+    data initialization. It is vital that any event trigger using the
+    <literal>login</literal> event checks whether or not the database 
is in
+    recovery.

Does any trigger really have to contain a pg_is_in_recovery() call? In 
this message 
(https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de) 
it was only about triggers on hot standby, which run not read-only 
queries

-- 
Andrey Sokolov
Postgres Professional: http://www.postgrespro.com



Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote:
> +    data initialization. It is vital that any event trigger using the
> +    <literal>login</literal> event checks whether or not the database is in
> +    recovery.
> 
> Does any trigger really have to contain a pg_is_in_recovery() call?

Not *any* trigger, just any trigger that writes.


> In this message
> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de)
> it was only about triggers on hot standby, which run not read-only queries

The problem precisely is that the login triggers run on hot standby nodes, and
that if they do writes, you can't login anymore.

Greetings,

Andres Freund



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote:
> On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote:

>> +    data initialization. It is vital that any event trigger using the
>> +    <literal>login</literal> event checks whether or not the database is in
>> +    recovery.
>>
>> Does any trigger really have to contain a pg_is_in_recovery() call?
>
> Not *any* trigger, just any trigger that writes.

Thats correct, the docs should be updated with something like the below I
reckon.

    It is vital that event trigger using the <literal>login</literal> event
    which has side-effects checks whether or not the database is in recovery to
    ensure they are not performing modifications to hot standby nodes.

>> In this message
>> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de)
>> it was only about triggers on hot standby, which run not read-only queries
>
> The problem precisely is that the login triggers run on hot standby nodes, and
> that if they do writes, you can't login anymore.

Do you think this potential foot-gun is scary enough to reject this patch?
There are lots of creative ways to cause Nagios alerts from ones database, but
this has the potential to do so with a small bug in userland code.  Still, I
kind of like the feature so I'm indecisive.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote:
> > On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote:
> > On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote:
> 
> >> +    data initialization. It is vital that any event trigger using the
> >> +    <literal>login</literal> event checks whether or not the database is in
> >> +    recovery.
> >> 
> >> Does any trigger really have to contain a pg_is_in_recovery() call?
> > 
> > Not *any* trigger, just any trigger that writes.
> 
> Thats correct, the docs should be updated with something like the below I
> reckon.
> 
>     It is vital that event trigger using the <literal>login</literal> event
>     which has side-effects checks whether or not the database is in recovery to
>     ensure they are not performing modifications to hot standby nodes.

Maybe side-effects is a bit too general? Emitting a log message, rejecting a
login, setting some GUCs, etc are all side-effects too.


> >> In this message
> >> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de)
> >> it was only about triggers on hot standby, which run not read-only queries
> > 
> > The problem precisely is that the login triggers run on hot standby nodes, and
> > that if they do writes, you can't login anymore.
> 
> Do you think this potential foot-gun is scary enough to reject this patch?
> There are lots of creative ways to cause Nagios alerts from ones database, but
> this has the potential to do so with a small bug in userland code.  Still, I
> kind of like the feature so I'm indecisive.

It does seem like a huge footgun. But also kinda useful. So I'm really +-0.

Greetings,

Andres Freund



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 28 Mar 2022, at 23:31, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote:
>>> On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote:
>>> On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote:
>>
>>>> +    data initialization. It is vital that any event trigger using the
>>>> +    <literal>login</literal> event checks whether or not the database is in
>>>> +    recovery.
>>>>
>>>> Does any trigger really have to contain a pg_is_in_recovery() call?
>>>
>>> Not *any* trigger, just any trigger that writes.
>>
>> Thats correct, the docs should be updated with something like the below I
>> reckon.
>>
>>    It is vital that event trigger using the <literal>login</literal> event
>>    which has side-effects checks whether or not the database is in recovery to
>>    ensure they are not performing modifications to hot standby nodes.
>
> Maybe side-effects is a bit too general? Emitting a log message, rejecting a
> login, setting some GUCs, etc are all side-effects too.

Good point, it needs to say that modifications that cause WAL to be generated
are prohibited, but in a more user-friendly readable way.  Perhaps in a big red
warning box.

>>>> In this message
>>>> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de)
>>>> it was only about triggers on hot standby, which run not read-only queries
>>>
>>> The problem precisely is that the login triggers run on hot standby nodes, and
>>> that if they do writes, you can't login anymore.
>>
>> Do you think this potential foot-gun is scary enough to reject this patch?
>> There are lots of creative ways to cause Nagios alerts from ones database, but
>> this has the potential to do so with a small bug in userland code.  Still, I
>> kind of like the feature so I'm indecisive.
>
> It does seem like a huge footgun. But also kinda useful. So I'm really +-0.

Looks like we are in agreement here.  I'm going to go over it again and sleep
on it some more before the deadline hits.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Tom Lane
Дата:
Andres Freund <andres@anarazel.de> writes:
> On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote:
>> Do you think this potential foot-gun is scary enough to reject this patch?
>> There are lots of creative ways to cause Nagios alerts from ones database, but
>> this has the potential to do so with a small bug in userland code.  Still, I
>> kind of like the feature so I'm indecisive.

> It does seem like a huge footgun. But also kinda useful. So I'm really +-0.

An on-login trigger is *necessarily* a foot-gun; I don't see that this
particular failure mode makes it any worse than it would be anyway.
There has to be some not-too-difficult-to-use way to bypass a broken
login trigger.  Assuming we are happy with the design for doing that,
might as well accept the hazards.

            regards, tom lane



Re[2]: On login trigger: take three

От
Ivan Panchenko
Дата:
 
Hi,
Tue, March 29, 2022, 0:31 +03:00 from Andres Freund <andres@anarazel.de>:
 
Hi,

On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote:
> > On 28 Mar 2022, at 19:10, Andres Freund <andres@anarazel.de> wrote:
> > On 2022-03-28 15:57:37 +0300, a.sokolov@postgrespro.ru wrote:
>
> >> + data initialization. It is vital that any event trigger using the
> >> + <literal>login</literal> event checks whether or not the database is in
> >> + recovery.
> »
 
> >> Does any trigger really have to contain a pg_is_in_recovery() call?
> >
> > Not *any* trigger, just any trigger that writes.
>
> Thats correct, the docs should be updated with something like the below I
> reckon.
>
> It is vital that event trigger using the <literal>login</literal> event
> which has side-effects checks whether or not the database is in recovery to
> ensure they are not performing modifications to hot standby nodes.

Maybe side-effects is a bit too general? Emitting a log message, rejecting a
login, setting some GUCs, etc are all side-effects too.
Something like this:
 
<important>
    <para>
      The <literal>login</literal> triggers fire also on standby servers.
      To keep them from becoming inaccessible, such triggers should
      avoid writing anything to the database when running on a standby.
      This can be achieved by checking <function>pg_is_in_recovery</function>(), see an example below.
    </para>
</important>
 
Also, please fix a typo in doc/src/sgml/ref/create_event_trigger.sgml :
 
- single-user mode and you'll be able to do that. Even triggers can also be
+ single-user mode and you'll be able to do that. Event triggers can also be
 
Regarding the trigger function example:
It does not do anything if run on a standby. To show that it can do something on a standby to, I propose to move throwing the night exception to the beginning.
So it will be:
 
CREATE OR REPLACE FUNCTION init_session() 
RETURNS event_trigger SECURITY DEFINER LANGUAGE plpgsql AS 
$$ 
DECLARE   hour integer = EXTRACT('hour' FROM current_time);   rec boolean;
BEGIN

-- 1) Forbid logging in late:
IF hour BETWEEN 2 AND 4 THEN  RAISE EXCEPTION 'Login forbidden';  -- do not allow to login these hours
END IF;

-- The remaining stuff cannot be done on standbys,
-- so ensure the database is not in recovery
SELECT pg_is_in_recovery() INTO rec;
IF rec THEN  RETURN;
END IF

-- 2) Assign some roles

IF hour BETWEEN 8 AND 20 THEN         -- at daytime grant the day_worker role  EXECUTE 'REVOKE night_worker FROM ' || quote_ident(session_user);  EXECUTE 'GRANT    day_worker TO '   || quote_ident(session_user);
ELSE                                  -- at other time grant the night_worker role  EXECUTE 'REVOKE day_worker FROM ' || quote_ident(session_user);  EXECUTE 'GRANT  night_worker TO ' || quote_ident(session_user);
END IF;

-- 3) Initialize some user session data

CREATE TEMP TABLE session_storage (x float, y integer);

-- 4) Log the connection time

INSERT INTO user_login_log VALUES (session_user, current_timestamp);

END;
$$;
Finally, let me propose to append to the regression test the following:
 
 
\c 
SELECT dathasloginevt FROM pg_database WHERE datname= :'DBNAME';
 
which should output:
 dathasloginevt 
---------------- f
(1 row)
 
So we can check that removal of the event trigger resets this flag in pg_database. Note that reconnect (\c) is necessary here.
 
Regards,
Ivan
 

> >> In this message
> >> (https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de)
> >> it was only about triggers on hot standby, which run not read-only queries
> >
> > The problem precisely is that the login triggers run on hot standby nodes, and
> > that if they do writes, you can't login anymore.
>
> Do you think this potential foot-gun is scary enough to reject this patch?
> There are lots of creative ways to cause Nagios alerts from ones database, but
> this has the potential to do so with a small bug in userland code. Still, I
> kind of like the feature so I'm indecisive.

It does seem like a huge footgun. But also kinda useful. So I'm really +-0.

Greetings,

Andres Freund
 

Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 30 Mar 2022, at 13:21, Ivan Panchenko <wao@mail.ru> wrote:
> Maybe side-effects is a bit too general? Emitting a log message, rejecting a
> login, setting some GUCs, etc are all side-effects too.
> Something like this:

I've reworded the docs close to what you suggested here.

> Also, please fix a typo in doc/src/sgml/ref/create_event_trigger.sgml :

Done.

> Regarding the trigger function example:
> It does not do anything if run on a standby. To show that it can do something on a standby to, I propose to move
throwingthe night exception to the beginning. 

Good idea, done.

> Finally, let me propose to append to the regression test the following:

Also a good idea, done.

--
Daniel Gustafsson        https://vmware.com/


Вложения

Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 29 Mar 2022, at 00:40, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Andres Freund <andres@anarazel.de> writes:
>> On 2022-03-28 23:27:56 +0200, Daniel Gustafsson wrote:
>>> Do you think this potential foot-gun is scary enough to reject this patch?
>>> There are lots of creative ways to cause Nagios alerts from ones database, but
>>> this has the potential to do so with a small bug in userland code.  Still, I
>>> kind of like the feature so I'm indecisive.
>
>> It does seem like a huge footgun. But also kinda useful. So I'm really +-0.
>
> An on-login trigger is *necessarily* a foot-gun; I don't see that this
> particular failure mode makes it any worse than it would be anyway.

Agreed.

> There has to be some not-too-difficult-to-use way to bypass a broken
> login trigger.  Assuming we are happy with the design for doing that,
> might as well accept the hazards.

The GUC in this patchset seems to be in line with what most in this thread have
preferred, and with that in place (and single-user mode which still works for
this) I think we have that covered.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
a.sokolov@postgrespro.ru
Дата:
Daniel Gustafsson писал 2022-03-30 16:48:
>> On 30 Mar 2022, at 13:21, Ivan Panchenko <wao@mail.ru> wrote:
>> Maybe side-effects is a bit too general? Emitting a log message, 
>> rejecting a
>> login, setting some GUCs, etc are all side-effects too.
>> Something like this:
> 
> I've reworded the docs close to what you suggested here.
> 
>> Also, please fix a typo in doc/src/sgml/ref/create_event_trigger.sgml 
>> :
> 
> Done.
> 
>> Regarding the trigger function example:
>> It does not do anything if run on a standby. To show that it can do 
>> something on a standby to, I propose to move throwing the night 
>> exception to the beginning.
> 
> Good idea, done.
> 
>> Finally, let me propose to append to the regression test the 
>> following:
> 
> Also a good idea, done.
> 
> --
> Daniel Gustafsson        https://vmware.com/

Please fix a typo in doc/src/sgml/event-trigger.sgml: "precvent"

-- 
Andrey Sokolov
Postgres Professional: http://www.postgrespro.com



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 1 Apr 2022, at 09:16, a.sokolov@postgrespro.ru wrote:

> Please fix a typo in doc/src/sgml/event-trigger.sgml: "precvent"

Will do.  With that fixed I think this is ready and unless I find something on
another read through and test pass I hope to be able to push this before the CF
closes today.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
This had bitrotted a fair bit, attached is a rebase along with (mostly)
documentation fixes.  0001 adds a generic GUC for ignoring event triggers and
0002 adds the login event with event trigger support, and hooks it up to the
GUC such that broken triggers wont require single-user mode.  Moving the CF
entry back to Needs Review.

--
Daniel Gustafsson        https://vmware.com/


Вложения

Re: On login trigger: take three

От
Zhihong Yu
Дата:


On Fri, Sep 2, 2022 at 8:37 AM Daniel Gustafsson <daniel@yesql.se> wrote:
This had bitrotted a fair bit, attached is a rebase along with (mostly)
documentation fixes.  0001 adds a generic GUC for ignoring event triggers and
0002 adds the login event with event trigger support, and hooks it up to the
GUC such that broken triggers wont require single-user mode.  Moving the CF
entry back to Needs Review.

--
Daniel Gustafsson               https://vmware.com/

Hi,
For EventTriggerOnLogin():

+           LockSharedObject(DatabaseRelationId, MyDatabaseId, 0, AccessExclusiveLock);
+
+           tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
+
+           if (!HeapTupleIsValid(tuple))
+               elog(ERROR, "cache lookup failed for database %u", MyDatabaseId);

Should the lock be taken after the check (HeapTupleIsValid) is done ?

Cheers

Re: On login trigger: take three

От
Sergey Shinderuk
Дата:
On 02.09.2022 18:36, Daniel Gustafsson wrote:
> This had bitrotted a fair bit, attached is a rebase along with (mostly)
> documentation fixes.  0001 adds a generic GUC for ignoring event triggers and
> 0002 adds the login event with event trigger support, and hooks it up to the
> GUC such that broken triggers wont require single-user mode.  Moving the CF
> entry back to Needs Review.


Hello!

There is a race around setting and clearing of dathasloginevt.

Steps to reproduce:

1. Create a trigger:

   CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$
   BEGIN
     RAISE NOTICE 'You are welcome!';
   END;
   $$ LANGUAGE plpgsql;

   CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE 
on_login_proc();

2. Then drop it, but don't start new sessions:

   DROP EVENT TRIGGER on_login_trigger;

3. Create another trigger, but don't commit yet:

   BEGIN;
   CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE 
on_login_proc();

4. Start a new session. This clears dathasloginevt.

5. Commit the transaction:

   COMMIT;

Now we have a trigger, but it doesn't fire, because dathasloginevt=false.


If two sessions create triggers concurrently, one of them will fail.

Steps:

1. In the first session, start a transaction and create a trigger:

   BEGIN;
   CREATE EVENT TRIGGER on_login_trigger1 ON login EXECUTE PROCEDURE 
on_login_proc();

2. In the second session, create another trigger (this query blocks):

   CREATE EVENT TRIGGER on_login_trigger2 ON login EXECUTE PROCEDURE 
on_login_proc();

3. Commit in the first session:

   COMMIT;

The second session fails:

   postgres=# CREATE EVENT TRIGGER on_login_trigger2 ON login EXECUTE 
PROCEDURE on_login_proc();
   ERROR:  tuple concurrently updated



What else bothers me is that login triggers execute in an environment 
under user control and one has to be very careful. The example trigger 
from the documentation

+DECLARE

+  hour integer = EXTRACT('hour' FROM current_time);

+  rec boolean;

+BEGIN

+-- 1. Forbid logging in between 2AM and 4AM.

+IF hour BETWEEN 2 AND 4 THEN

+  RAISE EXCEPTION 'Login forbidden';

+END IF;


can be bypassed with PGOPTIONS='-c timezone=...'. Probably this is 
nothing new and concerns any SECURITY DEFINER function, but still...


Best regards,

-- 
Sergey Shinderuk        https://postgrespro.com/



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 20 Sep 2022, at 15:43, Sergey Shinderuk <s.shinderuk@postgrespro.ru> wrote:
>
> On 02.09.2022 18:36, Daniel Gustafsson wrote:
>> This had bitrotted a fair bit, attached is a rebase along with (mostly)
>> documentation fixes.  0001 adds a generic GUC for ignoring event triggers and
>> 0002 adds the login event with event trigger support, and hooks it up to the
>> GUC such that broken triggers wont require single-user mode.  Moving the CF
>> entry back to Needs Review.

> There is a race around setting and clearing of dathasloginevt.

Thanks for the report!  The whole dathasloginevt mechanism is IMO too clunky
and unelegant to go ahead with, I wouldn't be surprised if there are other bugs
lurking there.  Since the original authors seems to have retired from the patch
(I've only rebased it to try and help) I am inclined to mark it as returned
with feedback.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Sergey Shinderuk
Дата:
On 20.09.2022 17:10, Daniel Gustafsson wrote:
>> On 20 Sep 2022, at 15:43, Sergey Shinderuk <s.shinderuk@postgrespro.ru> wrote: 
>> There is a race around setting and clearing of dathasloginevt.
> 
> Thanks for the report!  The whole dathasloginevt mechanism is IMO too clunky
> and unelegant to go ahead with, I wouldn't be surprised if there are other bugs
> lurking there.
Indeed.

CREATE DATABASE doesn't copy dathasloginevt from the template database.
ALTER EVENT TRIGGER ... ENABLE doesn't set dathasloginevt.

In both cases triggers are enabled according to \dy output, but don't 
fire. The admin may not notice it immediately.

-- 
Sergey Shinderuk        https://postgrespro.com/



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 20 Sep 2022, at 16:10, Daniel Gustafsson <daniel@yesql.se> wrote:

> Since the original authors seems to have retired from the patch
> (I've only rebased it to try and help) I am inclined to mark it as returned
> with feedback.

Doing so now since no updates have been posted for quite some time and holes
have been poked in the patch.

The GUC for temporarily disabling event triggers, to avoid the need for single-
user mode during troubleshooting, might however be of interest so I will break
that out and post to a new thread.

--
Daniel Gustafsson        https://vmware.com/




Re: On login trigger: take three

От
Ian Lawrence Barwick
Дата:
2022年11月4日(金) 5:23 Daniel Gustafsson <daniel@yesql.se>:
>
> > On 20 Sep 2022, at 16:10, Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > Since the original authors seems to have retired from the patch
> > (I've only rebased it to try and help) I am inclined to mark it as returned
> > with feedback.
>
> Doing so now since no updates have been posted for quite some time and holes
> have been poked in the patch.

I see it was still "Waiting on Author" so I went ahead and changed the status.

> The GUC for temporarily disabling event triggers, to avoid the need for single-
> user mode during troubleshooting, might however be of interest so I will break
> that out and post to a new thread.

For reference this is the new thread:

  https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se

Regards

Ian Barwick



Re: On login trigger: take three

От
Mikhail Gribkov
Дата:

Hi hackers,

 

Since the original authors, as Daniel said,  seems to have retired from the patch, I have allowed myself to continue the patch polishing.

Attached v32 includes fresh rebase and the following fixes:

- Copying dathasloginevt flag during DB creation from template;

- Restoring dathasloginevt flag after re-enabling a disabled trigger;

- Preserving dathasloginevt flag during not-running a trigger because of some filters (running with 'session_replication_role=replica' for example);

- Checking tags during the trigger creation;

The (en/dis)abling GUC was removed from the patch by default because it is now discussed and  supported in a separate thread by Daniel Gustaffson:

    https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9@yesql.se

Still the back-ported version of the Daniel’s patch is attached here too – just for convenience.

 

While the above changes represent the straight work continue, there is another version to consider. Most of the patch problems seem to originate from the dathasloginevt flag support. There are lots of known problems (partly fixed) and probably a lot more unfound. At the same time the flag itself is not a critical element, but just a performance optimizer for logging-in of users who are NOT using the on-login triggers.

I have made a simpler version without the flag and tried to compare the performance. The results show that life without dathasloginevt is still possible. When there is a small number of non-login event triggers, the difference is barely noticeable indeed. To show the difference, I have added 1000 sql_drop event triggers and used pgbench to continuously query the database for “select 1”  with reconnects for 3 seconds. Here are the results. For each version I recorded average connection time with 0 and with 1000 event triggers:

With dathasloginevt flag: 0.609 ms VS 0.611 ms

No-flag version: 0.617 ms VS 0.727 ms

You can see that the difference is noticeable, but not that critical as we can expect for 1000 triggers (while in a vast majority of real cases there would be a much lesser number of triggers). I.e. the flagless version of the patch introduces a huge reliability raise at the cost of a small performance drop.

What do you think? Maybe it’s better to use the flagless version? Or even a small performance drop is considered as unacceptable?

 

The attached files include the two versions of the patch: “v32” for the updated version with flag and “v31_nf” – for the flagless version. Both versions contain “0001“ file for the patch itself and “0002” for back-ported controlling GUC.

--
 best regards,
    Mikhail A. Gribkov



On Fri, Nov 4, 2022 at 3:58 AM Ian Lawrence Barwick <barwick@gmail.com> wrote:
2022年11月4日(金) 5:23 Daniel Gustafsson <daniel@yesql.se>:
>
> > On 20 Sep 2022, at 16:10, Daniel Gustafsson <daniel@yesql.se> wrote:
>
> > Since the original authors seems to have retired from the patch
> > (I've only rebased it to try and help) I am inclined to mark it as returned
> > with feedback.
>
> Doing so now since no updates have been posted for quite some time and holes
> have been poked in the patch.

I see it was still "Waiting on Author" so I went ahead and changed the status.

> The GUC for temporarily disabling event triggers, to avoid the need for single-
> user mode during troubleshooting, might however be of interest so I will break
> that out and post to a new thread.

For reference this is the new thread:

  https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se

Regards

Ian Barwick


Вложения

Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi hackers,
Attached v33 is a new rebase of the flagless version of the patch.  As there were no objections at first glance, I’ll try to post it to the upcoming commitfest, thus the brief recap of all the patch details is below.

v33-On_client_login_event_trigger
The patch introduces a trigger on login event, allowing to fire some actions right on the user connection. This can be useful for  logging or connection check purposes as well as for some personalization of the environment. Usage details are described in the documentation included, but shortly usage is the same as for other triggers: create function returning event_trigger and then create event trigger on login event.

The patch is prepared to be applied to the master branch and tested when applied after e52f8b301ed54aac5162b185b43f5f1e44b6b17e commit by Thomas Munro (Date:   Fri Dec 16 17:36:22 2022 +1300).
Regression tests and documentation included.

A couple of words about what and why I changed compared to the previous author's version.
First, the (en/dis)abling GUC was removed from the patch because ideologically it is a separate feature and nowadays it’s  discussed and  supported in a separate thread by Daniel Gustaffson:
https://www.postgresql.org/message-id/flat/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se
Second, I have removed the dathasloginevt flag. The flag was initially added to the patch for performance reasons and it did the job, although it was quite a clumsy construct causing tons of bugs and could potentially lead to tons more during further PostgreSQL evolution. I have removed the flag and found that the performance drop is not that significant.

And this is an aspect I should describe in more detail.
The possible performance drop is expected as an increased time used to login a user NOT using a login trigger.
First of all, the method of performance check:
    echo 'select 1;' > ./tst.sql
    pgbench -n -C -T3 -f tst.sql -U postgres postgres
The output value "average connection time" is the one I use to compare performance.
Now, what are the results.
* master branch: 0.641 ms
* patched version: 0.644 ms
No significant difference here and it is just what was expected. Based on the patch design the performance drop can be expected when there are lots of event triggers created, but not the login one. Thus I have created 1000 drop triggers (the script for creating triggers is attached too) in the database and repeated the test:
* master branch: 0.646 ms
* patched version: 0.754 ms
For 2000 triggers the patched version connection time is further increased to 0.862. Thus we have a login time rise in about 16.5% per 1000 event triggers in the database. It is a statistically noticeable value, still I don’t think it’s a critical one we should be afraid of.
N.B. The exact values of the login times  slightly differ from the ones I posted in the previous email. Well, that’s the repeatability level we have. This convinces me even more that the observed level of performance drop is acceptable.
--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick
Вложения

Re: On login trigger: take three

От
Nikita Malakhov
Дата:
Hi,

Mikhail, I've checked the patch and previous discussion,
the condition mentioned earlier is still actual:
>The example trigger from the documentation

+DECLARE

+  hour integer = EXTRACT('hour' FROM current_time);

+  rec boolean;

+BEGIN

+-- 1. Forbid logging in between 2AM and 4AM.

+IF hour BETWEEN 2 AND 4 THEN

+  RAISE EXCEPTION 'Login forbidden';

+END IF;


>can be bypassed with PGOPTIONS='-c timezone=...'. Probably this is
>nothing new and concerns any SECURITY DEFINER function, but still...

along with
+IF hour BETWEEN 8 AND 20 THEN
It seems to be a minor security issue, so just in case you haven't noticed it.

On Fri, Dec 16, 2022 at 9:14 PM Mikhail Gribkov <youzhick@gmail.com> wrote:
Hi hackers,
Attached v33 is a new rebase of the flagless version of the patch.  As there were no objections at first glance, I’ll try to post it to the upcoming commitfest, thus the brief recap of all the patch details is below.

v33-On_client_login_event_trigger
The patch introduces a trigger on login event, allowing to fire some actions right on the user connection. This can be useful for  logging or connection check purposes as well as for some personalization of the environment. Usage details are described in the documentation included, but shortly usage is the same as for other triggers: create function returning event_trigger and then create event trigger on login event.

The patch is prepared to be applied to the master branch and tested when applied after e52f8b301ed54aac5162b185b43f5f1e44b6b17e commit by Thomas Munro (Date:   Fri Dec 16 17:36:22 2022 +1300).
Regression tests and documentation included.

A couple of words about what and why I changed compared to the previous author's version.
First, the (en/dis)abling GUC was removed from the patch because ideologically it is a separate feature and nowadays it’s  discussed and  supported in a separate thread by Daniel Gustaffson:
https://www.postgresql.org/message-id/flat/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9%40yesql.se
Second, I have removed the dathasloginevt flag. The flag was initially added to the patch for performance reasons and it did the job, although it was quite a clumsy construct causing tons of bugs and could potentially lead to tons more during further PostgreSQL evolution. I have removed the flag and found that the performance drop is not that significant.

And this is an aspect I should describe in more detail.
The possible performance drop is expected as an increased time used to login a user NOT using a login trigger.
First of all, the method of performance check:
    echo 'select 1;' > ./tst.sql
    pgbench -n -C -T3 -f tst.sql -U postgres postgres
The output value "average connection time" is the one I use to compare performance.
Now, what are the results.
* master branch: 0.641 ms
* patched version: 0.644 ms
No significant difference here and it is just what was expected. Based on the patch design the performance drop can be expected when there are lots of event triggers created, but not the login one. Thus I have created 1000 drop triggers (the script for creating triggers is attached too) in the database and repeated the test:
* master branch: 0.646 ms
* patched version: 0.754 ms
For 2000 triggers the patched version connection time is further increased to 0.862. Thus we have a login time rise in about 16.5% per 1000 event triggers in the database. It is a statistically noticeable value, still I don’t think it’s a critical one we should be afraid of.
N.B. The exact values of the login times  slightly differ from the ones I posted in the previous email. Well, that’s the repeatability level we have. This convinces me even more that the observed level of performance drop is acceptable.
--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick


--
Regards,
Nikita Malakhov
Postgres Professional 

Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi Nikita,

> Mikhail, I've checked the patch and previous discussion,
> the condition mentioned earlier is still actual:

Thanks for pointing this out! My bad, I forgot to fix the documentation example.
Attached v34 has this issue fixed, as well as a couple other problems with the example code.

--
 best regards,
    Mikhail A. Gribkov

Вложения

Re: On login trigger: take three

От
Ted Yu
Дата:


On Sat, Dec 17, 2022 at 3:46 AM Mikhail Gribkov <youzhick@gmail.com> wrote:
Hi Nikita,

> Mikhail, I've checked the patch and previous discussion,
> the condition mentioned earlier is still actual:

Thanks for pointing this out! My bad, I forgot to fix the documentation example.
Attached v34 has this issue fixed, as well as a couple other problems with the example code.

--
 best regards,
    Mikhail A. Gribkov
Hi,

bq. in to the system

'in to' -> into

bq. Any bugs in a trigger procedure

Any bugs -> Any bug

+               if (event == EVT_Login)
+                       dbgtag = CMDTAG_LOGIN;
+               else
+                       dbgtag = CreateCommandTag(parsetree);

The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.

Cheers

Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi Ted,

> bq. in to the system
> 'in to' -> into
> bq. Any bugs in a trigger procedure
> Any bugs -> Any bug

Thanks!  Fixed typos in the attached v35.

>   +               if (event == EVT_Login)
>   +                       dbgtag = CMDTAG_LOGIN;
>   +               else
>   +                       dbgtag = CreateCommandTag(parsetree);
> The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.

It appears twice in fact, both cases are nearby: in the main code and under the assert-checking ifdef.
Moving CMDTAG_LOGIN to CreateCommandTag would change the CreateCommandTag function signature and ideology. CreateCommandTag finds a tag based on the command parse tree, while login event is a specific case when we do not have any command and the parse tree is NULL. Changing CreateCommandTag signature for these two calls looks a little bit overkill as it would lead to changing lots of other places the function is called from.
We could move this snippet to a separate function, but here are the similar concerns I think: it would make sense for a more common or a more complex snippet, but not for two cases of if-else one-liners.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Sat, Dec 17, 2022 at 3:29 PM Ted Yu <yuzhihong@gmail.com> wrote:


On Sat, Dec 17, 2022 at 3:46 AM Mikhail Gribkov <youzhick@gmail.com> wrote:
Hi Nikita,

> Mikhail, I've checked the patch and previous discussion,
> the condition mentioned earlier is still actual:

Thanks for pointing this out! My bad, I forgot to fix the documentation example.
Attached v34 has this issue fixed, as well as a couple other problems with the example code.

--
 best regards,
    Mikhail A. Gribkov
Hi,

bq. in to the system

'in to' -> into

bq. Any bugs in a trigger procedure

Any bugs -> Any bug

+               if (event == EVT_Login)
+                       dbgtag = CMDTAG_LOGIN;
+               else
+                       dbgtag = CreateCommandTag(parsetree);

The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.

Cheers
Вложения

Re: On login trigger: take three

От
Ted Yu
Дата:


On Mon, Dec 19, 2022 at 1:40 AM Mikhail Gribkov <youzhick@gmail.com> wrote:
Hi Ted,

> bq. in to the system
> 'in to' -> into
> bq. Any bugs in a trigger procedure
> Any bugs -> Any bug

Thanks!  Fixed typos in the attached v35.

>   +               if (event == EVT_Login)
>   +                       dbgtag = CMDTAG_LOGIN;
>   +               else
>   +                       dbgtag = CreateCommandTag(parsetree);
> The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.

It appears twice in fact, both cases are nearby: in the main code and under the assert-checking ifdef.
Moving CMDTAG_LOGIN to CreateCommandTag would change the CreateCommandTag function signature and ideology. CreateCommandTag finds a tag based on the command parse tree, while login event is a specific case when we do not have any command and the parse tree is NULL. Changing CreateCommandTag signature for these two calls looks a little bit overkill as it would lead to changing lots of other places the function is called from.
We could move this snippet to a separate function, but here are the similar concerns I think: it would make sense for a more common or a more complex snippet, but not for two cases of if-else one-liners.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick

Hi, Mikhail:
Thanks for the explanation.
It is Okay to keep the current formation of CMDTAG_LOGIN handling.

Cheers

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi


po 19. 12. 2022 v 10:40 odesílatel Mikhail Gribkov <youzhick@gmail.com> napsal:
Hi Ted,

> bq. in to the system
> 'in to' -> into
> bq. Any bugs in a trigger procedure
> Any bugs -> Any bug

Thanks!  Fixed typos in the attached v35.

>   +               if (event == EVT_Login)
>   +                       dbgtag = CMDTAG_LOGIN;
>   +               else
>   +                       dbgtag = CreateCommandTag(parsetree);
> The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.

It appears twice in fact, both cases are nearby: in the main code and under the assert-checking ifdef.
Moving CMDTAG_LOGIN to CreateCommandTag would change the CreateCommandTag function signature and ideology. CreateCommandTag finds a tag based on the command parse tree, while login event is a specific case when we do not have any command and the parse tree is NULL. Changing CreateCommandTag signature for these two calls looks a little bit overkill as it would lead to changing lots of other places the function is called from.
We could move this snippet to a separate function, but here are the similar concerns I think: it would make sense for a more common or a more complex snippet, but not for two cases of if-else one-liners.

I checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.

I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"

pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres

733 tps (master), 727 tps (patched).

I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.

Regards

Pavel


--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Sat, Dec 17, 2022 at 3:29 PM Ted Yu <yuzhihong@gmail.com> wrote:


On Sat, Dec 17, 2022 at 3:46 AM Mikhail Gribkov <youzhick@gmail.com> wrote:
Hi Nikita,

> Mikhail, I've checked the patch and previous discussion,
> the condition mentioned earlier is still actual:

Thanks for pointing this out! My bad, I forgot to fix the documentation example.
Attached v34 has this issue fixed, as well as a couple other problems with the example code.

--
 best regards,
    Mikhail A. Gribkov
Hi,

bq. in to the system

'in to' -> into

bq. Any bugs in a trigger procedure

Any bugs -> Any bug

+               if (event == EVT_Login)
+                       dbgtag = CMDTAG_LOGIN;
+               else
+                       dbgtag = CreateCommandTag(parsetree);

The same snippet appears more than once. It seems CMDTAG_LOGIN handling can be moved into `CreateCommandTag`.

Cheers

Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi Pavel,

Thanks for pointing out the tests. I completely agree that using an exception inside on-login trigger should be tested. It cannot be done via regular *.sql/*.out regress tests, thus I have added another perl test to authentication group doing this.
Attached v36 patch contains this test along with the fresh rebase on master.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
Hi

I checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.

I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"

pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres

733 tps (master), 727 tps (patched).

I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.

Regards

Pavel

Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:
Hi


so 14. 1. 2023 v 22:56 odesílatel Mikhail Gribkov <youzhick@gmail.com> napsal:
Hi Pavel,

Thanks for pointing out the tests. I completely agree that using an exception inside on-login trigger should be tested. It cannot be done via regular *.sql/*.out regress tests, thus I have added another perl test to authentication group doing this.
Attached v36 patch contains this test along with the fresh rebase on master.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
Hi

I checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.

I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"

pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres

733 tps (master), 727 tps (patched).

I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.

Thank you

check-world passed without problems
build doc passed without problems
I think so tests are now enough

I'll mark this patch as ready for committer

Regards

Pavel


 

Regards

Pavel

Re: On login trigger: take three

От
Pavel Stehule
Дата:


ne 15. 1. 2023 v 7:32 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Hi


On Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
Hi

I checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.

I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"

pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres

733 tps (master), 727 tps (patched).

I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.

Thank you

check-world passed without problems
build doc passed without problems
I think so tests are now enough

I'll mark this patch as ready for committer

Unfortunately, I  forgot one important point. There are not any tests related to backup.

I miss pg_dump related tests.

I mark this patch as waiting on the author.

Regards

Pavel



Regards

Pavel


 

Regards

Pavel

Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
  Hi Pavel,

On Mon, Jan 16, 2023 at 9:10 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:


ne 15. 1. 2023 v 7:32 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Hi


On Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
Hi

I checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.

I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"

pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres

733 tps (master), 727 tps (patched).

I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.

Thank you

check-world passed without problems
build doc passed without problems
I think so tests are now enough

I'll mark this patch as ready for committer

Unfortunately, I  forgot one important point. There are not any tests related to backup.

I miss pg_dump related tests.

I mark this patch as waiting on the author.

 

Thanks for noticing this.
I have added sections to pg_dump tests. Attached v37 patch contains these additions along with the fresh rebase on master.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



Вложения

Re: On login trigger: take three

От
Pavel Stehule
Дата:


pá 20. 1. 2023 v 19:46 odesílatel Mikhail Gribkov <youzhick@gmail.com> napsal:
  Hi Pavel,

On Mon, Jan 16, 2023 at 9:10 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:


ne 15. 1. 2023 v 7:32 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
Hi


On Thu, Jan 12, 2023 at 9:51 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
Hi

I checked this patch and it looks well. All tests passed. Together with https://commitfest.postgresql.org/41/4013/ it can be a good feature.

I re-tested impact on performance and for the worst case looks like less than 1% (0.8%). I think it is acceptable. Tested pgbench scenario "SELECT 1"

pgbench -f ~/test.sql -C -c 3 -j 5 -T 100 -P10 postgres

733 tps (master), 727 tps (patched).

I think raising an exception inside should be better tested - not it is only in 001_stream_rep.pl - generally more tests are welcome - there are no tested handling exceptions.

Thank you

check-world passed without problems
build doc passed without problems
I think so tests are now enough

I'll mark this patch as ready for committer

Unfortunately, I  forgot one important point. There are not any tests related to backup.

I miss pg_dump related tests.

I mark this patch as waiting on the author.

 

Thanks for noticing this.
I have added sections to pg_dump tests. Attached v37 patch contains these additions along with the fresh rebase on master.

Thank you

marked as ready for committer

Regards

Pavel
 

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi hackers,

The attached v38 patch is a fresh rebase on master branch.
Nothing has changed beyond rebasing.

And just for convenience, here is a link to the exact message of the thread describing the current approach:

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick


Thank you

marked as ready for committer

Regards

Pavel
 
Вложения

Re: On login trigger: take three

От
"Gregory Stark (as CFM)"
Дата:
It looks like Daniel Gustafsson, Andres, and Tom have all weighed in
on this patch with at least a neutral comment (+-0 from Andres :)

It looks like the main concern was breaking physical replicas and that
there was consensus that as long as single-user mode worked that it
was ok?

So maybe it's time after 2 1/2 years to get this one committed?

-- 
Gregory Stark
As Commitfest Manager



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 6 Mar 2023, at 21:55, Gregory Stark (as CFM) <stark.cfm@gmail.com> wrote:
>
> It looks like Daniel Gustafsson, Andres, and Tom have all weighed in
> on this patch with at least a neutral comment (+-0 from Andres :)

I think the concept of a login event trigger has merits, even though it's kind
of a niche use case.

> It looks like the main concern was breaking physical replicas and that
> there was consensus that as long as single-user mode worked that it
> was ok?

Having a way to not rely on single-user mode and not causing an unacceptable
performance hit (which judging by recent benchmarks might not be an issue?).  I
still intend to revisit this and I hope to get to it during this CF.

--
Daniel Gustafsson




Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2023-03-06 15:55:01 -0500, Gregory Stark (as CFM) wrote:
> It looks like Daniel Gustafsson, Andres, and Tom have all weighed in
> on this patch with at least a neutral comment (+-0 from Andres :)
> 
> It looks like the main concern was breaking physical replicas and that
> there was consensus that as long as single-user mode worked that it
> was ok?

I don't think it's OK with just single user mode as an scape hatch. We need
the GUC that was discussed as part of the thread (and I think there's a patch
for that too).

Greetings,

Andres Freund



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 6 Mar 2023, at 23:10, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2023-03-06 15:55:01 -0500, Gregory Stark (as CFM) wrote:
>> It looks like Daniel Gustafsson, Andres, and Tom have all weighed in
>> on this patch with at least a neutral comment (+-0 from Andres :)
>>
>> It looks like the main concern was breaking physical replicas and that
>> there was consensus that as long as single-user mode worked that it
>> was ok?
>
> I don't think it's OK with just single user mode as an scape hatch. We need
> the GUC that was discussed as part of the thread (and I think there's a patch
> for that too).

This is the patch which originated from this thread:

    https://commitfest.postgresql.org/42/4013/

--
Daniel Gustafsson




Re: On login trigger: take three

От
"Gregory Stark (as CFM)"
Дата:
It looks like the patch is failing a test by not dumping
login_event_trigger_func? I'm guessing there's a race condition in the
test but I don't know. I also don't see the tmp_test_jI6t output file
being preserved in the artifacts anywhere :(

https://cirrus-ci.com/task/6391161871400960?logs=test_world#L2671


[16:16:48.594] # Looks like you failed 1 test of 10350.


# Running: pg_dump --no-sync
--file=/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t/only_dump_measurement.sql
--table-and-children=dump_test.measurement --lock-wait-timeout=180000
postgres
[16:16:27.027](0.154s) ok 6765 - only_dump_measurement: pg_dump runs
.....
[16:16:27.035](0.000s) not ok 6870 - only_dump_measurement: should
dump CREATE FUNCTION dump_test.login_event_trigger_func
[16:16:27.035](0.000s)
[16:16:27.035](0.000s) #   Failed test 'only_dump_measurement: should
dump CREATE FUNCTION dump_test.login_event_trigger_func'
#   at t/002_pg_dump.pl line 4761.
.....
[16:16:48.594] +++ tap check in src/bin/pg_dump +++
[16:16:48.594] [16:16:05] t/001_basic.pl ................ ok 612 ms (
0.01 usr 0.00 sys + 0.24 cusr 0.26 csys = 0.51 CPU)
[16:16:48.594]
[16:16:48.594] # Failed test 'only_dump_measurement: should dump
CREATE FUNCTION dump_test.login_event_trigger_func'
[16:16:48.594] # at t/002_pg_dump.pl line 4761.
[16:16:48.594] # Review only_dump_measurement results in
/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t



Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi Gregory,

Thanks for the note. The problem was that the patch was not aware of yesterday Tom Lane's changes in the test.
It's fixed now: the attached v39 patch contains the updated version along with the freshest rebase on master branch.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Wed, Mar 15, 2023 at 8:45 PM Gregory Stark (as CFM) <stark.cfm@gmail.com> wrote:
It looks like the patch is failing a test by not dumping
login_event_trigger_func? I'm guessing there's a race condition in the
test but I don't know. I also don't see the tmp_test_jI6t output file
being preserved in the artifacts anywhere :(

https://cirrus-ci.com/task/6391161871400960?logs=test_world#L2671


[16:16:48.594] # Looks like you failed 1 test of 10350.


# Running: pg_dump --no-sync
--file=/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t/only_dump_measurement.sql
--table-and-children=dump_test.measurement --lock-wait-timeout=180000
postgres
[16:16:27.027](0.154s) ok 6765 - only_dump_measurement: pg_dump runs
.....
[16:16:27.035](0.000s) not ok 6870 - only_dump_measurement: should
dump CREATE FUNCTION dump_test.login_event_trigger_func
[16:16:27.035](0.000s)
[16:16:27.035](0.000s) #   Failed test 'only_dump_measurement: should
dump CREATE FUNCTION dump_test.login_event_trigger_func'
#   at t/002_pg_dump.pl line 4761.
.....
[16:16:48.594] +++ tap check in src/bin/pg_dump +++
[16:16:48.594] [16:16:05] t/001_basic.pl ................ ok 612 ms (
0.01 usr 0.00 sys + 0.24 cusr 0.26 csys = 0.51 CPU)
[16:16:48.594]
[16:16:48.594] # Failed test 'only_dump_measurement: should dump
CREATE FUNCTION dump_test.login_event_trigger_func'
[16:16:48.594] # at t/002_pg_dump.pl line 4761.
[16:16:48.594] # Review only_dump_measurement results in
/tmp/cirrus-ci-build/src/bin/pg_dump/tmp_check/tmp_test_jI6t
Вложения

Re: On login trigger: take three

От
Robert Haas
Дата:
On Tue, Mar 15, 2022 at 4:52 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> Yeah, that was the previously posted v25 from the author (who adopted it from
> the original author).  I took the liberty to quickly poke at the review
> comments you had left as well as the ones that I had found to try and progress
> the patch.  0001 should really go in it's own thread though to not hide it from
> anyone interested who isn't looking at this thread.

Some comments on 0001:

- In general, I think we should prefer to phrase options in terms of
what is done, rather than what is not done. For instance, the
corresponding GUC for row-level security is row_security={on|off}, not
ignore_row_security.

- I think it's odd that the GUC in question doesn't accept true and
false and our usual synonyms for those values. I suggest that it
should, even if we want to add more possible values later.

- "ignoreing" is mispleled. So is gux-ignore-event-trigger. "Even
triggers" -> "Event triggers".

- Perhaps the documentation for the GUC should mention that the GUC is
not relevant in single-user mode because event triggers don't fire
then anyway.

- "Disable event triggers during the session." isn't a very good
description because there is in theory nothing to prevent this from
being set in postgresql.conf.

Basically, I think 0001 is a good idea -- I'm much more nervous about
0002. I think we should get 0001 polished up and committed.

--
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 22 Mar 2023, at 18:54, Robert Haas <robertmhaas@gmail.com> wrote:

> Basically, I think 0001 is a good idea -- I'm much more nervous about
> 0002. I think we should get 0001 polished up and committed.

Correct me if I'm wrong, but I believe you commented on v27-0001 of the login
event trigger patch series?  Sorry about the confusion if so, this is a very
old and lengthy thread with many twists and turns.  This series was closed
downthread when the original authors of login EVT left, and the 0001 GUC patch
extracted into its own thread.  That patch now lives at:

    https://commitfest.postgresql.org/42/4013/

This thread was then later revived by Mikhail Gribkov but without 0001 instead
referring to the above patch for that part.

The new patch for 0001 is quite different, and I welcome your eyes on that
since I agree with you that it would be a good idea.

--
Daniel Gustafsson




Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi hackers,

The attached v40 patch is a fresh rebase on master branch to actualize the state before the upcoming commitfest.
Nothing has changed beyond rebasing.

And just for convenience, here is a link to the exact message of the thread
describing the current approach:
https://www.postgresql.org/message-id/CAMEv5_vg4aJOoUC74XJm%2B5B7%2BTF1nT-Yhtg%2BawtBOESXG5Grfg%40mail.gmail.com

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Wed, Mar 22, 2023 at 10:38 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> On 22 Mar 2023, at 18:54, Robert Haas <robertmhaas@gmail.com> wrote:

> Basically, I think 0001 is a good idea -- I'm much more nervous about
> 0002. I think we should get 0001 polished up and committed.

Correct me if I'm wrong, but I believe you commented on v27-0001 of the login
event trigger patch series?  Sorry about the confusion if so, this is a very
old and lengthy thread with many twists and turns.  This series was closed
downthread when the original authors of login EVT left, and the 0001 GUC patch
extracted into its own thread.  That patch now lives at:

        https://commitfest.postgresql.org/42/4013/

This thread was then later revived by Mikhail Gribkov but without 0001 instead
referring to the above patch for that part.

The new patch for 0001 is quite different, and I welcome your eyes on that
since I agree with you that it would be a good idea.

--
Daniel Gustafsson



Вложения

Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi!

On Wed, Jun 14, 2023 at 10:49 PM Mikhail Gribkov <youzhick@gmail.com> wrote:
> The attached v40 patch is a fresh rebase on master branch to actualize the state before the upcoming commitfest.
> Nothing has changed beyond rebasing.
>
> And just for convenience, here is a link to the exact message of the thread
> describing the current approach:
> https://www.postgresql.org/message-id/CAMEv5_vg4aJOoUC74XJm%2B5B7%2BTF1nT-Yhtg%2BawtBOESXG5Grfg%40mail.gmail.com

Thank you for the update.  I think the patch is interesting and
demanding.  The code, docs and tests seem to be quite polished
already.  Simultaneously, the thread is long and spread over a long
time.  So, it's easy to lose track.  I'd like to do a short summary of
design issues on this thread.

1. Initially the patch introduced "on login" triggers. It was unclear
if they need to rerun on RESET ALL or DISCARD ALL [1]. The new name is
"client connection" trigger, which seems fine [2].

2. Another question is how to deal with triggers, which hangs or fails
with error [1]. One possible way to workaround that is single-user
mode, which is already advised to workaround the errors in other event
triggers. However, in perspective single-user mode might be deleted.
Also, single-user mode is considered as a worst-case scenario recovery
tool, while it's very easy to block the database connections with
client connection triggers. As addition/alternative to single-user
mode, GUC options to disable all event triggers and/or client
connection triggers. Finally, the patch for the GUC option to disable
all event triggers resides in a separate thread [4]. Apparently that
patch should be committed first [5].

3. Yet another question is connection-time overhead introduced by this
patch. The overhead estimate varies from no measurable overhead [6] to
5% overhead [7]. In order to overcome that, [8] has introduced a
database-level flag indicating whether there are connection triggers.
Later this flag was removed [9] in a hope that the possible overhead
is acceptable.

4. [10] points that there is no clean way to store information about
unsuccessful connections (declined by either authentication or
trigger). However, this is considered out-of-scope for the current
patch, and could be implemented later if needed.

5. It was also pointed out [11] that ^C in psql doesn't cancel
long-running client connection triggers. That might be considered a
psql problem though.

6. It has been also pointed out that [12] all triggers, which write
data to the database, must check pg_is_in_recovery() to work correctly
on standby. That seems to be currently reflected in the documentation.

So, for me the open issues seem to be 2, 3 and 5.  My plan to revive
this patch is to commit the GUC patch [4], recheck the overhead and
probably leave "^C in psql" problem as a separate standalone issue.
Any thoughts?

Links.

1. https://www.postgresql.org/message-id/CAFj8pRBdqdqvkU3mVKzoOnO+jPz-6manRV47CDEa+1jD6x6LFg%40mail.gmail.com
2. https://www.postgresql.org/message-id/CAFj8pRCxdQgHy8Mynk3hz6pFsqQ9BN6Vfgy0MJLtQBAUhWDf3w%40mail.gmail.com
3. https://www.postgresql.org/message-id/E0D5DC61-C490-45BD-A984-E8D56493EC4F%40yesql.se
4. https://www.postgresql.org/message-id/9140106E-F9BF-4D85-8FC8-F2D3C094A6D9@yesql.se
5. https://www.postgresql.org/message-id/20230306221010.gszjoakt5jp7oqpd%40awork3.anarazel.de
6. https://www.postgresql.org/message-id/90760f2d-2f9c-12ab-d2c5-e8e6fb7d08de%40postgrespro.ru
7. https://www.postgresql.org/message-id/CAFj8pRChwu01VLx76nKBVyScHCsd1YnBGiKfDJ6h17g4CSnUBg%40mail.gmail.com
8. https://www.postgresql.org/message-id/4471d472-5dfc-f2b0-ad05-0ff8d0a3bb0c%40postgrespro.ru
9. https://www.postgresql.org/message-id/CAMEv5_vg4aJOoUC74XJm%2B5B7%2BTF1nT-Yhtg%2BawtBOESXG5Grfg%40mail.gmail.com
10. https://www.postgresql.org/message-id/9c897136-4755-dcfc-2d24-b12bcfe4467f%40sigaev.ru
11. https://www.postgresql.org/message-id/CA%2BTgmoZv9f1s797tihx-zXQN4AE4ZFBV5C0K%3DzngbgNu3xNNkg%40mail.gmail.com
12. https://www.postgresql.org/message-id/20220312024652.lvgehszwke4hhove%40alap3.anarazel.de

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 25 Sep 2023, at 11:13, Alexander Korotkov <aekorotkov@gmail.com> wrote:

> I'd like to do a short summary of
> design issues on this thread.

Thanks for summarizing this long thread!

> the patch for the GUC option to disable
> all event triggers resides in a separate thread [4]. Apparently that
> patch should be committed first [5].

I have committed the prerequisite patch for temporarily disabling EVTs via a
GUC in 7750fefdb2.  We should absolutely avoid creating any more dependencies
on single-user mode (yes, I have changed my mind since the beginning of the
thread).

> 3. Yet another question is connection-time overhead introduced by this
> patch. The overhead estimate varies from no measurable overhead [6] to
> 5% overhead [7]. In order to overcome that, [8] has introduced a
> database-level flag indicating whether there are connection triggers.
> Later this flag was removed [9] in a hope that the possible overhead
> is acceptable.

While I disliked the flag, I do think the overhead is problematic.  Last time I
profiled it I found it noticeable, and it seems expensive for such a niche
feature to impact such a hot path.  Maybe you can think of other ways to reduce
the cost here (if it indeed is an issue in the latest version of the patch,
which is not one I've benchmarked)?

> 5. It was also pointed out [11] that ^C in psql doesn't cancel
> long-running client connection triggers. That might be considered a
> psql problem though.

While it is a psql problem, it's exacerbated by a slow login EVT and it breaks
what I would guess is the mental model of many who press ^C in a stalled login.
At the very least we should probably document the risk?

--
Daniel Gustafsson




Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi, Daniel!

On Mon, Sep 25, 2023 at 3:42 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> > On 25 Sep 2023, at 11:13, Alexander Korotkov <aekorotkov@gmail.com> wrote:
>
> > I'd like to do a short summary of
> > design issues on this thread.
>
> Thanks for summarizing this long thread!
>
> > the patch for the GUC option to disable
> > all event triggers resides in a separate thread [4]. Apparently that
> > patch should be committed first [5].
>
> I have committed the prerequisite patch for temporarily disabling EVTs via a
> GUC in 7750fefdb2.  We should absolutely avoid creating any more dependencies
> on single-user mode (yes, I have changed my mind since the beginning of the
> thread).

Thank you for committing 7750fefdb2.  I've revised this patch
according to it.  I've resolved the conflicts, make use of
event_triggers GUC and adjusted some comments.

> > 3. Yet another question is connection-time overhead introduced by this
> > patch. The overhead estimate varies from no measurable overhead [6] to
> > 5% overhead [7]. In order to overcome that, [8] has introduced a
> > database-level flag indicating whether there are connection triggers.
> > Later this flag was removed [9] in a hope that the possible overhead
> > is acceptable.
>
> While I disliked the flag, I do think the overhead is problematic.  Last time I
> profiled it I found it noticeable, and it seems expensive for such a niche
> feature to impact such a hot path.  Maybe you can think of other ways to reduce
> the cost here (if it indeed is an issue in the latest version of the patch,
> which is not one I've benchmarked)?

I don't think I can reproduce the performance regression pointed out
by Pavel Stehule [1].

I run a simple ";" sql script (this script doesn't even get the
snapshot) on my laptop and run it multiple times with event_triggers =
on and event_triggers = off;

pgbench -c 10 -j 10 -M prepared -f 1.sql -P 1 -T 60 -C postgres
event_triggers = on
run1: 2261
run2: 2301
run3: 2281
event_triggers = off
run1: 2321
run2: 2277
run3: 2267

pgbench -c 10 -j 10 -M prepared -f 1.sql -P 1 -T 60 -C postgres
event_triggers = on
run1: 731
run2: 740
run3: 733
event_triggers = off
run1: 739
run2: 734
run3: 731

I can't confirm the measurable overhead.

> > 5. It was also pointed out [11] that ^C in psql doesn't cancel
> > long-running client connection triggers. That might be considered a
> > psql problem though.
>
> While it is a psql problem, it's exacerbated by a slow login EVT and it breaks
> what I would guess is the mental model of many who press ^C in a stalled login.
> At the very least we should probably document the risk?

Done in the attached patch.

Links
1. https://www.postgresql.org/message-id/CAFj8pRChwu01VLx76nKBVyScHCsd1YnBGiKfDJ6h17g4CSnUBg%40mail.gmail.com

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 28 Sep 2023, at 23:50, Alexander Korotkov <aekorotkov@gmail.com> wrote:

> I don't think I can reproduce the performance regression pointed out
> by Pavel Stehule [1].

> I can't confirm the measurable overhead.


Running the same pgbench command on my laptop looking at the average connection
times, and the averaging that over five runs (low/avg/high) I see ~5% increase
over master with the patched version (compiled without assertions and debug):

Patched event_triggers on:  6.858 ms/7.038 ms/7.434 ms
Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms
Master:                     6.676 ms/6.697 ms/6.760 ms

This is all quite unscientific with a lot of jitter so grains of salt are to be
applied, but I find it odd that you don't see any measurable effect.  Are you
seeing the same/similar connection times between master and with this patch
applied?

A few small comments on the patch:

+     prevent successful login to the system. Such bugs may be fixed by
+     restarting the system in single-user mode (as event triggers are
This paragraph should be reworded to recommend the GUC instead of single-user
mode (while retaining mention of single-user mode, just not as the primary
option).


+     Also, it's recommended to evade long-running queries in
s/evade/avoid/ perhaps?


Thanks for working on this!

--
Daniel Gustafsson




Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Fri, Sep 29, 2023 at 1:15 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> > On 28 Sep 2023, at 23:50, Alexander Korotkov <aekorotkov@gmail.com> wrote:
>
> > I don't think I can reproduce the performance regression pointed out
> > by Pavel Stehule [1].
>
> > I can't confirm the measurable overhead.
>
>
> Running the same pgbench command on my laptop looking at the average connection
> times, and the averaging that over five runs (low/avg/high) I see ~5% increase
> over master with the patched version (compiled without assertions and debug):
>
> Patched event_triggers on:  6.858 ms/7.038 ms/7.434 ms
> Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms
> Master:                     6.676 ms/6.697 ms/6.760 ms
>
> This is all quite unscientific with a lot of jitter so grains of salt are to be
> applied, but I find it odd that you don't see any measurable effect.  Are you
> seeing the same/similar connection times between master and with this patch
> applied?

Thank you for doing experiments on your side. I've rechecked. It
appears that I didn't do enough runs, thus I didn't see the overhead
as more than an error. Now, I also can confirm ~5% overhead.

I spent some time thinking about how to overcome this overhead, but I
didn't find a brilliant option.  Previously pg_database flag was
proposed but then criticized as complex and error-prone.  I can also
imagine shmem caching mechanism.  But it would require overcoming
possible race conditions between shared cache invalidation and
transaction commit etc.  So, that would be also complex and
error-prone.  Any better ideas?

> A few small comments on the patch:
>
> +     prevent successful login to the system. Such bugs may be fixed by
> +     restarting the system in single-user mode (as event triggers are
> This paragraph should be reworded to recommend the GUC instead of single-user
> mode (while retaining mention of single-user mode, just not as the primary
> option).
>
>
> +     Also, it's recommended to evade long-running queries in
> s/evade/avoid/ perhaps?

Fixed.

> Thanks for working on this!

Thank you as well!

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Robert Haas
Дата:
Sorry to have gone dark on this for a long time after having been
asked for my input back in March. I'm not having a great time trying
to keep up with email, and the threads getting split up makes it a lot
worse for me.

On Fri, Sep 29, 2023 at 6:15 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> Running the same pgbench command on my laptop looking at the average connection
> times, and the averaging that over five runs (low/avg/high) I see ~5% increase
> over master with the patched version (compiled without assertions and debug):
>
> Patched event_triggers on:  6.858 ms/7.038 ms/7.434 ms
> Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms
> Master:                     6.676 ms/6.697 ms/6.760 ms

This seems kind of crazy to me. Why does it happen? It sounds to me
like we must be doing a lot of extra catalog access to find out
whether there are any on-login event triggers. Like maybe a sequential
scan of pg_event_trigger. Maybe we need to engineer a way to avoid
that. I don't have a brilliant idea off-hand, but I feel like there
should be something we can do. I think a lot of users would say that
logins on PostgreSQL are too slow already.

--
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 2 Oct 2023, at 20:10, Robert Haas <robertmhaas@gmail.com> wrote:
>
> Sorry to have gone dark on this for a long time after having been
> asked for my input back in March. I'm not having a great time trying
> to keep up with email, and the threads getting split up makes it a lot
> worse for me.

Not a problem, thanks for chiming in.

> On Fri, Sep 29, 2023 at 6:15 AM Daniel Gustafsson <daniel@yesql.se> wrote:
>> Running the same pgbench command on my laptop looking at the average connection
>> times, and the averaging that over five runs (low/avg/high) I see ~5% increase
>> over master with the patched version (compiled without assertions and debug):
>>
>> Patched event_triggers on:  6.858 ms/7.038 ms/7.434 ms
>> Patched event_triggers off: 6.601 ms/6.958 ms/7.539 ms
>> Master:                     6.676 ms/6.697 ms/6.760 ms
>
> This seems kind of crazy to me. Why does it happen? It sounds to me
> like we must be doing a lot of extra catalog access to find out
> whether there are any on-login event triggers. Like maybe a sequential
> scan of pg_event_trigger.

That's exactly what happens, the patch is using BuildEventTriggerCache() to
build the hash for EVT which is then checked for login triggers.  This is
clearly the bottleneck and there needs to be a fast-path.  There used to be a
cache flag in an earlier version of the patch but it was a but klugy, a version
of that needs to be reimplemented for this patch to fly.

> I think a lot of users would say that logins on PostgreSQL are too slow already.

Agreed.

--
Daniel Gustafsson




Re: On login trigger: take three

От
Robert Haas
Дата:
On Tue, Oct 3, 2023 at 9:43 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> That's exactly what happens, the patch is using BuildEventTriggerCache() to
> build the hash for EVT which is then checked for login triggers.  This is
> clearly the bottleneck and there needs to be a fast-path.  There used to be a
> cache flag in an earlier version of the patch but it was a but klugy, a version
> of that needs to be reimplemented for this patch to fly.

So I haven't looked at this patch, but we basically saying that only
the superuser can create login triggers, and if they do, those
triggers apply to every single user on the system? That would seem to
be the logical extension of the existing event trigger mechanism, but
it isn't obviously as good of a fit for this case as it is for other
cases where event triggers are a thing.

Changing the catalog representation could be a way around this. What
if you only allowed one login trigger per database, and instead of
being stored in pg_event_trigger, the OID of the function gets
recorded in the pg_database row? Then this would be a lot cheaper
since we have to fetch the pg_database row anyway. Or change the SQL
syntax to something entirely new so you can have different login
triggers for different users -- and maybe users are allowed to create
their own -- but the relevant ones can be found by an index scan
instead of a sequential scan.

I'm just spitballing here. If you think the present design is good and
just want to try to speed it up, I'm not deeply opposed to that. But
it's also not obvious to me how to stick a cache in front of something
that's basically a full-table scan.

--
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi, Robert!

On Tue, Oct 3, 2023 at 5:21 PM Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Oct 3, 2023 at 9:43 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> > That's exactly what happens, the patch is using BuildEventTriggerCache() to
> > build the hash for EVT which is then checked for login triggers.  This is
> > clearly the bottleneck and there needs to be a fast-path.  There used to be a
> > cache flag in an earlier version of the patch but it was a but klugy, a version
> > of that needs to be reimplemented for this patch to fly.
>
> So I haven't looked at this patch, but we basically saying that only
> the superuser can create login triggers, and if they do, those
> triggers apply to every single user on the system? That would seem to
> be the logical extension of the existing event trigger mechanism, but
> it isn't obviously as good of a fit for this case as it is for other
> cases where event triggers are a thing.
>
> Changing the catalog representation could be a way around this. What
> if you only allowed one login trigger per database, and instead of
> being stored in pg_event_trigger, the OID of the function gets
> recorded in the pg_database row? Then this would be a lot cheaper
> since we have to fetch the pg_database row anyway. Or change the SQL
> syntax to something entirely new so you can have different login
> triggers for different users -- and maybe users are allowed to create
> their own -- but the relevant ones can be found by an index scan
> instead of a sequential scan.
>
> I'm just spitballing here. If you think the present design is good and
> just want to try to speed it up, I'm not deeply opposed to that. But
> it's also not obvious to me how to stick a cache in front of something
> that's basically a full-table scan.

Thank you for the interesting ideas. I'd like to try to revive the
version with the flag in pg_database.  Will use other ideas as backup
if no success.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi!

On Tue, Oct 3, 2023 at 8:35 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> Thank you for the interesting ideas. I'd like to try to revive the
> version with the flag in pg_database.  Will use other ideas as backup
> if no success.

I've revived the patch version with pg_database.dathasloginevt flag.
I took v32 version [1] and made the following changes.

 * Incorporate enchantments made on flagless version of patch.
 * Read dathasloginevt during InitPostgres() to prevent extra catalog
access and even more notable StartTransactionCommand() when there are
no login triggers.
 * Hold lock during setting of pg_database.dathasloginevt flag (v32
version actually didn't prevent race condition).
 * Fix AlterEventTrigger() to check event name not trigger name
 * Acquire conditional lock while resetting pg_database.dathasloginevt
flag to prevent new database connection to hang waiting another
transaction to finish.

This version should be good and has no overhead.  Any thoughts?
Daniel, could you please re-run the performance tests?

Links
1. https://www.postgresql.org/message-id/CAMEv5_vDjceLr54WUCNPPVsJs8WBWWsRW826VppNEFoLC1LAEw%40mail.gmail.com

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Andres Freund
Дата:
On 2023-10-09 17:11:25 +0300, Alexander Korotkov wrote:
> This version should be good and has no overhead.  Any thoughts?

I think you forgot to attach the patch?



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Mon, Oct 9, 2023 at 11:58 PM Andres Freund <andres@anarazel.de> wrote:
> On 2023-10-09 17:11:25 +0300, Alexander Korotkov wrote:
> > This version should be good and has no overhead.  Any thoughts?
>
> I think you forgot to attach the patch?

That's it!

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Robert Haas
Дата:
On Mon, Oct 9, 2023 at 10:11 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
>  * Hold lock during setting of pg_database.dathasloginevt flag (v32
> version actually didn't prevent race condition).

So ... how does getting this flag set actually work? And how does
clearing it work?

In the case of row-level security, you have to explicitly enable the
flag on the table level using DDL provided for that purpose. In the
case of relhas{rules,triggers,subclass} the flag is set automatically
as a side-effect of some other operation. I tend to consider that the
latter design is somewhat messy. It's potentially even messier here,
because at least when you add a rule or a trigger to a table you're
expecting to take a lock on the table anyway. I don't think you
necessarily expect creating a login trigger to take a lock on the
database. That's a bit odd and could have visible side effects. And if
you don't, then what happens is that if you create two login triggers
in overlapping transactions, then (1) if there were no login triggers
previously, one of the transactions fails with an internal-looking
error message about a concurrent tuple update and (2) if there were
login triggers previously, then it works fine. That's also a bit weird
and surprising. Now the counter-argument could be that adding new DDL
to enable login triggers for a database is too much cognitive burden
and it's better to have the kind of weird and surprising behavior that
I just discussed. I don't know that I would buy that argument, but it
could be made ... and my real point here is that I don't even see
these trade-offs being discussed. Apologies if they were discussed
earlier and I just missed that; I confess to not having read every
email message on this topic, and some of the ones I did read I read a
long time ago.

> This version should be good and has no overhead.  Any thoughts?
> Daniel, could you please re-run the performance tests?

Is "no overhead" an overly bold claim here?

--
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Andres Freund
Дата:
Hi,

On 2023-10-10 08:18:46 +0300, Alexander Korotkov wrote:
> @@ -968,7 +969,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
>  
>      if (!get_db_info(dbtemplate, ShareLock,
>                       &src_dboid, &src_owner, &src_encoding,
> -                     &src_istemplate, &src_allowconn,
> +                     &src_istemplate, &src_allowconn, &src_hasloginevt,
>                       &src_frozenxid, &src_minmxid, &src_deftablespace,
>                       &src_collate, &src_ctype, &src_iculocale, &src_icurules, &src_locprovider,
>                       &src_collversion))

This isn't your fault, but this imo has become unreadable. Think we ought to
move the information about a database to a struct.


> @@ -296,6 +306,13 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO
>      CatalogTupleInsert(tgrel, tuple);
>      heap_freetuple(tuple);
>  
> +    /*
> +     * Login event triggers have an additional flag in pg_database to allow
> +     * faster lookups in hot codepaths. Set the flag unless already True.
> +     */
> +    if (strcmp(eventname, "login") == 0)
> +        SetDatatabaseHasLoginEventTriggers();

It's not really faster lookups, it's no lookups, right?


>      /* Depend on owner. */
>      recordDependencyOnOwner(EventTriggerRelationId, trigoid, evtOwner);
>  
> @@ -357,6 +374,39 @@ filter_list_to_array(List *filterlist)
>      return PointerGetDatum(construct_array_builtin(data, l, TEXTOID));
>  }
>  
> +/*
> + * Set pg_database.dathasloginevt flag for current database indicating that
> + * current database has on login triggers.
> + */
> +void
> +SetDatatabaseHasLoginEventTriggers(void)
> +{
> +    /* Set dathasloginevt flag in pg_database */
> +    Form_pg_database db;
> +    Relation    pg_db = table_open(DatabaseRelationId, RowExclusiveLock);
> +    HeapTuple    tuple;
> +
> +    /*
> +     * Use shared lock to prevent a conflit with EventTriggerOnLogin() trying
> +     * to reset pg_database.dathasloginevt flag.  Note that we use
> +     * AccessShareLock allowing setters concurently.
> +     */
> +    LockSharedObject(DatabaseRelationId, MyDatabaseId, 0, AccessShareLock);

That seems like a very odd approach - how does this avoid concurrency issues
with one backend setting and another unsetting the flag?  And outside of that,
won't this just lead to concurrently updated tuples?


> +    tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
> +    if (!HeapTupleIsValid(tuple))
> +        elog(ERROR, "cache lookup failed for database %u", MyDatabaseId);
> +    db = (Form_pg_database) GETSTRUCT(tuple);
> +    if (!db->dathasloginevt)
> +    {
> +        db->dathasloginevt = true;
> +        CatalogTupleUpdate(pg_db, &tuple->t_self, tuple);
> +        CommandCounterIncrement();
> +    }
> +    table_close(pg_db, RowExclusiveLock);
> +    heap_freetuple(tuple);
> +}
> +
>  /*
>   * ALTER EVENT TRIGGER foo ENABLE|DISABLE|ENABLE ALWAYS|REPLICA
>   */
> @@ -391,6 +441,10 @@ AlterEventTrigger(AlterEventTrigStmt *stmt)
>  
>      CatalogTupleUpdate(tgrel, &tup->t_self, tup);
>  
> +    if (namestrcmp(&evtForm->evtevent, "login") == 0 &&
> +        tgenabled != TRIGGER_DISABLED)
> +        SetDatatabaseHasLoginEventTriggers();
> +
>      InvokeObjectPostAlterHook(EventTriggerRelationId,
>                                trigoid, 0);
>  
> @@ -557,7 +611,7 @@ filter_event_trigger(CommandTag tag, EventTriggerCacheItem *item)
>  static List *
>  EventTriggerCommonSetup(Node *parsetree,
>                          EventTriggerEvent event, const char *eventstr,
> -                        EventTriggerData *trigdata)
> +                        EventTriggerData *trigdata, bool unfiltered)
>  {
>      CommandTag    tag;
>      List       *cachelist;
> @@ -582,10 +636,15 @@ EventTriggerCommonSetup(Node *parsetree,
>      {
>          CommandTag    dbgtag;
>  
> -        dbgtag = CreateCommandTag(parsetree);
> +        if (event == EVT_Login)
> +            dbgtag = CMDTAG_LOGIN;
> +        else
> +            dbgtag = CreateCommandTag(parsetree);
> +
>          if (event == EVT_DDLCommandStart ||
>              event == EVT_DDLCommandEnd ||
> -            event == EVT_SQLDrop)
> +            event == EVT_SQLDrop ||
> +            event == EVT_Login)
>          {
>              if (!command_tag_event_trigger_ok(dbgtag))
>                  elog(ERROR, "unexpected command tag \"%s\"", GetCommandTagName(dbgtag));
> @@ -604,7 +663,10 @@ EventTriggerCommonSetup(Node *parsetree,
>          return NIL;
>  
>      /* Get the command tag. */
> -    tag = CreateCommandTag(parsetree);
> +    if (event == EVT_Login)
> +        tag = CMDTAG_LOGIN;
> +    else
> +        tag = CreateCommandTag(parsetree);

Seems this bit should instead be in a function, given that you have it in
multiple places.


>  
> +/*
> + * Fire login event triggers if any are present.  The dathasloginevt
> + * pg_database flag is left when an event trigger is dropped, to avoid
> + * complicating the codepath in the case of multiple event triggers.  This
> + * function will instead unset the flag if no trigger is defined.
> + */
> +void
> +EventTriggerOnLogin(void)
> +{
> +    List       *runlist;
> +    EventTriggerData trigdata;
> +
> +    /*
> +     * See EventTriggerDDLCommandStart for a discussion about why event
> +     * triggers are disabled in single user mode or via a GUC.  We also need a
> +     * database connection (some background workers doesn't have it).
> +     */
> +    if (!IsUnderPostmaster || !event_triggers ||
> +        !OidIsValid(MyDatabaseId) || !MyDatabaseHasLoginEventTriggers)
> +        return;
> +
> +    StartTransactionCommand();
> +    runlist = EventTriggerCommonSetup(NULL,
> +                                        EVT_Login, "login",
> +                                        &trigdata, false);
> +
> +    if (runlist != NIL)
> +    {
> +        /*
> +         * Event trigger execution may require an active snapshot.
> +         */
> +        PushActiveSnapshot(GetTransactionSnapshot());
> +
> +        /* Run the triggers. */
> +        EventTriggerInvoke(runlist, &trigdata);
> +
> +        /* Cleanup. */
> +        list_free(runlist);
> +
> +        PopActiveSnapshot();
> +    }
> +    /*
> +     * There is no active login event trigger, but our pg_database.dathasloginevt was set.
> +     * Try to unset this flag.  We use the lock to prevent concurrent
> +     * SetDatatabaseHasLoginEventTriggers(), but we don't want to hang the
> +     * connection waiting on the lock.  Thus, we are just trying to acquire
> +     * the lock conditionally.
> +     */
> +    else if (ConditionalLockSharedObject(DatabaseRelationId, MyDatabaseId,
> +                                         0, AccessExclusiveLock))

Eek. Why are we doing it this way?  I think this is a seriously bad
idea. Maybe it's obvious to you, but it seems much more reasonable to make the
pg_database column an integer and count the number of login event
triggers. When 0, then we don't need to look for login event triggers.



Greetings,

Andres Freund



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi!

Thank you for the review.

On Tue, Oct 10, 2023 at 7:37 PM Andres Freund <andres@anarazel.de> wrote:
> On 2023-10-10 08:18:46 +0300, Alexander Korotkov wrote:
> > @@ -968,7 +969,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
> >
> >       if (!get_db_info(dbtemplate, ShareLock,
> >                                        &src_dboid, &src_owner, &src_encoding,
> > -                                      &src_istemplate, &src_allowconn,
> > +                                      &src_istemplate, &src_allowconn, &src_hasloginevt,
> >                                        &src_frozenxid, &src_minmxid, &src_deftablespace,
> >                                        &src_collate, &src_ctype, &src_iculocale, &src_icurules, &src_locprovider,
> >                                        &src_collversion))
>
> This isn't your fault, but this imo has become unreadable. Think we ought to
> move the information about a database to a struct.

Should I do this in a separate patch?

> > @@ -296,6 +306,13 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO
> >       CatalogTupleInsert(tgrel, tuple);
> >       heap_freetuple(tuple);
> >
> > +     /*
> > +      * Login event triggers have an additional flag in pg_database to allow
> > +      * faster lookups in hot codepaths. Set the flag unless already True.
> > +      */
> > +     if (strcmp(eventname, "login") == 0)
> > +             SetDatatabaseHasLoginEventTriggers();
>
> It's not really faster lookups, it's no lookups, right?

Yes, no extra lookups. Fixed.

> >       /* Depend on owner. */
> >       recordDependencyOnOwner(EventTriggerRelationId, trigoid, evtOwner);
> >
> > @@ -357,6 +374,39 @@ filter_list_to_array(List *filterlist)
> >       return PointerGetDatum(construct_array_builtin(data, l, TEXTOID));
> >  }
> >
> > +/*
> > + * Set pg_database.dathasloginevt flag for current database indicating that
> > + * current database has on login triggers.
> > + */
> > +void
> > +SetDatatabaseHasLoginEventTriggers(void)
> > +{
> > +     /* Set dathasloginevt flag in pg_database */
> > +     Form_pg_database db;
> > +     Relation        pg_db = table_open(DatabaseRelationId, RowExclusiveLock);
> > +     HeapTuple       tuple;
> > +
> > +     /*
> > +      * Use shared lock to prevent a conflit with EventTriggerOnLogin() trying
> > +      * to reset pg_database.dathasloginevt flag.  Note that we use
> > +      * AccessShareLock allowing setters concurently.
> > +      */
> > +     LockSharedObject(DatabaseRelationId, MyDatabaseId, 0, AccessShareLock);
>
> That seems like a very odd approach - how does this avoid concurrency issues
> with one backend setting and another unsetting the flag?  And outside of that,
> won't this just lead to concurrently updated tuples?

When backend unsets the flag, it acquires the lock first.  If lock is
acquired successfully then no other backends hold it. If the
concurrent backend have already inserted an event trigger then we will
detect it by rechecking event list under lock. If the concurrent
backend inserted/enabled event trigger and waiting for the lock, then
it will set the flag once we release the lock.

> > +     tuple = SearchSysCacheCopy1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
> > +     if (!HeapTupleIsValid(tuple))
> > +             elog(ERROR, "cache lookup failed for database %u", MyDatabaseId);
> > +     db = (Form_pg_database) GETSTRUCT(tuple);
> > +     if (!db->dathasloginevt)
> > +     {
> > +             db->dathasloginevt = true;
> > +             CatalogTupleUpdate(pg_db, &tuple->t_self, tuple);
> > +             CommandCounterIncrement();
> > +     }
> > +     table_close(pg_db, RowExclusiveLock);
> > +     heap_freetuple(tuple);
> > +}
> > +
> >  /*
> >   * ALTER EVENT TRIGGER foo ENABLE|DISABLE|ENABLE ALWAYS|REPLICA
> >   */
> > @@ -391,6 +441,10 @@ AlterEventTrigger(AlterEventTrigStmt *stmt)
> >
> >       CatalogTupleUpdate(tgrel, &tup->t_self, tup);
> >
> > +     if (namestrcmp(&evtForm->evtevent, "login") == 0 &&
> > +             tgenabled != TRIGGER_DISABLED)
> > +             SetDatatabaseHasLoginEventTriggers();
> > +
> >       InvokeObjectPostAlterHook(EventTriggerRelationId,
> >                                                         trigoid, 0);
> >
> > @@ -557,7 +611,7 @@ filter_event_trigger(CommandTag tag, EventTriggerCacheItem *item)
> >  static List *
> >  EventTriggerCommonSetup(Node *parsetree,
> >                                               EventTriggerEvent event, const char *eventstr,
> > -                                             EventTriggerData *trigdata)
> > +                                             EventTriggerData *trigdata, bool unfiltered)
> >  {
> >       CommandTag      tag;
> >       List       *cachelist;
> > @@ -582,10 +636,15 @@ EventTriggerCommonSetup(Node *parsetree,
> >       {
> >               CommandTag      dbgtag;
> >
> > -             dbgtag = CreateCommandTag(parsetree);
> > +             if (event == EVT_Login)
> > +                     dbgtag = CMDTAG_LOGIN;
> > +             else
> > +                     dbgtag = CreateCommandTag(parsetree);
> > +
> >               if (event == EVT_DDLCommandStart ||
> >                       event == EVT_DDLCommandEnd ||
> > -                     event == EVT_SQLDrop)
> > +                     event == EVT_SQLDrop ||
> > +                     event == EVT_Login)
> >               {
> >                       if (!command_tag_event_trigger_ok(dbgtag))
> >                               elog(ERROR, "unexpected command tag \"%s\"", GetCommandTagName(dbgtag));
> > @@ -604,7 +663,10 @@ EventTriggerCommonSetup(Node *parsetree,
> >               return NIL;
> >
> >       /* Get the command tag. */
> > -     tag = CreateCommandTag(parsetree);
> > +     if (event == EVT_Login)
> > +             tag = CMDTAG_LOGIN;
> > +     else
> > +             tag = CreateCommandTag(parsetree);
>
> Seems this bit should instead be in a function, given that you have it in
> multiple places.

Done.

> > +/*
> > + * Fire login event triggers if any are present.  The dathasloginevt
> > + * pg_database flag is left when an event trigger is dropped, to avoid
> > + * complicating the codepath in the case of multiple event triggers.  This
> > + * function will instead unset the flag if no trigger is defined.
> > + */
> > +void
> > +EventTriggerOnLogin(void)
> > +{
> > +     List       *runlist;
> > +     EventTriggerData trigdata;
> > +
> > +     /*
> > +      * See EventTriggerDDLCommandStart for a discussion about why event
> > +      * triggers are disabled in single user mode or via a GUC.  We also need a
> > +      * database connection (some background workers doesn't have it).
> > +      */
> > +     if (!IsUnderPostmaster || !event_triggers ||
> > +             !OidIsValid(MyDatabaseId) || !MyDatabaseHasLoginEventTriggers)
> > +             return;
> > +
> > +     StartTransactionCommand();
> > +     runlist = EventTriggerCommonSetup(NULL,
> > +                                                                             EVT_Login, "login",
> > +                                                                             &trigdata, false);
> > +
> > +     if (runlist != NIL)
> > +     {
> > +             /*
> > +              * Event trigger execution may require an active snapshot.
> > +              */
> > +             PushActiveSnapshot(GetTransactionSnapshot());
> > +
> > +             /* Run the triggers. */
> > +             EventTriggerInvoke(runlist, &trigdata);
> > +
> > +             /* Cleanup. */
> > +             list_free(runlist);
> > +
> > +             PopActiveSnapshot();
> > +     }
> > +     /*
> > +      * There is no active login event trigger, but our pg_database.dathasloginevt was set.
> > +      * Try to unset this flag.  We use the lock to prevent concurrent
> > +      * SetDatatabaseHasLoginEventTriggers(), but we don't want to hang the
> > +      * connection waiting on the lock.  Thus, we are just trying to acquire
> > +      * the lock conditionally.
> > +      */
> > +     else if (ConditionalLockSharedObject(DatabaseRelationId, MyDatabaseId,
> > +                                                                              0, AccessExclusiveLock))
>
> Eek. Why are we doing it this way?  I think this is a seriously bad
> idea. Maybe it's obvious to you, but it seems much more reasonable to make the
> pg_database column an integer and count the number of login event
> triggers. When 0, then we don't need to look for login event triggers.

The approach with number of login event trigger obviously will also
work.  But this version with lazy flag cleaning avoids need to handle
every unobvious case we delete the event triggers (cascading deletion
etc).

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi, Robert!

Thank you for your feedback.

On Tue, Oct 10, 2023 at 5:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Oct 9, 2023 at 10:11 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> >  * Hold lock during setting of pg_database.dathasloginevt flag (v32
> > version actually didn't prevent race condition).
>
> So ... how does getting this flag set actually work? And how does
> clearing it work?

I hope I explained that in [1].

> In the case of row-level security, you have to explicitly enable the
> flag on the table level using DDL provided for that purpose. In the
> case of relhas{rules,triggers,subclass} the flag is set automatically
> as a side-effect of some other operation. I tend to consider that the
> latter design is somewhat messy. It's potentially even messier here,
> because at least when you add a rule or a trigger to a table you're
> expecting to take a lock on the table anyway. I don't think you
> necessarily expect creating a login trigger to take a lock on the
> database. That's a bit odd and could have visible side effects. And if
> you don't, then what happens is that if you create two login triggers
> in overlapping transactions, then (1) if there were no login triggers
> previously, one of the transactions fails with an internal-looking
> error message about a concurrent tuple update and (2) if there were
> login triggers previously, then it works fine.

Yep, in v43 it worked that way.  One transaction has to wait for
another finishing update of pg_database tuple, then fails.  This is
obviously ridiculous.  Since overlapping setters of flag will have to
wait anyway, I changed lock mode in v44 for them to
AccessExclusiveLock.  Now, waiting transaction then sees the updated
tuple and doesn't fail.

> That's also a bit weird
> and surprising. Now the counter-argument could be that adding new DDL
> to enable login triggers for a database is too much cognitive burden
> and it's better to have the kind of weird and surprising behavior that
> I just discussed. I don't know that I would buy that argument, but it
> could be made ... and my real point here is that I don't even see
> these trade-offs being discussed. Apologies if they were discussed
> earlier and I just missed that; I confess to not having read every
> email message on this topic, and some of the ones I did read I read a
> long time ago.

I have read the thread quite carefully.  I don't think manual setting
of the flag was discussed.  I do think it would be extra burden for
users, and I would prefer automatic flag as long as it works
transparently and reliably.

> > This version should be good and has no overhead.  Any thoughts?
> > Daniel, could you please re-run the performance tests?
>
> Is "no overhead" an overly bold claim here?

Yes, sure.  I meant no extra lookup.  Hopefully that would mean no
measurable overhead when is no enabled login triggers.

Links
1. https://www.postgresql.org/message-id/CAPpHfdtvvozi%3Dttp8kvJQwuLrP5Q0D_5c4Pw1U67MRXcROB9yA%40mail.gmail.com

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Robert Haas
Дата:
On Tue, Oct 10, 2023 at 3:43 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> Yep, in v43 it worked that way.  One transaction has to wait for
> another finishing update of pg_database tuple, then fails.  This is
> obviously ridiculous.  Since overlapping setters of flag will have to
> wait anyway, I changed lock mode in v44 for them to
> AccessExclusiveLock.  Now, waiting transaction then sees the updated
> tuple and doesn't fail.

Doesn't that mean that if you create the first login trigger in a
database and leave the transaction open, nobody can connect to that
database until the transaction ends?

--
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Oct 10, 2023 at 3:43 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> > Yep, in v43 it worked that way.  One transaction has to wait for
> > another finishing update of pg_database tuple, then fails.  This is
> > obviously ridiculous.  Since overlapping setters of flag will have to
> > wait anyway, I changed lock mode in v44 for them to
> > AccessExclusiveLock.  Now, waiting transaction then sees the updated
> > tuple and doesn't fail.
>
> Doesn't that mean that if you create the first login trigger in a
> database and leave the transaction open, nobody can connect to that
> database until the transaction ends?

It doesn't mean that, because when trying to reset the flag v44 does
conditional lock.  So, if another transaction is holding the log we
will just skip resetting the flag.  So, the flag will be cleared on
the first connection after that transaction ends.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Robert Haas
Дата:
On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote:

> > Doesn't that mean that if you create the first login trigger in a
> > database and leave the transaction open, nobody can connect to that
> > database until the transaction ends?
>
> It doesn't mean that, because when trying to reset the flag v44 does
> conditional lock.  So, if another transaction is holding the log we
> will just skip resetting the flag.  So, the flag will be cleared on
> the first connection after that transaction ends.

But in the scenario I am describing the flag is being set, not reset.

--
Robert Haas
EDB: http://www.enterprisedb.com



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Fri, Oct 13, 2023 at 4:18 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> > > Doesn't that mean that if you create the first login trigger in a
> > > database and leave the transaction open, nobody can connect to that
> > > database until the transaction ends?
> >
> > It doesn't mean that, because when trying to reset the flag v44 does
> > conditional lock.  So, if another transaction is holding the log we
> > will just skip resetting the flag.  So, the flag will be cleared on
> > the first connection after that transaction ends.
>
> But in the scenario I am describing the flag is being set, not reset.

Sorry, it seems I just missed some logical steps.  Imagine, that
transaction A created the first login trigger and hangs open.  Then
the new connection B sees no visible triggers yet, but dathasloginevt
flag is set.  Therefore, connection B tries conditional lock but just
gives up because the lock is held by transaction A.

Also, note that the lock has been just some lock with a custom tag.
It doesn't effectively block the database.  You may think about it as
of custom advisory lock.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Fri, Oct 13, 2023 at 11:26 AM Alexander Korotkov
<aekorotkov@gmail.com> wrote:
> On Fri, Oct 13, 2023 at 4:18 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> > > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > > > Doesn't that mean that if you create the first login trigger in a
> > > > database and leave the transaction open, nobody can connect to that
> > > > database until the transaction ends?
> > >
> > > It doesn't mean that, because when trying to reset the flag v44 does
> > > conditional lock.  So, if another transaction is holding the log we
> > > will just skip resetting the flag.  So, the flag will be cleared on
> > > the first connection after that transaction ends.
> >
> > But in the scenario I am describing the flag is being set, not reset.
>
> Sorry, it seems I just missed some logical steps.  Imagine, that
> transaction A created the first login trigger and hangs open.  Then
> the new connection B sees no visible triggers yet, but dathasloginevt
> flag is set.  Therefore, connection B tries conditional lock but just
> gives up because the lock is held by transaction A.
>
> Also, note that the lock has been just some lock with a custom tag.
> It doesn't effectively block the database.  You may think about it as
> of custom advisory lock.

I've revised the comments about the lock a bit.  I've also run some
tests regarding the connection time (5 runs).

v45, event_triggers=on: avg=3.081ms, min=2.992ms, max=3.146ms
v45, event_triggers=off: avg=3.132ms, min=3.048ms, max=3.186ms
master: 3.080ms, min=3.023ms, max=3.167ms

So, no measurable overhead (not surprising since no extra catalog lookup).
I think this patch is in the commitable shape.  Any objections?

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Sat, Oct 14, 2023 at 2:10 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> On Fri, Oct 13, 2023 at 11:26 AM Alexander Korotkov
> <aekorotkov@gmail.com> wrote:
> > On Fri, Oct 13, 2023 at 4:18 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > > On Thu, Oct 12, 2023 at 6:54 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
> > > > On Thu, Oct 12, 2023 at 8:35 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > >
> > > > > Doesn't that mean that if you create the first login trigger in a
> > > > > database and leave the transaction open, nobody can connect to that
> > > > > database until the transaction ends?
> > > >
> > > > It doesn't mean that, because when trying to reset the flag v44 does
> > > > conditional lock.  So, if another transaction is holding the log we
> > > > will just skip resetting the flag.  So, the flag will be cleared on
> > > > the first connection after that transaction ends.
> > >
> > > But in the scenario I am describing the flag is being set, not reset.
> >
> > Sorry, it seems I just missed some logical steps.  Imagine, that
> > transaction A created the first login trigger and hangs open.  Then
> > the new connection B sees no visible triggers yet, but dathasloginevt
> > flag is set.  Therefore, connection B tries conditional lock but just
> > gives up because the lock is held by transaction A.
> >
> > Also, note that the lock has been just some lock with a custom tag.
> > It doesn't effectively block the database.  You may think about it as
> > of custom advisory lock.
>
> I've revised the comments about the lock a bit.  I've also run some
> tests regarding the connection time (5 runs).
>
> v45, event_triggers=on: avg=3.081ms, min=2.992ms, max=3.146ms
> v45, event_triggers=off: avg=3.132ms, min=3.048ms, max=3.186ms
> master: 3.080ms, min=3.023ms, max=3.167ms
>
> So, no measurable overhead (not surprising since no extra catalog lookup).
> I think this patch is in the commitable shape.  Any objections?

The attached revision fixes test failures spotted by
commitfest.cputube.org.  Also, perl scripts passed perltidy.

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Michael Paquier
Дата:
On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote:
> The attached revision fixes test failures spotted by
> commitfest.cputube.org.  Also, perl scripts passed perltidy.

Still you've missed a few things.  At quick glance:
- The code indentation was off a bit in event_trigger.c.
- 005_login_trigger.pl fails if the code is compiled with
ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS because a WARNING is
reported in test "create tmp objects: err equals".
- 005_sspi.pl is older than the new test 005_login_trigger.pl, could
you rename it with a different number?
--
Michael

Вложения

Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Mon, Oct 16, 2023 at 4:00 AM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote:
> > The attached revision fixes test failures spotted by
> > commitfest.cputube.org.  Also, perl scripts passed perltidy.
>
> Still you've missed a few things.  At quick glance:
> - The code indentation was off a bit in event_trigger.c.
> - 005_login_trigger.pl fails if the code is compiled with
> ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS because a WARNING is
> reported in test "create tmp objects: err equals".
> - 005_sspi.pl is older than the new test 005_login_trigger.pl, could
> you rename it with a different number?

You are very fast and sharp eye!
Thank you for fixing the indentation.  I just pushed fixes for the rest.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi Alexander,

Sorry for my long offline and thanks for the activity. So should we close the patch on the commitfest page now?

By the way I had one more issue with the login trigger tests (quite a rare one though). A race condition may occur on some systems, when oidjoins test starts a moment later than normally and affects logins count for on-login trigger test. Thus I had to split event_trigger and oidjoins tests into separate parallel groups. I'll post this as an independent patch then.

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com
http://www.strava.com/athletes/5085772
phone: +7(916)604-71-12
Telegram: @youzhick



On Mon, Oct 16, 2023 at 4:05 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
On Mon, Oct 16, 2023 at 4:00 AM Michael Paquier <michael@paquier.xyz> wrote:
> On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote:
> > The attached revision fixes test failures spotted by
> > commitfest.cputube.org.  Also, perl scripts passed perltidy.
>
> Still you've missed a few things.  At quick glance:
> - The code indentation was off a bit in event_trigger.c.
> - 005_login_trigger.pl fails if the code is compiled with
> ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS because a WARNING is
> reported in test "create tmp objects: err equals".
> - 005_sspi.pl is older than the new test 005_login_trigger.pl, could
> you rename it with a different number?

You are very fast and sharp eye!
Thank you for fixing the indentation.  I just pushed fixes for the rest.

------
Regards,
Alexander Korotkov

Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi, Mikhail.

On Wed, Oct 18, 2023 at 1:30 PM Mikhail Gribkov <youzhick@gmail.com> wrote:
> Sorry for my long offline and thanks for the activity. So should we close the patch on the commitfest page now?

I have just done this.

> By the way I had one more issue with the login trigger tests (quite a rare one though). A race condition may occur on
somesystems, when oidjoins test starts a moment later than normally and affects logins count for on-login trigger test.
ThusI had to split event_trigger and oidjoins tests into separate parallel groups. I'll post this as an independent
patchthen. 

Thank you for catching it.  Please, post this.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Mikhail Gribkov
Дата:
Hi Alexander,

>> Thank you for catching it.  Please, post this.

Just for a more complete picture of the final state here.
I have posted the described fix (for avoiding race condition in the tests) separately:
https://commitfest.postgresql.org/45/4616/

--
 best regards,
    Mikhail A. Gribkov

e-mail: youzhick@gmail.com



Re: On login trigger: take three

От
Tom Lane
Дата:
Mikhail Gribkov <youzhick@gmail.com> writes:
> Just for a more complete picture of the final state here.
> I have posted the described fix (for avoiding race condition in the tests)
> separately:
> https://commitfest.postgresql.org/45/4616/

It turns out that the TAP test for this feature (006_login_trigger.pl)
also has a race condition:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba&dt=2023-10-28%2003%3A33%3A28

The critical bit of the log is

ack Broken pipe: write( 14, 'SELECT 1;' ) at /usr/pkg/lib/perl5/vendor_perl/5.36.0/IPC/Run/IO.pm line 550.

It looks to me that what happened here is that the backend completed the
authentication handshake, and then the login trigger caused a FATAL exit,
and after than the connected psql session tried to send "SELECT 1" on
an already-closed pipe.  That failed, causing IPC::Run to panic.

mamba is a fairly slow machine and doubtless has timing a bit different
from what this test was created on.  But I doubt there is any way to make
this behavior perfectly stable across a range of machines, so I recommend
just removing the test case involving a fatal exit.

            regards, tom lane



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Sun, Oct 29, 2023 at 6:16 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Mikhail Gribkov <youzhick@gmail.com> writes:
> > Just for a more complete picture of the final state here.
> > I have posted the described fix (for avoiding race condition in the tests)
> > separately:
> > https://commitfest.postgresql.org/45/4616/
>
> It turns out that the TAP test for this feature (006_login_trigger.pl)
> also has a race condition:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba&dt=2023-10-28%2003%3A33%3A28
>
> The critical bit of the log is
>
> ack Broken pipe: write( 14, 'SELECT 1;' ) at /usr/pkg/lib/perl5/vendor_perl/5.36.0/IPC/Run/IO.pm line 550.
>
> It looks to me that what happened here is that the backend completed the
> authentication handshake, and then the login trigger caused a FATAL exit,
> and after than the connected psql session tried to send "SELECT 1" on
> an already-closed pipe.  That failed, causing IPC::Run to panic.
>
> mamba is a fairly slow machine and doubtless has timing a bit different
> from what this test was created on.  But I doubt there is any way to make
> this behavior perfectly stable across a range of machines, so I recommend
> just removing the test case involving a fatal exit.

Makes sense.  Are you good with the attached patch?

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Tom Lane
Дата:
Alexander Korotkov <aekorotkov@gmail.com> writes:
> On Sun, Oct 29, 2023 at 6:16 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It looks to me that what happened here is that the backend completed the
>> authentication handshake, and then the login trigger caused a FATAL exit,
>> and after than the connected psql session tried to send "SELECT 1" on
>> an already-closed pipe.  That failed, causing IPC::Run to panic.

Looking closer, what must have happened is that the psql session ended
before IPC::Run could jam 'SELECT 1' into its stdin.  I wonder if this
could be stabilized by doing 'psql -c "SELECT 1"' and not having to
write anything to the child process stdin?  But I'm not sure this test
case is valuable enough to put a great deal of work into.

>> mamba is a fairly slow machine and doubtless has timing a bit different
>> from what this test was created on.  But I doubt there is any way to make
>> this behavior perfectly stable across a range of machines, so I recommend
>> just removing the test case involving a fatal exit.

> Makes sense.  Are you good with the attached patch?

OK by me, though I note you misspelled "mallory" in the comment.

            regards, tom lane



Re: On login trigger: take three

От
Alexander Lakhin
Дата:
Hello Alexander,

I've discovered one more instability in the event_trigger_login test.
Please look for example at case [1]:
ok 213       + event_trigger                           28946 ms
not ok 214   - event_trigger_login                      6430 ms
ok 215       - fast_default                            19349 ms
ok 216       - tablespace                              44789 ms
1..216
# 1 of 216 tests failed.

--- /home/bf/bf-build/skink-master/HEAD/pgsql/src/test/regress/expected/event_trigger_login.out 2023-10-27 
22:55:12.574139524 +0000
+++ /home/bf/bf-build/skink-master/HEAD/pgsql.build/src/test/regress/results/event_trigger_login.out 2024-01-03 
23:49:50.177461816 +0000
@@ -40,6 +40,6 @@
  SELECT dathasloginevt FROM pg_database WHERE datname= :'DBNAME';
   dathasloginevt
  ----------------
- f
+ t
  (1 row)

2024-01-03 23:49:40.378 UTC [2235175][client backend][3/5949:0] STATEMENT:  REINDEX INDEX CONCURRENTLY
concur_reindex_ind;
...
2024-01-03 23:49:50.340 UTC [2260974][autovacuum worker][5/5439:18812] LOG:  automatic vacuum of table 
"regression.pg_catalog.pg_statistic": index scans: 1
(operations logged around 23:49:50.177461816)

I suspected that this failure was caused by autovacuum, and I've managed to
reproduce it without Valgrind or slowing down a machine.
With /tmp/extra.config:
log_autovacuum_min_duration = 0
autovacuum_naptime = 1
autovacuum = on

I run:
for ((i=1;i<10;i++)); do echo "ITERATION $i"; \
TEMP_CONFIG=/tmp/extra.config \
TESTS=$(printf 'event_trigger_login %.0s' `seq 1000`) \
make -s check-tests || break; done
and get failures on iterations 1, 2, 1...
ok 693       - event_trigger_login                        15 ms
not ok 694   - event_trigger_login                        15 ms
ok 695       - event_trigger_login                        21 ms

Also, I've observed an anomaly after dropping a login event trigger:
CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$
BEGIN RAISE NOTICE 'You are welcome!'; END; $$ LANGUAGE plpgsql;
CREATE EVENT TRIGGER olt ON login EXECUTE PROCEDURE on_login_proc();
SELECT dathasloginevt FROM pg_database WHERE datname= current_database();
  dathasloginevt
----------------
  t
(1 row)

DROP EVENT TRIGGER olt;
SELECT dathasloginevt FROM pg_database WHERE datname= current_database();
  dathasloginevt
----------------
  t
(1 row)

Although after reconnection (\c, as done in the event_trigger_login test),
this query returns 'f'.

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2024-01-03%2023%3A04%3A20
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2023-12-26%2019%3A33%3A02

Best regards,
Alexander



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi, Alexander!

On Sat, Jan 13, 2024 at 6:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> I've discovered one more instability in the event_trigger_login test.
> Please look for example at case [1]:
> ok 213       + event_trigger                           28946 ms
> not ok 214   - event_trigger_login                      6430 ms
> ok 215       - fast_default                            19349 ms
> ok 216       - tablespace                              44789 ms
> 1..216
> # 1 of 216 tests failed.

Thank you for reporting this.

I'm going to take a closer look at this tomorrow.  But I doubt I would
find a solution other than removing the flaky parts of the test.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 13 Jan 2024, at 17:00, Alexander Lakhin <exclusion@gmail.com> wrote:

> DROP EVENT TRIGGER olt;
> SELECT dathasloginevt FROM pg_database WHERE datname= current_database();
>  dathasloginevt
> ----------------
>  t
> (1 row)
>
> Although after reconnection (\c, as done in the event_trigger_login test),
> this query returns 'f'.

This is by design in the patch.  The flag isn't changed on DROP, it is only
cleared on logins iff there is no login event trigger.  The description in the
docs is worded to indicate that it shouldn't be taken as truth for monitoring
purposes:

    "This flag is used internally by PostgreSQL and should not be manually
    altered or read for monitoring purposes."

--
Daniel Gustafsson




Re: On login trigger: take three

От
Daniel Gustafsson
Дата:
> On 13 Jan 2024, at 17:00, Alexander Lakhin <exclusion@gmail.com> wrote:

> I suspected that this failure was caused by autovacuum, and I've managed to
> reproduce it without Valgrind or slowing down a machine.

This might be due to the fact that the cleanup codepath to remove the flag when
there is no login event trigger doesn't block on locking pg_database to avoid
stalling connections.  There are no guarantees when the flag is cleared, so a
test like this will always have potential to be flaky it seems.

--
Daniel Gustafsson




Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Mon, Jan 15, 2024 at 11:29 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> > On 13 Jan 2024, at 17:00, Alexander Lakhin <exclusion@gmail.com> wrote:
>
> > I suspected that this failure was caused by autovacuum, and I've managed to
> > reproduce it without Valgrind or slowing down a machine.
>
> This might be due to the fact that the cleanup codepath to remove the flag when
> there is no login event trigger doesn't block on locking pg_database to avoid
> stalling connections.  There are no guarantees when the flag is cleared, so a
> test like this will always have potential to be flaky it seems.

+1
Thank you, Daniel.  I think you described everything absolutely
correctly.  As I wrote upthread, it doesn't seem much could be done
with this, at least within a regression test.  So, I just removed this
query from the test.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Alexander Lakhin
Дата:
Hello Alexander and Daniel,

Please look at the following query, which triggers an assertion failure on
updating the field dathasloginevt for an entry in pg_database:
SELECT format('CREATE DATABASE db1 LOCALE_PROVIDER icu ICU_LOCALE en ENCODING utf8
ICU_RULES ''' || repeat(' ', 200000) || ''' TEMPLATE template0;')
\gexec
\c db1 -

CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$
BEGIN
   RAISE NOTICE 'You are welcome!';
END;
$$ LANGUAGE plpgsql;

CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc();
DROP EVENT TRIGGER on_login_trigger;

\c

\connect: connection to server on socket "/tmp/.s.PGSQL.5432" failed: server closed the connection unexpectedly

The stack trace of the assertion failure is:
...
#5  0x000055c8699b9b8d in ExceptionalCondition (
     conditionName=conditionName@entry=0x55c869a1f7c0 "HaveRegisteredOrActiveSnapshot()",
     fileName=fileName@entry=0x55c869a1f4c6 "toast_internals.c", lineNumber=lineNumber@entry=669) at assert.c:66
#6  0x000055c86945df0a in init_toast_snapshot (...) at toast_internals.c:669
#7  0x000055c86945dfbe in toast_delete_datum (...) at toast_internals.c:429
#8  0x000055c8694fd1da in toast_tuple_cleanup (...) at toast_helper.c:309
#9  0x000055c8694b55a1 in heap_toast_insert_or_update (...) at heaptoast.c:333
#10 0x000055c8694a8c6c in heap_update (... at heapam.c:3604
#11 0x000055c8694a96cb in simple_heap_update (...) at heapam.c:4062
#12 0x000055c869555b7b in CatalogTupleUpdate (...) at indexing.c:322
#13 0x000055c8695f0725 in EventTriggerOnLogin () at event_trigger.c:957
...

Funnily enough, when Tom Lane was wondering, whether pg_database's toast
table could pose a risk [1], I came to the conclusion that it's impossible,
but that was before the login triggers introduction...

[1] https://www.postgresql.org/message-id/1284094.1695479962%40sss.pgh.pa.us

Best regards,
Alexander



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Tue, Jan 23, 2024 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> Please look at the following query, which triggers an assertion failure on
> updating the field dathasloginevt for an entry in pg_database:
> SELECT format('CREATE DATABASE db1 LOCALE_PROVIDER icu ICU_LOCALE en ENCODING utf8
> ICU_RULES ''' || repeat(' ', 200000) || ''' TEMPLATE template0;')
> \gexec
> \c db1 -
>
> CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$
> BEGIN
>    RAISE NOTICE 'You are welcome!';
> END;
> $$ LANGUAGE plpgsql;
>
> CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc();
> DROP EVENT TRIGGER on_login_trigger;
>
> \c
>
> \connect: connection to server on socket "/tmp/.s.PGSQL.5432" failed: server closed the connection unexpectedly
>
> The stack trace of the assertion failure is:
> ...
> #5  0x000055c8699b9b8d in ExceptionalCondition (
>      conditionName=conditionName@entry=0x55c869a1f7c0 "HaveRegisteredOrActiveSnapshot()",
>      fileName=fileName@entry=0x55c869a1f4c6 "toast_internals.c", lineNumber=lineNumber@entry=669) at assert.c:66
> #6  0x000055c86945df0a in init_toast_snapshot (...) at toast_internals.c:669
> #7  0x000055c86945dfbe in toast_delete_datum (...) at toast_internals.c:429
> #8  0x000055c8694fd1da in toast_tuple_cleanup (...) at toast_helper.c:309
> #9  0x000055c8694b55a1 in heap_toast_insert_or_update (...) at heaptoast.c:333
> #10 0x000055c8694a8c6c in heap_update (... at heapam.c:3604
> #11 0x000055c8694a96cb in simple_heap_update (...) at heapam.c:4062
> #12 0x000055c869555b7b in CatalogTupleUpdate (...) at indexing.c:322
> #13 0x000055c8695f0725 in EventTriggerOnLogin () at event_trigger.c:957
> ...
>
> Funnily enough, when Tom Lane was wondering, whether pg_database's toast
> table could pose a risk [1], I came to the conclusion that it's impossible,
> but that was before the login triggers introduction...
>
> [1] https://www.postgresql.org/message-id/1284094.1695479962%40sss.pgh.pa.us

Thank you for reporting.  I'm looking into this...
I wonder if there is a way to avoid toast update given that we don't
touch the toasted field here.

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
On Tue, Jan 23, 2024 at 11:52 PM Alexander Korotkov
<aekorotkov@gmail.com> wrote:
> On Tue, Jan 23, 2024 at 8:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> > Please look at the following query, which triggers an assertion failure on
> > updating the field dathasloginevt for an entry in pg_database:
> > SELECT format('CREATE DATABASE db1 LOCALE_PROVIDER icu ICU_LOCALE en ENCODING utf8
> > ICU_RULES ''' || repeat(' ', 200000) || ''' TEMPLATE template0;')
> > \gexec
> > \c db1 -
> >
> > CREATE FUNCTION on_login_proc() RETURNS event_trigger AS $$
> > BEGIN
> >    RAISE NOTICE 'You are welcome!';
> > END;
> > $$ LANGUAGE plpgsql;
> >
> > CREATE EVENT TRIGGER on_login_trigger ON login EXECUTE PROCEDURE on_login_proc();
> > DROP EVENT TRIGGER on_login_trigger;
> >
> > \c
> >
> > \connect: connection to server on socket "/tmp/.s.PGSQL.5432" failed: server closed the connection unexpectedly
> >
> > The stack trace of the assertion failure is:
> > ...
> > #5  0x000055c8699b9b8d in ExceptionalCondition (
> >      conditionName=conditionName@entry=0x55c869a1f7c0 "HaveRegisteredOrActiveSnapshot()",
> >      fileName=fileName@entry=0x55c869a1f4c6 "toast_internals.c", lineNumber=lineNumber@entry=669) at assert.c:66
> > #6  0x000055c86945df0a in init_toast_snapshot (...) at toast_internals.c:669
> > #7  0x000055c86945dfbe in toast_delete_datum (...) at toast_internals.c:429
> > #8  0x000055c8694fd1da in toast_tuple_cleanup (...) at toast_helper.c:309
> > #9  0x000055c8694b55a1 in heap_toast_insert_or_update (...) at heaptoast.c:333
> > #10 0x000055c8694a8c6c in heap_update (... at heapam.c:3604
> > #11 0x000055c8694a96cb in simple_heap_update (...) at heapam.c:4062
> > #12 0x000055c869555b7b in CatalogTupleUpdate (...) at indexing.c:322
> > #13 0x000055c8695f0725 in EventTriggerOnLogin () at event_trigger.c:957
> > ...
> >
> > Funnily enough, when Tom Lane was wondering, whether pg_database's toast
> > table could pose a risk [1], I came to the conclusion that it's impossible,
> > but that was before the login triggers introduction...
> >
> > [1] https://www.postgresql.org/message-id/1284094.1695479962%40sss.pgh.pa.us
>
> Thank you for reporting.  I'm looking into this...
> I wonder if there is a way to avoid toast update given that we don't
> touch the toasted field here.

Usage of heap_inplace_update() seems appropriate for me here.  It
avoids trouble with both TOAST and row-level locks.  Alexander, could
you please recheck this fixes the problem.

------
Regards,
Alexander Korotkov

Вложения

Re: On login trigger: take three

От
Alexander Lakhin
Дата:
Hello Alexander,

05.02.2024 02:51, Alexander Korotkov wrote:

> Usage of heap_inplace_update() seems appropriate for me here.  It
> avoids trouble with both TOAST and row-level locks.  Alexander, could
> you please recheck this fixes the problem.

I've re-tested the last problematic scenario and can confirm that the fix
works for it (though it still doesn't prevent the autovacuum issue (with
4b885d01 reverted)), but using heap_inplace_update() was considered risky
in a recent discussion:
https://www.postgresql.org/message-id/1596629.1698435146%40sss.pgh.pa.us
So maybe it's worth to postpone such a fix till that discussion is
finished or to look for another approach...

Best regards,
Alexander



Re: On login trigger: take three

От
Alexander Korotkov
Дата:
Hi, Alexander!

On Mon, Feb 5, 2024 at 7:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> 05.02.2024 02:51, Alexander Korotkov wrote:
>
> > Usage of heap_inplace_update() seems appropriate for me here.  It
> > avoids trouble with both TOAST and row-level locks.  Alexander, could
> > you please recheck this fixes the problem.
>
> I've re-tested the last problematic scenario and can confirm that the fix
> works for it (though it still doesn't prevent the autovacuum issue (with
> 4b885d01 reverted)), but using heap_inplace_update() was considered risky
> in a recent discussion:
> https://www.postgresql.org/message-id/1596629.1698435146%40sss.pgh.pa.us
> So maybe it's worth to postpone such a fix till that discussion is
> finished or to look for another approach...

Thank you for pointing this out.  I don't think there is a particular
problem with this use case for the following reasons.
1) Race conditions over pg_database.dathasloginevt are protected with lock tag.
2) Unsetting pg_database.dathasloginevt of old tuple version shouldn't
cause a problem.  The next tuple version will be updated by further
connections.
However, I agree that it's better to wait for the discussion you've
pointed out before introducing another use case of
heap_inplace_update().

------
Regards,
Alexander Korotkov



Re: On login trigger: take three

От
Bharath Rupireddy
Дата:
On Mon, Oct 16, 2023 at 6:36 AM Alexander Korotkov <aekorotkov@gmail.com> wrote:
>
> > On Mon, Oct 16, 2023 at 02:47:03AM +0300, Alexander Korotkov wrote:
> > > The attached revision fixes test failures spotted by
> > > commitfest.cputube.org.  Also, perl scripts passed perltidy.
>
> You are very fast and sharp eye!
> Thank you for fixing the indentation.  I just pushed fixes for the rest.

I noticed some typos related to this feature.

While on this, the event trigger example for C programming language
shown in the docs doesn't compile as-is. fmgr.h is needed.

Please see the attached patch.

[1]
2024-02-06 05:31:24.453 UTC [1401414] LOG:  tag LOGIN, event login

#include "postgres.h"

#include "commands/event_trigger.h"
#include "fmgr.h"

PG_MODULE_MAGIC;

PG_FUNCTION_INFO_V1(onlogin);

Datum
onlogin(PG_FUNCTION_ARGS)
{
    EventTriggerData *trigdata;

    if (!CALLED_AS_EVENT_TRIGGER(fcinfo))  /* internal error */
        elog(ERROR, "not fired by event trigger manager");

    trigdata = (EventTriggerData *) fcinfo->context;

    elog(LOG, "tag %s, event %s", GetCommandTagName(trigdata->tag),
trigdata->event);

    PG_RETURN_NULL();
}

gcc -shared -fPIC -I /home/ubuntu/postgres/pg17/include/server -o
login_trigger.so login_trigger.c

LOAD '/home/ubuntu/postgres/login_trigger.so';

CREATE FUNCTION onlogin() RETURNS event_trigger
    AS '/home/ubuntu/postgres/login_trigger' LANGUAGE C;

CREATE EVENT TRIGGER onlogin ON login
    EXECUTE FUNCTION onlogin();

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

Вложения