Обсуждение: CVE-2019-9193 about COPY FROM/TO PROGRAM

От:
"Daniel Verite"
Дата:

Hi,

I've noticed this post being currently shared on social media:


https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/cve-2019-9193-authenticated-arbitrary-command-execution-on-postgresql-9-3/

The claim that COPY FROM PROGRAM warrants a CVE seems groundless
because you need to be superuser in the first place to do that.

Apparently these guys have not figured out that a superuser can
also inject arbitrary code with CREATE EXTENSION or even CREATE
FUNCTION since forever, or maybe that will be for a future post?

The CVE itself has not been published, in the sense that it's not
on https://cve.mitre.org, but the ID is reserved.

I don't know if there are precedents of people claiming
CVE entries on Postgres without seemingly reaching out to the
community first. Should something be done proactively about
that particular claim?


Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite



От:
Tom Lane
Дата:

"Daniel Verite" <> writes:
> I've noticed this post being currently shared on social media:

>
https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/cve-2019-9193-authenticated-arbitrary-command-execution-on-postgresql-9-3/

> The claim that COPY FROM PROGRAM warrants a CVE seems groundless
> because you need to be superuser in the first place to do that.

Yeah; this is supposing that there is a security boundary between
Postgres superusers and the OS account running the server, which
there is not.  We could hardly have features like untrusted PLs
if we were trying to maintain such a boundary.

> I don't know if there are precedents of people claiming
> CVE entries on Postgres without seemingly reaching out to the
> community first. Should something be done proactively about
> that particular claim?

Well, it's odd, because somebody at trustwave (not the actual
author of this "research") did reach out to the pgsql-security
list, and we discussed with him that it wasn't a violation of
Postgres' security model, and he agreed.  But then they've
posted this anyway.  Left hand doesn't talk to right hand there,
apparently.

            regards, tom lane



От:
Magnus Hagander
Дата:



On Sat, Mar 30, 2019 at 10:16 PM Tom Lane <> wrote:
"Daniel Verite" <> writes:
> I've noticed this post being currently shared on social media:

> https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/cve-2019-9193-authenticated-arbitrary-command-execution-on-postgresql-9-3/

> The claim that COPY FROM PROGRAM warrants a CVE seems groundless
> because you need to be superuser in the first place to do that.

Yeah; this is supposing that there is a security boundary between
Postgres superusers and the OS account running the server, which
there is not.  We could hardly have features like untrusted PLs
if we were trying to maintain such a boundary.

> I don't know if there are precedents of people claiming
> CVE entries on Postgres without seemingly reaching out to the
> community first. Should something be done proactively about
> that particular claim?

Well, it's odd, because somebody at trustwave (not the actual
author of this "research") did reach out to the pgsql-security
list, and we discussed with him that it wasn't a violation of
Postgres' security model, and he agreed.  But then they've
posted this anyway.  Left hand doesn't talk to right hand there,
apparently.

I wonder if we need to prepare some sort of official response to that.

I was considering writing up a blog post about it, but maybe we need something more official?
 
--
От:
Tom Lane
Дата:

Magnus Hagander <> writes:
> On Sat, Mar 30, 2019 at 10:16 PM Tom Lane <> wrote:
>> Yeah; this is supposing that there is a security boundary between
>> Postgres superusers and the OS account running the server, which
>> there is not.  We could hardly have features like untrusted PLs
>> if we were trying to maintain such a boundary.

> I wonder if we need to prepare some sort of official response to that.
> I was considering writing up a blog post about it, but maybe we need
> something more official?

Blog post seems like a good idea.  As for an "official" response,
it strikes me that maybe we need better documentation.  I'm not sure
that we spell out anywhere what we think the security model is.
There are plenty of scattered warnings about unsafe things, but
if there's any specific statement equivalent to what I just
wrote above, I can't remember where.

(By the same token, I'm not sure where would be a good place
to put it ...)

            regards, tom lane



От:
"Jonathan S. Katz"
Дата:

> On Apr 1, 2019, at 9:55 AM, Tom Lane <> wrote:
>
> Magnus Hagander <> writes:
>>> On Sat, Mar 30, 2019 at 10:16 PM Tom Lane <> wrote:
>>> Yeah; this is supposing that there is a security boundary between
>>> Postgres superusers and the OS account running the server, which
>>> there is not.  We could hardly have features like untrusted PLs
>>> if we were trying to maintain such a boundary.
>
>> I wonder if we need to prepare some sort of official response to that.
>> I was considering writing up a blog post about it, but maybe we need
>> something more official?
>
> Blog post seems like a good idea.  As for an "official" response,
> it strikes me that maybe we need better documentation.

+1, though I’d want to see if people get noisier about it before we rule
out an official response.

A blog post from a reputable author who can speak to security should
be good enough and we can make noise through our various channels.

Jonathan



От:
Alvaro Herrera
Дата:

On 2019-Apr-01, Tom Lane wrote:

> Magnus Hagander <> writes:
> > On Sat, Mar 30, 2019 at 10:16 PM Tom Lane <> wrote:
> >> Yeah; this is supposing that there is a security boundary between
> >> Postgres superusers and the OS account running the server, which
> >> there is not.  We could hardly have features like untrusted PLs
> >> if we were trying to maintain such a boundary.
> 
> > I wonder if we need to prepare some sort of official response to that.
> > I was considering writing up a blog post about it, but maybe we need
> > something more official?
> 
> Blog post seems like a good idea.  As for an "official" response,
> it strikes me that maybe we need better documentation.  I'm not sure
> that we spell out anywhere what we think the security model is.
> There are plenty of scattered warnings about unsafe things, but
> if there's any specific statement equivalent to what I just
> wrote above, I can't remember where.
> 
> (By the same token, I'm not sure where would be a good place
> to put it ...)

Apparently we had a "Security" chapter in version 7.0, which got
removed, and we recently got a complaint about that:
https://postgr.es/m/

I think Peter is right that we may not want to duplicate the contents of
each section, but I think it makes sense to have a chapter in "Part III.
Server Administration", maybe just after chapters 26 or 27, where some
security considerations are put forth with references to the detailed
docs on security for each aspect of the system.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



От:
Michael Paquier
Дата:

On Mon, Apr 01, 2019 at 10:04:32AM -0400, Jonathan S. Katz wrote:
> +1, though I’d want to see if people get noisier about it before we rule
> out an official response.
>
> A blog post from a reputable author who can speak to security should
> be good enough and we can make noise through our various channels.

Need a hand?  Not sure if I am reputable enough though :)

By the way, it could be the occasion to consider an official
PostgreSQL blog on the main website.  News are not really a model
adapted for problem analysis and for entering into technical details.
--
Michael

От:
"Brad Nicholson"
Дата:

Michael Paquier <> wrote on 04/02/2019 01:05:01 AM:

> From: Michael Paquier <>

> To: "Jonathan S. Katz" <>
> Cc: Tom Lane <>, Magnus Hagander
> <>, Daniel Verite <>,
> pgsql-general <>

> Date: 04/02/2019 01:05 AM
> Subject: Re: CVE-2019-9193 about COPY FROM/TO PROGRAM
>
> On Mon, Apr 01, 2019 at 10:04:32AM -0400, Jonathan S. Katz wrote:
> > +1, though I’d want to see if people get noisier about it before we rule
> > out an official response.
> >
> > A blog post from a reputable author who can speak to security should
> > be good enough and we can make noise through our various channels.
>
> Need a hand?  Not sure if I am reputable enough though :)
>
> By the way, it could be the occasion to consider an official
> PostgreSQL blog on the main website.  News are not really a model
> adapted for problem analysis and for entering into technical details.

A blog post would be nice, but it seems to me have something about this clearly in the manual would be best, assuming it's not there already.  I took a quick look, and couldn't find anything.

Brad

От:
"Jonathan S. Katz"
Дата:

On 4/2/19 1:05 AM, Michael Paquier wrote:
> On Mon, Apr 01, 2019 at 10:04:32AM -0400, Jonathan S. Katz wrote:
>> +1, though I’d want to see if people get noisier about it before we rule
>> out an official response.
>>
>> A blog post from a reputable author who can speak to security should
>> be good enough and we can make noise through our various channels.
>
> Need a hand?  Not sure if I am reputable enough though :)

I believe you are, and any blog entries helping the matter are welcome :)

> By the way, it could be the occasion to consider an official
> PostgreSQL blog on the main website.  News are not really a model
> adapted for problem analysis and for entering into technical details.

I think this is warrants a longer discussion, albeit for a different day.

Jonathan


От:
Andres Freund
Дата:

Hi,

On 2019-04-02 07:35:02 -0500, Brad Nicholson wrote:
> Michael Paquier <> wrote on 04/02/2019 01:05:01 AM:
> 
> > From: Michael Paquier <>
> > To: "Jonathan S. Katz" <>
> > Cc: Tom Lane <>, Magnus Hagander
> > <>, Daniel Verite <>,
> > pgsql-general <>
> > Date: 04/02/2019 01:05 AM
> > Subject: Re: CVE-2019-9193 about COPY FROM/TO PROGRAM
> >
> > On Mon, Apr 01, 2019 at 10:04:32AM -0400, Jonathan S. Katz wrote:
> > > +1, though I’d want to see if people get noisier about it before we
> rule
> > > out an official response.
> > >
> > > A blog post from a reputable author who can speak to security should
> > > be good enough and we can make noise through our various channels.
> >
> > Need a hand?  Not sure if I am reputable enough though :)
> >
> > By the way, it could be the occasion to consider an official
> > PostgreSQL blog on the main website.  News are not really a model
> > adapted for problem analysis and for entering into technical details.
> 
> A blog post would be nice, but it seems to me have something about this
> clearly in the manual would be best, assuming it's not there already.  I
> took a quick look, and couldn't find anything.

https://www.postgresql.org/docs/devel/sql-copy.html

"Note that the command is invoked by the shell, so if you need to pass
any arguments to shell command that come from an untrusted source, you
must be careful to strip or escape any special characters that might
have a special meaning for the shell. For security reasons, it is best
to use a fixed command string, or at least avoid passing any user input
in it."

"Similarly, the command specified with PROGRAM is executed directly by
the server, not by the client application, must be executable by the
PostgreSQL user. COPY naming a file or command is only allowed to
database superusers or users who are granted one of the default roles
pg_read_server_files, pg_write_server_files, or
pg_execute_server_program, since it allows reading or writing any file
or running a program that the server has privileges to access."

Those seem reasonable to me?

Greetings,

Andres Freund



От:
Magnus Hagander
Дата:

On Tue, Apr 2, 2019 at 5:31 PM Andres Freund <> wrote:
Hi,

On 2019-04-02 07:35:02 -0500, Brad Nicholson wrote:
> Michael Paquier <> wrote on 04/02/2019 01:05:01 AM:
>
> > From: Michael Paquier <>
> > To: "Jonathan S. Katz" <>
> > Cc: Tom Lane <>, Magnus Hagander
> > <>, Daniel Verite <>,
> > pgsql-general <>
> > Date: 04/02/2019 01:05 AM
> > Subject: Re: CVE-2019-9193 about COPY FROM/TO PROGRAM
> >
> > On Mon, Apr 01, 2019 at 10:04:32AM -0400, Jonathan S. Katz wrote:
> > > +1, though I’d want to see if people get noisier about it before we
> rule
> > > out an official response.
> > >
> > > A blog post from a reputable author who can speak to security should
> > > be good enough and we can make noise through our various channels.
> >
> > Need a hand?  Not sure if I am reputable enough though :)
> >
> > By the way, it could be the occasion to consider an official
> > PostgreSQL blog on the main website.  News are not really a model
> > adapted for problem analysis and for entering into technical details.
>
> A blog post would be nice, but it seems to me have something about this
> clearly in the manual would be best, assuming it's not there already.  I
> took a quick look, and couldn't find anything.

https://www.postgresql.org/docs/devel/sql-copy.html

"Note that the command is invoked by the shell, so if you need to pass
any arguments to shell command that come from an untrusted source, you
must be careful to strip or escape any special characters that might
have a special meaning for the shell. For security reasons, it is best
to use a fixed command string, or at least avoid passing any user input
in it."

"Similarly, the command specified with PROGRAM is executed directly by
the server, not by the client application, must be executable by the
PostgreSQL user. COPY naming a file or command is only allowed to
database superusers or users who are granted one of the default roles
pg_read_server_files, pg_write_server_files, or
pg_execute_server_program, since it allows reading or writing any file
or running a program that the server has privileges to access."

Those seem reasonable to me?

Agreed, that part can't really be much clearer.

But perhaps we should add a warning box to https://www.postgresql.org/docs/11/sql-createrole.html that basically says "creating a superuser means they can x, y and z"?
 
--
От:
"Jonathan S. Katz"
Дата:

On 4/2/19 2:08 PM, Magnus Hagander wrote:
> On Tue, Apr 2, 2019 at 5:31 PM Andres Freund <
> <mailto:>> wrote:
>
>     Hi,
>
>     On 2019-04-02 07:35:02 -0500, Brad Nicholson wrote:
>     > Michael Paquier < <mailto:>>
>     wrote on 04/02/2019 01:05:01 AM:
>     >
>     > > From: Michael Paquier <
>     <mailto:>>
>     > > To: "Jonathan S. Katz" <
>     <mailto:>>
>     > > Cc: Tom Lane < <mailto:>>,
>     Magnus Hagander
>     > > < <mailto:>>, Daniel
>     Verite < <mailto:>>,
>     > > pgsql-general <
>     <mailto:>>
>     > > Date: 04/02/2019 01:05 AM
>     > > Subject: Re: CVE-2019-9193 about COPY FROM/TO PROGRAM
>     > >
>     > > On Mon, Apr 01, 2019 at 10:04:32AM -0400, Jonathan S. Katz wrote:
>     > > > +1, though I’d want to see if people get noisier about it
>     before we
>     > rule
>     > > > out an official response.
>     > > >
>     > > > A blog post from a reputable author who can speak to security
>     should
>     > > > be good enough and we can make noise through our various channels.
>     > >
>     > > Need a hand?  Not sure if I am reputable enough though :)
>     > >
>     > > By the way, it could be the occasion to consider an official
>     > > PostgreSQL blog on the main website.  News are not really a model
>     > > adapted for problem analysis and for entering into technical
>     details.
>     >
>     > A blog post would be nice, but it seems to me have something about
>     this
>     > clearly in the manual would be best, assuming it's not there
>     already.  I
>     > took a quick look, and couldn't find anything.
>
>     https://www.postgresql.org/docs/devel/sql-copy.html
>
>     "Note that the command is invoked by the shell, so if you need to pass
>     any arguments to shell command that come from an untrusted source, you
>     must be careful to strip or escape any special characters that might
>     have a special meaning for the shell. For security reasons, it is best
>     to use a fixed command string, or at least avoid passing any user input
>     in it."
>
>     "Similarly, the command specified with PROGRAM is executed directly by
>     the server, not by the client application, must be executable by the
>     PostgreSQL user. COPY naming a file or command is only allowed to
>     database superusers or users who are granted one of the default roles
>     pg_read_server_files, pg_write_server_files, or
>     pg_execute_server_program, since it allows reading or writing any file
>     or running a program that the server has privileges to access."
>
>     Those seem reasonable to me?
>
>
> Agreed, that part can't really be much clearer.
>
> But perhaps we should add a warning box
> to https://www.postgresql.org/docs/11/sql-createrole.html that basically
> says "creating a superuser means they can x, y and z"?

Yeah, I think that's the path forward -- make it much clearer by putting
it in the warning box and just re-stating that this is what it means.

Jonathan


От:
Magnus Hagander
Дата:

On Mon, Apr 1, 2019 at 4:04 PM Jonathan S. Katz <> wrote:

> On Apr 1, 2019, at 9:55 AM, Tom Lane <> wrote:
>
> Magnus Hagander <> writes:
>>> On Sat, Mar 30, 2019 at 10:16 PM Tom Lane <> wrote:
>>> Yeah; this is supposing that there is a security boundary between
>>> Postgres superusers and the OS account running the server, which
>>> there is not.  We could hardly have features like untrusted PLs
>>> if we were trying to maintain such a boundary.
>
>> I wonder if we need to prepare some sort of official response to that.
>> I was considering writing up a blog post about it, but maybe we need
>> something more official?
>
> Blog post seems like a good idea.  As for an "official" response,
> it strikes me that maybe we need better documentation.

+1, though I’d want to see if people get noisier about it before we rule
out an official response.

A blog post from a reputable author who can speak to security should
be good enough and we can make noise through our various channels.


--
От:
Jeremy Schneider
Дата:

On 4/2/19 05:35, Brad Nicholson wrote:
> A blog post would be nice, but it seems to me have something about this
> clearly in the manual would be best, assuming it's not there already.  I
> took a quick look, and couldn't find anything.

For the record, I don't see any warnings at all in the Oracle docs about
this. Maybe I'm remembering wrong, but I think it's exactly the same
situation there - anyone with full administrative privileges can use
DBMS_SCHEDULER to run OS executables. And I don't think there's a way to
configure Oracle to disable this for people logging in over the network
with administrative privileges.


https://docs.oracle.com/en/database/oracle/oracle-database/19/arpls/DBMS_SCHEDULER.html#GUID-F41A5779-1915-4D5D-A7F5-87727320B742

I'm all for having clear documentation about the security model in
PostgreSQL, but I personally wouldn't be in favor of adding extra
wording to the docs just to pacify concerns about a CVE which may have
been erroneously granted by an assigning authority, who possibly should
have done better due diligence reviewing the content. Particularly if
there's any possibility that the decision to assign the number can be
appealed/changed, though admittedly I know very little about the CVE
process.

Or if this is a legitimate CVE, and if I'm remembering correctly about
Oracle, then maybe the CVE needs to be expanded to cover that database too?

-Jeremy

-- 
Jeremy Schneider
Database Engineer
Amazon Web Services



От:
Tom Lane
Дата:

Jeremy Schneider <> writes:
> I'm all for having clear documentation about the security model in
> PostgreSQL, but I personally wouldn't be in favor of adding extra
> wording to the docs just to pacify concerns about a CVE which may have
> been erroneously granted by an assigning authority, who possibly should
> have done better due diligence reviewing the content. Particularly if
> there's any possibility that the decision to assign the number can be
> appealed/changed, though admittedly I know very little about the CVE
> process.

Just FYI, we have filed a dispute with Mitre about the CVE, and also
reached out to trustwave to try to find out why they filed the CVE
despite the earlier private discussion.

            regards, tom lane



От:
Magnus Hagander
Дата:

On Thu, Apr 4, 2019 at 9:45 PM Tom Lane <> wrote:
Jeremy Schneider <> writes:
> I'm all for having clear documentation about the security model in
> PostgreSQL, but I personally wouldn't be in favor of adding extra
> wording to the docs just to pacify concerns about a CVE which may have
> been erroneously granted by an assigning authority, who possibly should
> have done better due diligence reviewing the content. Particularly if
> there's any possibility that the decision to assign the number can be
> appealed/changed, though admittedly I know very little about the CVE
> process.

Just FYI, we have filed a dispute with Mitre about the CVE, and also
reached out to trustwave to try to find out why they filed the CVE
despite the earlier private discussion.

The original author has also pretty much acknowledged in comments on his blog and on twitter that it's not actually a vulnerability. (He doesn't agree with the design decision, which is apparently enough for a high scoring CVE registration).
 
--
От:
Andres Freund
Дата:

Hi

On 2019-04-04 21:50:41 +0200, Magnus Hagander wrote:
> On Thu, Apr 4, 2019 at 9:45 PM Tom Lane <> wrote:
> 
> > Jeremy Schneider <> writes:
> > > I'm all for having clear documentation about the security model in
> > > PostgreSQL, but I personally wouldn't be in favor of adding extra
> > > wording to the docs just to pacify concerns about a CVE which may have
> > > been erroneously granted by an assigning authority, who possibly should
> > > have done better due diligence reviewing the content. Particularly if
> > > there's any possibility that the decision to assign the number can be
> > > appealed/changed, though admittedly I know very little about the CVE
> > > process.
> >
> > Just FYI, we have filed a dispute with Mitre about the CVE, and also
> > reached out to trustwave to try to find out why they filed the CVE
> > despite the earlier private discussion.
> >
> 
> The original author has also pretty much acknowledged in comments on his
> blog and on twitter that it's not actually a vulnerability. (He doesn't
> agree with the design decision, which is apparently enough for a high
> scoring CVE registration).

Btw, the xp_cmdshell thing the author references several times?
It can be enabled via tsql if you have a privileged account.


https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/xp-cmdshell-server-configuration-option?view=sql-server-2017

and it allows to execute shell code (as a specified user) even when not
a sysadmin:

https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql?view=sql-server-2017#xp_cmdshell-proxy-account

Greetings,

Andres Freund



От:
Jeff Janes
Дата:

On Tue, Apr 2, 2019 at 11:31 AM Andres Freund <> wrote:
Hi,

On 2019-04-02 07:35:02 -0500, Brad Nicholson wrote:

> A blog post would be nice, but it seems to me have something about this
> clearly in the manual would be best, assuming it's not there already.  I
> took a quick look, and couldn't find anything.

https://www.postgresql.org/docs/devel/sql-copy.html

"Note that the command is invoked by the shell, so if you need to pass
any arguments to shell command that come from an untrusted source, you
must be careful to strip or escape any special characters that might
have a special meaning for the shell. For security reasons, it is best
to use a fixed command string, or at least avoid passing any user input
in it."

"Similarly, the command specified with PROGRAM is executed directly by
the server, not by the client application, must be executable by the
PostgreSQL user. COPY naming a file or command is only allowed to
database superusers or users who are granted one of the default roles
pg_read_server_files, pg_write_server_files, or
pg_execute_server_program, since it allows reading or writing any file
or running a program that the server has privileges to access."

Those seem reasonable to me?

Yes, but I think that the use of the phrase "default roles" here is unfortunate.  I know it means that the role exists by default, but it is easy to read that to mean they are granted by default.  They should probably be called something like 'built-in roles' or 'system roles'.

And even with the understanding that we are referring to existence, not grant status, "default roles" is still not really correct. If it exists by default, that means I can make it not exist by taking action.  But these roles cannot be dropped.
 
We don't have 'default functions' or 'default types' in the user-facing documentation.  We shouldn't call these 'default roles'.

Cheers,

Jeff
От:
Robert Treat
Дата:

On Fri, Apr 5, 2019 at 8:35 AM Jeff Janes <> wrote:
> On Tue, Apr 2, 2019 at 11:31 AM Andres Freund <> wrote:
>> On 2019-04-02 07:35:02 -0500, Brad Nicholson wrote:
>>
>> > A blog post would be nice, but it seems to me have something about this
>> > clearly in the manual would be best, assuming it's not there already.  I
>> > took a quick look, and couldn't find anything.
>>
>> https://www.postgresql.org/docs/devel/sql-copy.html
>>
>> "Note that the command is invoked by the shell, so if you need to pass
>> any arguments to shell command that come from an untrusted source, you
>> must be careful to strip or escape any special characters that might
>> have a special meaning for the shell. For security reasons, it is best
>> to use a fixed command string, or at least avoid passing any user input
>> in it."
>>
>> "Similarly, the command specified with PROGRAM is executed directly by
>> the server, not by the client application, must be executable by the
>> PostgreSQL user. COPY naming a file or command is only allowed to
>> database superusers or users who are granted one of the default roles
>> pg_read_server_files, pg_write_server_files, or
>> pg_execute_server_program, since it allows reading or writing any file
>> or running a program that the server has privileges to access."
>>
>> Those seem reasonable to me?
>
>
> Yes, but I think that the use of the phrase "default roles" here is unfortunate.  I know it means that the role
existsby default, but it is easy to read that to mean they are granted by default.  They should probably be called
somethinglike 'built-in roles' or 'system roles'. 
>
> And even with the understanding that we are referring to existence, not grant status, "default roles" is still not
reallycorrect. If it exists by default, that means I can make it not exist by taking action.  But these roles cannot be
dropped.
>
> We don't have 'default functions' or 'default types' in the user-facing documentation.  We shouldn't call these
'defaultroles'. 
>

As someone who likes to break systems in interesting ways, I do find
it interesting that you can actually remove all superuser roles and/or
the superuser bit from all roles (not that I would recommend that to
anyone) but that these roles cannot be removed without some serious
heavy lifting.

Given that, I think I would tend to agree, describing them more
consistently as "system roles" is probably warranted.

Robert Treat
https://xzilla.net
https://credativ.com